Unicode Regex Data Structure for Fast IT Performance Monitoring
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current methods for monitoring and analyzing large volumes of performance data from IT systems and other sources, such as server virtualization and sensor data, face challenges including high bandwidth requirements, scalability issues, and inefficiencies in data storage and pattern analysis using relational databases and SQL queries.
Innovation Solution
A system that uses a non-relational database approach, storing performance data in a special directory structure and encoding it using Unicode to facilitate fast pattern searches with regular expressions, eliminating the need for complex custom programming and reducing storage costs.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Ease of operation
If relational databases with SQL queries are used to store and search performance data, then data can be organized and queried using structured tables, but storage costs and time overhead increase significantly for large datasets
Solution Approach 1:
The patent uses regular expressions as a simplified copy or alternative representation of complex SQL query logic. Instead of storing and processing data in traditional relational database tables with complex joins and indexes, the invention encodes performance data into regex patterns that can be searched directly, eliminating the need for expensive database infrastructure while maintaining query capabilities
Solution Approach 2:
The patent replaces expensive, long-lived relational database systems with lighter, more disposable regex-based data structures. The encoded data files can be created, searched, and discarded more easily than traditional database systems, reducing both storage costs and computational overhead for large performance datasets
2Difficulty of detecting and measuring
If custom stored procedures are written to search for patterns in performance data, then pattern matching capability is achieved, but programming complexity and development time increase
Solution Approach 1:
The patent replaces the mechanical process of writing custom stored procedures and complex SQL queries with a more elegant regex-based system. Instead of manually programming pattern matching logic in SQL, the invention directly encodes performance data into regex patterns that automatically perform pattern matching, eliminating the need for custom programming while maintaining or enhancing pattern detection capabilities
3Adaptability or versatility
If remote servers are monitored using virtual private networks, then no changes are needed on the monitored server, but bandwidth requirements become prohibitively high
Solution Approach 1:
The patent extracts only the essential performance data from the monitored server environment and encodes it into compact regex patterns for transmission and storage. Instead of transmitting complete performance logs or graphical data over the network, the invention extracts key metrics and represents them in a compressed regex format, dramatically reducing bandwidth requirements while maintaining monitoring capabilities
Data Source
Figure 1
Figure 2~3
Figure 4~5
AI summary
A system to collect and store in a special data structure arranged for rapid searching massive amounts of data. Performance metric data is one example. The performance metric data is recorded in time-series measurements, converted into unicode, and arranged into a special data structure having one directory for every day which stores all the metric data collected that day. The performance metric data is collected by one or more probes running on machines about which data is being collected. The performance metric data is compressed prior to transmission to a server over any data path. The data structure at the server where analysis is done has a subdirectory for every resource type. Each subdirectory contains text files of performance metric data values measured for attributes in a group of attributes to which said text file is dedicated. Each attribute has its own section and the performance metric data values are recorded in time series as unicode hex numbers as a comma delimited list. Analysis of the performance metric data is done using regular expressions. For speed, cache memory is used. Performance metric data outside the start time and end time harried by the user on a query screen is eliminated before the regular expression is applied.