Time-Series Data Structure for Fast Regex Search at Scale
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current methods for monitoring and analyzing performance data from IT systems and other sources face challenges such as high bandwidth requirements, scalability issues, and inefficiencies in data storage and search processes, particularly when using relational databases and SQL queries, which become cumbersome with large volumes of data.
Innovation Solution
The system employs a directory structure for storing performance data, using regular expressions to encode and search data, eliminating the need for relational databases and reducing bandwidth requirements by storing data in a file system with time-series organization, allowing for efficient pattern analysis without the need for complex custom programming.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If relational databases and SQL queries are used to store and search performance data, then data can be organized and searched, but storage overhead and search complexity increase significantly with large volumes of data
Solution Approach 1:
The patent segments performance data into time-series partitions stored in flat files organized by time periods (e.g., daily, hourly). Each partition contains data for a specific time range, allowing independent storage and retrieval without affecting the entire dataset. This segmentation eliminates the need for complex database indexing while enabling efficient time-range queries.
Solution Approach 2:
The patent extracts the indexing function from traditional relational databases and implements a simpler time-based partitioning scheme. Instead of maintaining complex B-tree indexes and foreign key relationships, the system extracts only the essential time-series ordering capability, storing data in chronologically organized flat files that can be directly searched by time ranges.
2Measurement precision
If relational databases are used to store time-series performance data, then data can be queried by key columns, but storage space requirements and costs increase significantly
Solution Approach 1:
The patent creates simplified copies of performance data in a time-series optimized format, copying only the essential timestamp and metric value pairs into flat files. This copying approach eliminates redundant database overhead structures such as index trees, transaction logs, and metadata tables, retaining only the core data needed for time-series analysis.
Solution Approach 2:
The patent changes the storage parameters from relational database formats (with indexes, constraints, and normalization) to flat file time-series formats. This parameter change involves switching from structured query language (SQL) to simple text-based or binary time-series formats, reducing storage overhead while maintaining query capability for time-range and pattern searches.
3Measurement precision
If SQL queries with stored procedures are used to search for patterns in performance data, then pattern matching can be performed, but programming complexity and development time increase
Solution Approach 1:
The patent enables self-service pattern matching by providing built-in time-series analysis capabilities directly in the query interface. Instead of requiring developers to write custom stored procedures for pattern matching, the system offers native functions for detecting trends, anomalies, and temporal patterns, allowing users to perform complex analysis without programming expertise.
Solution Approach 2:
The patent replaces the mechanical system of writing and maintaining complex SQL stored procedures with a simpler declarative query language optimized for time-series data. This substitution eliminates the need for iterative program development, debugging, and maintenance of custom analysis logic, replacing it with straightforward queries that leverage the system's native time-series processing capabilities.
4Reliability
If remote servers are monitored using virtual private networks with high bandwidth connections, then performance data can be transmitted, but bandwidth costs and infrastructure requirements increase
Solution Approach 1:
The patent applies partial action by transmitting only essential performance metric data rather than complete system states or graphical interfaces. Instead of sending all possible data streams, the system selectively transmits only the critical time-series metrics needed for monitoring and analysis, reducing bandwidth consumption while maintaining monitoring effectiveness.
Solution Approach 2:
The patent extracts only the necessary performance data elements from the monitored system and transmits them to the remote monitoring server. By extracting and transmitting only the essential metrics (CPU usage, memory utilization, disk I/O, etc.) in a compact time-series format, the system eliminates the need for high-bandwidth connections required by traditional remote desktop or full-data transmission approaches.
Data Source
AI summary
A system to collect and store in a special data structure arranged for rapid searching massive amounts of data. Performance metric data is one example. The performance metric data is recorded in time-series measurements, converted into unicode, and arranged into a special data structure having one directory for every day which stores all the metric data collected that day. The data structure at the server where analysis is done has a subdirectory for every resource type. Each subdirectory contains text files of performance metric data values measured for attributes in a group of attributes to which said text file is dedicated. Each attribute has its own section and the performance metric data values are recorded in time series as unicode hex numbers. Analysis of the performance metric data is done using regular expressions. Performance metric data outside the start time and end time named by the user on a query screen is eliminated before the regular expression is applied. A first regular expression finds data which qualifies and which is used as a starting point for simultaneous searches by second and third regular expressions. Data which qualifies the second and third regular expressions is merged with data which qualified the first regular expression.