Time-Series File Structure for Fast Search of Performance Data
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current methods for managing and analyzing large amounts of performance data, such as in server virtualization and bridge monitoring, face challenges with high bandwidth requirements, scalability issues, and inefficiencies in data storage and search processes, particularly when using relational databases and SQL queries.
Innovation Solution
The system employs a non-relational database approach, storing data in a file system with a special directory structure and using regular expressions for pattern searching, eliminating the need for complex custom programming and reducing bandwidth requirements by converting performance metric data into Unicode strings for efficient storage and analysis.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Ease of operation
If relational databases and SQL queries are used to store and search performance data, then data can be organized and searched, but storage overhead and search complexity increase significantly with large data volumes
Solution Approach 1:
The patent segments performance data into fixed-size buckets (e.g., 1-minute intervals) and organizes them in a time-series file structure. Each bucket contains data points for multiple metrics, creating a hierarchical organization that eliminates the need for complex database indexing while enabling efficient range queries through simple file operations.
Solution Approach 2:
The patent uses simple text-based files instead of expensive commercial databases. These files are lightweight, can be easily created and deleted, and require minimal processing power to search. The file-based approach replaces costly database infrastructure with inexpensive storage operations.
2Productivity
If high bandwidth data links are used to transmit performance data between monitored and monitoring servers, then data can be collected in real-time, but the number of supported remote servers decreases due to bandwidth constraints
Solution Approach 1:
The patent performs data aggregation and preprocessing at the monitored server using lightweight agent programs. Data is consolidated into compact time-series buckets before transmission, reducing the volume of data that needs to be sent over the network while preserving all necessary information for analysis.
Solution Approach 2:
The patent changes the data representation format from detailed individual measurements to compressed time-series buckets with fixed intervals. This parameter transformation reduces data transmission requirements while maintaining the ability to perform meaningful analysis on the aggregated data.
3Adaptability or versatility
If custom stored procedures are written to search for patterns in performance data using SQL queries, then pattern analysis can be performed, but programming complexity and development time increase
Solution Approach 1:
The patent enables pattern analysis through direct inspection of time-series data files without requiring custom programming. The structured file format allows analysts to query patterns using simple text processing or visualization tools, making the system self-sufficient for common analysis tasks without needing complex stored procedures.
Solution Approach 2:
The patent replaces the mechanical process of writing and maintaining complex SQL stored procedures with a simpler file-based query approach. Pattern analysis is achieved through direct data inspection and visualization rather than programmed database operations, eliminating the need for specialized programming expertise.
4Loss of time
If large amounts of performance data are stored in relational databases with indexes, then data can be searched efficiently, but storage costs and infrastructure requirements increase
Solution Approach 1:
The patent segments data into time-based buckets organized in hierarchical directories, enabling efficient range queries without requiring global indexes. Each bucket is self-contained and can be searched independently, reducing the overhead of maintaining large-scale database indexes while preserving fast search capabilities for temporal patterns.
Data Source
AI summary
A system to collect and store in a special data structure arranged for rapid searching massive amounts of data. Performance metric data is one example. The performance metric data is recorded in time-series measurements, converted into Unicode, and arranged into a special data structure having one directory for every day which stores all the metric data collected that day. The data structure at the server where analysis is done has a subdirectory for every resource type. Each subdirectory contains text files of performance metric data values measured for attributes in a group of attributes to which said text file is dedicated. Each attribute has its own section and the performance metric data values are recorded in time series as Uinicode hex numbers as a comma delimited list. Analysis of the performance metric data is done using regular expressions.