File-Based Metric Data Search Using Unicode and Regex
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current methods for managing and analyzing large amounts of performance data, such as those from virtual servers and sensors, face challenges including high bandwidth requirements, scalability issues, and inefficiencies in data storage and search processes, particularly when using relational databases and SQL queries.
Innovation Solution
The system employs a non-relational database approach, storing data in a file system with a special directory structure and using regular expressions for pattern searching, eliminating the need for complex custom programming and reducing bandwidth requirements by converting performance metric data into Unicode strings for efficient storage and analysis.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Ease of operation
If relational databases and SQL queries are used to store and search performance data, then data can be organized and searched, but storage overhead and search complexity increase significantly for large datasets
Solution Approach 1:
The patent segments the monolithic relational database into multiple distributed file-based storage units organized in a hierarchical directory structure. Data is divided into time-series partitions stored as plain text files, eliminating the need for complex database indexing while maintaining efficient access through logical organization of segmented data files.
Solution Approach 2:
The patent replaces the mechanical relational database system with complex indexing and query mechanisms with a simpler file-based storage system using standard text files and regular expressions. This substitution eliminates database overhead while maintaining search functionality through pattern-matching operations on structured text data.
2Productivity
If high bandwidth connections are used to transmit performance data from remote servers, then data can be collected in real-time, but transmission costs and bandwidth requirements increase
Solution Approach 1:
The patent extracts only the essential performance metric values from remote servers, transmitting minimal data in a compact format. By taking out only the necessary numerical data points and storing them in space-efficient representations, the system reduces transmission bandwidth requirements while maintaining the ability to collect real-time performance information.
Solution Approach 2:
The patent changes the parameter representation of performance data by storing values as compact Unicode strings rather than traditional database formats. This parameter transformation reduces the size of transmitted and stored data, thereby reducing bandwidth consumption and transmission costs while preserving data integrity and accessibility.
3Measurement precision
If custom programming is used to search for patterns in performance data, then specific patterns can be detected, but programming complexity and development time increase
Solution Approach 1:
The patent replaces complex custom programming for pattern detection with regular expression-based searching on structured text files. This substitution maintains the ability to detect specific patterns in performance data while dramatically reducing programming complexity, as regular expressions provide a standardized, built-in mechanism for pattern matching without requiring custom algorithm development.
Data Source
AI summary
A system to collect and store in a special data structure arranged for rapid searching massive amounts of data. Performance metric data is one example. The performance metric data is recorded in time-series measurements, converted into unicode, and arranged into a special data structure having one directory for every day which stores all the metric data collected that day. The data structure at the server where analysis is done has a subdirectory for every resource type. Each subdirectory contains text files of performance metric data values measured for attributes in a group of attributes to which said text file is dedicated. Each attribute has its own section and the performance metric data values are recorded in time series as unicode hex numbers as a comma delimited list. Analysis of the performance metric data is done using regular expressions.