Time-Series Data Search Using Flat Files and Regular Expressions

Resolve Bottlenecks,
Find Innovative SolutionsGenerate Solutions

Solution Overview

Problem

Current methods for monitoring and analyzing performance data from IT systems and other data sources face challenges such as high bandwidth requirements, scalability issues, and inefficiencies in data storage and search processes, particularly when using relational databases and SQL queries.

Innovation Solution

The system employs a directory structure for storing performance data, using regular expressions to encode and search data, eliminating the need for relational databases and reducing bandwidth requirements by storing data in a non-relational database format, allowing for efficient pattern analysis without complex custom programming.

Engineering Contradictions & Design Principles

VSEngineering Contradiction Analysis

1Ease of operation

If relational databases and SQL queries are used to store and search performance data, then data can be organized and searched, but storage overhead and search complexity increase significantly with large data volumes

Engineering Contradiction:

Improvedata storage and search capabilityVSAvoidstorage overhead and search complexity

Core Design Contradiction:

Ease of operationVSDevice complexity

Solution Approach 1:

The patent segments performance data into fixed-size blocks and organizes them in a hierarchical file system structure. Instead of using relational tables, data is divided into manageable units stored in flat files, eliminating the complexity of database indexes and joins while maintaining efficient access patterns.

Inventive Principle:

Principle #1Segmentation

Solution Approach 2:

The patent replaces the mechanical database system (relational databases with indexes and SQL queries) with a simplified file-based system using regular expressions for pattern matching. This substitution eliminates the overhead of database management while providing equivalent or superior search capabilities for time-series data.

Inventive Principle:

Principle #28Mechanics substitution (Replace mechanical system)

2Productivity

If high bandwidth data links are used to transmit performance data from monitored servers to monitoring servers, then data can be collected in real-time, but the number of supported remote servers is limited and costs increase

Engineering Contradiction:

Improvereal-time data collection capabilityVSAvoidnumber of supported remote servers

Core Design Contradiction:

ProductivityVSQuantity of substance

Solution Approach 1:

The patent performs preliminary data processing and filtering at the source (monitored servers) before transmission. By pre-processing data locally and only transmitting essential information, the bandwidth requirement per connection is reduced, allowing more servers to be monitored within the same network capacity.

Inventive Principle:

Principle #10Preliminary action

Solution Approach 2:

The patent implements selective data collection where only relevant performance metrics are gathered and transmitted. By applying partial action (collecting only necessary data rather than all possible data), the system reduces transmission volume and supports a larger number of remote servers.

Inventive Principle:

Principle #16Partial or excessive action

3Adaptability or versatility

If custom programming in the form of stored procedures is used to search for patterns in performance data, then pattern analysis can be performed, but the process becomes burdensome, time-consuming and expensive

Engineering Contradiction:

Improvepattern analysis capabilityVSAvoidtime and effort required for pattern searching

Core Design Contradiction:

Adaptability or versatilityVSLoss of time

Solution Approach 1:

The patent enables self-service pattern searching by providing users with regular expression tools that can be directly applied to the flat-file data structure. Users can independently perform complex pattern matching without requiring database administrators to write custom stored procedures, significantly reducing the time and expertise required.

Inventive Principle:

Principle #25Self-service

Solution Approach 2:

Instead of requiring complex programming to extract patterns from structured database tables, the patent inverts the approach by using simple text-based files that can be directly searched with regular expressions. This reversal of the traditional data storage paradigm simplifies the search process while maintaining pattern analysis versatility.

Inventive Principle:

Principle #13The other way round (Inversion)

Data Source

PatentUS11366844B2System for organizing and fast searching of massive amounts of data

Publication Date: 2022.06.21 HITACHI VANTARA LLC

AI summary

A system to collect and store in a special data structure arranged for rapid searching massive amounts of data. Performance metric data is one example. The performance metric data is recorded in time-series measurements, converted into unicode, and arranged into a special data structure having one directory for every day which stores all the metric data collected that day. The performance metric data is collected by one or more probes running on machines about which data is being collected. The performance metric data is compressed prior to transmission to a server over any data path. The data structure at the server where analysis is done has a subdirectory for every resource type. Each subdirectory contains text files of performance metric data values measured for attributes in a group of attributes to which said text file is dedicated. Each attribute has its own section and the performance metric data values are recorded in time series as unicode hex numbers as a comma delimited list. Analysis of the performance metric data is done using regular expressions. For speed, cache memory is used. Performance metric data outside the start time and end time named by the user on a query screen is eliminated before the regular expression is applied.