Log Record Indexing via Template and Attribute Segmentation
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing systems face challenges in efficiently storing and searching log records generated at high rates in data centers, as they require significant storage space and computational resources for full-text indexing and querying.
Innovation Solution
The method involves partitioning log streams into records with defined intervals, selecting templates from a templates index repository, identifying attributes within these records, and storing them in an attributes index repository, allowing for efficient retrieval and search operations using both repositories in parallel.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Ease of operation
If log records are stored in a full-text index (FTI), then complex text queries can be performed on the log records, but the storage requirements become proportional to the number of terms in the log records, which becomes significant at millions of records per second
Solution Approach 1:
The patent segments the full-text index into two separate indexes: a templates index containing unique log record patterns and an attributes index containing term-frequency information for each template. This segmentation reduces storage by avoiding redundant term storage across similar log records while preserving full-text query capabilities through the structured relationship between templates and attributes.
Solution Approach 2:
The patent changes the indexing parameter from storing complete log record text to storing structured template-attribute pairs. Each log record is represented by its template identifier and a subset of relevant attributes, transforming the storage model from raw text to structured metadata that enables efficient querying with reduced storage footprint.
2Ease of operation
If iterative full-text queries are performed on the FTI of log records, then operators can search for specific log patterns, but the computational resources required increase significantly with the volume of log records
Solution Approach 1:
The patent performs preliminary action by pre-processing log records into templates and attributes during ingestion, organizing data into a structured format with pre-computed term frequencies and template hierarchies. This preliminary structuring enables faster query execution by eliminating the need for iterative full-text scanning, as queries can directly leverage the pre-organized template-attribute relationships.
3Productivity
If log records are generated at high rates (millions per second), then comprehensive monitoring data is captured, but efficient storage and searching become significant challenges
Solution Approach 1:
The patent uses copying by creating a structured representation (template-attribute model) that copies only the essential characteristics of log records rather than storing complete copies of every record. The templates index stores unique patterns once, while the attributes index stores term-frequency information that can be efficiently queried, reducing the complexity of handling high-volume log data.
Data Source
AI summary
A computer partitions a log stream into records having defined intervals. For each of the records, the computer selects a template from among a set of templates in a templates index repository based on an identifier for the record being associated in the templates index repository with the template, identifies an attribute contained in the record based on the template, and stores the attribute associated with an identifier for the record and an identifier for the template in an attributes index repository. The attributes index repository and the templates index repository are used to retrieve records responsive to search queries.