Log Record Storage Using Template and Attribute Repositories
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing systems face challenges in efficiently storing and searching log records from data centers due to high volumes and rates of log generation, requiring significant storage space and time while maintaining efficient search capabilities.
Innovation Solution
The method involves identifying template and attribute identifiers for log records, partitioning them into batches using a data structure that includes these identifiers, and storing them in a log repository, allowing for efficient retrieval and search through template and attribute repositories.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Ease of operation
If log records are stored in a full-text index to enable complex text queries, then search functionality is improved, but storage space requirements increase proportionally to the number of terms in the log records
Solution Approach 1:
The log record storage system is segmented into multiple repositories: a template repository storing unique template strings, an attribute repository storing unique attribute values, and a log repository storing compact log records with references. This segmentation eliminates redundant storage of repeated strings while maintaining search capabilities through the structured reference system.
Solution Approach 2:
Instead of storing complete log record texts in the log repository, the system stores compact references (template identifiers and attribute identifiers) that point to the actual template and attribute strings stored in separate repositories. This copying approach significantly reduces storage space while enabling efficient retrieval and search operations.
2Ease of operation
If log records are stored in full-text index format, then complex text queries can be performed, but processing time increases due to the volume of log records generated at millions per second
Solution Approach 1:
The system performs preliminary actions by pre-processing log records into a normalized format with template identifiers and attribute identifiers during ingestion. This pre-structuring enables faster query processing later, as searches can operate on the compact reference format rather than parsing full text records at query time.
Solution Approach 2:
The system changes the parameter representation from storing complete text strings to storing compact integer identifiers (template IDs and attribute IDs). This parameter transformation dramatically reduces the amount of data that needs to be processed during queries while maintaining the ability to perform text searches by retrieving and searching template and attribute strings from their respective repositories.
3Loss of information
If all log records are stored in detail, then complete information is preserved, but storage efficiency decreases significantly
Solution Approach 1:
The system creates a compact copy representation where each log record is stored as a tuple of (template_id, attribute_id, timestamp, log_id) in the log repository, while the actual template and attribute strings are stored once in their respective repositories. This copying mechanism preserves complete information while achieving significant storage efficiency through deduplication of repeated strings.
Solution Approach 2:
The information is segmented into three separate repositories: templates (invariant parts), attributes (variable parts), and log records (references). This segmentation allows the system to store only unique template and attribute strings once, while log records merely reference them, thereby preserving complete information without the redundancy of storing full text repeatedly.
Data Source
AI summary
A method by a computer includes, for each of a plurality of log records received as part of a log stream from a host machine node, identifying a template identifier within a template repository for a template string matching an invariant string of the log record, and identifying an attribute identifier in an attribute repository for an attribute string matching a variant string of the log record. The log records are partitioned into batches. Each of the batches are defined by a data structure that includes the template identifier and the attribute identifier for each of the log records within the batch. The data structures for each of the batches are stored into a log repository.