Log Data Compression via Columnar Indexing for Real-Time Analytics
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current log management platforms face challenges in efficiently compressing and querying large volumes of log data, particularly in storing and analyzing compressed data logs, which affects storage capacity and query performance.
Innovation Solution
A computing system that employs machine-learned models to compress log data into a columnar format with an index type, allowing for efficient storage and querying by parsing log templates and variables, and using user-defined functions to filter and reconstruct log statements.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Quantity of substance
If log data is compressed using traditional compression algorithms, then storage capacity is improved, but query performance deteriorates due to the need for decompression
Solution Approach 1:
The patent segments log data into fixed-width columnar records with defined schemas, allowing the storage system to organize compressed data into structured columns that can be queried directly without full decompression. Each log statement is parsed into fixed-position fields (e.g., timestamp at positions 0-19, message at positions 20-200) that map to queryable columns.
Solution Approach 2:
The patent introduces an intermediate columnar representation layer between the compressed data and the query interface. This intermediate format stores parsed log components (timestamp, level, message, etc.) as separate columns in a compressed columnar structure, enabling direct querying of specific fields without decompressing the entire log statement.
2Adaptability or versatility
If log data is stored in raw format, then query flexibility is improved, but storage efficiency deteriorates
Solution Approach 1:
The patent performs preliminary parsing and schema extraction during the data ingestion phase, organizing raw log statements into structured columnar format with predefined schemas before storage. This preliminary structuring enables flexible querying later without requiring decompression or re-parsing of the original raw data.
Solution Approach 2:
The patent transforms the parameter representation of log data from variable-length raw strings to fixed-width columnar formats with standardized schemas. By changing the data representation parameters (fixed positions, data types, column structures), the system achieves both storage efficiency through compression and query flexibility through structured access.
3Quantity of substance
If data is compressed into a compact format, then storage density is improved, but processing power requirements worsen due to decompression overhead
Solution Approach 1:
The patent segments compressed data into fixed-width columnar records where each segment represents a parsed log component stored in a standardized format. This segmentation allows the storage engine to process and query specific columns directly from the compressed format without requiring decompression of the entire data structure.
Solution Approach 2:
The columnar format is designed to be self-service for querying operations, where the compressed columnar structure inherently supports direct access to specific fields through its organized layout. The data structure itself enables efficient scanning and filtering without requiring external decompression processing.
Data Source
AI summary
Systems and methods for compressing and querying data for real-time analytics. The system can receive log data and generate an intermediate representation by parsing a log template and variables from the log data into a columnar format. The method includes generating a compressed intermediate representation associated with an index type and storing the compressed intermediate representation in the columnar format based on the index type. The method includes receiving a search query and analyzing the search query to identify a user defined function. The method includes parsing the search query to convert the search query into one or more predicates that satisfy the search query. The method includes filtering the compressed log data based on the one or more predicates and providing a query result.