Columnar Log Compression for Real-Time Analytics Queries
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing data logging systems face challenges in efficiently compressing and querying large volumes of log data without impacting latency, especially when dealing with high variance and diverse log types, which affects storage capacity and computing efficiency.
Innovation Solution
A computing system employs machine-learned models to compress log data into a columnar format with indexed dictionary values, using a universal schema and native database operators for efficient storage and querying, allowing for flexible and efficient retrieval of compressed data.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Quantity of substance
If log data is compressed using traditional compression algorithms, then storage capacity is improved, but querying and analytics performance deteriorates due to decompression requirements
Solution Approach 1:
The patent segments log data into fixed-width columnar records with defined schemas, organizing data by columns rather than rows. This segmentation enables independent column processing during queries, allowing analytics to operate on compressed data without full decompression, thus resolving the contradiction between storage efficiency and query performance
Solution Approach 2:
The patent introduces an intermediary compressed columnar format that sits between raw log data and query processing. This intermediate representation maintains queryability through columnar organization and indexing while achieving high compression ratios, acting as a mediator that preserves both storage efficiency and analytics capability
2Speed
If log data is stored in raw format, then querying speed is improved, but storage capacity deteriorates due to lack of compression
Solution Approach 1:
The patent performs preliminary action by pre-compressing log data into a query-optimized columnar format during ingestion, rather than compressing at query time. This advance preparation includes creating columnar structures, indexes, and dictionaries that enable fast querying on compressed data, resolving the contradiction between storage efficiency and query speed
Solution Approach 2:
The patent changes the structural parameters of log data from raw text format to a compressed columnar binary format with fixed-width fields. This parameter transformation enables both high compression ratios and efficient query processing by organizing data according to analytical requirements rather than ingestion simplicity
3Device complexity
If diverse log types are handled with a single compression approach, then system complexity is reduced, but compression ratios deteriorate due to high variance in log structures
Solution Approach 1:
The patent creates a universal schema-based compression framework that can handle diverse log types through a common columnar structure. The system uses type inference and schema evolution to accommodate different log formats while maintaining a unified compressed representation, achieving both low system complexity and high compression ratios through multi-functional design
Data Source
AI summary
Systems and methods for compressing and querying data for real-time analytics. The system can receive log data and generate an intermediate representation by parsing a log template and variables from the log data into a columnar format. The method includes generating a compressed intermediate representation associated with an index type and storing the compressed intermediate representation in the columnar format based on the index type. The method includes receiving a search query and analyzing the search query to identify a user defined function. The method includes parsing the search query to convert the search query into one or more predicates that satisfy the search query. The method includes filtering the compressed log data based on the one or more predicates and providing a query result.