Dictionary-Based Log Compression for Search Without Decompression
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
The large quantity of log data generated by technology companies poses challenges in storage and search efficiency, as conventional methods require significant resources and are impractical for long-term retention due to the need for decompression and access to the program that generated the logs.
Innovation Solution
A system that compresses log messages using tokenization and dictionaries to differentiate between numeric and non-numeric expressions, allowing for efficient storage and search without decompression, and supports complex queries without requiring access to the program that generated the logs.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Loss of substance
If conventional compression methods are used to reduce storage space, then storage costs decrease, but search efficiency deteriorates because decompression is required
Solution Approach 1:
The log message is segmented into multiple tokens using delimiter-based tokenization. Each token is then classified as either a numeric expression or a non-numeric expression. This segmentation allows selective compression where non-numeric tokens are replaced with dictionary keys while numeric tokens are retained in original form, enabling efficient storage without compromising searchability.
Solution Approach 2:
The patent changes the representation parameter of log message tokens from original text form to compressed form. Non-numeric tokens undergo parameter transformation by replacing them with compact dictionary keys, while numeric tokens maintain their numerical parameters. This parameter change enables space-efficient storage while preserving the ability to search through the compressed structure.
2Loss of substance
If log data is compressed to reduce storage resources, then storage costs decrease, but access complexity increases due to need for decompression and program access
Solution Approach 1:
The patent performs preliminary tokenization and classification of log messages during the compression phase. The log message is pre-processing into tokens, classified as numeric or non-numeric, and non-numeric tokens are pre-replaced with dictionary keys. This preliminary action eliminates the need for complex decompression operations later, as the compressed format is inherently searchable.
Solution Approach 2:
The patent introduces an intermediary dictionary structure that maps compressed keys back to original non-numeric token values. This intermediary dictionary serves as a mediator between the compressed storage format and the original log message content, enabling efficient access without full decompression by allowing direct key-based lookups.
3Measurement precision
If full log messages are stored to maintain searchability, then search accuracy is improved, but storage efficiency deteriorates
Solution Approach 1:
The patent segments the log message into tokens and applies different compression strategies to different segments. Numeric token segments are stored in their original form to preserve precise numerical values for accurate searching, while non-numeric token segments are replaced with compact dictionary keys. This selective segmentation maintains search accuracy for numeric values while achieving storage efficiency through compression of text segments.
Data Source
AI summary
Log messages are compressed, searched, and decompressed. A dictionary is used to store non-numeric expressions found in log messages. Both numeric and non-numeric expressions found in log messages are represented by placeholders in a string of log “type” information. Another dictionary is used to store the log type information. A compressed log message contains a key to the log-type dictionary and a sequence of values that are keys to the non-numeric dictionary and/or numeric values. Searching may be performed by parsing a search query into subqueries that target the dictionaries and/or content of the compressed log messages. A dictionary may reference segments that contain a number of log messages, so that all log message need not be considered for some searches.