Searchable Log Compression Using Token and Type Dictionaries
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
The large volume of log data generated by technology companies poses significant challenges in terms of storage costs and resource-intensive searching, with conventional compression methods making analysis impractical due to the need for decompression and the inability to search compressed logs without access to the program that generated them.
Innovation Solution
A lossless compression method that separates log messages into tokens, using dictionaries for static text and variable values, allowing direct searching without decompression, with support for complex queries and APIs for custom analysis.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Quantity of substance
If conventional compression methods are used on log data, then storage space is reduced, but the ability to search and analyze the logs is lost without decompression
Solution Approach 1:
The log message is segmented into distinct components: static text portions and variable portions. Each segment is processed differently - static text is compressed using dictionaries while variable portions are preserved in a searchable format. This segmentation allows simultaneous compression and maintainability of search capabilities.
Solution Approach 2:
An intermediary data structure is introduced between the compressed log data and the search function. This intermediary maintains the relationship between compressed representations and their original meanings, enabling searches to be performed on the compressed data without full decompression by translating search queries into the compressed domain.
2Quantity of substance
If log data is compressed to reduce storage costs, then storage efficiency improves, but search speed and analysis capability deteriorate
Solution Approach 1:
During the compression phase, the system performs preliminary actions by building dictionaries of static text portions and organizing variable portions in a structured format. This preliminary organization during compression enables fast search operations later without requiring decompression, as the search-optimized structure is created upfront.
Solution Approach 2:
The invention changes the parameter representation of log data by transforming static text into dictionary-encoded forms while maintaining variable portions in their original or simplified numeric forms. This parameter transformation allows the compressed data to retain searchable properties, achieving both compression and fast search capability.
3Ease of operation
If complete decompression is performed to enable searching, then search capability is restored, but processing time and resource consumption increase
Solution Approach 1:
Instead of performing complete decompression to enable searching, the system applies partial action by maintaining only the portions of log data that are necessary for search operations in an accessible format. The static compressed portions remain compressed while variable portions are kept in a search-friendly structure, avoiding the time cost of full decompression.
4Quantity of substance
If log data is stored in compressed format, then storage costs are reduced, but the complexity of implementing search and analysis increases
Solution Approach 1:
The invention creates simplified copies or representations of the compressed log data that are optimized for search operations. Rather than implementing complex search algorithms on fully compressed data, it maintains searchable copies of variable portions and uses dictionary lookups for static portions, reducing implementation complexity while preserving compression benefits.
Data Source
AI summary
Log messages are compressed, searched, and decompressed. A dictionary is used to store non-numeric expressions found in log messages. Both numeric and non-numeric expressions found in log messages are represented by placeholders in a string of log “type” information. Another dictionary is used to store the log type information. A compressed log message contains a key to the log-type dictionary and a sequence of values that are keys to the non-numeric dictionary and/or numeric values. Searching may be performed by parsing a search query into subqueries that target the dictionaries and/or content of the compressed log messages. A dictionary may reference segments that contain a number of log messages, so that all log message need not be considered for some searches. Compressed logs may be stored and queried on the basis of log type.