Hierarchical Index Compression for JSON Logs
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing log compression methods, such as the Microsoft Xpress Compression Algorithm, are not tailored for hierarchical index logs and fail to preserve structural information and global indexes, leading to inefficient storage usage for large datasets like user search-engine interaction logs.
Innovation Solution
A hierarchical index-based compression method is developed, where an index tree is constructed from the log data, allowing for efficient compression and decompression of JSON-formatted logs, and can be further compressed using conventional schemes, thereby reducing storage needs.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Quantity of substance
If general compression schemes like Microsoft Xpress Compression Algorithm are used, then compression is achieved, but structural information and global indexes are not preserved
Solution Approach 1:
The patent segments the log data into hierarchical components (global indexes, structural information, and data rows) and applies different compression strategies to each segment. The index tree structure separates the hierarchical framework from the actual data, allowing independent optimization of each part while preserving their relationships.
Solution Approach 2:
The patent performs preliminary actions by constructing an index tree from the log data before compression. This index tree captures the hierarchical structure and global indexes in advance, enabling the compression process to reference this pre-built structure and preserve it in the compressed output.
2Quantity of substance
If log size is reduced through compression, then storage costs decrease, but compression and decompression time increases
Solution Approach 1:
The patent applies local quality by optimizing compression for different parts of the log data. The index tree structure allows frequently accessed global indexes and structural information to be compressed with higher priority, while data rows can use more efficient compression methods. This localized optimization reduces overall processing time while maintaining storage efficiency.
Data Source
AI summary
Computer-readable media, systems, and methods for hierarchical index based compression are described. In embodiments, a hierarchical data log or key-value pair based data log, such as a JSON log, is received and a tree-structured index (index tree) is recursively constructed. In one embodiment, the log comprises search-engine user interaction information. Structural information of the log is preserved by the index tree structure; for example, each node of the log has a corresponding index-tree node. Frequently repeating keys, values, and correlated key-value pairs may be stored in the index-tree node, which may be indexed using multiple levels of detail including a raw-string level for raw string representations of the node, a first level for indexing keys and common values, and a second level for indexing correlated key-value pairs. The index tree may be used to compress rows of the data log and also used to decompress and restore the log.