Machine-Generated Data Compression With Reusable Shared Dictionaries
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing data compression methods for machine-generated data, such as log data, consume significant computing resources and network bandwidth, as they require calculating a compression dictionary for each batch of data, which is inefficient due to the repetitive nature of the data.
Innovation Solution
Implementing a pre-shared compression dictionary that is reused across batches of data, with an additional batch-specific compression to achieve higher efficiency, allowing for more effective compression with reduced compute resources.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Quantity of substance
If a compression dictionary is calculated for each batch of data, then compression effectiveness is improved, but computing resource consumption increases
Solution Approach 1:
The patent pre-calculates a compression dictionary from historical machine-generated data before actual compression is needed. This pre-computed dictionary is stored and reused for compressing subsequent batches of data, eliminating the need to recalculate it for each batch. The preliminary action of building the dictionary in advance resolves the contradiction by maintaining high compression effectiveness while significantly reducing per-batch computing resource consumption.
Solution Approach 2:
The patent creates a copy of the compression dictionary from historical data and reuses this copied dictionary for multiple compression operations. Instead of regenerating the dictionary each time, the system makes copies or references to the pre-built dictionary structure, allowing efficient reuse across different data batches without repeating the expensive computation of dictionary construction.
2Loss of energy
If data is compressed to reduce size, then network bandwidth usage is reduced, but computing resources are consumed during compression
Solution Approach 1:
The compression dictionary is built in advance from historical data before the actual data transmission phase. By performing the computationally intensive dictionary construction operation beforehand, the system enables subsequent compression operations to achieve network bandwidth savings with minimal additional computing resources, as only the lightweight compression encoding phase remains.
Solution Approach 2:
The patent changes the computational parameters by using a pre-established dictionary with fixed structure and vocabulary, transforming the compression process from a computationally intensive optimization problem into a simpler encoding operation. This parameter change allows the system to achieve network bandwidth reduction without proportionally increasing computing resource consumption during data transmission.
3Productivity
If compression efficiency is improved by reusing dictionaries, then computing resources are saved, but adaptability to new data patterns may be reduced
Solution Approach 1:
The patent implements a dynamic dictionary update mechanism where the compression dictionary can be periodically refreshed or updated with new historical data. This allows the system to maintain high compression efficiency through dictionary reuse while adapting to changing data patterns over time. The dictionary evolves dynamically rather than remaining static, balancing efficiency and adaptability.
Solution Approach 2:
The system employs periodic dictionary updates or refreshes at predetermined intervals or based on data volume thresholds. This periodic action allows the compression dictionary to be rebuilt or augmented with newer data patterns, ensuring adaptability to changing machine-generated data characteristics while maintaining the efficiency benefits of dictionary reuse between updates.
Data Source
AI summary
A pre-shared compression dictionary is received. The pre-shared compression dictionary was generated based on an analysis of sample data for use in compression of other data. A compressed version of a batch of machine-generated data is received. The batch of machine-generated data has been compressed at least in part using the pre-shared compression dictionary and a batch-specific compression dictionary. The received compressed batch is uncompressed using the batch-specific compression dictionary to determine an intermediate version. The intermediate version is uncompressed using the pre-shared compression dictionary to determine an uncompressed version of the batch of machine-generated data.