Structured Record Compression via State Machine Segmentation
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current compression techniques for log files generated by computing devices are inefficient, leading to storage of repetitive data and requiring the decompression of entire files to retrieve portions of data.
Innovation Solution
The method involves parsing structured records into data buffers based on field tags, generating a state machine, and encoding a control sequence to efficiently compress and store data, allowing for selective decompression of only the requested data.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Quantity of substance
If current compression techniques are used on log files, then data size is reduced, but the compression is inefficient and results in storage of repetitive data
Solution Approach 1:
The patent segments the log file data into fixed-size blocks and further divides each block into multiple bands (e.g., 4 bands of 64KB each). This segmentation allows independent processing and compression of different data portions, improving compression efficiency by applying specialized techniques to each segment based on its characteristics while reducing the storage of repetitive data through targeted compression strategies.
2Quantity of substance
If current compression techniques are used on log files, then data size is reduced, but retrieving portions of data requires decompression of the entire compressed file
Solution Approach 1:
The compressed data is organized into multiple independent bands where each band represents a segment of the original data. This segmentation enables random access to specific bands without requiring decompression of the entire file. The system can retrieve portions of data by decompressing only the necessary bands, significantly reducing retrieval time while maintaining efficient compression ratios.
Solution Approach 2:
The patent introduces a new dimensional organization of compressed data by creating a multi-band structure with independent access paths. Instead of a single linear compressed stream, the data is arranged in a multi-dimensional structure where bands can be independently accessed and decompressed, enabling partial file retrieval without full decompression.
3Quantity of substance
If data is compressed to reduce storage space, then storage efficiency improves, but the compression process becomes more complex
Solution Approach 1:
The compression system is divided into modular components that process data in segments. Each band is processed independently through standardized compression routines, which simplifies the overall system architecture. The segmentation allows the use of simpler, more efficient compression algorithms on each segment rather than requiring a single complex algorithm to handle the entire data set.
Solution Approach 2:
The patent employs parameter changes by adjusting compression settings and data representation formats for different bands based on their specific characteristics. This allows optimization of compression ratios for each segment while maintaining system simplicity through standardized processing pipelines that can be configured with different parameters rather than requiring fundamentally different compression mechanisms.
Data Source
AI summary
This technology relates to encoding data. For example, a sequence of one or more structured records as input data, at least one of the structured records including one or more field tags and associated field data. The input data may be parsed into data buffers, each data buffer corresponding to a field tag in the one or more field tags, wherein each data buffer includes the associated field data of the corresponding field tag. A control sequence specifying a sequence of the one or more fields tags may be encoded into a transition record. A state machine comprising nodes and transitions may be generated, with each node corresponding to occurrences of the one or more field tags and each transition corresponding to successive pairs of the one or more field tags. The data buffers, a representation of the state machine, and the encoded control sequence may be output.