Prefix Tree Compression for Fast Keyed String Deduplication
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing data compression techniques are slow for high compression rates and result in large decompressed datasets, requiring lengthy processing and duplication of repeated string data, which hampers data processing and storage efficiency.
Innovation Solution
A tree data structure is used for string compression, where nodes are associated with character sequences and unique identifiers, allowing for efficient storage and retrieval of strings by traversing the tree and sharing external buffers, reducing memory footprint and processing time.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Quantity of substance
If existing compression techniques are used for high compression rates, then compression ratio is improved, but processing speed deteriorates
Solution Approach 1:
The patent segments the data processing task by separating prefix extraction from suffix processing. The tree data structure pre-processes and stores only the unique prefixes of strings, while the suffixes (which contain the actual varying information) are processed separately. This segmentation allows high compression ratios for the common prefix portions while maintaining fast processing speeds for the suffix portions, resolving the contradiction between compression ratio and processing speed.
2Productivity
If decompression produces the original large dataset, then data transfer efficiency is improved, but memory usage and processing time worsen
Solution Approach 1:
The patent extracts and stores only the unique prefixes in the tree data structure, separating them from the full original strings. During decompression, instead of reconstructing entire large datasets, the system retrieves only the necessary prefix information from the compact tree structure and combines it with the suffix portions. This extraction approach maintains data transfer efficiency while dramatically reducing memory usage by eliminating redundant prefix duplications.
3Quantity of substance
If string interning is used to eliminate duplicates, then memory efficiency is improved, but processing complexity worsens
Solution Approach 1:
The patent performs preliminary action by pre-processing the string data during the compression phase to identify and extract unique prefixes, organizing them in a tree data structure with unique identifiers. This preliminary organization eliminates the need for complex runtime processing when handling duplicate strings, as the system can directly reference the pre-computed prefix structure. The complexity is shifted to the initial compression phase, simplifying subsequent operations and improving overall memory efficiency.
Data Source
AI summary
Systems and techniques are described for compressing strings by using a tree data structure. Specifically, for each string in a sequence of strings, the embodiments can traverse the tree data structure by matching characters of the string with characters associated with nodes of the tree data structure until either (1) all characters in the string have been processed, or (2) a current character in the string does not match a corresponding character in a current node of the tree data structure. Next, a first node identifier associated with the current node can be returned if all characters have been processed. Otherwise, a new node can be created in the tree data structure to store the remaining characters in the string, and a second node identifier associated with the new node in the tree data structure can be returned.