Semi-Sorting Prefix Compression for High-Associativity Fingerprints
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing methods for compressing fingerprints in semi-sorting algorithms, such as cuckoo filters and d-left counting Bloom filters, face limitations due to the rapid growth of encoding and decoding tables, which restricts associativity in hardware caches and memory storage, typically allowing only four or less buckets due to the large size of lookup tables.
Innovation Solution
A data processing platform that selects suffix and prefix data items based on content, sorts them, and uses multiple encoding tables to create a code word representing compressed prefix values, allowing for efficient storage and decompression, reducing the size of tables through semi-sorting and combinatorial encoding schemes.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Loss of information
If prior work uses a pair of encoding and decoding tables to encode and decode fingerprints, then compression is achieved, but the size of these tables grows rapidly, limiting associativity to four or less
Solution Approach 1:
The patent divides the single large encoding/decoding table into multiple smaller tables organized in a tree structure. Each level of the tree represents a segment of the prefix space, with root level tables handling most common prefixes and child level tables handling less common prefixes. This segmentation reduces the memory footprint of any single table while maintaining comprehensive coverage of all possible prefixes.
Solution Approach 2:
The patent transitions from a flat, single-level table structure to a multi-level hierarchical tree structure. This adds a dimensional aspect to the table organization, where tables are arranged in levels from root to leaves. The hierarchical arrangement allows the system to navigate through multiple dimensions of prefix matching, reducing the cardinality required at each level while achieving the same compression functionality.
2Adaptability or versatility
If table size is reduced to improve storage in hardware caches, then associativity can be increased, but compression effectiveness may be compromised
Solution Approach 1:
The patent applies local quality by making different parts of the prefix space have different table allocations. Frequently occurring prefixes at the root level have dedicated small tables for fast encoding/decoding, while less common prefixes are handled by tables at deeper levels. This localized optimization ensures that the most important prefixes (those appearing most frequently) get the most efficient treatment, maintaining high compression effectiveness for the majority of cases while using minimal resources.
3Quantity of substance
If multiple encoding tables are used to reduce table size, then storage requirements decrease, but system complexity increases
Solution Approach 1:
The patent performs preliminary action by pre-organizing the encoding tables into a fixed hierarchical tree structure during system initialization. The tree structure and table assignments are predetermined based on statistical analysis of prefix frequencies. This preliminary organization eliminates the need for complex runtime decisions about table selection and structure, reducing operational complexity despite having multiple tables. The system simply follows the pre-established tree paths during encoding and decoding operations.
Data Source
AI summary
A data processing platform, method, and program product perform compression and decompression of a set of data items. Suffix data and a prefix are selected for each respective data item in the set of data items based on data content of the respective data item. The set of data items is sorted based on the prefixes. The prefixes are encoded by querying multiple encoding tables to create a code word containing compressed information representing values of all prefixes for the set of data items. The code word and suffix data for each of the data items are stored in memory. The code word is decompressed to recover the prefixes. The recovered prefixes are paired with their respective suffix data.