Semi-Sorted Prefix Encoding for High-Associativity Compression
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing semi-sorting methods for compressing fingerprints in cuckoo filters and d-left counting Bloom filters face limitations due to the rapid growth of encoding and decoding tables, which restricts associativity to four or less, leading to large lookup tables that are impractical for storage in hardware caches or memory.
Innovation Solution
The proposed solution involves a data processing platform that selects suffix and prefix data items based on content, sorts them, and uses multiple encoding tables to create a code word representing compressed prefix values, allowing for efficient compression and decompression while reducing the size of the tables through combinatorial expressions and precomputed lookup tables.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Loss of information
If prior work uses a pair of encoding and decoding tables to encode and decode fingerprints, then compression is achieved, but the size of these tables grows rapidly relative to associativity, limiting associativity to four or less
Solution Approach 1:
The patent divides the single large encoding/decoding table into multiple smaller tables organized in a tree structure. Each level of the tree represents a segment of the prefix space, with root level tables handling most common prefixes and child level tables handling less common prefixes. This segmentation reduces the memory footprint of individual tables while maintaining comprehensive encoding capability.
Solution Approach 2:
The patent transitions from a two-dimensional table structure (single table with row-column indices) to a multi-level hierarchical structure adding a vertical dimension. The tree organization introduces levels as a new dimension, where each level contains multiple tables that collectively cover the entire prefix space. This dimensional change allows distributed storage of encoding information across multiple smaller tables.
2Adaptability or versatility
If the associativity of a bucket is increased beyond four, then more fingerprints can be stored, but the lookup tables become several kilobytes in size, making them impractical for hardware cache storage
Solution Approach 1:
The patent segments the lookup functionality across multiple tables organized in a tree structure. Instead of requiring one large table for high associativity, the system uses many small tables distributed across memory. Each table in the tree handles a subset of prefixes, and the combined capacity of all tables supports high associativity values while keeping individual table sizes small enough for cache storage.
Solution Approach 2:
The patent adds the dimension of tree depth and level organization to the traditional single-table approach. This multi-level hierarchical structure allows the system to achieve high associativity by distributing lookup operations across multiple levels and tables, reducing the memory area requirement for any single cache structure while maintaining the ability to handle large numbers of fingerprints.
Data Source
AI summary
A data processing platform, method, and program product perform compression and decompression of a set of data items. Suffix data and a prefix are selected for each respective data item in the set of data items based on data content of the respective data item. The set of data items is sorted based on the prefixes. The prefixes are encoded by querying multiple encoding tables to create a code word containing compressed information representing values of all prefixes for the set of data items. The code word and suffix data for each of the data items are stored in memory. The code word is decompressed to recover the prefixes. The recovered prefixes are paired with their respective suffix data.