Memory-Efficient Perfect Hashing for Large Records
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Traditional dictionary implementations, such as hash tables, require significant memory overhead for storing large keys like container IDs or SHA1 fingerprints, making them inefficient for large-scale deduplication systems.
Innovation Solution
Implementing a memory-efficient perfect hashing process using perfect hash functions that map a set of keys to a smaller range of positions, reducing memory overhead and enabling fast lookups, by dividing the key set into fixed-size ranges and creating perfect hash functions for each bucket.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If traditional hash tables are used to store container IDs or fingerprints, then lookup functionality is provided, but memory overhead becomes very high
Solution Approach 1:
The patent extracts the key storage requirement from the traditional hash table implementation. Instead of storing both keys and values in the hash table, it uses perfect hash functions to compute key positions on-the-fly, storing only the perfect hash function parameters and the values. This extraction of key storage eliminates the memory overhead associated with storing billions of keys while maintaining O(1) lookup capability.
Solution Approach 2:
The patent uses perfect hash functions as a computational copy mechanism. Rather than physically storing keys in memory, the perfect hash function serves as a computational replica that can reconstruct the key-to-position mapping whenever needed. This functional copy approach maintains lookup functionality while avoiding the memory cost of actual key storage.
2Quantity of substance
If perfect hash functions are used to reduce memory overhead, then memory efficiency is improved, but the complexity of creating and managing perfect hash functions increases
Solution Approach 1:
The patent applies preliminary action by pre-computing and storing the perfect hash function parameters during a setup phase. The perfect hash function is constructed offline using the complete set of keys, and its parameters (hash constants, bucket configurations) are stored in a compact form. This preliminary computation eliminates the need for runtime key storage while enabling fast O(1) lookups, and the complexity is shifted to the offline setup phase rather than ongoing operations.
3Quantity of substance
If keys are not stored in the dictionary structure, then memory overhead is reduced, but the ability to perform lookups efficiently is compromised
Solution Approach 1:
The patent replaces the mechanical storage system (physically storing keys in memory arrays) with a computational system (perfect hash functions). Instead of mechanically accessing stored keys to determine positions, the system uses mathematical hash functions to compute positions directly from keys. This substitution maintains O(1) lookup speed while eliminating the memory overhead of key storage, as the computational mechanism requires no additional memory beyond the function parameters.
Data Source
AI summary
Embodiments for a memory efficient perfect hashing for large records. A container ID set is divided into multiple fixed range sizes. These ranges are then mapped into perfect hash buckets until each bucket is filled to uniformly distribute the container IDs across different perfect hash buckets so that the number of CIDs in every perfect hash bucket is the same or nearly the same. Individual perfect hash functions are created for each perfect hash bucket. With container IDs as keys, the process maps n keys to n positions to reduce any extra memory overhead. The perfect hash function is implemented using a compress, hash, displace (CHD) algorithm using two levels of hash functions. The level 1 hash functions divides the keys into multiple internal buckets with a defined average number of keys per bucket. The CHD algorithm iteratively tries different level 2 hash variables to achieve collision-free mapping.