Rolling Hash Lookup Tables for Fast Data Boundary Detection
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing data de-duplication algorithms, such as Rabin-Karp fingerprinting, face computational overhead challenges in identifying natural data boundaries, which affects the speed and efficiency of data de-duplication processes.
Innovation Solution
A computing device and method for computing rolling hashes that reduce computational overhead by using a hash window of fixed size, incremental hash calculation, and bitmask comparison to identify data boundaries, allowing for efficient data de-duplication by constructing data block boundaries based on computed rolling hashes that match trigger criteria.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If rolling hash computation is performed at every location to identify data boundaries, then identification quality is improved, but computational overhead increases
Solution Approach 1:
The patent pre-computes hash values for all possible bytes within the hash window and stores them in lookup tables before processing. This preliminary action allows the rolling hash to be computed incrementally during data processing by simply adding new byte contributions and removing old byte contributions, rather than computing the entire hash from scratch at each position.
Solution Approach 2:
The patent divides the hash computation into discrete byte-level segments, where each byte's contribution to the hash is independently calculated and stored in separate lookup tables. This segmentation allows the rolling hash to be updated by combining individual byte contributions rather than re-computing the entire hash function, significantly reducing computational overhead while maintaining identification quality.
2Productivity
If incremental hash calculation is used to reduce computational overhead, then processing speed is improved, but identification accuracy may deteriorate
Solution Approach 1:
The patent implements a feedback mechanism where the rolling hash computation continuously incorporates new byte contributions and removes old byte contributions in an incremental manner. The lookup tables provide pre-computed feedback values that ensure the incremental updates maintain mathematical accuracy, allowing fast processing without sacrificing identification precision.
Solution Approach 2:
The patent changes the parameter representation by pre-computing and storing hash contributions for all possible byte values (0-255) in lookup tables. This parameter transformation allows the system to switch from expensive on-the-fly hash computation to efficient table lookup and combination operations, achieving both high processing speed and maintained accuracy through the mathematical properties of the hash function.
Data Source
Figure 1
Figure 2
Figure 3
AI summary
Technologies for computing rolling hashes include a computing device having a first hash table that includes a first plurality of random-valued entries and a second hash table that includes a second plurality of random-valued entries. The computing device retrieves a block of data from a data buffer and generates a hash based on the block of data, a previously generated hash, the first hash table, and the second hash table. The computing device further determines whether the generated hash matches a predefined trigger and records a data boundary in response to a determination that the generated hash matches the trigger.