Hardware-Accelerated Substring Search Using Rolling Hash XOR Shift
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing substring search algorithms, such as the Rabin-Karp algorithm, face challenges when implemented in hardware-based systems like ASICs or FPGAs due to the complexity of integer multiplication and hash recomputation, leading to inefficiencies in CPU-intensive tasks like virus signature detection.
Innovation Solution
The implementation of a rolling hash using XOR and shift operations, combined with an index table or linear feedback shift registers, to generate hash values efficiently, allowing for hardware-based substring search without the need for frequent data re-reads and minimizing hash collisions.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If Rabin-Karp rolling hash with multiplication operations is used, then hash value computation is mathematically accurate, but hardware implementation complexity increases significantly
Solution Approach 1:
The patent changes the mathematical parameters of the hash function from multiplication-based (Rabin-Karp) to XOR and shift operations. This parameter change maintains the rolling hash functionality while making the operations suitable for hardware implementation, reducing complexity from multipliers to simple bitwise operations that can be executed in parallel in FPGA/ASIC circuits.
Solution Approach 2:
The patent substitutes the mechanical multiplication operation with bitwise XOR and shift operations. These bitwise operations can be implemented more efficiently in hardware circuits, replacing complex arithmetic logic with simpler digital logic gates that consume less power and occupy less area in FPGA/ASIC implementations.
2Reliability
If hash function re-reads L-1 bytes of data file, then hash value can be updated for sliding window, but computational overhead increases
Solution Approach 1:
The patent applies preliminary action by pre-processing the data file into fixed-size blocks and pre-computing initial hash values for each block. This preliminary preparation allows the sliding window hash updates to proceed efficiently using only XOR and shift operations on the pre-blocked data, avoiding the need to re-read bytes during the main processing phase.
Solution Approach 2:
The patent segments the data file into fixed-size blocks before applying the rolling hash. This segmentation allows the hash function to operate on discrete, manageable units, enabling efficient block-based processing in hardware where entire blocks can be loaded into registers or memory buffers once and reused for multiple sliding window positions without repeated I/O operations.
3Measurement precision
If L is larger than available cache, then entire L-1 bytes must be reread as sliding window advances, but memory access time increases
Solution Approach 1:
The patent performs preliminary blocking of the data file into fixed-size chunks that can be loaded into memory or registers in advance. This preliminary action ensures that when the sliding window moves, the required bytes are already available in fast memory or registers, eliminating the need for repeated memory accesses even when L exceeds cache size.
Solution Approach 2:
The patent enables continuous hash computation by maintaining the data blocks in memory and using efficient XOR/shift operations that can be executed continuously as the sliding window advances. This continuity eliminates idle time associated with repeated memory reads, allowing the hash function to operate continuously on data already resident in fast memory.
Data Source
AI summary
Techniques are provided for generating a hash value for searching for substrings in a data stream without reading more than one element (e.g. one byte) at a time. According to one technique, a before a next element is added to an old hash value, the old hash value is circularly shifted one or more bits. The first original element is shifted a number of bits and XOR'ed against the old hash value. The next element is added to the old hash value. In one embodiment, an entry value is retrieved for each element from an index table and the XOR and shift operations are performed on the entry values. According to another technique, each Linear Feedback Shift Register (LFSR) of a plurality of LFSRs read in one element at a time beginning at different offsets. Each LFSR uses the same state machine. The result of reading a number of elements into an LFSR is used as the hash value.