Hardware Compression Hash Tables for Collision Mitigation
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
High performance lossless data compression in hardware is computationally intensive due to challenges in implementing efficient compression algorithms like Deflate, which are affected by hash collisions that reduce data processing speed and increase storage demands in data centers.
Innovation Solution
A processor-implemented method that mitigates hash collisions by hashing input data byte sequences, storing hash values and addresses in a hash table, and using parallel independent storage banks to distribute memory access, while also randomizing data sequences to avoid ghost matches and extend data match searches.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Speed
If a hash table is used for data compression in hardware, then data processing speed is improved, but hash collisions occur that reduce compression effectiveness and increase storage demands
Solution Approach 1:
The hash table is divided into multiple segments or regions, allowing the system to distribute hash entries across different segments. This segmentation reduces the probability of hash collisions within individual segments while maintaining high processing speed through parallel access to multiple segments.
Solution Approach 2:
The patent implements a hierarchical hash table structure where multiple levels of hashing are nested. When a hash collision occurs at one level, the system nests an additional hashing operation to resolve the collision, thereby maintaining compression effectiveness without sacrificing the initial fast lookup capability.
2Speed
If a hash table is used for data compression in hardware, then data processing speed is improved, but storage capacity requirements increase
Solution Approach 1:
The system uses partial hashing where only a portion of the data sequence is hashed rather than the entire sequence. This partial action reduces the storage requirements for the hash table while maintaining sufficient discrimination to achieve high data processing speed for compression operations.
Solution Approach 2:
The patent dynamically adjusts hash table parameters such as table size, hash function characteristics, and entry storage format based on the specific compression workload. This parameter optimization allows the system to achieve high processing speed with minimized storage capacity requirements.
Data Source
AI summary
Systems and methods are provided for mitigating effects of hash collisions in hardware data compression, for example reducing or avoiding the side effects of hash collisions, or reducing or avoiding slow downs caused by hash collisions. In an aspect, a processor-implemented method includes: hashing an input data byte sequence to produce a hash value, the input data byte sequence being located at a sequence address within an input data stream; and storing, in a hash table at a hash address corresponding to the hash value, the sequence address and a portion of the input data byte sequence. In an aspect, to further avoid hash collisions, hash memory accesses are distributed among a plurality of parallel hash banks to increase the throughput. Another aspect virtually extends a hash depth by extending a data match search around broken hash links, going backward in the data sequence.