Data Compression Hash Selection to Reduce Dictionary Collisions
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
The existing methods for lossless data compression, such as those using dictionary encoding and entropy encoding, face challenges due to hash collisions, which reduce the accuracy of data retrieval and compression efficiency in storage systems like SSDs and HDDs, leading to increased costs and storage capacity issues.
Innovation Solution
A compression device is designed with a search unit that determines a hash function based on the input data characteristics, such as the presence of bits always 0 or 1, to minimize hash collisions by excluding these bits from the hash calculation, thereby improving search accuracy and compression efficiency.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Speed
If a hash function is used to search the dictionary buffer, then the search speed is improved, but hash collisions occur which reduce the accuracy of data retrieval
Solution Approach 1:
The patent dynamically determines the hash function based on the characteristics of the input data. Specifically, it analyzes whether certain bits in the data are always 0 or always 1, and adjusts the hash function selection accordingly to minimize hash collisions while maintaining search speed efficiency.
Solution Approach 2:
The patent changes the parameters of the hash function by selecting different functions based on data characteristics. It modifies the hash calculation process to exclude bits that are always 0 or always 1, thereby changing the input parameters to the hash function to reduce collision probability and improve retrieval accuracy.
2Productivity
If dictionary encoding is used for data compression, then the compression efficiency is improved, but hash collisions reduce the overall compression performance
Solution Approach 1:
The system dynamically adapts the hash function selection during the dictionary encoding process based on the actual data being processed. This dynamic adjustment ensures that the most appropriate hash function is used for each data segment, maintaining high compression efficiency while minimizing the negative impact of hash collisions on overall performance.
Solution Approach 2:
The patent modifies the hash function parameters by excluding specific bit positions that cause collisions. This parameter change is applied throughout the compression process to ensure that dictionary encoding operates with minimal collision interference, thereby maintaining both efficiency and reliability.
3Quantity of substance
If more storage capacity is allocated to store uncompressed data, then the data can be stored without compression, but the storage cost increases
Solution Approach 1:
The patent optimizes the compression process by changing the hash function parameters to reduce collisions, thereby improving compression ratios. This allows more data to be stored in the same physical space, effectively increasing storage capacity utilization without requiring additional storage resources, thus reducing storage costs.
Data Source
AI summary
According to one embodiment, a compression device includes a first storage unit, a second storage unit, a calculation unit, and a comparison unit. The first storage unit stores addresses associated with hash values, respectively. The second storage unit includes storage areas specified by the addresses, respectively. The calculation unit determines a hash function to be used for first data in accordance with at least a part of the first data, and calculates a hash value using the hash function and at least a part of second data included in the first data. The comparison unit acquires third data from a storage area in the second storage unit specified by a first address, and compares the second data with the third data. The first address is stored in the first storage unit and is associated with the hash value.