Hash-Based Data Block Compression for Read/Write Overhead Balance
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing data compression methods face challenges in balancing data reduction ratio and system overheads, as increasing the number of combined and compressed data blocks improves data reduction but increases compression and decompression overheads, degrading read/write performance.
Innovation Solution
A data processing method that evaluates the similarity between data blocks using hash values and combines those with similar characteristics, applying a hash algorithm like locality-sensitive hash to determine suitable blocks for compression, ensuring a balance between data reduction ratio and system overheads.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Loss of substance
If a larger quantity of data blocks is combined and compressed, then data reduction ratio increases, but compression and decompression overheads increase and read/write performance decreases
Solution Approach 1:
The patent changes the parameter of data block selection by using hash value similarity comparison. Instead of combining all data blocks or using fixed-size combinations, the system dynamically selects data blocks for combination based on their hash value similarity, adjusting the combination strategy to optimize both compression ratio and performance
Solution Approach 2:
The patent introduces hash values as an intermediary mechanism to evaluate data block similarity. By comparing hash values rather than directly analyzing data block content, the system efficiently identifies suitable candidates for combination compression, reducing computational overhead while maintaining effective compression ratios
2Loss of substance
If a larger quantity of data blocks is combined and compressed, then data reduction ratio increases, but system overheads increase
Solution Approach 1:
The patent replaces the mechanical approach of direct data block comparison with a computational hash-based similarity assessment. By substituting content-based comparison with hash value comparison, the system significantly reduces computational complexity and system overheads while maintaining the ability to identify similar data blocks for compression
3Productivity
If data blocks are selected for combination compression without similarity evaluation, then processing speed increases, but compression efficiency decreases
Solution Approach 1:
The patent performs preliminary hash value calculation and similarity evaluation before combining and compressing data blocks. This preliminary action of pre-assessing similarity through hash comparison ensures that only suitable data blocks are selected for combination, maintaining compression efficiency while avoiding the need for more complex post-selection optimizations
Data Source
AI summary
A data processing method includes obtaining a plurality of data blocks, determining a first data block and a second data block from the data blocks, where the first data block has a first hash value, and the second data block has a second hash value, where the first hash value is obtained by performing calculation on the first data block based on a hash algorithm and the second hash value is obtained by performing calculation on the second data block based on the hash algorithm, and combining and compressing the first data block and the second data block based on a degree of similarity of the first data block and the second data block.