File Block Compression Using Hash Coalescing for Redundancy Removal
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing file compression methods using windows for data analysis are inefficient in achieving significant compression, especially when dealing with large files or streaming data, as they may not fully utilize the potential for redundancy reduction and require extensive processing.
Innovation Solution
A method involving the use of strong collision-resistant cryptographic hash representations to identify unique blocks of data, storing only the hash and offset in tables, allowing for efficient compression and decompression by eliminating redundant data storage and leveraging hash tables for quick lookups.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If a window of certain size is used to review data for compression, then the compression process speed is improved, but the compression efficiency and amount of compression are reduced
Solution Approach 1:
The patent divides the file into fixed-size blocks and processes them individually through hashing. Each block is segmented and its hash is computed independently, allowing parallel processing while maintaining the ability to detect redundancies across the entire file. This segmentation approach enables both speed (through independent block processing) and compression efficiency (through comprehensive redundancy detection via hash comparison).
Solution Approach 2:
The patent introduces a hash table as an intermediary structure to store block hashes and their positions. This intermediary enables efficient lookup and comparison of blocks without requiring direct comparison of the actual data blocks, thus speeding up the compression process while maintaining high compression efficiency through accurate redundancy detection.
2Quantity of substance
If a small window of data is reviewed at once for streaming data compression, then the processing memory usage is reduced, but the compression efficiency is limited
Solution Approach 1:
The patent extracts the essential identifying feature of each data block (its hash value) and stores only that in memory, rather than storing the entire block data. This extraction approach dramatically reduces memory usage while preserving the ability to detect duplications, thereby maintaining compression efficiency with minimal memory requirements.
Solution Approach 2:
The patent transforms the data blocks into a different parameter representation (hash values) that is much more compact. By changing from storing actual block data to storing hash values, the memory requirement is reduced from block-size to fixed-hash-size, enabling efficient processing of streaming data with limited memory while maintaining compression capability.
3Loss of substance
If the whole file is searched for redundancies without using a window, then the compression efficiency is improved, but the compression process speed is reduced
Solution Approach 1:
The patent replaces the mechanical approach of comparing entire data blocks (which is computationally expensive and slow) with a hash-based comparison system. By substituting the direct block comparison mechanism with hash value comparison, the system achieves both full-file redundancy detection (maintaining compression efficiency) and improved processing speed (since hash comparison is much faster than block comparison).
Data Source
AI summary
A method for processing blocks of data in a file is provided. The method includes determining a plurality of blocks of data in the file for a compression process used for compressing data in the file. This process is iterated over all the plurality of blocks of data in the file. A block of data is read into memory, and the offset from which it is read is also stored. A strong collision-resistant cryptographic hash representation, such as a strong collision-resistant cryptographic hash, of the block is determined. The cryptographic hash representation (CHR) of the block of data as well as the data itself, is added to a table with the CHR as a unique index. The offset of the block of data in the source file, as well as the CHR for that block are appended to a separate table.