Data Block De-Duplication Using Lightweight Hashes and Linked Lists
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current data storage systems face inefficiencies in managing large data volumes due to high storage costs, mechanical limitations in data retrieval, and the need for specialized hardware to implement computationally heavy hash functions, which increases costs and reduces flexibility and speed in data de-duplication processes.
Innovation Solution
The implementation of data processing methods that support data de-duplication and compression using lighter-weight hash functions, such as 22 to 31 bit hash values, allowing for software-based hash function execution without specialized hardware, and employing linked lists and tables to efficiently track and retrieve data blocks, thereby reducing storage requirements and retrieval time.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If computationally heavy hash functions (MD-5 with 128 bits) are used to minimize hash collisions, then collision resistance is improved, but hardware complexity and cost increase due to specialized hardware requirements
Solution Approach 1:
The patent replaces expensive, specialized hardware for MD-5 hash computation with inexpensive general-purpose processors running software-based hash functions. The system accepts a trade-off between hash length (using 32-bit instead of 128-bit) and hardware cost, achieving adequate collision resistance through software implementation on commodity hardware.
Solution Approach 2:
The patent changes the hash value parameter from 128 bits to 32 bits, reducing the computational complexity and hardware requirements while maintaining acceptable collision resistance for the application. This parameter change enables the use of software-based hash functions instead of specialized hardware.
2Quantity of substance
If de-duplication processes are implemented to reduce storage requirements, then storage efficiency is improved, but data retrieval time increases due to the need to retrieve and compare previously stored blocks
Solution Approach 1:
The patent pre-computes and stores hash values for all data blocks before the actual de-duplication comparison process. When a new block arrives, its hash is computed and compared against the pre-stored hash table, avoiding the need to retrieve and compare entire data blocks. This preliminary hashing action significantly reduces retrieval time while maintaining de-duplication efficiency.
Solution Approach 2:
The patent extracts the essential identifying feature (hash value) from each data block and stores it separately in a hash table. This extracted hash serves as a surrogate for the entire block, enabling fast comparison without retrieving the actual data blocks, thus reducing retrieval time while maintaining storage efficiency.
3Quantity of substance
If data compression is applied to minimize storage requirements, then storage capacity is improved, but processing time increases due to additional compression and decompression operations
Solution Approach 1:
The patent performs data compression before the de-duplication hashing process. By compressing data blocks in advance and storing the compressed versions, the system reduces the amount of data that needs to be hashed and compared. The hash function operates on the compressed data, and during retrieval, the same hash is computed on incoming compressed blocks, eliminating the need for decompression during comparison operations.
Data Source
AI summary
Data processing methods and apparatus for efficiently storing and retrieving data, e.g., blocks of data, to and from memory. The data processing including, e.g., techniques such as using linked lists and/or tables for tracking duplicate data blocks received for storage, the use of lossless data compression, and de-duplication based on comparing hash values, compressed data block sizes, and/or bit by bit comparisons of the block of data to be stored and previously stored blocks of data.