Unified Storage and Communication De-duplication via Hashing
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Traditional techniques for alleviating data storage and communication burdens are often disjointed and resource-intensive, limiting the sophistication of algorithms that can be employed.
Innovation Solution
A system that utilizes algorithms to calculate hashes of data chunks, replacing chunks with identifiers for storage and communication, allowing for efficient data reconstruction and reducing redundancy by checking for local availability of chunks using these identifiers.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Loss of energy
If traditional techniques are used to alleviate data storage and communication burdens, then some strain is reduced, but the techniques are disjointed and consume significant resources
Solution Approach 1:
The patent combines storage de-duplication and communication de-duplication into a unified system. The sending device and receiving device both perform hash calculation and chunk identification, merging previously separate storage and communication optimization functions into a single integrated approach that reduces overall resource consumption.
Solution Approach 2:
The data is divided into chunks and each chunk is identified by a hash value. This segmentation allows the system to work with individual chunks rather than entire files, enabling precise de-duplication at the chunk level and reducing the computational overhead compared to traditional whole-file approaches.
2Productivity
If computationally expensive algorithms are employed for de-duplication, then de-duplication effectiveness is improved, but resource consumption increases
Solution Approach 1:
The patent extracts only the essential identifying feature (hash value) from each data chunk, rather than processing or transmitting the entire chunk. This extraction approach allows sophisticated de-duplication to be achieved by comparing compact hash representations rather than the full data, significantly reducing computational resources while maintaining effectiveness.
3Reliability
If all data chunks are communicated via network, then data availability is ensured, but network bandwidth is wasted on redundant data
Solution Approach 1:
The receiving device performs preliminary hash calculation on locally stored chunks before network communication occurs. By pre-computing hashes and creating a local index of available chunks, the system can quickly identify which chunks are already present and only request missing chunks, avoiding redundant network traffic while ensuring data availability.
Data Source
AI summary
Storage and communication de-duplication are described. In one or more implementations, a system comprises one or more modules that are implemented at least partially in hardware, the one or more modules configured to utilize one or more algorithms to calculate hashes of chunks of data, the hashes used to replace the chunks in the data for storage locally in the system as well as to communicate the hashes in response to a request received via a network for the data to avoid communicating at least one of the chunks of the data via the network.