Data Deduplication via Reception-Side Hash Prediction
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing data deduplication systems incur high processing loads on the transmission-side apparatus due to chunk division, SHA1 calculation, and hash search, which can negate the bandwidth reduction benefits, especially when data is not duplicated, and are inefficient for low-performance devices like smartphones.
Innovation Solution
The system employs a dual-mode operation in the transmission-side apparatus, switching to a duplication prediction mode based on notification from the reception-side apparatus, where it performs memory comparisons rather than full SHA1 calculations, reducing processing load by offloading chunk division and hash search to the reception-side apparatus.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If the transmission-side apparatus performs data duplication determination processing using SHA1 calculation and hash search, then data duplication can be accurately detected, but the processing load on the transmission-side apparatus becomes excessively high
Solution Approach 1:
The patent extracts the complex SHA1 calculation and hash search operations from the transmission-side apparatus and relocates them to the reception-side apparatus. The transmission-side apparatus only performs simple memory comparison operations, while the reception-side apparatus performs the intensive processing of dividing received data into chunks, calculating SHA1 hashes, and searching the cache for duplicates.
Solution Approach 2:
The patent inverts the traditional deduplication architecture by making the reception-side apparatus perform the active duplication detection role previously held by the transmission-side apparatus. Instead of the transmitter analyzing outgoing data, the receiver analyzes incoming data and provides feedback, fundamentally reversing the direction of the duplication detection process.
2Reliability
If the transmission-side apparatus performs full SHA1 calculation and hash search for every data transmission, then complete duplication detection is achieved, but the processing time and overhead increase significantly
Solution Approach 1:
The patent implements a two-level detection approach where the transmission-side apparatus performs a partial, quick memory comparison check first. Only when this preliminary check indicates potential duplication does the system engage the full SHA1 calculation and hash search process at the reception-side apparatus, avoiding unnecessary intensive processing for non-duplicate data.
Solution Approach 2:
The transmission-side apparatus performs preliminary duplication checking by comparing transmission scheduled data with cached data before actual transmission occurs. This preliminary action filters out obvious duplicates early in the process, reducing the volume of data that requires the more time-consuming reception-side analysis.
3Extent of automation
If the transmission-side apparatus performs duplication detection independently, then it maintains full control over the process, but it cannot leverage the reception-side cache to improve detection efficiency
Solution Approach 1:
The patent merges the duplication detection capabilities of both transmission-side and reception-side apparatuses into a coordinated system. The transmission-side apparatus maintains independence by initiating the detection process and making final transmission decisions, while the reception-side apparatus contributes its cache and processing power to enhance overall detection efficiency through collaborative feedback.
Solution Approach 2:
The reception-side apparatus provides feedback to the transmission-side apparatus about detected duplications in the received data. This feedback mechanism allows the transmission-side apparatus to adjust its transmission strategy, sending only non-duplicate data or appropriate notifications, thereby improving overall data transfer efficiency while maintaining independent control.
Data Source
AI summary
A reception-side apparatus determines whether or not data duplicating a part of received data from a transmission-side apparatus is stored in the first storage that stores first data which has been received, and notifies, when data duplicating a part of the received data is stored in the first storage, the transmission-side apparatus of prediction information on duplicate reception of the first data. A transmission-side apparatus compares, when the prediction information is received, second data to be transmitted and a part of the first data in a second storage that stores the first data which has been transmitted based on the prediction information, determines whether or not there is a first part of the first data matching the second data in the second storage, and transmits, when there is the first part of the first data in the second storage, outline information on the second data.