Network Storage Deduplication via Hash Splitting Cost
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current data deduplication methods face challenges in identifying redundant data files and require significant memory resources for effective data management.
Innovation Solution
A computer-implemented method that generates hashes for remote and local data, compares them to identify unmatched data, estimates the splitting cost, and sends requests to a server based on a threshold to determine whether to send data to a storage controller, optimizing memory usage and transmission decisions.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If current data deduplication methods are used to identify redundant data files, then data deduplication can be performed, but significant memory resources are required
Solution Approach 1:
The patent segments the data comparison process into multiple phases: first comparing hash values to identify potential matches, then selectively performing full data comparisons only for candidates that pass the hash filter. This segmentation reduces memory requirements by avoiding loading all data into memory simultaneously while maintaining deduplication effectiveness through multi-stage filtering.
Solution Approach 2:
The patent introduces hash values as an intermediary mechanism between the original data files. Instead of directly comparing large data files in memory, the system first compares compact hash representations, which serve as a memory-efficient filter. Only when hash values match does the system proceed to full data comparison, significantly reducing overall memory consumption while preserving deduplication accuracy.
2Reliability
If all unmatched hash data is transmitted to the server, then complete data management is achieved, but network transmission volume increases
Solution Approach 1:
The patent applies partial action by transmitting only a subset of unmatched hash data to the server rather than all unmatched data. The system performs local filtering and preliminary analysis to identify the most critical unmatched hash entries, then transmits only those to the server. This approach maintains data management completeness for critical data while reducing overall network transmission volume through selective partial transmission.
Solution Approach 2:
The patent implements local quality by performing different levels of data management at different locations. Local systems perform initial hash comparison and filtering, transmitting only specific types of unmatched data to the server. The server then performs centralized deduplication on this filtered subset. This distributed approach with varying local quality levels reduces network traffic while maintaining overall system effectiveness.
Data Source
AI summary
A computer controls deduplication of data. The computer generates a hash of a remote data and a hash of a local data. The computer generates a set of unmatched hash data based on a comparison of the hash of the remote data against the hash of the local data. The computer generates a splitting cost that is associated with splitting the set of unmatched hash data. The computer sends a request to a server based on a comparison of the splitting cost to a threshold. The request dictates sending of the remote data to a storage controller.