Storage Deduplication Using Host Fingerprints and Range Queries
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing deduplication methods in storage systems are inefficient due to the need for frequent inquiries to the storage apparatus for duplication determination, leading to increased processing load and reduced system performance as the number of host computers increases.
Innovation Solution
A computer system where the host computer and storage apparatus cooperate to minimize data transfer by calculating and managing identification information for deduplication, allowing the host to determine duplicate data without relying on the storage apparatus for every chunk, thereby reducing the load on the storage system and improving deduplication processing efficiency.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If duplication determination is conducted on the storage apparatus for every chunk, then the host computer can receive accurate duplication results, but the system performance drops due to increased round-trip data communication and processing load on the storage apparatus
Solution Approach 1:
The patent divides the deduplication system into two segments: the host computer performs initial duplication determination using fingerprints, while the storage apparatus performs final verification using full hash values. This segmentation allows the host to handle most deduplication decisions locally, reducing communication overhead, while the storage apparatus performs precise verification only for borderline cases, thus resolving the contradiction between measurement precision and productivity.
Solution Approach 2:
The patent introduces fingerprints as an intermediary mechanism between the host computer and storage apparatus. Instead of directly comparing full data chunks, the system uses lightweight fingerprints for initial matching, which significantly reduces communication data volume and processing load while maintaining adequate accuracy for most deduplication decisions.
2Measurement precision
If the host computer transmits information for duplication determination to the storage apparatus for every chunk, then the storage apparatus can perform accurate duplication checks, but the amount of data transferred increases
Solution Approach 1:
The patent extracts the essential identifying feature (fingerprint) from the complete data chunk and uses only this extracted feature for initial duplication determination. This allows the host computer to perform deduplication checks without transmitting the entire chunk data to the storage apparatus, dramatically reducing data transfer volume while maintaining sufficient accuracy for most deduplication decisions.
3Reliability
If an inquiry is made to the storage apparatus for every chunk's duplication determination, then complete deduplication coverage is achieved, but the processing load on the storage apparatus increases
Solution Approach 1:
The patent applies partial action by having the host computer perform the majority of duplication determination work using fingerprints, and only invoking the storage apparatus for verification when needed. This partial engagement of the storage apparatus significantly reduces its processing load while maintaining reliable deduplication coverage through the two-stage verification approach.
Data Source
AI summary
It is provided a computer system comprising at least one storage apparatus and a computer, wherein the each of the at least one storage apparatus is configured to manage identification information indicating specifics of the stored data, and wherein the computer determines whether the data to be written to the one of the at least one storage apparatus has duplicate data, which is the same data already stored in any one of the at least one storage apparatus, transmits deduplicated data, and uses at least one of individual pieces of identification information or a range of pieces of identification information, depending on how many pieces of identification information appear in succession, to request the information indicating whether the data that is associated with the calculated identification information is stored from the one of the at least one storage apparatus.