Collaborative Storage De-duplication via Host Pre-calculation
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current methods for redundancy determination in data de-duplication either require significant communication overhead between host computers and storage apparatuses, leading to performance issues, or burden the host computer with large information storage needs, making de-duplication inefficient as data volume increases.
Innovation Solution
A storage system that collaborates between an external apparatus and a storage apparatus, using a backup apparatus to determine redundancy in data storage, where the backup apparatus and storage apparatus share redundancy information to efficiently manage data de-duplication, reducing the need for repeated data transmission and storage.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If the host computer requests the storage apparatus to make redundancy determination for all chunks, then the storage apparatus can accurately determine redundant data, but the performance is lowered due to data roundtrip between host computer and storage apparatus
Solution Approach 1:
The host computer performs redundancy determination in advance before data transmission to the storage apparatus. By pre-calculating which chunks are redundant and which are new, the host computer eliminates the need for post-transmission redundancy checks and roundtrip communications, thereby maintaining accuracy while improving processing speed
Solution Approach 2:
The host computer acts as an intermediary that performs redundancy determination locally using its own resources and previously stored index information. This intermediary function eliminates the need for the storage apparatus to perform exhaustive redundancy checks on all incoming chunks, reducing the communication overhead and processing time
2Ease of operation
If the host computer stores information for use in redundancy determination for all data, then the host computer can perform local redundancy determination, but the disk capacity burden on the host computer increases as stored data increases
Solution Approach 1:
The patent extracts and transmits only the necessary redundancy determination results (information about which chunks are redundant and which are new) from the host computer to the storage apparatus, rather than transferring all stored data. This extraction approach enables the host computer to maintain local determination capability while minimizing the data volume transmitted and stored, thereby reducing the disk capacity burden
Data Source
AI summary
In a storage system for backing up data of an external apparatus, the external apparatus and a storage apparatus collaboratively perform efficient de-duplication. A storage system stores data from the external apparatus in a unit of content, and includes a backup apparatus configured to execute backup processing to create backup data of the data from the external apparatus in the unit of content; and a storage apparatus coupled to the backup apparatus in a communication-enabled manner and configured to store the backup data received from the backup apparatus. A first backup processing part of the backup apparatus determines whether or not a content is already stored in the storage apparatus by using first redundancy determination information that is information for determining whether or not each of contents of the backup data is already stored in the storage apparatus.