Serial-Access Storage De-duplication via Hash-Based Chunk Tracking
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing data de-duplication methods are inefficient for serial-access storage media like tape drives due to the time-consuming process of repositioning the media to retrieve full chunk data and the requirement for large, expensive cache memory to avoid repositioning.
Innovation Solution
A data storage method and apparatus that uses a chunk processing subsystem to identify and replace duplicate data chunks with hashes, storing the first occurrence of each chunk and its hash on a serial-access medium, along with a database to track chunk locations, allowing for efficient reconstruction of the original data stream during retrieval without extensive repositioning.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Quantity of substance
If data de-duplication is applied to serial-access storage media, then storage efficiency is improved, but media repositioning time increases
Solution Approach 1:
The patent applies preliminary action by storing the first occurrence of each data chunk and its hash value in advance during the writing phase. During retrieval, the hash value allows immediate identification of duplicate chunks without needing to reposition the media to compare entire chunks, thus resolving the contradiction between storage efficiency and repositioning time.
Solution Approach 2:
The patent extracts the essential identifying feature (hash value) from the complete data chunk and stores it separately. This extraction allows the system to identify duplicates using only the compact hash rather than retrieving and comparing full chunks, eliminating media repositioning for duplicate detection while maintaining storage efficiency.
2Speed
If all chunk data is stored in random access cache memory to avoid media repositioning, then data retrieval speed is improved, but cache memory size and cost increase
Solution Approach 1:
The patent creates a simplified copy (hash value) of the chunk data that serves as a unique identifier. Instead of storing entire chunks in cache, the system stores only these compact hash copies, which occupy minimal cache space while still enabling rapid duplicate identification and retrieval speed improvement.
Solution Approach 2:
The patent transforms the chunk data into a different parameter representation (hash value) that is much more compact. This parameter change allows the system to maintain retrieval speed by working with small hash values in cache rather than large chunk data, thus reducing cache memory requirements while preserving performance.
3Reliability
If the same data chunk occurs at both beginning and end of subject data, then data completeness is improved, but media repositioning requirements increase
Solution Approach 1:
The patent introduces the hash value as an intermediary that bridges the beginning and end of the data stream. When a chunk appears at both beginning and end, the hash stored during the first occurrence allows the system to immediately recognize and reconstruct the duplicate at the end without repositioning the media to compare full chunks, thus maintaining data completeness while eliminating repositioning time.
Data Source
AI summary
Data storage and retrieval methods and apparatus are provided for facilitating data de-duplication for serial-access storage media such as tape. During data storage, input data is divided into a succession of chunks and, for each chunk, a corresponding data item is written to the storage media. The data item comprises the chunk data itself where it is the first occurrence of that data, and otherwise comprises a chunk-data identifier identifying that chunk of subject data. To facilitate reconstruction of the original data on read-back from the storage media a cache (50) is used together with a database (35R), stored on the media, that includes for each duplicated chunk, the location of the corresponding chunk of subject data.