Chunk Store Backup and Restore for Data Deduplication
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Data deduplication techniques face challenges such as latency in accessing files, resource consumption issues, and scalability problems due to data fragmentation, especially in primary storage where seamless and fast access is expected, and difficulties in backing up and restoring deduplicated data.
Innovation Solution
A method for efficiently backing up and restoring optimized data streams by using a chunk store that includes optimized data streams with metadata mapping identifiers to data chunks, allowing for various backup techniques such as optimized, un-optimized, item-level, and data chunk identifier backups, and utilizing heuristics to select the most appropriate technique based on data stream characteristics.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Quantity of substance
If data is stored using deduplication techniques to reduce storage capacity, then storage hardware costs are reduced, but latency in accessing files increases
Solution Approach 1:
The patent segments deduplicated data into discrete chunks with unique identifiers, allowing individual chunks to be accessed independently. This segmentation enables the system to retrieve only specific data portions rather than scanning through entire deduplicated structures, thereby reducing access latency while maintaining storage efficiency.
Solution Approach 2:
The patent introduces an intermediary indexing structure that maps data chunk identifiers to their physical locations on storage devices. This intermediary layer acts as a bridge between logical data access requests and physical storage locations, eliminating the need to traverse fragmented deduplicated data structures and significantly reducing access latency.
2Quantity of substance
If data deduplication is performed on a file server to eliminate redundancy, then storage efficiency improves, but device resources such as memory, I/O mechanisms, and CPU capacity are over-consumed
Solution Approach 1:
The patent performs preliminary data deduplication and chunking during the data write operation, before data needs to be read or accessed. By pre-processing data into deduplicated chunks with identifiers during ingestion, the system eliminates the need for resource-intensive real-time deduplication during read operations, thereby reducing overall device resource consumption while maintaining storage efficiency.
3Reliability
If traditional backup methods are used on deduplicated data, then backup completeness is achieved, but backup time increases and restore speed decreases
Solution Approach 1:
The patent implements a backup mechanism that copies only the unique data chunks and their identifiers rather than entire deduplicated data structures. This selective copying approach ensures backup completeness by capturing all unique data portions while dramatically reducing backup time and restore speed requirements, as the backup process operates on compact chunk representations rather than full data sets.
Data Source
Figure 1
Figure 2
Figure 3
AI summary
Techniques for backup and restore of optimized data streams are described. A chunk store includes each optimized data stream as a plurality of chunks including at least one data chunk and corresponding optimized stream metadata. The chunk store includes data chunks in a deduplicated manner. Optimized data streams stored in the chunk store are identified for backup. At least a portion of the chunk store is stored in backup storage according to an optimized backup technique, an un-optimized backup technique, an item level backup technique, or a data chunk identifier backup technique. Optimized data streams stored in the backup storage may be restored. A file reconstructor includes a callback module that generates calls to a restore application to request optimized stream metadata and any referenced data chunks from the backup storage. The file reconstructor reconstructs the data streams from the referenced data chunks.