Sparse Chunk Index Deduplication for Backup Storage
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing data storage methods are inefficient due to repeated storage of identical data during backup processes, leading to unnecessary usage of storage medium capacity.
Innovation Solution
A data processing apparatus comprising a chunk store, a manifest store, and a sparse chunk index, which processes input data into segments, identifies and prioritizes manifests referencing specimen data chunks, and compares input data chunks with these to deduplicate and efficiently store data, using a sparse chunk index to reduce storage redundancy.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If full backups are performed regularly to ensure data recovery capability, then data reliability is improved, but storage capacity consumption increases due to repeated storage of identical data chunks
Solution Approach 1:
The patent segments data into fixed-size chunks and creates a chunk index that tracks unique data chunks across multiple backups. By segmenting the backup process into chunk-level operations rather than file-level or full-disk operations, the system can identify and eliminate redundant data chunks while maintaining complete data recovery capability.
Solution Approach 2:
The patent creates a virtual copy mechanism where instead of physically storing duplicate data chunks multiple times, it maintains references to unique chunks in a chunk index. The backup system copies only the metadata (chunk indices) rather than the actual data chunks, allowing multiple backup versions to reference the same underlying data without duplicating storage.
2Quantity of substance
If incremental backups are used to reduce storage consumption, then storage capacity efficiency is improved, but data reliability may be compromised if previous backup versions are needed
Solution Approach 1:
The chunk index serves multiple functions simultaneously: it tracks unique data chunks for deduplication purposes, maintains version history for recovery operations, and enables both incremental and full backup strategies. This universal data structure supports multiple backup modes without sacrificing either storage efficiency or data recovery completeness.
Solution Approach 2:
The system implements feedback mechanisms where the chunk index continuously tracks which data chunks have been backed up and their locations. This feedback information is used to determine what new data needs to be backed up in subsequent operations, ensuring that incremental backups maintain complete recovery capability while optimizing storage usage through intelligent deduplication.
3Quantity of substance
If all data chunks are indexed to enable complete tracking and deduplication, then storage efficiency is improved, but processing time and computational resources increase
Solution Approach 1:
The patent applies partial indexing where not every single data chunk requires a full index entry. Instead, the system uses a combination of full chunk indexing for critical data and reference-based tracking for redundant chunks. This partial action approach achieves effective deduplication while reducing the computational overhead of maintaining complete indexes for all data.
Solution Approach 2:
The system performs preliminary hashing and chunk identification during the data ingestion phase, preparing chunk signatures and metadata before actual backup operations. This preliminary action allows the backup process to quickly compare incoming data against existing chunks without performing expensive full comparisons, significantly reducing processing time while maintaining deduplication effectiveness.
Data Source
AI summary
One embodiment is a data processing apparatus that has a chunk store containing specimen data chunks, a manifest store containing a plurality of manifests, each of which represents at least a part of previously processed data and includes at least one reference to at least one of the specimen data chunks, and a sparse chunk index containing information on only some specimen data chunks. Input data is processed into a plurality of input data segments. Each manifest of the first set has at least one reference to one of said specimen data chunks that corresponds to one of the input data chunks of a first input data segment. Specimen data chunks corresponding to other input data chunks of the first input data segment are identified by using the identified first set of manifests and at least one manifest identified when processing previous data.