Data Processing Apparatus Sparse Chunk Index Deduplication
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing data backup methods inefficiently store duplicate data, leading to excessive use of storage medium capacity due to repeated storage of identical data parts during backup processes.
Innovation Solution
A data processing apparatus comprising a chunk store, a manifest store, and a sparse chunk index that identifies and stores only unique data chunks with a predetermined characteristic, using hash identifiers to deduplicate data by referencing unique specimen data chunks and populating the sparse chunk index with information on these chunks.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If full backups are carried out repeatedly, then data recovery capability is improved, but storage medium capacity is wasted due to repeated storage of identical data parts
Solution Approach 1:
The patent segments data into fixed-size chunks and creates a chunk index structure that tracks unique data blocks across multiple backups. Instead of storing complete duplicate datasets, the system divides data into manageable segments (chunks) and maintains only unique segments in the backup repository, with an index mapping logical chunk identifiers to physical storage locations.
Solution Approach 2:
The patent implements a copying mechanism where instead of storing actual data chunks repeatedly, it stores references (pointers) to unique chunks. The backup system creates copies of chunk index entries rather than data copies, allowing multiple logical representations of the same physical data blocks, thereby reducing storage consumption while maintaining data recovery capability.
2Quantity of substance
If incremental backups are performed, then storage capacity is reduced compared to full backups, but processing time and complexity increase due to change detection requirements
Solution Approach 1:
The patent performs preliminary hashing of data chunks during the backup process to generate chunk identifiers before actual storage operations. By pre-computing hash values and comparing them against the existing chunk index, the system quickly identifies unchanged data blocks without needing to perform complex change detection algorithms, thereby reducing processing time while maintaining incremental backup efficiency.
3Speed
If all data chunks are indexed, then data retrieval speed is improved, but index size and memory requirements increase significantly
Solution Approach 1:
The patent extracts only the essential indexing information (chunk identifiers and physical location pointers) from the actual data chunks. Instead of indexing entire data blocks or complex metadata, the system maintains a minimal index structure that contains only the necessary references to locate unique chunks, significantly reducing index size while preserving fast retrieval capability through efficient hash-based lookup.
Data Source
AI summary
Data processing apparatus comprising: a chunk store containing specimen data chunks, a manifest store containing at least one manifest that represents at least a part of a data set and that comprises at least one reference to at least one of said specimen data chunks, a sparse chunk index containing information on only those specimen data chunks having a predetermined characteristic, the processing apparatus being operable to process input data into input data chunks and to use the sparse chunk index to identify at least one of said at least one manifest that includes at least one reference to one of said specimen data chunks that corresponds to one of said input data chunks having the predetermined characteristic.