Chunk Store and Manifest Reference for Data Deduplication
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing data backup methods inefficiently store duplicate data, leading to excessive use of storage medium capacity as the same data is repeatedly backed up, especially during full backups.
Innovation Solution
A data processing apparatus comprising a chunk store and a manifest store that processes input data into chunks, identifies corresponding specimen data chunks, and generates chunk identifiers to efficiently store and retrieve data by referencing existing specimen data chunks, thereby avoiding redundant storage.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If full back-ups are carried out, then data recovery capability is improved, but storage medium capacity is excessively consumed due to duplicate data
Solution Approach 1:
The patent divides data into discrete chunks and stores them in a chunk store, with manifests referencing these chunks. This segmentation allows the system to identify and store only unique data portions, avoiding duplication while maintaining complete data recovery capability through the manifest references.
Solution Approach 2:
The patent creates references to data chunks rather than storing complete copies of all data. The manifest store contains references (pointers) to actual data chunks in the chunk store, allowing multiple data sets to reference the same underlying data without physical duplication, thus reducing storage capacity requirements.
2Adaptability or versatility
If data is stored numerous times in backup, then data recovery at specified points in time is enabled, but storage efficiency deteriorates
Solution Approach 1:
The patent creates a universal indexing mechanism where manifests serve multiple purposes: they reference data chunks for storage efficiency, enable recovery at any specified point in time, and provide a history of data changes. This multi-functional approach allows the system to maintain storage efficiency while supporting versatile recovery operations.
Solution Approach 2:
The patent performs preliminary processing of data into chunks and creation of manifests before actual backup operations. This preliminary action establishes an efficient storage structure in advance, allowing subsequent backup operations to simply reference existing chunks rather than storing new data, thereby maintaining high storage efficiency while enabling point-in-time recovery.
Data Source
AI summary
Data processing apparatus comprising: a chunk store configured for containing specimen data chunks, a discrete manifest store, configured for containing at least one manifest that represents at least a part of a data set and that comprises at least one reference to at least one of said specimen data chunks, the processing apparatus being operable to process input data into input data chunks and to identify at least one manifest in the manifest store that includes at least one said reference to a said specimen data chunk corresponding to at least one of said input data chunks.