Chunk Store Data Deduplication for Backup Storage Efficiency
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing data backup strategies inefficiently store duplicate data, leading to wasteful use of storage medium capacity, as the same data is often stored multiple times, especially during full backups.
Innovation Solution
A data processing apparatus that partitions data into chunk sections, identifies and stores specimen data chunks efficiently by relocating them to other sections based on predetermined characteristics such as fragmentation, non-contiguity, and reference frequency, thereby avoiding redundant storage and optimizing storage capacity.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If data is backed up using traditional full backup strategies, then data recovery capability is ensured, but storage medium capacity is wasted due to redundant duplicate data storage
Solution Approach 1:
The patent segments data into fixed-size chunks and organizes them in a chunk store with multiple chunk sections. Each chunk is independently addressable and can be stored once and referenced multiple times across different backups, eliminating redundant storage while maintaining complete data recovery capability through manifest files that track chunk locations.
Solution Approach 2:
Instead of copying entire data sets for each backup, the patent creates references (pointers) to existing chunks in the chunk store. When data is backed up, only new or changed chunks are added to the chunk store, and manifest files are updated with references to these chunks, dramatically reducing storage requirements while preserving full backup functionality.
2Adaptability or versatility
If specimen data chunks are stored in fragmented chunk sections, then storage flexibility is improved, but data restoration efficiency deteriorates due to multiple read/write operations
Solution Approach 1:
The patent performs preliminary consolidation of specimen chunks during the backup process. When a chunk section becomes fragmented (exceeds a threshold number of chunks), the system proactively consolidates these chunks into a dedicated specimen chunk section before restoration is needed, ensuring optimal read performance is maintained without impacting backup flexibility.
Solution Approach 2:
The patent implements dynamic management of chunk sections where sections can transition between containing general backup chunks and consolidated specimen chunks. The system dynamically monitors fragmentation levels and automatically triggers consolidation operations when thresholds are exceeded, adapting the storage structure to maintain performance while preserving flexibility.
Data Source
AI summary
Data processing apparatus comprising: a chunk store partitioned into a plurality of chunk sections, at least one section storing specimen data chunks, the processing apparatus being operable to: process input data into one or more input data chunks; identify a chunk section already containing a specimen data chunk corresponding to at least one input data chunk; and store the at least one input data chunk in another chunk section as a specimen data chunk if the identified chunk section has a predetermined characteristic.