De-duplicated Storage Reference Count Minimization
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Conventional de-duplicated data storage systems face inefficiencies in maintaining reference counts and performing garbage collection, which can be time-consuming and resource-intensive, especially when dealing with large data sets.
Innovation Solution
The system minimizes reference count updates by only adding or removing reference counts for unique data chunks and eliminates the need for garbage collection by storing de-duplicated chunks without updating reference counts for most chunks, using a manifest file to track fingerprints and manage chunk presence in storage.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If reference counts are maintained for all data chunks in conventional de-duplication systems, then data can be properly tracked and removed when no longer needed, but the time and computing resources required for backup operations increase significantly
Solution Approach 1:
The patent extracts the reference count maintenance function from the backup operation. Instead of updating reference counts for all chunks during backup, the system uses a separate mechanism where only chunks not present in previous backups need tracking. This is achieved by maintaining a set of unique chunks and only updating reference counts for those, rather than all chunks in the backup set.
Solution Approach 2:
The patent applies partial action by only updating reference counts for a subset of chunks (those not already in previous backups) rather than all chunks. This partial update approach reduces the computational burden while still maintaining accurate tracking for chunks that need to be removed later.
2Reliability
If reference counts are updated for all data chunks during each backup operation, then complete tracking is maintained, but computing resources are consumed excessively
Solution Approach 1:
The system extracts the essential tracking function from full reference count maintenance. By separating the tracking mechanism and only maintaining reference counts for chunks that need to be removed (those not in previous backups), the system reduces computing resource consumption while preserving necessary tracking accuracy.
Solution Approach 2:
Instead of updating reference counts for all chunks (excessive action), the patent updates only for the partial set of chunks that are candidates for removal. This selective updating approach maintains sufficient reliability while dramatically reducing computing resource consumption.
3Quantity of substance
If garbage collection is performed to remove unreferenced data chunks, then storage efficiency is improved, but the process can take several hours or days to complete
Solution Approach 1:
The patent applies preliminary action by identifying and removing unreferenced chunks at the time of backup operations rather than waiting for a separate garbage collection phase. When a backup is performed, the system immediately identifies chunks that will not be referenced by future backups and removes them, eliminating the need for time-consuming post-backup garbage collection.
Solution Approach 2:
The system makes garbage collection continuous by integrating the removal of unreferenced chunks into the normal backup operation flow. Instead of a periodic or batch garbage collection process, the action occurs continuously during each backup when chunks are added or removed, eliminating idle time and extending the useful action throughout the system operation.
Data Source
AI summary
Systems and methods for adding backups to de-duplicated storage and for removing backups from de-duplicated storage are disclosed. Backups can be added to the de-duplicated storage with minimal reference count updates. Backup data sets can be removed without garbage collection processes being performed on the de-duplicated storage.