Deduplication Chunk Retention via Reference Measurements
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Deduplication systems face inefficiencies in managing and retaining dereferenced chunks, leading to increased storage and transmission bandwidth usage due to the need to retransmit and recreate index entries for chunks that are later added to data objects.
Innovation Solution
Implementing a deduplication system that maintains reference measurements for chunks, allowing for the selection and retention of chunks based on criteria such as reference counts, recent usage, and accumulated references to minimize storage and transmission bandwidth by identifying and removing dereferenced chunks that are unlikely to be referenced again.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Volume of stationary object
If dereferenced chunks are removed from storage space, then storage capacity is freed up, but bandwidth usage increases due to retransmission needs
Solution Approach 1:
The system performs preliminary actions by maintaining dereferenced chunks in storage and tracking their reference status through measurement values. Instead of immediately removing chunks when dereferenced, the system proactively monitors and predicts future reference needs, keeping chunks available to avoid retransmission costs.
Solution Approach 2:
The system implements feedback mechanisms by continuously measuring reference levels of chunks and using these measurements to make informed decisions about chunk retention. The measurement values provide feedback on chunk usage patterns, enabling the system to optimize storage allocation and minimize retransmission bandwidth usage.
2Loss of energy
If dereferenced chunks are retained in storage, then bandwidth usage is reduced by avoiding retransmission, but storage capacity is consumed
Solution Approach 1:
The system changes parameters by introducing measurement values that quantify the reference status of chunks. These measurement values serve as dynamic parameters that guide retention decisions, allowing the system to adapt storage allocation based on observed reference patterns rather than using static retention policies.
Solution Approach 2:
The system implements a selective discarding and recovering mechanism where dereferenced chunks are temporarily retained and monitored. When measurement values indicate low probability of future reference, chunks are discarded to free storage. When references are detected, chunks are recovered or retransmitted, optimizing the balance between storage usage and bandwidth consumption.
3Measurement precision
If reference measurements are maintained for all chunks, then chunk selection accuracy improves, but system complexity increases
Solution Approach 1:
The system applies local quality by maintaining measurement values selectively for specific chunks based on their reference status and importance, rather than uniformly tracking all chunks. This localized approach to measurement maintains accuracy for critical chunks while reducing overall system complexity.
Data Source
AI summary
A chunk index has information on chunks in a storage space referenced in objects in the storage space. The chunk index includes a reference count for each chunk indicating a number of objects in which the chunk is referenced and a reference measurement representing a level of data object references to the chunk. One chunk is selected to remove from the storage space based on a criteria applied to the reference measurements of chunks having reference counts indicating that the chunks are not referenced in one object in the storage space.