Reference Block Management in Data Deduplication
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing storage systems face inefficiencies due to the storage of multiple redundant data copies, outdated reference blocks, and challenges in managing storage space, leading to increased costs, latency, and reduced memory capacity.
Innovation Solution
A non-volatile cache memory system with a buffer memory storing comparison blocks and a storage controller that performs data deduplication and garbage collection, maintaining hit counts for comparison blocks and deleting outdated ones, while relocating valid user data blocks to optimize storage usage.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If multiple copies of the same data are stored in the storage network, then data loss prevention is improved, but storage resource utilization deteriorates
Solution Approach 1:
The patent implements data deduplication by storing only unique data blocks and replacing redundant copies with references or pointers to the original block. When data is written to storage, the system compares incoming data blocks against existing blocks using hash functions or other comparison algorithms. If a match is found, the redundant block is not stored; instead, a reference to the existing block is created. This maintains data availability (multiple references point to the same data) while eliminating actual redundant storage consumption.
2Quantity of substance
If exact match algorithms are used to identify redundant data, then storage efficiency is improved, but similarity detection capability deteriorates
Solution Approach 1:
The patent employs hash functions that transform data blocks into fixed-size representations, enabling efficient comparison. By adjusting hash algorithm parameters and using different hash functions, the system can balance between exact match performance and similarity detection. Some implementations use rolling hashes or checksums that can detect similar but not identical blocks, allowing the system to adapt to different deduplication requirements based on the specific use case and data characteristics.
3Measurement precision
If reference blocks are retained in limited storage space, then match determination accuracy is improved, but space availability for updated reference blocks deteriorates
Solution Approach 1:
The patent implements reference block management through eviction policies and aging mechanisms. When the reference block storage reaches capacity, the system identifies and removes outdated or least recently used reference blocks to make space for new ones. This is done through garbage collection processes that track reference block usage and automatically purge obsolete entries. The system maintains a dynamic balance between retaining enough reference blocks for accurate matching and freeing space for updated blocks, ensuring continuous deduplication effectiveness.
Data Source
AI summary
Various aspects for managing data blocks in a storage system are provided. For instance, a method may include storing, in a buffer memory, a plurality of comparison blocks, initiating a data deduplication process utilizing the plurality of comparison blocks, and performing garbage collection in conjunction with the data deduplication process. Garbage collection may include maintaining a hit count for comparison blocks of a passive set of comparison blocks in the buffer memory and deleting the passive set from the buffer memory when the hit count is decremented to a predetermined value. The hit count may be incremented and decremented based on utilization of a comparison block in the data deduplication process.