Weak References in Deduplication Hash Tables for Data Integrity
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Identifying and handling redundant data stored by a file system is resource-intensive and may introduce additional complexity to read and write operations, with existing deduplication techniques potentially causing data corruption if not properly implemented.
Innovation Solution
The use of a deduplication hash table with weak references to associate block hashes and block addresses, where a weak reference is generated on a block when added to the hash table, and is broken or removed upon modification, ensuring data integrity by confirming the data has not changed before using existing blocks for deduplication.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Loss of substance
If deduplication techniques are implemented to identify and handle redundant data, then data storage efficiency is improved, but code complexity increases and file system performance deteriorates
Solution Approach 1:
A deduplication hash table is introduced as an intermediary data structure to store associations between block hashes and block addresses. This hash table serves as a mediator that automatically identifies duplicate blocks without requiring complex comparison logic, thereby reducing code complexity while maintaining deduplication functionality.
Solution Approach 2:
Instead of implementing complex deduplication logic that compares and analyzes data blocks directly, the patent uses hash copies of block data as keys in the hash table. This copying approach allows for efficient identification of duplicate blocks through simple hash lookups rather than complex data comparison, improving performance while simplifying the deduplication mechanism.
2Loss of substance
If deduplication techniques are implemented to identify and handle redundant data, then data storage efficiency is improved, but file system performance deteriorates
Solution Approach 1:
The deduplication hash table is built in advance by scanning and hashing all existing blocks before normal file system operations resume. This preliminary action pre-computes hash associations, so that subsequent deduplication checks during read/write operations can proceed quickly through simple hash lookups rather than complex data analysis, thereby maintaining file system performance.
Solution Approach 2:
The patent replaces complex mechanical data comparison operations with hash-based lookup operations. Instead of comparing actual block data to identify duplicates (which is computationally intensive), the system substitutes this with a faster hash table lookup mechanism that compares hash values, significantly improving file system performance during deduplication operations.
3Reliability
If hash table entries are maintained contemporaneously to ensure data integrity, then data corruption is prevented, but overhead increases and performance suffers
Solution Approach 1:
Instead of maintaining hash table entries contemporaneously with every data modification (which creates excessive overhead), the patent uses a periodic approach where the hash table is rebuilt or updated at scheduled intervals or triggered by specific events. Weak references allow the system to defer full hash table maintenance until necessary, reducing overhead while still ensuring data integrity when deduplication operations are performed.
Solution Approach 2:
The weak reference mechanism enables the hash table to effectively self-manage its own validity. When a block is modified, the weak reference is automatically broken, providing a self-service indication that the corresponding hash table entry may be stale. This eliminates the need for complex contemporaneous maintenance logic, reducing overhead while preserving data integrity through automatic detection of changed blocks.
4Reliability
If weak references are used to track block changes, then data corruption is prevented, but additional data structures are required
Solution Approach 1:
The weak reference mechanism is merged with the existing reference count data structure that already tracks block usage. By integrating the weak reference functionality into the existing metadata infrastructure, the patent avoids adding completely separate data structures. The weak reference information is combined with the reference counting mechanism, reducing overall data structure complexity while maintaining the ability to track block changes for data integrity.
Data Source
AI summary
Aspects of the present disclosure relate to weak references of allocated logical clusters. In examples, a deduplication hash table stores associations between block hashes and block addresses, such that a block can be identified and used in place of another block storing the same data. A weak reference may be generated on a block, such that an intact weak reference indicates that the block is unchanged. When a block is identified that has a hash matching that of an existing block in the hash table, it may be determined whether a weak reference on the existing block is intact. If there is not an intact weak reference on the existing block, the existing block may have changed, such that it is not used for deduplication. By contrast, if the weak reference is intact, the existing block has not changed, such that it may be used for deduplication accordingly.