CKD Hash Table Deduplication for Variable-Length Records
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Variable-length allocation units, such as count key data (CKD) records, are ill-suited for deduplication in data storage systems due to unique information they contain, which complicates the process of reducing storage requirements.
Innovation Solution
A data storage system that includes a compute node configured to store meta data in a first data structure and user data in a second data structure, using a CKD hash table to reversibly decouple unique information from user data, enabling deduplication by creating or updating entries based on hash keys for meta and user data.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Adaptability or versatility
If variable length allocation units such as CKD records are used in data storage systems, then storage flexibility and adaptability are improved, but deduplication effectiveness deteriorates due to unique information contained in the records
Solution Approach 1:
The CKD record is divided into two separate components: a hash table entry containing only the hash key (derived from unique identifiers like cylinder-head-sector) and a data structure containing the actual user data. This segmentation allows the unique identifying information to be separated from the user data, enabling deduplication of the data while preserving the flexibility of variable-length records.
Solution Approach 2:
A hash table is introduced as an intermediary structure between the unique identifiers in CKD records and the user data. The hash table maps hash keys to data structures, serving as a mediator that enables the system to recognize duplicates without requiring the entire CKD record to be identical, thus resolving the contradiction between flexibility and deduplication effectiveness.
2Quantity of substance
If traditional deduplication methods are applied to CKD records, then storage reduction is attempted, but the process becomes complex and inefficient due to the unique information in each record
Solution Approach 1:
The unique identifying information (hash key) is extracted from the CKD record and placed in a separate hash table structure. This extraction simplifies the deduplication process by allowing the system to compare only the hash keys for duplication detection, rather than comparing entire CKD records, thereby reducing the complexity of the deduplication process while achieving storage reduction.
3Reliability
If meta data and user data are stored together in CKD records, then data integrity is maintained, but deduplication efficiency is reduced due to the coupling of unique and duplicable information
Solution Approach 1:
The CKD record structure is segmented into a hash table entry (containing only the hash key for integrity verification) and a data structure (containing user data for deduplication). This segmentation allows the system to maintain data integrity through the hash key while improving deduplication efficiency by comparing only the user data portions, resolving the contradiction between integrity and efficiency.
Data Source
AI summary
The meta data containing count and key fields of CKD records are reversibly decoupled from the user data of the data field so that the data can be deduplicated. Multiple CKD records may be coalesced into a larger size CKD track. The coalesced meta data is compressed and stored in a CKD hash table. The user data is hashed, and the hash is used as a hash key that is associated with the compressed meta data in the CKD hash table. When the hash of user data associated with a CKD write IO matches the hash key of an existing entry in the table, data duplication is indicated. The compressed meta data is added to the entry and the user data is deduplicated by creating storage system meta data that points to the pre-existing copy of the user data. The storage system metadata includes unique information that enables the corresponding compressed metadata to be subsequently located in the hash table to reassemble the CKD records.