Content-Aware Deduplication for Reversibly-Transformed Data
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Conventional deduplication systems are unable to recognize and deduplicate files or objects that are closely related from an end-user information content perspective, such as plain text, compressed, and encrypted versions, as they have different bit patterns and are thus treated as distinct data.
Innovation Solution
A content-aware deduplication system that calculates hash values for data, maintains object and deduplication maps to identify and convert between plain, compressed, and encrypted data forms, allowing for deduplication of reversibly-transformed versions of the same data by tracking conversion steps and updating maps accordingly.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Quantity of substance
If conventional pattern matching deduplication is used, then storage space is reduced for identical files, but related data forms (compressed, encrypted versions) cannot be recognized as duplicates
Solution Approach 1:
The system performs preliminary actions by converting data to a standardized form before hashing and comparison. This preprocessing step transforms compressed, encrypted, or formatted data into a common representation, enabling the hash function to recognize related data forms as duplicates. The conversion happens in advance of the deduplication comparison, allowing the system to identify relationships that would otherwise be hidden by different bit patterns.
Solution Approach 2:
The invention changes the parameter of data representation by transforming data from its original form (compressed, encrypted, or formatted) into a standardized form. This parameter change occurs through conversion processes that normalize the data before hashing, allowing the system to recognize that different representations (e.g., compressed vs. uncompressed) of the same underlying information should be treated as duplicates.
2Quantity of substance
If data is converted to standard form for deduplication, then related data forms can be deduplicated, but additional processing steps are required
Solution Approach 1:
The system introduces an intermediary conversion process that acts as a mediator between the original data forms and the hashing operation. This intermediary step transforms diverse data representations into a standardized intermediate form that can be efficiently hashed and compared. The mediator handles the complexity of format detection and conversion, shielding the core deduplication logic from dealing with multiple data formats directly.
3Productivity
If hash values are calculated on original data, then processing is simple, but compressed and encrypted versions are treated as different data
Solution Approach 1:
The system performs preliminary conversion of data to standard form before calculating hash values. This ensures that compressed, encrypted, or formatted data is transformed into a recognizable standard representation, allowing the hash function to produce identical values for related data forms. The preliminary action maintains processing efficiency while enabling proper deduplication recognition.
Data Source
AI summary
A method for writing data to a storage device including a processor, a volatile memory, and a non-volatile memory, the storage device being in communication with a host, the method includes receiving, by the processor, a command to write host data to the non-volatile memory, the host data being associated with a host identification, calculating, by the processor, a hash value associated with the host data, adding, by the processor, an object entry in an object map associated with the hash value, the object entry including the host identification and the hash value, and identifying, by the processor, whether there is a dedup entry in a deduplication map associated with the hash value, and writing, by the processor, stored data to the non-volatile memory based on the identification, the stored data being associated with and different from the host data.