NVM Data Deduplication Using ECC-Derived Feature Matching
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
In flash memory storage systems, repeated writing of the same encoded data leads to increased write counts, reducing the endurance and efficiency of the storage device, as existing data compression methods are not universally effective and can overload the Flash Translation Layer due to varying data compressibility.
Innovation Solution
A non-volatile memory apparatus and data de-duplication method that uses error checking and correcting (ECC) methods to generate feature information for raw data, allowing the controller to look up a feature list and avoid rewriting encoded data already stored, thereby reducing redundant writes.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If data compression technique is used to reduce data volume of duplicated data, then the amount of writes is reduced, but the Flash Translation Layer is overloaded due to managing compressed data of different lengths
Solution Approach 1:
The patent extracts only the essential feature information (hash values) from the raw data for duplication detection, rather than processing the entire compressed data. This separation allows the FTL to manage only the compact feature information while the actual data compression is handled independently, reducing FTL management complexity while maintaining write reduction benefits
Solution Approach 2:
The patent transforms the data representation from full compressed data to a condensed parameter form (hash values or feature information). This parameter transformation enables efficient comparison and deduplication while significantly reducing the management burden on the FTL, as hash values are much smaller and faster to process than compressed data
2Quantity of substance
If the same data is written multiple times into the flash memory, then the data volume increases, but the write count increases reducing endurance and efficiency
Solution Approach 1:
The patent performs preliminary deduplication by comparing feature information (hash values) before writing data to flash memory. By detecting duplicates in advance through feature matching, the system prevents redundant writes from occurring, thereby maintaining write efficiency while managing data volume
Solution Approach 2:
The patent creates and stores only unique data copies in the flash memory by identifying and eliminating duplicates through feature information comparison. This ensures that each unique data set is written only once, optimizing both storage utilization and write efficiency while preserving data integrity
Data Source
AI summary
A non-volatile memory (NVM) apparatus and a data de-duplication method thereof are provided. The NVM apparatus includes a NVM and a controller. The controller performs an error checking and correcting (ECC) method to convert a raw data into an encoded data. The controller performs the data de-duplication method to reduce a number of times that the same encoded data is repeatedly written into the NVM. The controller generates the feature information corresponding to the raw data by reusing the ECC method. When the feature information is found in a feature list, the encoded data corresponding to the raw data will not be written into the NVM. When the feature information is not found in the feature list, the feature information is added into the feature list, and the encoded data corresponding to the raw data is written into the NVM.