Flash Storage Deduplication via Controller Feature Value Calculation
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
In large-scale storage systems using flash memories, the calculation of feature values for deduplication increases overhead, and existing methods do not efficiently handle deduplication across multiple flash packages, leading to performance deterioration and increased costs.
Innovation Solution
A method where an 8-byte assurance code including a 2-byte feature value is added to data, allowing the storage controller to calculate and transmit the feature value, thereby reducing the need for feature value calculation in flash packages and enabling efficient deduplication across multiple flash packages.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If the flash package calculates the feature value for deduplication, then the deduplication function is implemented, but the calculation overhead increases and performance deteriorates
Solution Approach 1:
The feature value calculation function is extracted from the flash package and transferred to the storage controller. The storage controller calculates the feature value of write data and transmits it along with the data to the flash package, eliminating the computational burden from the flash package and improving its performance while maintaining the deduplication function.
Solution Approach 2:
The storage controller acts as an intermediary that performs the feature value calculation and passes both the calculated feature value and the original data to the flash package. This intermediary approach allows the flash package to perform deduplication checks without bearing the computational overhead of feature value calculation.
2Measurement precision
If all data are checked for deduplication, then deduplication accuracy is improved, but the amount of calculation becomes excessive
Solution Approach 1:
The deduplication check is segmented into two stages: first, the storage controller compares feature values (a small segment of data) to quickly identify potential duplicates, and second, only data with matching feature values undergo full content comparison. This segmentation reduces the overall calculation amount while maintaining deduplication accuracy.
Solution Approach 2:
Instead of performing full data comparison for all write operations, the system performs partial comparison by first checking feature values. Only when feature values match does the system perform the excessive action of full data comparison, significantly reducing the total computational burden while maintaining accurate deduplication.
3Quantity of substance
If multiple flash packages are used to increase storage capacity, then the storage capacity is improved, but the deduplication efficiency across packages deteriorates
Solution Approach 1:
The storage controller merges the deduplication management of multiple flash packages into a centralized system. It calculates feature values for data destined for any flash package and coordinates the deduplication process across all packages, enabling efficient cross-package deduplication without reducing overall storage capacity.
Solution Approach 2:
The storage controller is designed with universal functionality to manage deduplication across multiple flash packages simultaneously. It can calculate feature values, compare data, and coordinate writes to any flash package in the system, making the deduplication mechanism multi-functional and efficient across the entire storage array.
Data Source
AI summary
A storage system according to the present invention has a plurality of flash packages equipped with a deduplication function. When a storage controller transmits a write data and a feature value of write data to a flash package, the flash package compares contents of the write data with data having a same feature value as the feature value of the write data. As a result of the comparison, if there is no corresponding data, the write data is stored in the flash memory, but if there is a corresponding data, the new data will not be stored. Thus, a greater number of data can be stored in the flash memory while preventing deterioration of performance.