Storage Controller Deduplication for Flash Memory Lifespan
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current solid state memory systems do not effectively implement data deduplication for application-specific data structures, leading to inefficient management of write and erase operations due to the limited lifespan of flash memory.
Innovation Solution
A storage system that separates application-specific data into headers and bodies, calculates a deduplication fingerprint, and stores them separately to improve deduplication ratios by using a controller to manage data sets and apply deduplication functions, including padding data when necessary, to optimize storage and retrieval operations.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If data deduplication is applied to application-specific data structures without separation, then the deduplication process is simple, but the data deduplication ratio is low and storage efficiency is poor
Solution Approach 1:
The patent segments application-specific data structures into application header portions and application data body portions. The controller separately processes these segments, calculating fingerprints only for the data body portions while preserving the header portions. This segmentation enables higher deduplication ratios by focusing on the duplicative data body while maintaining the necessary header information, directly resolving the contradiction between simplicity and effectiveness.
2Quantity of substance
If flash memory is used for storage, then storage capacity is high, but the lifespan is limited due to erase page operation
Solution Approach 1:
The patent implements discarding of redundant data by identifying and eliminating duplicate data body portions through fingerprint matching. By discarding only the redundant portions and preserving unique data and necessary headers, the system reduces the total number of write and erase operations on flash memory, thereby extending memory lifespan while maintaining storage capacity for unique data.
3Reliability
If application metadata is stored with data sets, then data integrity is maintained, but deduplication efficiency is reduced due to inability to data deduplicate the metadata portion
Solution Approach 1:
The patent extracts the application header portion (metadata) from the application data body portion. The controller calculates fingerprints only for the data body portions and stores headers separately or in a non-deduplicated area. This extraction maintains data integrity by preserving headers while enabling efficient deduplication of the data body portions, resolving the contradiction between reliability and productivity.
Data Source
AI summary
According to one embodiment, a storage system comprises a controller being configured to: receive a write data of an application from a computer; manage the write data as a plurality of data sets, each of the plurality of data sets having a size of a deduplication unit; and apply a deduplication function to each of the plurality of data sets. When a data set of the plurality of data sets has an application metadata portion containing application metadata, the controller separates the application metadata from the data set and fills the application metadata portion by a padding data and applies the deduplication function to the data set including the padding data.