Flash Memory Deduplication via Hash Comparison
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Duplicate data sets in memory, such as flash memory arrays, occupy unnecessary space and increase processing overhead, leading to wear and reduced storage capacity due to redundant data management.
Innovation Solution
A deduplication process that identifies and removes duplicate data sets by applying a hash function to generate hash values, comparing them to group identical data, and evaluating variable parameters to select survivor and victim data sets, where survivor data sets are retained and victim data sets are scheduled for removal based on performance metrics.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If duplicate data sets are stored in memory, then data redundancy is maintained for potential retrieval, but memory space is wasted and processing overhead increases
Solution Approach 1:
The patent uses hash function copying to create compact representations of data sets. Instead of storing full duplicate data, the system generates hash values that serve as compact copies for identification and comparison, significantly reducing memory space while maintaining the ability to detect and manage duplicates
Solution Approach 2:
The patent transforms data sets into hash values through a parameter change (hashing process). This transformation converts large amounts of data into compact fixed-size values, enabling efficient storage and comparison while reducing the quantity of stored substance
2Adaptability or versatility
If duplicate data sets are stored in memory, then data availability is maintained, but processing overhead increases due to redundant data management
Solution Approach 1:
The patent extracts the essential identifying feature of data sets (their hash values) and separates this from the full data content. This extraction allows the system to manage duplicate identification efficiently using only the hash values, while the actual data remains accessible when needed
Solution Approach 2:
The hash function serves as an intermediary between the full data sets and the duplicate detection process. Instead of directly comparing large data sets, the system uses hash values as intermediaries for identification, significantly reducing processing overhead while maintaining data availability
3Reliability
If duplicate data sets are stored in memory, then storage capacity is reduced, but data redundancy provides backup capability
Solution Approach 1:
The patent implements a selective copying strategy where only unique data sets are stored in the memory array, while duplicate representations are identified through hash comparison. This eliminates redundant copies from the main storage, maximizing available capacity while maintaining backup capability through the deduplication mechanism
Solution Approach 2:
The patent discards duplicate data sets from storage by identifying them through hash comparison and marking them as stale. The system recovers storage space by removing these duplicates while maintaining the ability to recover original data through the surviving copy and associated metadata
4Reliability
If duplicate data sets are managed in memory, then wear on memory cells increases due to additional write/erase cycles, but data redundancy ensures data integrity
Solution Approach 1:
The patent extracts duplicate identification through hash comparison, separating this management function from actual data write/erase operations. By managing duplicates at the hash level rather than the full data level, the system minimizes write/erase cycles on memory cells while maintaining data integrity through the surviving copy
Solution Approach 2:
The patent changes the state of duplicate data from active to stale through parameter changes (marking), avoiding additional write/erase cycles. This parameter change approach maintains data integrity by preserving the surviving copy while extending memory device life by eliminating redundant wear-inducing operations
Data Source
AI summary
Apparatus and method for data management in a memory, such as but not limited to a flash memory array. In accordance with some embodiments, a plurality of data sets in a memory are identified as having a common data content and different physical addresses in the memory. A selected one of the data sets is marked as valid data and the remaining data sets are marked as stale data responsive to evaluation of at least one variable parameter associated with the physical addresses at which the data sets are respectively stored.