Storage Data Clustering for Lossy Deduplication Capacity Gains
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing storage devices face challenges in increasing their effective storage capacity without adding additional physical storage capacity, particularly when dealing with large amounts of data that have varying degrees of similarity, as conventional deduplication methods are inefficient for non-identical data sets.
Innovation Solution
Implementing a lossy storage approach using neural networks to classify and store representative datasets in clusters, allowing for dynamic quantization and storing only one representative dataset per cluster, thereby reducing the physical storage needed.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If data is stored at full fidelity with each data being stored independently, then data accuracy is maintained, but storage capacity is insufficient and the number of storage devices must increase
Solution Approach 1:
The patent merges multiple similar data sets into a single representative data set by identifying similarities and storing common characteristics. This combining approach allows the storage system to represent multiple data sets with one stored instance, thereby increasing effective storage capacity while maintaining acceptable data accuracy for similar datasets.
Solution Approach 2:
The patent changes the storage parameter from full-fidelity storage to lossy compression storage. By accepting controlled data loss and storing approximate representations rather than exact copies, the system achieves higher storage capacity utilization while maintaining sufficient accuracy for applications that can tolerate minor variations.
2Quantity of substance
If data deduplication is performed to eliminate redundant copies, then storage capacity is increased, but it cannot be optimally performed when there is any variation between two data sets
Solution Approach 1:
The patent extends deduplication from exact matching to approximate matching by introducing similarity thresholds and tolerance parameters. This allows the deduplication system to identify and merge data sets that are similar but not identical, significantly improving deduplication effectiveness for varied datasets while maintaining storage capacity benefits.
Solution Approach 2:
The patent applies different deduplication strategies to different data regions or characteristics. By analyzing specific attributes and applying targeted compression techniques to similar portions of data while preserving unique characteristics, the system achieves both high storage efficiency and adaptability to various data types and variation levels.
Data Source
Figure 1
Figure 2
Figure 3
AI summary
A storage device (125) is disclosed. The storage device (125) may include storage (415) for data. A host interface (405) may receive a write request from a host (105) at the storage device (125). The write request may include a first data chunk and a first data identifier (ID). A first class ID determiner circuitry (430) may determine a first class ID for the first data chunk. A first mapping table (505) may map the first data ID to the first class ID.