Storage Optimization via Partial Data Deduplication
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current deduplication techniques fail to efficiently utilize storage space when data units have substantial identical parts with only small non-identical differences, as they require complete identity for deduplication, leading to redundant storage.
Innovation Solution
A storage system with a control layer that identifies partly-identical data units by comparing data and metadata parts, associates logical addresses with physical addresses, and discards the data part of one unit while storing the metadata part, thereby reducing physical storage volume.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If complete identity between data units is required for deduplication, then data integrity is ensured, but storage efficiency deteriorates when data units have substantial identical parts with only small differences
Solution Approach 1:
The data unit is divided into a data part and a metadata part. The deduplication process compares only the data parts for identity, while metadata parts are handled separately. This segmentation allows parts of data units to be deduplicated even when metadata differs, improving storage efficiency while maintaining integrity through separate metadata validation.
Solution Approach 2:
The metadata part is extracted and separated from the data part. The deduplication process focuses only on the data part for identity comparison, while the metadata part is stored separately in a metadata storage area. This extraction enables partial deduplication of data units that have identical data content but different metadata, resolving the contradiction between integrity and storage efficiency.
2Manufacturing precision
If all different data portions are stored to ensure complete data representation, then data accuracy is maintained, but redundant storage increases
Solution Approach 1:
Instead of storing complete duplicate data units, the system stores only the metadata part as a separate copy when the data part is identified as identical to existing data. The metadata copy contains sufficient information to reconstruct or validate the original data unit, maintaining data accuracy while eliminating redundant storage of identical data portions.
Solution Approach 2:
The system discards redundant data parts that are identical to existing stored data, keeping only the metadata part for recovery purposes. When data is needed, the metadata is used to recover or reconstruct the complete data unit by combining the stored metadata with the identified identical data part from existing storage, maintaining accuracy while reducing redundancy.
Data Source
AI summary
A method and system are disclosed for storage optimization. Data parts and metadata within a source data unit are identified and the data parts are compared with data which is already stored in the physical storage space. In case identical data parts are found within the physical storage, the data parts from the source data unit are linked to the identified data, while the data parts can be discarded, thereby reducing the required storage capacity. The metadata parts can be separately stored in a designated storage area.