Object Storage Deduplication Using Slice Granularity and Soft Links
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Large-scale object storage systems lack deduplication functionality, leading to high performance consumption, inaccurate duplicate data search, and potential security risks, while existing solutions outside the system impose complex data management and maintenance burdens on users.
Innovation Solution
Implement deduplication within the object storage system by determining target granularities of object or slice granularity, performing duplicate data screening, and adding soft links in metadata to recycle storage space, reducing performance overhead and improving stability and security without additional management complexity.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Loss of substance
If deduplication function is not implemented in object storage system, then system complexity remains low, but storage space is wasted and performance consumption increases
Solution Approach 1:
The patent segments object data into multiple slices at a specified granularity. Each slice can be independently deduplicated, allowing the system to manage large objects in smaller, more manageable units. This segmentation enables efficient deduplication while maintaining object integrity and reducing the computational burden of processing entire objects at once.
Solution Approach 2:
The patent introduces an intermediary deduplication layer between the object storage system and the underlying storage resources. This intermediary component handles duplicate detection and elimination without requiring changes to the core object storage architecture, thus improving storage efficiency while minimizing system complexity.
2Loss of substance
If external deduplication solution is added, then storage space is saved, but data management and maintenance complexity increases for users
Solution Approach 1:
The patent merges the deduplication function directly into the object storage system, combining storage and deduplication operations into a unified system. This integration eliminates the need for separate external deduplication solutions, reducing user management overhead while maintaining storage efficiency benefits.
Solution Approach 2:
The deduplication system automatically performs duplicate detection and elimination without requiring user intervention. The system self-manages the deduplication process, including slice creation, duplicate identification, and space reclamation, thereby simplifying user operations while achieving storage space savings.
3Ease of manufacture
If object granularity deduplication is used, then implementation is simple, but deduplication effectiveness is reduced due to large data processing units
Solution Approach 1:
The patent segments objects into slices at a configurable granularity level. This segmentation allows the system to achieve finer deduplication effectiveness compared to whole-object deduplication, while still maintaining implementation simplicity by processing manageable slice units rather than entire large objects.
Solution Approach 2:
The patent introduces granularity as a configurable parameter that can be adjusted to optimize between implementation simplicity and deduplication effectiveness. By changing the slice granularity parameter, the system can adapt to different workloads and storage patterns, achieving effective deduplication without overly complex implementation.
4Loss of substance
If slice granularity deduplication is implemented, then deduplication effectiveness improves, but performance overhead increases
Solution Approach 1:
The patent implements deduplication at slice granularity rather than processing entire objects, applying partial action to only the necessary data units. This approach improves deduplication effectiveness by identifying duplicates at the slice level while reducing overall performance overhead compared to processing complete objects, as only relevant slices require deduplication operations.
Data Source
Figure 1a~1b
Figure 2
Figure 3
AI summary
The disclosed embodiment provides a method, device and storage medium for deduplication of an object storage system, which determines (S201) the target granularity for deduplication of the object storage system, which is the object granularity or the slice granularity; obtains (S202) the first check value of each data of the target granularity in the object storage system, and screens out the data with the same first check value from the object storage system to determine as duplicate data; adds (S203), by taking one data in any group of duplicate data as the reference data, a soft link to the metadata of other data except the reference data in any group of duplicate data to point to the reference data, and recycles the storage space of other data.