Object Storage Deduplication Using Slice Granularity and Soft Links
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Large-scale object storage systems lack deduplication functionality, leading to high performance consumption, inaccurate duplicate data search, large data granularity, and potential security risks, while existing solutions outside the system impose complex management and maintenance burdens on users.
Innovation Solution
Implement deduplication within the object storage system by determining target granularities of object or slice granularity, performing duplicate data screening, and adding soft links to metadata to recycle storage space, reducing performance overhead and improving stability and security without additional management complexity.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Quantity of substance
If deduplication is implemented in large-scale object storage systems, then storage space is saved, but performance consumption increases
Solution Approach 1:
The patent segments object data into multiple slices and performs deduplication at the slice level rather than at the entire object level. This allows finer-grained duplicate detection and reduces the computational overhead per operation, as smaller slice comparisons are more efficient than large object comparisons, thereby saving storage space while controlling performance consumption.
Solution Approach 2:
The patent implements selective deduplication by identifying and processing only duplicate slices rather than entire objects. By performing partial deduplication at the slice level and using soft links to reference existing data, the system achieves storage savings without requiring full deduplication of all data, thus reducing overall performance consumption while still eliminating redundant storage.
2Quantity of substance
If external deduplication solutions are used, then deduplication functionality is added, but management and maintenance complexity increases
Solution Approach 1:
The patent merges the deduplication functionality directly into the object storage system's internal architecture rather than using external solutions. By integrating slice-level deduplication, soft link management, and duplicate detection within the storage system itself, the patent eliminates the need for separate external deduplication systems, thereby adding functionality without increasing user-facing management complexity.
Solution Approach 2:
The patent implements self-service deduplication where the storage system automatically performs duplicate detection, slice identification, and soft link creation without requiring user intervention. The system autonomously manages the deduplication process, including automatic recycling of storage space from duplicate slices, which eliminates the maintenance burden that would otherwise fall on users.
3Ease of manufacture
If object granularity is used for deduplication, then implementation is simpler, but data granularity is too large to save space
Solution Approach 1:
The patent segments objects into multiple slices to enable finer-grained deduplication. Instead of treating entire objects as single units, the system divides objects into smaller slice components that can be independently analyzed for duplicates. This segmentation allows the system to identify and eliminate duplicate slices within and across objects, achieving significant storage space savings while maintaining manageable implementation complexity through the use of soft links.
4Quantity of substance
If slice granularity is used for deduplication, then storage space is saved, but system complexity increases
Solution Approach 1:
The patent introduces soft links as an intermediary mechanism to manage slice-level deduplication. Soft links act as references that point to duplicate slices without requiring full data duplication, thereby enabling fine-grained storage savings while abstracting the complexity of slice management from the system. This intermediary approach allows the system to track and manage duplicate slices efficiently without proportionally increasing overall system complexity.
Data Source
AI summary
The disclosed embodiment provides a method, device and storage medium for deduplication of an object storage system, which determines the target granularity for deduplication of the object storage system, which is the object granularity or the slice granularity; obtains the first check value of each data of the target granularity in the object storage system, and screens out the data with the same first check value from the object storage system to determine as duplicate data; adds, by taking one data in any group of duplicate data as the reference data, a soft link to the metadata of other data except the reference data in any group of duplicate data to point to the reference data, and recycles the storage space of other data.