Custom Object Paths for Distributed Storage Deduplication
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Conventional object storage systems face performance bottlenecks and resource inefficiencies due to the need for global content indexing and updates during deduplication, which slows down I/O operations and limits scalability.
Innovation Solution
Implementing custom object paths that use hash algorithms to create meaningful paths for objects, allowing strategic storage of similar data in the same devices, thereby reducing the scope of deduplication operations and improving I/O performance by limiting analysis to specific groups of storage devices.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Loss of substance
If global content indexing table is used for deduplication in distributed object storage system, then data redundancy can be removed, but I/O performance is significantly slowed down and system scalability is limited
Solution Approach 1:
The patent segments the storage system into multiple storage devices with local content indexing tables instead of using a single global table. Each storage device maintains its own deduplication index, allowing parallel processing and eliminating the performance bottleneck of centralized table lookups and updates during deduplication operations.
2Loss of substance
If global content indexing table is used for deduplication, then data redundancy can be removed, but resource consumption increases significantly
Solution Approach 1:
The patent divides the centralized indexing resource into distributed local indexes across multiple storage devices. This segmentation reduces the resource burden on any single device, enables parallel deduplication processing, and lowers overall system resource consumption while maintaining effective redundancy removal.
3Loss of substance
If global content indexing table is used for deduplication, then data redundancy can be removed, but system scalability is limited
Solution Approach 1:
The patent implements a segmented distributed architecture where each storage device operates independently with its own content index. This segmentation removes the scalability bottleneck of centralized table management, allowing the system to expand by simply adding more storage devices with local indexes, thereby improving adaptability and versatility.
4Device complexity
If traditional opaque object paths are used, then object storage system is simple to implement, but data storage efficiency and deduplication are reduced
Solution Approach 1:
The patent transforms opaque object paths into meaningful custom paths by incorporating hash values of object data. This parameter change in the path structure enables efficient data grouping and strategic storage placement, improving data storage efficiency and deduplication capabilities while maintaining reasonable system implementation complexity.
Data Source
AI summary
A processing device receives a request to perform an operation for an object in a storage system. The request includes a custom path for the object that represents the content of the object. The custom path for the object includes a hash value resulting from a hashing algorithm. The processing device determines a data store that stores the one or more other objects having a respective custom path that corresponds to the custom path in the request. Responsive to determining the data store, the processing device performs the operation for the object using the data store.