Ordered Parallel Replication for Distributed Object Storage
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing distributed object storage systems face challenges in efficiently replicating data objects of varying sizes, leading to suboptimal completion times and resource utilization, particularly due to inflexible replication processes and high overhead costs associated with redundancy schemes like RAID and replication.
Innovation Solution
A system comprising processors, memory, and storage nodes with replication engines that generate an ordered list of data parts by size, allowing for sequential assignment and parallel processing across available engines, optimizing replication commands and resource allocation.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If data objects are replicated using traditional parallel GET-PUT processes, then replication can be performed concurrently across multiple storage elements, but the completion time cannot be reliably met due to fixed resource allocation and varying object sizes
Solution Approach 1:
The patent segments data objects into multiple data parts before replication. This segmentation allows the replication system to process smaller, uniform units across multiple replication engines simultaneously, improving load balancing and resource utilization while meeting completion time requirements
Solution Approach 2:
The patent dynamically adjusts the replication strategy by sorting data parts and assigning them to replication engines based on current system state and object size characteristics. This dynamic allocation optimizes resource utilization and ensures completion within the quality of service time boundary
2Quantity of substance
If redundant array of independent disks (RAID) systems are used for redundancy, then storage capacity overhead is reduced compared to full replication, but the system requires strict synchronization and identical storage element types reducing flexibility
Solution Approach 1:
The patent creates a universal replication framework that can handle both full replication and RAID-like redundancy scenarios. The system can adapt to different storage element types and configurations while maintaining the same replication mechanism, achieving both efficiency and flexibility
3Ease of operation
If first in first out (FIFO) ordering is used to assign data objects to parallel replication processes, then simple allocation is achieved, but optimal completion time is not realized due to varying object sizes
Solution Approach 1:
The patent changes the allocation parameter from FIFO (time-based) to size-based ordering. By sorting data parts according to their size before assignment to replication engines, the system achieves better load balancing and optimizes completion time while maintaining operational simplicity
Data Source
AI summary
Example distributed object storage systems and methods provide ordered parallel replication of data object parts. Batch lists of data objects that identify data parts for replication are generated. Available replication engines are identified and the data parts are sorted according to a predetermined order, such as largest to smallest by part size, to create an ordered list. The data parts are then sequentially assigned to the available replication engines based on the ordered list of data parts for parallel processing of replication commands. Initiate-multipart commands may be executed before the data parts are processed and complete-multipart commands may be executed after the data parts are processed.