Bulk-Mode Object Replication in Data Storage Systems
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing data storage systems face inefficiencies in object-level replication, particularly in geographically dispersed systems where replication over long distances using less stable network channels is inefficient, and duplicating data protection leads to extra traffic.
Innovation Solution
Implementing region-based replication queues that segment replication tasks by destination region, packing objects into batches, and using round-robin policies to distribute data efficiently, with online encoding of chunks for direct protection once complete, avoiding preliminary protection schemes when enough data is available.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If individual objects are replicated to remote regions, then data protection is improved, but replication efficiency deteriorates due to long-distance transmission over unstable network channels
Solution Approach 1:
The patent combines multiple small individual objects into a single bulk data structure for replication. Instead of transmitting each object separately across long-distance unstable network channels, the system packs multiple objects into one bulk structure, reducing the number of transmission operations and improving overall replication efficiency while maintaining data protection through the bulk replication process
2Reliability
If data is protected by preliminary protection scheme until chunk is filled, then data safety is improved, but data traffic increases due to duplicate protection of same data
Solution Approach 1:
The patent extracts and eliminates the redundant preliminary protection step for objects that will be replicated as part of a bulk operation. When objects are selected for bulk replication, the system removes the need for preliminary protection schemes that would otherwise duplicate protection efforts, thereby reducing data traffic while maintaining data safety through the bulk replication process itself
Data Source
AI summary
The described technology is generally directed towards bulk-mode replication of data objects to a destination replication site. When replicating a group of objects, and there is enough object data to fill one or more chunk data structures, bulk-mode data replication is performed, with the destination site replicating the objects into chunk data structures and performing erasure coding to protect the data. This avoids the need for inefficient preliminary data protection scheme at the replication destination. When there are multiple objects to be replicated but not enough object data to fill a chunk data structure, bulk-mode data replication transfers the objects to the destination site for replicating as individual objects. Standard object-level replication can be used for single object replication. The technology thus facilitates high-performance data transfer to destination replication sites and efficient handling of data at the replication destination site.