Replication Request Prioritization in Distributed Storage
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current techniques lack effective methods for managing replicas of objects in planet-wide distributed storage systems with trillions of objects and petabytes of data across dozens of data centers, particularly in prioritizing and executing replication requests efficiently under high loads.
Innovation Solution
A computer-implemented method that calculates the priority of replication requests as the difference between benefits and costs, sorts requests, and distributes them to instances in the distributed storage system, considering current network state, user quotas, storage space, and replication policies to ensure timely execution and compliance with replication policies.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If replication requests are executed without prioritization in a planet-wide distributed storage system, then all objects can be replicated, but the probability of data loss increases for newly uploaded objects and existing objects with multiple replicas cannot be replicated in a timely manner under high loads
Solution Approach 1:
The system changes the parameter of replication request handling by introducing dynamic priority assignment based on object characteristics (newly uploaded vs. existing objects, number of current replicas). This allows the system to adapt replication execution order to current system state and object importance, resolving the contradiction between reliable data protection and efficient replication under high loads
2Reliability
If the number of replicas for hit videos is increased quickly to handle increased demand, then service availability improves, but system resources are consumed faster
Solution Approach 1:
The system dynamically adjusts replication priorities based on real-time object popularity and demand metrics. Hit videos automatically receive higher replication priority without manual intervention, allowing the system to respond flexibly to changing access patterns while optimizing resource consumption by focusing replication efforts on objects that actually need them
3Reliability
If replication requests are processed in priority order with benefit-cost calculation, then important objects are replicated first maximizing data protection, but the complexity of the replication management system increases
Solution Approach 1:
The system uses parameter changes (priority assignment based on simple object attributes and current replica count) to achieve complex behavior (optimal replication ordering). The benefit-cost calculation framework provides a systematic way to evaluate and prioritize replication requests without requiring complex decision logic, resolving the contradiction between effective prioritization and management simplicity
Data Source
Figure 1A
Figure 1B
Figure 1C
AI summary
A system and method for generating replication requests for objects in a distributed storage system is provided. Replication requests for objects in a distributed storage system are generated based at least in part on replication policies for the objects and a current state of the distributed storage system, wherein a respective replication request for a respective object instructs a respective instance of the distributed storage system to replicate the respective object so as to at least partially satisfy a replication policy for the respective object, wherein a respective replication policy includes criteria specifying at least storage device types on which replicas of object are to be stored. At least a subset of the replication requests is then distributed to the respective instances of the distributed storage system for execution.