Distributed Storage Volume Recovery via Throttled Restore Operations
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
In distributed computing environments, data centers face challenges in minimizing downtime during storage server failures, such as power outages, where simultaneous restore operations can overwhelm resources and lead to data loss or corruption.
Innovation Solution
A control plane manages storage servers to identify authoritative data copies, pause I/O requests, restore volumes in a read-only state, and merge data from high-durability snapshots with 'dirty' data, allowing customers to verify and enable volumes, while throttling restore operations to prevent resource overload.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If simultaneous restore operations are performed on multiple storage servers after a failure, then data recovery speed is improved, but system resources become overwhelmed and data loss or corruption may occur
Solution Approach 1:
The system implements throttling of restore operations to control the rate at which data is restored. Instead of allowing simultaneous unrestricted restore operations, the system periodically limits the number of restore operations that can occur, preventing resource overload while maintaining steady progress on data recovery. This is achieved through rate-limiting mechanisms that regulate the flow of restore operations across multiple storage servers.
Solution Approach 2:
The system monitors system resource utilization during restore operations and dynamically adjusts the rate of restore operations based on current system state. When resources become overwhelmed, the system reduces the rate of restore operations; when resources are available, it increases the rate. This feedback loop ensures data integrity is maintained while maximizing recovery speed within safe operational limits.
2Reliability
If restore operations are throttled to prevent resource overload, then system stability is improved, but data recovery time increases
Solution Approach 1:
The throttling mechanism is dynamic rather than static. The system continuously adjusts the restore operation rate based on real-time system conditions, allowing maximum throughput when resources are available and reducing throughput only when necessary to maintain stability. This dynamic approach minimizes the impact on recovery time while preserving system stability during critical periods.
Solution Approach 2:
The system changes operational parameters (such as the rate limit for restore operations) based on system state. Rather than using a fixed throttling rate, the system adjusts parameters like the number of concurrent restore operations, the rate of data transfer, and the priority of different restore operations to optimize both stability and recovery time under varying conditions.
3Productivity
If authoritative data copies are identified and restored first, then data availability is improved, but system complexity increases
Solution Approach 1:
Storage servers autonomously identify and restore their own authoritative data copies without requiring complex centralized coordination. Each server independently determines which of its data copies are authoritative and initiates restore operations accordingly. This self-service approach reduces the complexity of recovery management while ensuring critical data is restored first for improved availability.
Data Source
AI summary
A computing system recovers volumes in a distributed computing environment while reducing downtime of storage servers. In an embodiment, a storage server contacts a control plane after a storage failure has occurred. If the storage server hosts an authoritative copy of an offline volume, the storage server is requested to restore the volume. Non-authoritative volumes are removed from the storage server and the storage server provides read access to the restored volume while resuming storage services.