Storage System Proactive Data Rebuilding
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing data storage systems face challenges in efficiently managing data recovery across multiple storage nodes, particularly in ensuring data availability and integrity in the event of node failures.
Innovation Solution
The implementation of a storage system that utilizes non-volatile solid state storage units with embedded CPUs and controllers, coupled with a distributed architecture that employs erasure coding and redundant copies of metadata, allows for proactive data rebuilding and recovery across storage nodes.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If data is stored across multiple storage nodes with redundancy, then data availability and integrity are improved, but system complexity and storage overhead increase
Solution Approach 1:
The patent segments data into multiple shards distributed across different storage nodes. Each shard is a discrete unit that can be independently managed, retrieved, and replicated. This segmentation enables the system to provide data availability through distribution while managing complexity through standardized shard handling procedures and metadata tracking.
Solution Approach 2:
The patent creates redundant copies of data shards and stores them across multiple storage nodes. When data is written, multiple copies are simultaneously created and distributed to different nodes, ensuring data availability even if some nodes fail. This copying approach improves reliability while the systematic management of copies through metadata reduces the perceived complexity.
2Loss of time
If data recovery is performed on-demand when read access is requested, then storage resources are conserved, but data availability time increases
Solution Approach 1:
The patent performs data recovery operations in advance by proactively rebuilding data on standby or underutilized storage nodes before read access is requested. The system monitors storage node status and triggers proactive recovery when resources are available, reducing data recovery time for future access requests while optimizing storage resource utilization through load-aware scheduling.
Solution Approach 2:
The system implements self-service data recovery where the storage system automatically detects missing or corrupted data and triggers recovery operations without external intervention. The metadata system automatically identifies which shards are missing and initiates reconstruction from remaining copies, reducing both recovery time and manual operational overhead.
3Reliability
If proactive data rebuilding is performed continuously, then data availability is improved, but energy consumption and system load increase
Solution Approach 1:
The patent implements periodic data integrity checks and selective proactive rebuilding instead of continuous operations. The system schedules regular verification of data shards and triggers rebuilding operations only when integrity issues are detected or during scheduled maintenance windows when system load is lower. This periodic approach maintains data integrity while significantly reducing energy consumption compared to continuous rebuilding.
Solution Approach 2:
The system dynamically adjusts recovery parameters such as rebuild speed, priority levels, and triggering thresholds based on current system conditions including energy availability, load levels, and criticality of data. This allows the system to maintain data integrity through proactive rebuilding while adapting energy consumption to available resources and operational requirements.
Data Source
AI summary
An illustrative method includes accessing, by a controller operating within a global control plane, a recovery policy that specifies parameters for generating recovery data associated with one or more resources of a first cluster comprising a first one or more containers, wherein the first cluster comprises a first local control plane separate from the global control plane; determining, by the controller and based on the recovery policy, a second cluster comprising a second one or more containers, wherein the second cluster comprises a second local control plane separate from the global control plane; and generating, by the controller and with respect to the second cluster, the recovery data associated with the one or more resources of the first cluster.