Deferred Bulk Maintenance for Distributed Storage Reliability
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
The increasing volume of data stored in enterprises, particularly inactive data, poses challenges due to the high failure rates of low-cost storage devices, which conventional RAID systems struggle to manage effectively, leading to reliability issues and the need for frequent maintenance.
Innovation Solution
A distributed storage system employing deferred, bulk maintenance and erasure coding techniques to assess and manage data object fragments, allowing for predictive scheduling of maintenance based on projected failure rates and usage trends, thereby reducing the need for immediate repairs and expanding storage capacity as needed.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Quantity of substance
If low-cost storage devices are used to store large amounts of inactive data, then storage cost is reduced, but failure rate increases
Solution Approach 1:
The patent segments data into multiple fragments using erasure coding, distributing them across multiple low-cost storage devices. This allows the system to tolerate individual device failures while maintaining data availability, effectively resolving the contradiction between using cheap storage and maintaining reliability.
Solution Approach 2:
The system changes the parameter of data representation from complete copies to encoded fragments. By transforming data into k data fragments and m parity fragments through erasure coding, the system enables reliable storage on unreliable devices, allowing recovery even when some devices fail.
2Reliability
If conventional RAID systems are used to manage storage devices, then data protection is provided, but maintenance frequency increases
Solution Approach 1:
The system performs preliminary actions by proactively identifying at-risk data objects before actual failures occur. It schedules rebuild operations during off-peak hours and batches multiple rebuilds together, reducing the frequency and impact of maintenance operations compared to reactive RAID systems.
Solution Approach 2:
The system implements periodic health monitoring and risk assessment of storage devices, scheduling maintenance operations at optimal intervals rather than reacting to failures. This periodic approach reduces overall maintenance frequency by preventing cascading failures and consolidating maintenance tasks.
3Reliability
If erasure coding with m>2 is used to increase data reliability, then fault tolerance improves, but storage overhead increases
Solution Approach 1:
The system applies erasure coding selectively rather than uniformly to all data. It prioritizes encoding for data objects with higher risk profiles or critical importance, applying the full m>2 redundancy only where necessary. This partial application reduces overall storage overhead while maintaining fault tolerance for critical data.
4Ease of operation
If centralized database is used to track data object locations, then data location management is simplified, but system performance deteriorates
Solution Approach 1:
The patent extracts the location tracking function from a centralized database and distributes it across the storage nodes themselves. Each node maintains information about the data fragments it holds, eliminating the performance bottleneck of centralized lookups while preserving location management capabilities through distributed queries.
Data Source
AI summary
Failed capacity of a distributed storage system is determined. The distributed storage system includes a plurality of storage nodes, wherein the plurality of storage nodes include at least one storage device to store data objects, wherein the data objects have been divided into constituent fragments in the distributed storage system. Protection capacity of the distributed storage system is determined. Protection capacity includes the data fragments generated to allow the data objects to be rebuilt in response to at least a part of the data objects being either lost or corrupted. A probability is determined that the failed capacity overlaps with the used capacity of the distributed storage system prior to a next periodically scheduled maintenance of the distributed storage system. In response to the probability exceeding a risk threshold, a next maintenance of the distributed storage system is scheduled that comprises reducing the failed capacity.