Hierarchical Erasure Coding Repair for Multi-Cloud Storage
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Multi-cloud storage systems face challenges in maintaining low operational costs while ensuring data reliability, particularly due to the high data transfer volumes during erasure coding data repair, which incur significant operational costs, especially when connecting public clouds.
Innovation Solution
The proposed solution involves a data processing and management apparatus and method that includes hierarchical data repair, multiple data-object-level management configurations, and extensions to new cloud storage services and data management mechanisms for improved reliability and security, aiming to reduce network bandwidth usage and data repair time.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Quantity of substance
If erasure coding is applied for data redundancy, then storage overhead is reduced, but repair bandwidth increases significantly
Solution Approach 1:
The patent segments the repair process into two phases: a fast local repair phase that repairs single-node failures using cached parity data, and a slower remote repair phase for multiple failures. This segmentation reduces the bandwidth required for common single-node repair scenarios.
Solution Approach 2:
The patent pre-generates and caches parity data locally at storage nodes before failures occur. When a node fails, the cached parity data is immediately available for repair without requiring extensive data transfer from remote nodes, thus reducing repair bandwidth requirements.
2Quantity of substance
If erasure coding is used for data redundancy, then storage overhead remains low, but operational cost increases due to high data transfer volumes
Solution Approach 1:
The patent divides repair operations into local repairs (using cached data within the same data center) and remote repairs (across data centers). By handling single-node failures locally, the system avoids expensive cross-cloud data transfers for the majority of repair scenarios.
Solution Approach 2:
The patent dynamically adjusts the repair strategy based on failure patterns and data access frequencies. For frequently accessed data, it maintains more aggressive caching strategies, while for less critical data, it uses more conservative approaches, optimizing the balance between storage overhead and operational costs.
3Loss of energy
If replication is used instead of erasure coding, then repair bandwidth is reduced, but storage overhead increases
Solution Approach 1:
The patent implements partial replication by caching only the necessary parity data locally rather than maintaining full replicas. This partial caching approach provides sufficient repair capability for single-node failures while avoiding the full storage overhead of complete replication.
4Loss of time
If hierarchical data repair is implemented, then data repair time is reduced, but system complexity increases
Solution Approach 1:
The patent segments the repair system into hierarchical levels: local storage nodes with cached parity data, regional storage pools, and remote cloud storage. This segmentation enables faster local repairs while maintaining a structured approach to handling more complex failure scenarios at higher hierarchical levels.
Solution Approach 2:
The patent introduces intermediary components (such as repair coordination services and caching layers) that manage the complexity of hierarchical repair operations. These intermediaries abstract the complexity from individual storage nodes, enabling fast repair execution without requiring each node to understand the entire hierarchical repair strategy.
Data Source
AI summary
A distributed storage system includes one or multiple proxies, a metadata store, multiple agents, and multiple data repositories, which provisions storage space within a cloud or a cloud region from one or multiple cloud storage services. A proxy encodes data objects to generate coded chunks for fault tolerance and security, stores coded chunks in data repositories via agents, and persists the states and metadata of data objects in the metadata store. An agent manages the coded chunks in one or multiple data repositories and encodes coded chunks during data repair. The system utilizes data processing and management methods for improved data repair and management, including (i) hierarchical data repair for reducing both the cross-cloud network bandwidth usage and the data repair time, (ii) various data management configurations across objects, and (iii) extensions for new data management mechanisms and new cloud storage services.