Geo-Distributed Erasure Coding for Balanced Storage Node Loads
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing geo-distributed storage systems face inefficiencies in data ingestion, retrieval, and repair due to inadequate handling of erasure-coded fragments and parity fragments, leading to higher input/output resource utilization and client device delays.
Innovation Solution
A method for distributing erasure-coded fragments in geo-distributed storage nodes involves receiving an object and storage reliability requirements, determining an appropriate erasure coding group, and performing erasure coding using a selected scheme, thereby distributing the coded object across geographically distributed nodes to balance load and improve efficiency.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If erasure coding is used to distribute data across geographically distributed storage nodes, then data reliability is improved, but system efficiency deteriorates due to higher I/O resource utilization
Solution Approach 1:
The patent segments storage nodes into geographically distributed data centers and further segments data into erasure-coded fragments with associated parity fragments. This hierarchical segmentation allows reliable distributed storage while enabling parallel processing across multiple locations, improving both reliability and efficiency simultaneously
Solution Approach 2:
The patent performs preliminary actions by pre-calculating and distributing parity fragments along with data fragments during the initial erasure coding process. This preliminary distribution of all necessary fragments (data and parity) across storage nodes enables efficient data retrieval and repair operations without requiring additional real-time computation, thus improving system efficiency while maintaining reliability
2Adaptability or versatility
If traditional erasure coding distribution is used, then data can be stored across multiple nodes, but inefficiencies occur during data ingestion, retrieval and repair operations
Solution Approach 1:
The patent applies local quality by organizing storage nodes into geographically distributed data centers with specific roles. Data fragments and parity fragments are strategically placed in different data centers based on local capacity and requirements. This localized organization enables faster data ingestion, retrieval, and repair operations by reducing cross-geographic traffic while maintaining versatile data distribution capabilities
Solution Approach 2:
The patent introduces data center intermediaries that manage erasure-coded fragments locally. These intermediaries coordinate data ingestion, retrieval, and repair operations within and between data centers, reducing direct client-to-node communication overhead. This intermediary layer eliminates operation delays by localizing management functions while preserving flexible data distribution across multiple nodes
3Reliability
If more storage nodes are used to improve reliability, then data redundancy increases, but resource utilization becomes less efficient
Solution Approach 1:
The patent transitions from a flat storage node architecture to a multi-dimensional hierarchical structure with geographically distributed data centers as one dimension and erasure-coded fragment groups as another. This dimensional organization allows data redundancy to be achieved across multiple dimensions simultaneously, improving reliability while optimizing I/O resource utilization by localizing operations within each dimension
Data Source
AI summary
A method, non-transitory computer readable medium, and storage management computing device that assists with distributing erasure coded fragments in geo-distributed storage nodes includes receiving an object and a storage reliability requirement from a client computing device. Erasure coding is performed on the received object using an erasure coding scheme. An erasure coding group from a plurality of erasure coding groups present in a plurality of geographically distributed storage nodes is determined based on the received storage reliability requirement and the erasure coding scheme. The erasure coded object is distributed to the determined erasure coding group from the plurality of erasure coding groups in the plurality of geographically distributed storage nodes.