Erasure Coding in Distributed Storage Nodes
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing implementations of erasure coding in distributed storage systems are inefficient in terms of generated network traffic and elapsed encoding time.
Innovation Solution
Scheduling and executing erasure coding tasks on storage nodes that have local copies of the data, allowing for parallel execution of tasks across multiple nodes to reduce unnecessary network traffic and encoding time.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If erasure coding is implemented in distributed storage systems, then data protection is achieved, but network traffic increases and encoding time increases
Solution Approach 1:
The system performs preliminary actions by storing complete data copies on multiple storage nodes before erasure coding is executed. This allows encoding tasks to be performed locally on nodes that already have the data, eliminating the need to transfer data across the network for encoding operations, thus reducing network traffic while maintaining data protection
2Reliability
If erasure coding is implemented in distributed storage systems, then data protection is achieved, but encoding time increases
Solution Approach 1:
The system prepares data in advance by distributing complete copies to multiple storage nodes before the encoding operation. This preliminary distribution enables parallel encoding execution across multiple nodes, significantly reducing the total encoding time compared to sequential processing or centralized encoding
Solution Approach 2:
The erasure coding process is segmented into multiple independent encoding tasks that can be executed in parallel on different storage nodes. Each node processes its local data copy independently, dividing the overall encoding workload and reducing total encoding time through concurrent execution
3Productivity
If data copies are stored on multiple storage nodes before encoding, then parallel encoding is enabled, but storage space requirements increase
Solution Approach 1:
The system temporarily stores complete data copies on storage nodes only during the encoding phase. After the erasure coding tasks are successfully executed and coded fragments are generated, the original complete data copies are deleted from the storage nodes. This approach enables parallel encoding execution while minimizing long-term storage space requirements, as the additional space is only needed temporarily during the encoding operation
Data Source
AI summary
Systems and methods for efficiently protecting data within a distributed storage system using erasure coding. Unnecessary network traffic can be eliminated by scheduling executing erasure coding tasks on storage nodes that have local copies of data. Encoding may be performed in parallel by multiple nodes to reduce elapsed encoding time.