Distributed Storage Erasure Coding for Secure Low-Latency Recovery
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Distributed storage systems face inefficiencies in data protection and storage efficiency, particularly with high computational complexity and low storage efficiency in cloud storage systems, where replication methods like triple replication are inefficient, and error-correcting codes like locally decodable codes and regenerating codes have limitations in practical applications.
Innovation Solution
A method involving encoding data into codeword chunks with encoding parameters and identifiers, using erasure coding for redundancy, and distributing these chunks across multiple storage nodes while optimizing data retrieval latency and workload distribution using availability coefficients, along with encryption for security, to ensure data integrity and availability.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If triple replication is used to protect data against storage node failures, then data reliability is improved, but storage efficiency deteriorates
Solution Approach 1:
The data is divided into multiple segments and encoded using erasure coding schemes, where the original data is split into k segments and distributed across n storage nodes. This allows the system to tolerate up to n-k failed nodes while using less storage space than full replication, thus improving storage efficiency while maintaining data reliability.
Solution Approach 2:
The patent employs configurable erasure coding parameters (k, n) that allow flexible adjustment between storage efficiency and data reliability. By changing these parameters, the system can adapt to different requirements - for example, using (3,5) coding for higher efficiency or (2,3) for higher reliability - resolving the contradiction between storage efficiency and data protection.
2Quantity of substance
If erasure coding is used to improve storage efficiency, then storage efficiency is improved, but computational complexity increases
Solution Approach 1:
The system pre-generates and caches coding coefficients and decoding matrices during system initialization or low-load periods. When data needs to be stored or retrieved, these pre-computed parameters are reused, significantly reducing the real-time computational complexity of encoding and decoding operations while maintaining high storage efficiency.
3Productivity
If locally decodable codes are used to minimize I/O overhead, then I/O overhead is reduced, but the number of practical constructions is limited
Solution Approach 1:
The patent implements a universal erasure coding framework that can accommodate multiple coding schemes (Reed-Solomon, RDP, EVENODD, and other locally decodable codes) through a common architecture. This universal design allows the system to select the most appropriate code for each specific application scenario, minimizing I/O overhead while providing access to a wide variety of practical code constructions.
4Loss of energy
If regenerating codes are used to reduce data transmission during repair, then bandwidth is optimized, but advantages are limited to partial read operations
Solution Approach 1:
The system dynamically selects between regenerating codes and other erasure coding schemes based on the operation type. For partial read operations, regenerating codes are used to minimize bandwidth consumption. For full read operations or other scenarios, different coding schemes are employed. This dynamic adaptation extends the versatility of the system beyond just partial reads while maintaining bandwidth optimization where applicable.
Data Source
Figure 1
Figure 2
Figure 3
AI summary
A system and method for distributing data over a plurality of remote storage nodes. Data are split into segments and each segment is encoded into a number of codeword chunks. None of the codeword chunks contains any of the segments. Each codeword chunk is packaged with at least one encoding parameter and identifier, and metadata are generated for at least one file and for related segments of the at least one file. The metadata contains information to reconstruct from the segments, and information for reconstructing from corresponding packages. Further, metadata are encoded into package(s), and correspond to a respective security level and a protection against storage node failure. A plurality of packages are assigned to remote storage nodes to optimize workload distribution. Each package is transmitted to at least one respective storage node as a function iteratively accessing and retrieving the packages of metadata and file data.