Distributed Storage Erasure Coding for Secure Low-Overhead Repair
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Distributed storage systems face challenges in achieving high storage efficiency, minimizing I/O overhead, and ensuring data security and availability, particularly in cloud storage systems, where redundancy measures like replication and error-correcting codes incur high costs and computational complexity, and data encryption is resource-intensive.
Innovation Solution
A system that splits data into segments, encodes them into slices, and packages these slices with metadata for secure distribution across remote storage nodes, using erasure coding to maximize storage efficiency and security, while optimizing node selection based on network bandwidth and availability, and providing a graphical interface for managing storage nodes.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If replication is used to protect data against storage node failures, then data reliability is improved, but storage efficiency deteriorates due to low storage efficiency of triple replication
Solution Approach 1:
The patent uses erasure coding to create redundant copies of data in an optimized manner. Instead of simple replication that stores complete copies, the system encodes data into multiple fragments where any subset of fragments can reconstruct the original data, achieving better storage efficiency while maintaining reliability
Solution Approach 2:
The patent combines multiple coding techniques including MDS codes for basic redundancy, LDC codes for localized repair, and regenerating codes for bandwidth optimization. This composite approach integrates different error-correcting code families to simultaneously achieve high storage efficiency, fast repair, and optimized bandwidth usage
2Quantity of substance
If MDS codes are used to achieve high storage efficiency, then storage efficiency is improved, but repair cost deteriorates due to prohibitively high repair costs
Solution Approach 1:
The patent divides data into segments and applies different coding schemes to different segments. LDC codes are applied to create localized repair groups where failed nodes can be repaired by accessing only a subset of other nodes rather than all nodes, significantly reducing repair cost while maintaining the storage efficiency benefits of MDS codes
Solution Approach 2:
The patent implements localized repair by organizing codewords into groups where each group can independently repair failures within that group. This local quality approach allows repairs to be performed with minimal I/O overhead by only accessing nodes within the same local group, rather than requiring access to all nodes in the system
3Quantity of substance
If LDC codes are used to minimize I/O overhead, then I/O overhead is reduced, but the number of practical constructions deteriorates due to low number of practical constructions
Solution Approach 1:
The patent develops a unified framework that integrates multiple coding techniques (MDS, LDC, regenerating codes) into a single versatile system. This universal approach allows the system to achieve the I/O overhead benefits of LDC codes while providing practical constructions by combining LDC with well-established MDS and regenerating code families, making the solution both theoretically optimal and practically implementable
4Quantity of substance
If regenerating codes are used to reduce data transmission during repair, then bandwidth is optimized, but the advantages deteriorate when limited to partial read operations
Solution Approach 1:
The patent implements a dynamic system that can adaptively select between different read operation modes (partial reads, full reads, repair operations) and switch between different coding techniques based on the specific operation requirements. This dynamic approach allows the system to achieve bandwidth optimization through regenerating codes while maintaining adaptability to various read operation types by selecting the appropriate coding scheme for each operation
5Reliability
If data encryption is used to ensure security, then data security is improved, but computational complexity deteriorates due to high computation complexity
Solution Approach 1:
The patent performs data encoding and security-related processing during the initial data ingestion phase rather than during every read/write operation. By preliminarily encoding data with erasure codes and establishing security parameters upfront, the system reduces the computational complexity of subsequent operations while maintaining strong security guarantees
Solution Approach 2:
The patent extracts security-critical functions from the main data processing path and handles them separately through dedicated encoding and key management mechanisms. This separation allows the system to maintain data security through encryption and secure key management while reducing the computational complexity of routine data operations by pre-establishing security parameters
Data Source
AI summary
Secure distributed storage and transmission of electronic content is provided over at least one communication network. At least one data file is received and parsed into a plurality of segments, wherein each one of the segments has a respective size. Thereafter, each of the plurality of segments is divided into a plurality of slices, wherein each one of the slices has a respective size. A plurality of data chunks are encoded, each data chunk comprising a portion of at least two of the slices, wherein no portion comprises an entire slice. The data chunks are packaged with at least metadata, and each of the packages is assigned to respective remote storage nodes. Each of the packages is transmitted to the respectively assigned remote storage node.