Distributed Storage Erasure Coding for Secure Low-Overhead Recovery
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Distributed storage systems face challenges in achieving high storage efficiency, minimizing I/O overhead, and ensuring data security and availability, particularly in cloud storage systems, where redundancy measures like replication and error-correcting codes incur high costs and computational complexity, and data encryption is resource-intensive.
Innovation Solution
A system that divides data into segments, encodes them into slices, and packages these slices with metadata for secure distribution across remote storage nodes, using erasure coding to maximize storage efficiency and security, while optimizing node selection based on network bandwidth and availability, and providing a graphical user interface for managing storage nodes and data recovery.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If replication is used to protect data against storage node failures, then data reliability is improved, but storage efficiency deteriorates due to very low storage efficiency of triple replication
Solution Approach 1:
The patent uses erasure coding to create redundant copies of data in an optimized manner. Instead of simple replication that stores identical copies, the system encodes data into multiple fragments with redundancy information, allowing recovery from any subset of fragments. This resolves the contradiction by providing data reliability through redundancy while improving storage efficiency compared to traditional replication methods.
Solution Approach 2:
The patent changes the redundancy parameter from fixed triple replication to flexible erasure coding schemes where the redundancy factor can be adjusted. By using MDS codes and configurable encoding parameters, the system can optimize the balance between storage efficiency and data reliability based on specific requirements, rather than being locked into inefficient fixed replication ratios.
2Quantity of substance
If erasure coding is used to achieve high storage efficiency, then storage efficiency is improved, but repair costs worsen due to prohibitively high repair costs for long MDS codes
Solution Approach 1:
The patent segments the erasure coding process into manageable components with specific code lengths optimized for different scenarios. By breaking down long MDS codes into shorter, more efficient code segments that can be processed independently, the system maintains high storage efficiency while reducing the computational burden and cost of repair operations.
Solution Approach 2:
The patent implements partial repair strategies where not all code fragments need to be processed during repair operations. By using properties of the erasure coding scheme to identify and repair only the necessary portions of damaged data, the system reduces repair costs while maintaining the high storage efficiency benefits of erasure coding.
3Reliability
If data encryption is used to ensure data security, then data security is improved, but computational complexity worsens due to high computation complexity of data encryption
Solution Approach 1:
The patent performs encryption operations in advance during the data ingestion phase, rather than encrypting data on-the-fly during every access operation. By pre-encrypting data before storage and using secure key management mechanisms, the system ensures data security while avoiding the repeated computational overhead of encryption during normal read/write operations.
Solution Approach 2:
The patent introduces cryptographic intermediaries such as trusted execution environments and secure key management services that handle the complex encryption operations. These intermediaries shield the main storage system from direct computational complexity while maintaining strong security through standardized cryptographic protocols.
4Productivity
If locally decodable codes are used to minimize I/O overhead, then I/O overhead is improved, but the number of practical constructions worsens due to low number of practical LDC code constructions
Solution Approach 1:
The patent adopts a universal erasure coding framework based on MDS codes that can serve multiple functions including error correction, data recovery, and I/O optimization. By using a well-established, widely-implemented coding scheme with numerous practical constructions available, the system achieves low I/O overhead through optimized read operations while benefiting from the maturity and availability of practical implementations.
Data Source
AI summary
Secure distributed storage and transmission of electronic content is provided over at least one communication network. At least one data file is received and parsed into a plurality of segments, wherein each one of the segments has a respective size. Thereafter, each of the plurality of segments is divided into a plurality of slices, wherein each one of the slices has a respective size. A plurality of data chunks are encoded, each data chunk comprising a portion of at least two of the slices, wherein no portion comprises an entire slice. The data chunks are packaged with at least metadata, and each of the packages is assigned to respective remote storage nodes. Each of the packages is transmitted to the respectively assigned remote storage node.