Erasure-Coded Distributed Storage With Incentivized Node Redundancy
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Distributed data-storage systems face challenges in maintaining data integrity and redundancy due to hardware failures and the high costs associated with data replication and erasure coding, which require significant storage space and additional infrastructure.
Innovation Solution
A system and method that incentivizes third-party storage providers to increase the number of storage nodes by offering an incentive mechanism, using erasure coding to generate and allocate data blocks and parity blocks across a network of voluntary and non-voluntary storage nodes, ensuring data integrity and redundancy while reducing storage costs.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If data replication is used to ensure reliability, then data access speed and availability are improved, but storage space requirements and costs increase significantly
Solution Approach 1:
The patent uses erasure coding to create n chunks from a data block, where any k chunks can reconstruct the original data. This is a form of intelligent copying that reduces redundancy compared to traditional replication. Instead of creating full copies, the system creates mathematical derivatives that collectively preserve the original information with less overhead.
Solution Approach 2:
The system changes the parameter of data representation by transforming a single data block into n encoded chunks using mathematical algorithms. This transformation allows the same reliability level to be achieved with fewer storage resources, as the encoded chunks contain distributed information that can be reconstructed from any k of them.
2Reliability
If the number of storage nodes is increased to improve redundancy, then data storage reliability is enhanced, but system complexity and infrastructure costs increase
Solution Approach 1:
The patent segments the storage system into independent storage nodes that can be added or removed without affecting the entire system. Each node stores specific chunks of encoded data, and the modular architecture allows incremental expansion of redundancy by simply adding more nodes to the network.
Solution Approach 2:
The system creates a universal storage network where storage nodes can be provided by third parties and integrated into the distributed storage system. These nodes serve multiple purposes: storing data chunks, providing redundancy, and enabling flexible scaling without requiring specialized infrastructure.
3Quantity of substance
If erasure coding is used to reduce storage overhead, then storage efficiency is improved, but the computational complexity of data encoding and decoding increases
Solution Approach 1:
The patent performs erasure coding in advance, transforming the original data block into n encoded chunks before distribution. This preliminary encoding action eliminates the need for real-time computation during data access, as the redundant information is already prepared and distributed across storage nodes.
Solution Approach 2:
The system replaces complex real-time computational processes with pre-computed mathematical transformations. The erasure coding algorithms are executed once during data ingestion, substituting ongoing computational complexity with static encoded data that can be directly stored and retrieved.
Data Source
AI summary
A system and method is provided for supporting integrity of distributed data storage with erasure coding. An exemplary method includes receiving a request to create a backup or archive of electronic data using erasure coding, contacting storage nodes and providing an incentive mechanism to incentivize each of the storage nodes to support the distributed data storage, receiving confirmation from at least some of the storage nodes indicating an agreement to store data, generating K data blocks and M parity blocks from the electronic data using erasure coding, and allocating the K data blocks and the M parity blocks to the storage nodes that have agreed to store data.