Decentralized Edge Storage Network File Sharding
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Traditional sharding techniques are difficult to implement in fully decentralized networks, as they require centralized control and are not efficient for resource sharing among users in a dynamically changing peer-to-peer environment.
Innovation Solution
A decentralized edge storage network that divides files into portions and uses a probabilistic approach to determine storage nodes based on distance thresholds, leveraging blockchain and smart contracts for efficient resource allocation and data distribution, with flexible sharding and erasure code encoding for high availability.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If traditional sharding techniques are used in decentralized networks, then centralized control is achieved, but peer-to-peer resource sharing efficiency deteriorates
Solution Approach 1:
The system segments files into multiple portions and distributes them across different peer nodes in the decentralized network. Each node stores and manages specific file portions independently, enabling efficient peer-to-peer resource sharing while maintaining system reliability through distributed storage architecture
Solution Approach 2:
Peer nodes autonomously locate and retrieve file portions from other nodes in the network using distributed hash table (DHT) technology and distance-based proximity queries. The system enables self-service file retrieval without centralized coordination, improving peer-to-peer resource sharing efficiency while maintaining data integrity
2Reliability
If files are divided into multiple portions and distributed across peer nodes, then data availability is improved, but system complexity increases
Solution Approach 1:
The system introduces a distributed hash table (DHT) as an intermediary mechanism that simplifies file portion location and retrieval. Nodes can efficiently find and access file portions stored on peer nodes through DHT lookups based on proximity, reducing the complexity of managing distributed file portions while maintaining high data availability
Solution Approach 2:
The system uses distance-based proximity parameters to determine optimal peer nodes for storing and retrieving file portions. By changing the selection criterion from arbitrary distribution to distance-based proximity, the system simplifies node selection and file retrieval processes while improving data availability through localized storage
3Productivity
If distance-based proximity queries are used for file portion location, then resource allocation efficiency is improved, but computational overhead increases
Solution Approach 1:
The system performs distance-based proximity queries only for file portion location and retrieval operations, rather than continuously monitoring or optimizing all network interactions. This partial application of distance-based selection achieves efficient resource allocation for critical operations while minimizing unnecessary computational overhead
Data Source
AI summary
Methods and systems for decentralized storage are provided. To store a file in a decentralized network, a source node in the decentralized network first divides the file into a plurality of file portions. For each given file portion, a corresponding peer storage node is located in the decentralized network, where a distance between the corresponding peer storage node and the given file portion is below a given threshold, where the distance is computed based on an identifier of the corresponding peer storage node, and an identifier of the given file portion, and where the given threshold is associated with a storage capacity of the corresponding peer storage node. The source node then transmits each given file portion to the corresponding peer storage node, and generates a file identifier for the file, based on the identifiers of the plurality of file portions.