Decentralized Edge Storage File Sharding and Node Selection
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing technologies face challenges in implementing efficient data partitioning and distribution techniques in fully decentralized networks, particularly for Web3 applications that require fast and efficient resource sharing.
Innovation Solution
The proposed solution involves a computer-implemented method for decentralized storage, where a source node in a decentralized network divides a file into multiple file portions and locates corresponding peer storage nodes based on a computed distance threshold associated with each node's storage capacity, before transmitting and generating a file identifier for the file.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Adaptability or versatility
If traditional centralized sharding techniques are used, then data partitioning can be controlled by a central authority, but implementation in fully decentralized networks becomes difficult
Solution Approach 1:
The system segments files into multiple sharded copies and distributes them across different peer nodes in the decentralized network. Each peer stores only portions of files rather than complete files, enabling decentralized storage without requiring centralized control over the partitioning process.
Solution Approach 2:
Peer nodes autonomously select and store sharded copies of files based on their own storage capacity and proximity metrics. The system enables self-organizing distribution where nodes independently determine their role in storing file portions without central coordination.
2Productivity
If file sharding is implemented across distributed peer storage nodes, then resource sharing efficiency improves, but locating suitable storage nodes becomes challenging
Solution Approach 1:
The system transforms the node selection problem into a parameter-based matching problem. It computes proximity metrics based on storage capacity parameters and node identifiers, converting qualitative suitability into quantitative distance measurements that can be efficiently calculated and compared.
Solution Approach 2:
The system replaces manual or centralized node selection mechanisms with automated computational methods. Distance computation algorithms automatically identify suitable storage nodes based on mathematical proximity metrics, substituting mechanical search processes with efficient computational matching.
3Reliability
If peer nodes store complete files, then data integrity is maintained, but storage capacity requirements increase significantly
Solution Approach 1:
Files are divided into multiple smaller sharded copies that can be distributed across numerous peer nodes. Each node stores only a portion of the file rather than the complete file, dramatically reducing individual storage requirements while maintaining overall data availability through redundancy.
Solution Approach 2:
The system changes the storage parameter from storing complete files at each node to storing fragmented portions. This parameter transformation allows the network to achieve the same data integrity goals with significantly reduced total storage capacity requirements across the network.
Data Source
AI summary
Methods and systems for decentralized storage are provided. To store a file in a decentralized network, a source node in the decentralized network first divides the file into a plurality of file portions. For each given file portion, a corresponding peer storage node is located in the decentralized network, where a distance between the corresponding peer storage node and the given file portion is below a given threshold, where the distance is computed based on an identifier of the corresponding peer storage node, and an identifier of the given file portion, and where the given threshold is associated with a storage capacity of the corresponding peer storage node. The source node then transmits each given file portion to the corresponding peer storage node, and generates a file identifier for the file, based on the identifiers of the plurality of file portions.