Random Linear Coding for Distributed Data Storage
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Conventional file transfer mechanisms, such as client-server and peer-to-peer networks, face inefficiencies in distributing large or popular files, particularly in terms of bandwidth usage and resource management, with peer-to-peer systems experiencing protocol overhead and limited availability of obscure files.
Innovation Solution
The implementation of a random linear coding approach for distributed data storage in peer-to-peer networks, where a file is broken into pieces and each peer stores a random combination of these pieces with an associated code vector, allowing for efficient storage and retrieval without the need for a central controller.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If a file is broken into multiple pieces and stored in a peer-to-peer network using conventional methods, then the file can be distributed across multiple peers, but it becomes difficult to determine an efficient way to store and retrieve the pieces without a central controller
Solution Approach 1:
The file is divided into multiple pieces that are distributed across different peers in the network. Each peer stores only a portion of the file, which enables efficient distribution while reducing the load on any single node. This segmentation allows the system to scale horizontally across multiple peers without requiring a central controller to manage the entire file.
Solution Approach 2:
The system enables peers to autonomously store and retrieve file pieces without requiring a central controller. Each peer can independently contribute storage space and retrieve pieces it needs, with the system self-organizing through the properties of linear coding. This eliminates the coordination overhead associated with centralized management while maintaining efficient file distribution.
2Ease of operation
If random uncoded storage is used where each peer stores k different pieces out of m pieces at random, then storage is simplified, but the number of peers required for efficient file reconstruction increases
Solution Approach 1:
The system changes the storage parameter from storing raw file pieces to storing linear combinations of file pieces with associated code vectors. This parameter change allows for more efficient file reconstruction because the coded pieces contain redundant information that enables recovery with fewer peers. The code vectors provide mathematical structure that reduces the number of peers needed compared to random uncoded storage.
3Extent of automation
If conventional peer-to-peer systems are used, then files can be shared without a server, but there is significant protocol overhead for passing search queries amongst peers
Solution Approach 1:
The system extracts and eliminates the need for complex search query protocols by using a mathematical approach based on linear coding. Instead of peers needing to communicate extensively to find and coordinate file pieces, the system uses the properties of linear codes to enable direct retrieval. This extraction of the search coordination function reduces protocol overhead while maintaining serverless operation.
Data Source
AI summary
A method and computer program product for providing a random linear coding approach to distributed data storage is presented. A file is broken into a plurality of pieces. For every peer (peer means storage-location with limited storage space), the number of coded-pieces the peer can store is determined. Each of the coded-piece is determined by taking random linear combination of all the pieces of the entire file. The associate code-vector is stored for every coded-piece. The file is retrieved by collecting code-vectors and the coded-pieces from the peers and viewing the collected code-vectors as a matrix. When a dimension of the matrix is equal to the number of pieces of the file, the file is recovered using the collection of code vectors in the matrix.