Coded Fragment Updates for Distributed Storage Version Control
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing distributed storage systems face inefficiencies in updating coded fragments, requiring full recoding and network transmission, especially when files are modified, leading to high network strain and storage usage, and lack seamless update capabilities.
Innovation Solution
The use of linear erasure codes to efficiently update coded fragments by calculating and transmitting only the coded difference between original and modified data, allowing for partial recovery without full reupload, and enabling efficient version control in distributed and cloud storage systems.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If the entire file is recoded and all coded fragments are transmitted through the network when any portion of a file is changed, then the distributed storage system maintains data reliability and consistency, but the network strain and storage usage increase significantly
Solution Approach 1:
The patent segments the file update process by dividing the file into fragments and applying erasure coding to generate coded fragments. When updates are needed, only the affected coded fragments are identified and transmitted, rather than recoding the entire file. This segmentation allows selective updates that reduce network strain while maintaining data reliability through the distributed coded fragment storage.
2Reliability
If the entire file is recoded and all coded fragments are transmitted through the network when any portion of a file is changed, then the distributed storage system maintains data reliability and consistency, but the storage space requirements increase
Solution Approach 1:
The patent changes the parameter representation by storing coded fragments instead of complete file copies. When updates occur, the system modifies only the necessary coded fragments using erasure code mathematics, rather than storing entire new file versions. This parameter change in the storage representation reduces storage space requirements while preserving data reliability through the mathematical properties of erasure codes.
3Quantity of substance
If differences between original and modified data are stored in unencoded form, then storage space is reduced, but the system loses the benefits of coding and cannot update files seamlessly
Solution Approach 1:
The patent creates a composite storage system that combines encoded and unencoded representations strategically. Coded fragments are stored in encoded form for efficient retrieval and update operations, while difference information is maintained in unencoded form for space efficiency. This composite approach allows the system to leverage both the space-saving benefits of unencoded differences and the operational benefits of encoded storage for seamless file updates.
4Adaptability or versatility
If multiple versions of a file are maintained, then version control capability is improved, but the network use and storage use of the system increase
Solution Approach 1:
The patent implements version control by nesting difference information within the coded fragment structure. Instead of storing complete separate versions of files, the system stores coded fragments that can represent multiple versions through layered difference encoding. This nested structure allows version control capability while reducing storage use, as each coded fragment can serve multiple version requirements through mathematical transformations rather than requiring separate full-file storage for each version.
Data Source
AI summary
A method for use in a distributed storage system having a plurality of nodes, the method including receiving, at a source node, original data, encoding the original data into plurality of coded fragments using a linear code, transmitting at least a first one of the coded fragments from the source node to a first sink node. The method further includes receiving, at the source node, modified data, calculating, at the source node, a coded difference between the original data and the modified data, transmitting the coded difference from the source node to the first sink node; and recovering, at the first sink node, at least a first portion of the modified data using the coded difference and the at least a first one of the coded fragments.


