HAIL Distributed Storage Parity Block Update Mechanism
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Conventional approaches to verifying file availability and integrity in cloud storage environments are inadequate, particularly against a 'mobile adversary' that can progressively attack multiple storage providers, and they do not efficiently handle file updates.
Innovation Solution
A distributed storage system with a High-Availability and Integrity Layer (HAIL) that uses parity groups and Reed-Solomon codes to ensure file integrity and security, where files are encoded into multiple sets of blocks across servers, with parity blocks computed and updated efficiently to prevent adversary manipulation.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If conventional POR or PDP approaches are used to verify file integrity, then verification can be performed, but the system is vulnerable to mobile adversaries that can progressively attack and corrupt all storage providers
Solution Approach 1:
The system segments the verification process into multiple independent challenge-response interactions with different storage providers. Each provider is challenged independently, and the client aggregates results. This segmentation prevents a mobile adversary from compromising the entire verification system by attacking single providers sequentially, as each challenge uses fresh randomness and targets different file portions.
Solution Approach 2:
The client performs preliminary actions by generating challenge values and selecting which storage providers to challenge before the adversary can execute corruption. The system proactively verifies file integrity across distributed providers rather than reactively responding to corruption, preventing mobile adversaries from establishing dominance before detection.
2Reliability
If traditional erasure coding with Reed-Solomon codes is used for distributed storage, then file redundancy and recoverability are achieved, but file updates require extensive recomputation of all parity blocks
Solution Approach 1:
The system applies local quality by enabling selective updating of only those parity blocks that are directly affected by file modifications. When a file update occurs, the client identifies and recomputes only the specific parity blocks associated with the changed data portions, rather than regenerating the entire set of parity blocks. This localized approach maintains file recoverability while dramatically improving update efficiency.
Solution Approach 2:
The parity blocks are segmented and organized such that each parity block is associated with specific file blocks. This segmentation allows the system to update only the relevant subset of parity blocks when file content changes, rather than treating the entire parity set as a monolithic unit that requires complete recomputation.
3Reliability
If file blocks are distributed across multiple servers for redundancy, then file availability is improved, but the complexity of managing parity groups and tracking block locations increases
Solution Approach 1:
The system applies universality by using a unified metadata structure that simultaneously tracks file block locations, parity group associations, and server assignments. This multi-functional metadata approach consolidates what would otherwise require separate tracking mechanisms, reducing overall system complexity while maintaining distributed file availability across multiple servers.
Data Source
AI summary
A client device or other processing device comprises a file encoding module, with the file encoding module being configured to separate a file into a plurality of sets of file blocks, to assign sets of the file blocks to respective ones of a plurality of servers, to define a plurality of parity groups each comprising a different subset of the plurality of servers, to assign, for each of the servers, each of its file blocks to at least one of the defined parity groups, and to compute one or more parity blocks for each of the parity groups. The file blocks are stored on their associated servers, and the parity blocks computed for each of the parity groups are stored on respective ones of the servers other than those within that parity group. Such an arrangement advantageously ensures that only a limited number of parity block recomputations are required in response to file block updates.