Distributed Parity Protection Conglomerates for Multi-Server Failure
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing data storage systems lack the ability to perform a put operation on a new data chunk with parity protection using only the data bandwidth required for a single multicast transmission, while efficiently creating parity protected data, especially for 'cold' data that is unlikely to be accessed frequently.
Innovation Solution
The introduction of a Parity Protection Conglomerate (PPC) system that allows for distributed creation of coordinated PPCs to protect against the concurrent loss of multiple failure domains in a distributed storage cluster, using a method that requires no additional network transmissions beyond those needed for creating whole replicas, and allows for efficient storage and recovery of data chunks.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If whole replicas are created for each data chunk, then data reliability is improved, but storage requirements increase significantly
Solution Approach 1:
The patent segments data into chunks and creates parity protection conglomerates (PPCs) that protect multiple chunks simultaneously. Instead of replicating entire data sets, the system divides data into manageable chunks and applies parity protection selectively, reducing overall storage requirements while maintaining reliability.
Solution Approach 2:
The system changes the protection parameter from full replication to parity-based protection. By using parity protection conglomerates that can protect against the concurrent loss of multiple failure domains, the system achieves the same reliability level with reduced storage overhead compared to creating multiple whole replicas.
2Quantity of substance
If parity protection is implemented for new data chunks, then storage efficiency is improved, but network bandwidth requirements increase
Solution Approach 1:
The system performs preliminary actions by creating parity protection conglomerates during the initial put operation. By pre-calculating and storing parity information along with data chunks in a coordinated manner, the system avoids the need for additional network transmissions later when parity protection is needed, thus reducing overall network bandwidth requirements.
3Device complexity
If distributed parity protection is implemented without centralized control, then system complexity is reduced, but coordination of parity protection becomes difficult
Solution Approach 1:
The patent implements self-service mechanisms where storage servers autonomously participate in creating and maintaining parity protection conglomerates. Each server independently contributes to the distributed parity protection system by storing data chunks and parity information, eliminating the need for complex centralized coordination while maintaining ease of operation through standardized protocols.
Data Source
AI summary
The present invention introduces a specific form of parity protection chunk (PPC) that allows for distributed creation of coordinated PPCs that can reliably provide protection against the concurrent loss of two or more storage servers of devices. Coordinated PPCs can protect against the concurrent loss of multiple lost chunks by ensuring that the PPCSs protecting any specific chunk have at most a single overlapping failure domain. This is done without requiring full centralized control over the assignment of protected chunks to specific PPCs. The PPCs are created as part of the put transaction for a chunk. The chunk itself is stored as a whole replica, and the PPCs provide parity protection for the chunk.