Layered Error Correction for Distributed Object Storage Recovery
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Large scale distributed object storage systems face inefficiencies in data recovery due to excessive network transportation of fragments and excessive parity calculation, leading to high computational and storage overheads.
Innovation Solution
An improved encoding and recovery process that calculates parity information over accumulated fragments rather than individual ones, providing thinner parity coverage sufficient for common data loss events, with additional parity calculation at servers and network-based recovery for insufficient local recovery.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If parity information is calculated for every individual fragment to ensure data integrity, then data reliability is improved, but computational overhead and storage space increase significantly
Solution Approach 1:
The patent combines multiple data fragments into a single encoded stream and calculates parity information for the entire combined data rather than for each individual fragment. This merging approach reduces the total amount of parity calculation required while maintaining data integrity across all fragments.
Solution Approach 2:
The encoded stream serves multiple functions simultaneously: it protects multiple different fragments from corruption, enables recovery of any single lost fragment, and reduces overall computational overhead. The single parity calculation provides universal protection across the entire data set.
2Reliability
If extensive parity information is calculated for each fragment to protect against data loss, then data recovery capability is improved, but storage space is wasted due to parity overkill
Solution Approach 1:
Multiple fragments are merged into a single encoded stream where parity information is calculated once for the entire combination. This eliminates the redundancy of calculating separate parity for each fragment, reducing total storage requirements while maintaining recovery capability.
Solution Approach 2:
The patent applies parity protection at the optimal level - not excessive individual fragment protection, but sufficient combined protection. The single parity calculation for the encoded stream provides exactly the right amount of protection needed without waste.
3Reliability
If parity information is calculated at the source before network transmission, then data protection is improved, but network inefficiency increases due to excessive data transportation
Solution Approach 1:
The patent merges multiple fragments into a single encoded stream with integrated parity information, reducing the total volume of data that needs to be transmitted across the network compared to sending separate protected fragments.
Solution Approach 2:
The data is segmented into an optimized encoded stream format that combines multiple fragments efficiently, allowing for reduced network transmission while maintaining protection. The segmentation enables selective recovery without requiring full data reconstruction.
Data Source
AI summary
A method is described. The method includes fragmenting data of an object for storage into an object storage system into multiple data fragments and performing a first error correction encoding process on the data to generate one or more parity fragments for the object. The method also includes sending the multiple data fragments and the one or more parity fragments over a network to different storage servers of the object storage system. The method also includes performing the following at each of the different storage servers: i) incorporating the received one of the multiple data fragments and one or more parity fragments into an extent comprising multiple fragments of other objects; ii) performing a second error correction encoding process on multiple extents including the extent to generate parity information for the multiple extents; and, iii) storing the multiple extents and the parity information.