Erasure Code Parity Grouping for Efficient Node Failure Recovery
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current data storage systems face challenges in efficiently adapting to varying data sizes and ensuring safe storage, particularly in handling big data, where existing error correction codes are not effective in managing node failures and data recovery.
Innovation Solution
The method involves generating local and global parity groups for data nodes, calculating an average erasure probability, and dynamically converting global parity nodes into local parity groups to optimize data storage and recovery, using a combination of Local Repairable Codes (LRC) and Enhanced Local Repairable Codes (E-LRC) to manage node failures effectively.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If conventional error correction codes are used for data storage, then data can be stored with basic protection, but the system cannot efficiently handle node failures and data recovery in big data environments
Solution Approach 1:
The patent divides the storage system into data nodes and parity nodes, further organizing parity nodes into local parity groups and global parity groups. This segmentation allows for localized recovery operations that don't require access to all nodes, improving both reliability and storage efficiency by enabling faster, more targeted data recovery.
Solution Approach 2:
The patent implements dynamic conversion between global parity groups and local parity groups based on the average erasure probability. When failure probability exceeds a threshold, global parity nodes are converted into local parity groups, allowing the system to adapt its structure in real-time to maintain optimal reliability and efficiency under varying conditions.
2Reliability
If multiple global parity nodes are generated for all data nodes, then data recovery capability is improved, but storage overhead and reconstruction costs increase
Solution Approach 1:
The patent creates local parity groups where each group serves a specific subset of data nodes with localized failure patterns. This local quality approach ensures that recovery operations for local failures only require accessing nodes within the same local group, reducing storage overhead while maintaining recovery capability for the most common failure scenarios.
Solution Approach 2:
The system dynamically adjusts the number of local parity groups based on monitored erasure probabilities. When failure rates are low, fewer local parity groups are maintained, reducing storage overhead. When failure rates increase, the system converts global parity nodes to local parity groups, optimizing the balance between reliability and storage efficiency in real-time.
3Adaptability or versatility
If the storage system structure is fixed, then implementation is simple, but the system cannot adapt to varying data sizes and failure patterns in big data environments
Solution Approach 1:
The patent implements a dynamic storage system that monitors erasure probabilities and automatically reconfigures the parity group structure. The controller converts between global and local parity groups based on monitored conditions, allowing the system to adapt to varying data sizes and failure patterns without requiring complex manual reconfiguration, thus achieving adaptability with manageable complexity.
Solution Approach 2:
The storage system performs self-monitoring and self-reconfiguration based on observed erasure patterns. The controller automatically detects when erasure probability exceeds thresholds and initiates conversions between parity group types without external intervention, enabling the system to adapt to changing conditions autonomously while maintaining operational simplicity.
Data Source
AI summary
Disclosed is a method and apparatus for encoding an erasure code for storing data. The disclosed method for encoding an erasure code comprises the steps of: (a) generating a first local parity group including two or more local parity nodes for data nodes; (b) generating at least one global parity node for the data nodes; (c) generating at least one second local parity group including two or more local parity nodes for the data nodes; and (d) storing the data nodes, the first local parity group, the second local parity group, and the global parity node. According to the disclosed method, it is possible to store and recover data safely and efficiently.