Erasure Coding Layout for Scalable Fault-Tolerant Storage
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing storage systems face challenges in providing incremental scalability, efficient fault tolerance, and optimal data distribution across a large number of disks, particularly in handling failures and adding/removing storage devices without disrupting operations.
Innovation Solution
A method for encoding data into smaller components that can be reconstructed from any set of original and new components, using non-cyclic shift operations and bitwise XOR operations to distribute redundant information across different storage devices, ensuring fault tolerance and efficient storage capacity utilization.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If RAID5 systems store redundant information across multiple disks, then data safety is improved, but storage space overhead increases
Solution Approach 1:
The patent segments data into multiple components and distributes them across different storage devices using erasure coding. Instead of storing complete redundant copies (RAID1) or simple parity (RAID5), the data is divided into N components with M redundant components, allowing flexible reconstruction while optimizing storage utilization. This segmentation approach enables the system to tolerate up to M failures while using less storage overhead than traditional replication methods.
2Quantity of substance
If subsystems are added to increase storage capacity, then total capacity is improved, but system reliability deteriorates
Solution Approach 1:
The patent implements dynamic erasure coding where the number of data components N and redundant components M can be adjusted based on system configuration and failure scenarios. This dynamic approach allows the system to optimize the balance between storage capacity and reliability by configuring different (N, M) pairs depending on whether priority is given to maximizing capacity or ensuring higher fault tolerance. The system can adaptively reconfigure redundancy levels when subsystems are added or removed.
3Reliability
If data components are rigidly assigned between RAID group elements, then fault tolerance is achieved, but incremental scaling becomes difficult
Solution Approach 1:
The patent introduces a new dimension of flexibility by allowing erasure-coded components to be distributed across storage devices beyond traditional RAID group boundaries. Instead of being constrained to fixed RAID group structures, components can span multiple devices and groups, enabling incremental scaling where devices can be added to the erasure coding set dynamically. This dimensional expansion of the storage architecture allows both fault tolerance and scalability to coexist.
4Quantity of substance
If the number of disks increases in large-scale storage systems, then storage capacity is improved, but fault tolerance management becomes more complex
Solution Approach 1:
The patent implements a universal erasure coding framework that works across varying numbers of storage devices and failure scenarios. The same (N, M) erasure coding configuration can protect data across different scales of the storage system, from small to large numbers of disks. This universal approach simplifies management by providing consistent fault tolerance mechanisms regardless of system size, eliminating the need for complex, scale-specific configurations.
Data Source
AI summary
A method for encoding a block of data to allow it to be stored or transmitted correctly in the face of accidental or deliberate modifications, the method including constructing a number n greater than one of original components, each of which is derived from the block and each of which is smaller than the block, and combining original components to construct a number m greater than one of new components, wherein each of the new components is smaller than the sum of the sizes of the original components combined to produce it, wherein the block can be reconstructed from any set of n different components selected from the original components and new components, and wherein a set of n different components selected from the original components and new components contains more redundant information about the block than the set of n original components.