Erasure-Coded Distributed Storage for Independent Block Updates
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Distributed data storage systems face high storage costs and inefficiencies due to data replication and fragmentation, as well as the need for simultaneous modification of all data blocks within a protection group when one block is written or read.
Innovation Solution
The implementation of an erasure resilient coding (ERC) system that uses software to create multiple protection groups across and within storage nodes, allowing independent read and write operations and load balancing by interleaving original and ERC data blocks, reducing hardware requirements and storage costs.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If data is replicated multiple times to ensure reliability, then data availability is improved, but storage cost increases
Solution Approach 1:
The patent uses erasure coding to create parity data blocks that are mathematical copies of the original data blocks. These parity blocks can be used to reconstruct lost data blocks, providing redundancy without requiring full duplicate copies of the entire dataset. This reduces storage requirements compared to traditional replication while maintaining data availability.
Solution Approach 2:
The patent changes the redundancy parameter from full replication (100% duplicate storage) to erasure coding with configurable redundancy ratios (e.g., storing k data blocks and m parity blocks where m < k). This allows flexible adjustment of the balance between storage efficiency and data reliability.
2Productivity
If data is fragmented into multiple blocks for distributed storage, then storage scalability is improved, but data recovery complexity increases
Solution Approach 1:
The patent segments data into fixed-size blocks and organizes them into superblocks that contain both data blocks and parity blocks. This hierarchical segmentation allows efficient distributed storage while simplifying recovery operations, as the superblock structure provides a clear framework for identifying and reconstructing lost blocks.
Solution Approach 2:
The patent introduces superblocks as intermediary structures that group related data blocks and parity blocks together. These superblocks act as organizational units that simplify the recovery process by providing a structured context for applying erasure decoding algorithms, reducing the complexity of managing individual fragmented blocks.
3Stability of the object's composition
If all data blocks in a protection group must be modified when one block is written, then data consistency is improved, but write operation performance deteriorates
Solution Approach 1:
The patent pre-calculates and stores the relationships between data blocks and parity blocks within superblocks during the data initialization phase. This preliminary organization allows write operations to efficiently identify only the specific parity blocks that need updating, rather than requiring modification of all blocks in a protection group, thus improving write performance while maintaining consistency.
Data Source
AI summary
An erasure resilient coding (ERC) distributed data storage system and method for storing data in a reliable and survivable fashion while minimizing hardware and associated costs. The system and method includes forming multiple protection groups both within and across storage nodes of the storage system. Data is segmented into original data blocks and ERC data blocks. Load balancing occurs by interleaving storage nodes with equal numbers of original data blocks and ERC data blocks while ensuring each node has an equal number of combined read and write operations. Unique read and write operations on data block can be performed independent of other data blocks in a protection group. The write operation uses Galois field arithmetic and ERC transform to either write or append a new data block to a storage node. The read operation recovers data in a variety of ways using ERC decoding.