Distributed Erasure Coded Virtual File System for Fault Tolerance
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Conventional data storage systems face challenges in providing efficient, scalable, and fault-tolerant solutions for distributed storage, particularly in ensuring data reliability and availability across multiple failure domains.
Innovation Solution
The implementation of a distributed, erasure-coded virtual file system that organizes data into multiple failure resilient address spaces (DFRAS) across a network of storage devices, utilizing forward error correction schemes and intelligent data management techniques to ensure data integrity and availability.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If data is distributed across multiple storage devices without erasure coding, then storage capacity and accessibility are improved, but data reliability and fault tolerance deteriorate
Solution Approach 1:
The system segments data into multiple chunks and distributes them across different storage devices. Each chunk is further divided into data digits and protection digits, with data digits stored on primary devices and protection digits stored on secondary devices. This segmentation enables both improved accessibility (data can be retrieved from multiple locations) and maintained reliability (protection digits enable recovery from failures).
Solution Approach 2:
The system applies erasure coding to generate protection digits beforehand and stores them alongside data digits across the distributed storage network. These protection digits act as a cushion against potential failures, allowing the system to recover from single or concurrent failures without data loss. The protection is prepared in advance rather than reacting to failures after they occur.
2Reliability
If erasure coding is applied to all data blocks, then data reliability is improved, but processing overhead and complexity increase
Solution Approach 1:
The system applies erasure coding selectively rather than uniformly to all data. It identifies which data blocks require protection based on local conditions such as storage device reliability, data criticality, and failure risk assessment. This local quality approach ensures high reliability for critical data while reducing processing complexity for less critical data, optimizing the balance between protection and overhead.
Solution Approach 2:
The system applies erasure coding to only the necessary portion of data blocks rather than all blocks. It determines the optimal level of protection needed for each data block and applies相应的 protection digits generation and storage only where required. This partial action reduces overall processing complexity while maintaining sufficient reliability for the most critical data portions.
3Reliability
If data is replicated across multiple storage devices, then data availability is improved, but storage efficiency and space utilization deteriorate
Solution Approach 1:
The system changes the representation parameters of stored data by using erasure coding to transform original data into data digits and protection digits. This parameter transformation allows the system to achieve data availability equivalent to replication while using less storage space. The encoded representation enables recovery from failures without requiring full replicas of the original data, thus improving storage efficiency while maintaining availability.
Data Source
AI summary
A plurality of computing devices are communicatively coupled to each other via a network, and each of the plurality of computing devices comprises one or more of a plurality of storage devices. A plurality of failure resilient address spaces are distributed across the plurality of storage devices such that each of the plurality of failure resilient address spaces spans a plurality of the storage devices. Each one of the plurality of failure resilient address spaces is organized into a plurality of stripes. Each one or more stripes of the plurality of stripes is part of a respective one of a plurality of forward error correction (FEC) protection domains. Each of the plurality of stripes may comprise a plurality of storage blocks. Each block of a particular one of the plurality of stripes may reside on a different one of the plurality of storage devices.