Distributed File Storage Architecture for Fast Failure Reconstruction
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing storage systems with error correction code redundancy schemes have single points of failure and cannot reconstruct failed drives quickly, as responsibility for data storage and maintenance resides in a single component, leading to inefficiencies in fault tolerance and recovery.
Innovation Solution
A dynamically scalable redundant distributed storage system that separates logical responsibility for data from actual control, using distributed hash tables and components that can be dynamically added or removed, employing error correction codes or replication for redundancy, allowing for rapid recovery and reconstruction of failed data.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If a single component is used to store and maintain data with error correction code redundancy, then data integrity can be ensured, but the system has single points of failure and cannot reconstruct failed drives quickly
Solution Approach 1:
The patent segments the storage system into multiple independent storage nodes distributed across the network. Each node stores portions of data with error correction codes, eliminating the single point of failure. The segmentation allows the system to tolerate individual node failures while maintaining overall data integrity and enabling parallel reconstruction of failed drives from multiple sources.
2Ease of operation
If data is stored in a centralized manner with single component responsibility, then system control is simplified, but reconstruction of failed drives takes the time to rewrite the entire drive
Solution Approach 1:
Data is segmented into multiple chunks distributed across different storage nodes. When a drive fails, reconstruction can proceed in parallel by retrieving different segments from different nodes simultaneously, dramatically reducing total reconstruction time compared to sequential rewriting from a single source.
Solution Approach 2:
Error correction codes are pre-calculated and stored with the data segments during the initial storage phase. This preliminary action enables rapid reconstruction without requiring complex real-time calculations, allowing the system to quickly recover failed drives by simply reassembling pre-encoded segments from surviving nodes.
3Reliability
If traditional replication methods are used for fault tolerance, then data can be recovered from failures, but the system cannot scale dynamically and requires special hardware
Solution Approach 1:
The system uses universal software-based storage nodes that can run on standard hardware platforms. The error correction code mechanism works uniformly across all nodes regardless of hardware specifics, allowing the system to scale dynamically by adding or removing nodes without requiring special hardware or reconfiguring fault tolerance mechanisms.
Solution Approach 2:
The distributed storage system dynamically adapts to changes in the number of available nodes. Error correction codes are automatically reconfigured based on the current number of participating nodes, allowing the system to maintain optimal fault tolerance and scalability as nodes are added or removed from the distributed network.
Data Source
AI summary
A distributed heterogeneous or homogeneous file storage system divides logical responsibility for data from actual control of the data by using separate file and storage managers. Each file manager provides access to stored files and stores metadata associated with each of the files for which it is responsible. A file manager control component allocates responsibilities among file managers, assigns management responsibilities for individual files, and maintains a record of which files are assigned to each manager. Each storage manager is responsible for managing storage of files on data storage devices and maintains a record of the location of each stored file. A storage manager control component allocates responsibilities among storage managers, assigns responsibilities for individual files, and maintains a record of which files and storage devices are assigned to each manager. A client provides user access, initiates actions within the system, and implements a redundant data recovery and reconstruction scheme.