Distributed Electronic Storage with SSD Agents and Failure Resilience
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Conventional file systems face limitations in providing high availability, reliability, and resiliency against failures, especially in distributed environments, and often require complex software implementations for data migration and load balancing.
Innovation Solution
A virtualized distributed file system that utilizes erasure coding for resiliency, policy-based tiered storage, and flexible core allocation to ensure high performance and reliability, with automatic load balancing and data redistribution across SSDs and object stores.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If conventional file systems are used in distributed environments, then basic file storage is achieved, but high availability, reliability, and resiliency against failures cannot be provided
Solution Approach 1:
The system divides the distributed file system into independent backend components, each responsible for specific storage operations. Multiple backends can operate autonomously, providing fault isolation and enabling the system to maintain high availability even when individual components fail, without requiring complex inter-component coordination software.
Solution Approach 2:
The storage backend is designed to handle multiple functions including data storage, load balancing, and failure recovery through a unified interface. The system can operate with different configurations (single backend or multiple backends) and automatically adapts to failure scenarios, providing universal functionality that simplifies software implementation while maintaining reliability.
2Speed
If data is stored on SSDs for high performance, then data access speed is improved, but the system becomes vulnerable to power failures and requires complex protection mechanisms
Solution Approach 1:
The system implements prior cushioning by creating redundant copies of data across multiple backends before any failure can occur. When data is written to an SSD, it is simultaneously replicated to other backends, ensuring that power failures or SSD malfunctions do not result in data loss. This proactive redundancy eliminates the need for complex post-failure recovery mechanisms.
3Reliability
If multiple backends are used for load balancing, then system reliability is improved, but data redistribution complexity increases
Solution Approach 1:
The backend components perform self-service by automatically detecting failures and redistributing their data loads to remaining healthy backends without external intervention. When a backend fails, the system automatically rebalances data across available backends, eliminating the need for complex manual data redistribution procedures while maintaining high reliability.
4Device complexity
If SSDs are used without uninterruptible power supplies, then system complexity is reduced, but data integrity during power failures is compromised
Solution Approach 1:
The system uses copying to create redundant data copies across multiple backends immediately upon writing. This eliminates the need for uninterruptible power supplies because data is already replicated and protected against power failures before they can occur. The copying mechanism provides data integrity protection through software rather than requiring complex hardware infrastructure.
Data Source
Figure 1
Figure 2
AI summary
Methods and systems for electronic storage are provided. A storage system comprises a plurality of storage system front ends, a plurality of storage system back ends, and a plurality of solid state drive (SSD) agents. Each storage system front end resides on a server of a plurality of servers. Each server of the plurality of servers comprises one or more storage system back ends of the plurality of storage system back ends. Each storage system front end is able to receive I/O requests and relay information associated with the I/O requests to a relevant storage system back end. The relevant storage system back end communicates metadata associated with the I/O request to an SSD via an SSD agent.