Decoupled Storage System Using Consumer SSDs
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Traditional data center storage systems are inefficient due to coupled computing and storage resources, high costs of enterprise-grade SSDs, and potential single points of failure in server-attached storage configurations, which limit scalability and performance.
Innovation Solution
A large-scale storage system with decoupled computing and storage resources, utilizing consumer-grade SSDs and NVRAM, where compute nodes manage storage operations and communicate with storage nodes via a fabric, reducing write-erase cycles and enabling independent operation and scalability.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If enterprise-grade SSDs are used to ensure high write-erase cycles, then reliability is improved, but cost increases
Solution Approach 1:
The system segments storage operations into two distinct paths: a first storage path for frequently accessed data using consumer-grade SSDs, and a second storage path for archival data using lower-cost storage media. This segmentation allows the system to use expensive enterprise-grade SSDs only when necessary while using cheaper storage for bulk data, thereby reducing overall cost while maintaining reliability for critical operations.
Solution Approach 2:
The system dynamically changes operational parameters by monitoring data access patterns and automatically transitioning data between the first storage path (consumer-grade SSDs with higher write-erase cycle limits) and the second storage path (lower-cost storage). This parameter change allows the system to optimize both cost and reliability based on actual usage patterns rather than statically assigning all data to expensive enterprise-grade storage.
2Ease of operation
If computing and storage resources are coupled in server-attached storage, then ease of operation is improved, but adaptability deteriorates
Solution Approach 1:
The system extracts the storage management function from traditional server-attached storage and implements it as a separate, centralized software-defined storage layer. This extracted storage management system can independently provision, manage, and allocate storage resources across multiple compute nodes without requiring physical reconfiguration of server hardware, thereby enabling independent scaling of both compute and storage resources while maintaining ease of operation through software abstraction.
Solution Approach 2:
The software-defined storage system provides universal storage management capabilities that can serve multiple different compute nodes and workloads through a single unified interface. This multi-functional storage management layer can dynamically allocate storage resources to different compute nodes based on demand, enabling both ease of operation through centralized management and adaptability through independent scaling of storage capacity without requiring corresponding increases in compute nodes.
3Ease of operation
If storage drives are attached directly to servers, then ease of operation is improved, but device complexity increases due to single point of failure
Solution Approach 1:
The system implements beforehand cushioning by creating redundant storage paths and using software-defined storage abstraction that eliminates single points of failure. Instead of directly attaching storage drives to servers where a single server failure would render all attached drives inaccessible, the system uses a distributed storage architecture where data is replicated and can be accessed through multiple paths, cushioning against failures before they impact data availability.
Solution Approach 2:
The system introduces an intermediary software-defined storage layer between the physical storage drives and the compute nodes. This intermediary manages storage operations, handles failure recovery, and provides abstracted access to storage resources. By inserting this intermediary layer, the system maintains ease of operation for users while reducing device complexity and failure risk through centralized management and redundancy.
Data Source
AI summary
A large-scale storage system is provided. The large-scale system includes a plurality of compute nodes; a plurality of storage nodes; and a communication fabric for providing a communication infrastructure between the plurality of compute nodes and the plurality of storage nodes; wherein each compute node of the plurality of compute nodes is configured to independently perform at least a storage operation on any of the storage nodes in a persistent manner, and wherein each storage node of the plurality of storage nodes provides physical storage space of the large-scale storage system.