Accelerator Pool Data Storage Acknowledgment Latency
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current data storage systems face challenges in providing high throughput data storage with low latency due to the time required for redundant data storage across multiple fault domains, and they often require all computing devices in a distributed system to have high performance capabilities, which is costly and inefficient.
Innovation Solution
A data storage system comprising an accelerator pool that provides high-performance data storage services by acknowledging data storage before the data is fully redundantly stored across separate fault domains, using non-volatile memory and RDMA to store and migrate data efficiently, while relegating lower performance data silos to store secondary copies.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If data is stored redundantly across multiple fault domains before acknowledging storage completion, then data reliability is improved, but storage latency increases
Solution Approach 1:
The system performs preliminary actions by storing the first copy of data in the accelerator pool and immediately acknowledging storage completion to the client before the redundant copy is written to the data silo. This preliminary acknowledgment allows the client to proceed without waiting for full redundancy, while the system continues to complete the redundant storage in the background.
Solution Approach 2:
The storage system is segmented into two distinct components with different performance characteristics: an accelerator pool for high-speed initial storage and acknowledgment, and data silos for slower redundant storage. This segmentation allows the critical path (acknowledgment) to be decoupled from the non-critical path (redundant storage), resolving the contradiction between reliability and latency.
2Productivity
If all computing devices in the distributed system have high performance capabilities, then data storage throughput is improved, but system cost increases
Solution Approach 1:
The system applies local quality by assigning different performance levels to different components based on their functional requirements. The accelerator pool, which handles high-speed data ingestion and acknowledgment, is equipped with high-performance computing resources. The data silos, which handle redundant storage, use lower-performance, more cost-effective hardware. This heterogeneous architecture optimizes throughput where needed while minimizing overall system cost.
Solution Approach 2:
The system creates a copy of the data storage function across two different types of infrastructure: high-performance accelerator pool devices and lower-performance data silo devices. This copying approach allows the system to achieve high throughput for the critical storage operation while using cheaper hardware for the non-critical redundant storage, thereby reducing overall system cost.
Data Source
Figure 1.1
Figure 1.2
Figure 2.1
AI summary
A data storage system includes an accelerator pool and data silos. The accelerator pool obtains a data storage request for first data; stores a copy of the first data locally in a memory of the accelerator pool; in response to storing the copy of the first data: sends an acknowledgement to a requesting entity that generated the data storage request; and, after sending the acknowledgement, stores at least one second copy of the first data in the data silos. The acknowledgement indicates that the first data is redundantly stored in at least two different fault domains.