Intermediate Storage Manager for Low-Latency Data Writes
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
In multi-node computer systems, the direct communication between computing nodes and storage nodes for data access leads to increased load and latency due to the need for multiple requests, particularly when writing data to storage spaces.
Innovation Solution
Implementing an intermediate device that manages metadata of storage spaces, allowing computing nodes to send write requests directly to it, which allocates addresses and handles data operations based on stored metadata, reducing the need for direct communication with storage nodes.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If the storage node directly manages the storage space of the application, then the storage space can be effectively managed, but the load of the computing node and data processing latency increase when the computing node accesses application data
Solution Approach 1:
The patent introduces a storage manager as an intermediary component that sits between the computing node and the storage node. The storage manager receives storage space management requests from the computing node, allocates storage addresses, and communicates with the storage node on behalf of the computing node. This intermediary approach allows the computing node to offload storage management tasks, reducing its load and improving data processing efficiency while maintaining effective storage space management through the storage manager's coordination with the storage node
2Reliability
If the computing node sends multiple requests to the storage node for data writing operations, then the storage operations can be completed, but the load of the computing node and data write latency increase
Solution Approach 1:
The storage manager pre-allocates storage addresses and prepares storage space before the computing node needs to write data. When the computing node sends a write request, the storage manager has already prepared the target storage location, allowing the data to be written directly without requiring multiple back-and-forth requests between the computing node and storage node. This preliminary preparation significantly reduces data write latency
Solution Approach 2:
The storage manager acts as an intermediary that consolidates multiple storage operations into single coordinated actions. Instead of the computing node sending separate requests for address allocation, data writing, and status confirmation to the storage node, the storage manager handles these operations in a streamlined manner, reducing the number of communication rounds and minimizing data write latency
Data Source
Figure 1~2
Figure 3
Figure 4
AI summary
Embodiments of this application provide a method for processing data by using an intermediate device, a computer system, and an intermediate device. The computer system includes a computing node, a storage node, and an intermediate device. A service runs in the computing node. The storage node stores data of the service. The service is provided with a storage space, and the service performs an operation on the data of the service in the storage node by accessing the storage space. For example, the storage space is a persistence LOG (PLOG) space. In embodiments of this application, metadata of the storage space is stored in the intermediate device. The intermediate device manages the metadata of the storage space, and implements a data operation between the computing node and the storage node based on the metadata of the storage space. For example, the intermediate device is a switch. According to the solutions provided in this application, a load of the computing node and a data access latency can be reduced.