Compute-Node Metadata Routing for Low-Latency Storage Reads
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current data reading processes in data storage systems are hindered by cross-node data forwarding and multiple metadata searches, leading to delays and reduced reading speed.
Innovation Solution
A data storage system with a compute node that stores metadata indicating the location of target data, allowing direct reading from storage nodes without cross-node forwarding or multiple metadata searches, utilizing an intelligent network interface card (DPU) to manage metadata and reduce computational workload.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If cross-node data forwarding is performed to read deduplication data, then data can be retrieved from storage nodes, but data reading delay increases and reading speed decreases
Solution Approach 1:
The system pre-establishes metadata that maps logical addresses to physical addresses and pre-determines the locations of deduplication data across storage nodes. When a read request arrives, the compute node already has the necessary routing information, eliminating the need for cross-node data forwarding and multiple metadata searches during the actual read operation.
2Reliability
If multiple metadata searches are performed to access snapshot data, then the required snapshot data can be located, but reading delay increases and reading speed decreases
Solution Approach 1:
The system merges multiple levels of snapshot metadata into a unified metadata structure maintained by the compute node. This consolidated metadata contains direct mappings from logical addresses to physical addresses across all snapshot levels, allowing the compute node to locate snapshot data in a single lookup operation rather than performing multiple sequential searches across different storage nodes.
3Productivity
If compute node directly reads data using stored metadata, then data reading speed improves, but compute node must maintain and manage metadata
Solution Approach 1:
The intelligent network interface card serves as an intermediary component between the compute node and storage nodes. It is equipped with dedicated storage units to hold metadata and compute units to perform address translation and data routing operations. This offloads metadata management and processing from the compute node's main processor, reducing computational workload while enabling direct data reading.
Data Source
AI summary
A data storage system, an intelligent network interface card, and a compute node are provided, and belong to the field of data storage. The data storage system includes a compute node and a plurality of storage nodes. The compute node stores metadata, and the metadata may indicate an address at which target data is stored in the plurality of storage nodes. According to this application, the compute node may directly read the target data from a corresponding storage node based on the address indicated by the metadata stored in the compute node, does not need to perform cross-node data forwarding by using the plurality of storage nodes, and does not need to perform a plurality of times of metadata searching. This effectively reduces a data reading delay and improves a data reading speed.