Storage Engine Data Path Bypassing Host Memory for NVMe Write Efficiency
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current data storage methods in storage engines are inefficient due to the complexity and high costs associated with quality of service (QOS) optimization, leading to low data storage efficiency.
Innovation Solution
The method involves transmitting target data directly from a persistent memory region of a storage device to its internal memory, bypassing the host memory or Apache Pass (AEP), using a Non-Volatile Memory Express (NVMe) protocol and Direct Memory Access (DMA) to optimize data writing, thereby eliminating the need for additional PCIe traffic and reducing bandwidth usage.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If data is transmitted through host memory or AEP before reaching storage device, then data storage can be achieved, but data storage efficiency is low and costs are high
Solution Approach 1:
The patent extracts and removes the host memory or AEP from the data transmission path. By enabling direct data transmission between the application and storage device, it eliminates the intermediate memory components that were causing efficiency losses and increased complexity, directly resolving the technical contradiction.
Solution Approach 2:
The patent introduces a persistent memory region as an intermediary buffer that enables direct data transmission between application and storage device. This mediator allows data to be temporarily stored in a persistent manner without requiring traditional host memory or AEP, thus improving efficiency while maintaining data integrity.
2Loss of time
If traditional data transmission path through host memory is used, then data can be stored, but latency is high
Solution Approach 1:
The patent removes host memory from the critical data transmission path, eliminating the latency associated with memory access and data copying operations. This direct transmission approach significantly reduces the time loss while maintaining high data storage speed.
Solution Approach 2:
The patent performs preliminary data preparation in the persistent memory region before final storage, allowing data to be staged and optimized for storage operations. This preliminary action in a persistent buffer reduces the latency of actual storage operations while maintaining high throughput.
3Reliability
If QOS optimization is implemented using host memory or AEP, then service quality can be maintained, but costs increase
Solution Approach 1:
The patent uses persistent memory region as an intermediary that provides QOS optimization capabilities without requiring traditional host memory or AEP resources. This mediator maintains service quality through persistent buffering while eliminating the need for expensive memory resources, directly resolving the contradiction between reliability and resource consumption.
Solution Approach 2:
The patent replaces expensive, long-lived memory resources (host memory or AEP) with a more efficient persistent storage solution. By using the persistent memory region for temporary data holding, the system achieves QOS optimization without consuming valuable memory resources, effectively substituting expensive resources with a more cost-effective approach.
Data Source
AI summary
A data processing method, a storage medium, and a processor are provided that solve the technical problem of low efficient of data storage. The data processing method is applied to a storage engine, and involves determining that target data is transmitted from a target application to a first persistent memory region of a storage device, and sending a target instruction to the storage device. The target instruction is used for instructing the storage device to write the target data from the first persistent memory region into a memory of the storage device.