Zero-Copy Data Assembly in Storage Acceleration Modules
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Traditional distributed storage systems face increased data transfer latency and resource consumption due to multiple copying processes required for data storage and the inclusion of check information for data reliability, leading to a write amplification effect.
Innovation Solution
A data processing system comprising a network card module, a data management module, and a storage acceleration module, all with read-write access permission to a target memory space, allowing for zero-copy data transmission by directly accessing and assembling data in memory without additional copying or replication.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If multiple copying processes are used to store data from network card to hard drive, then data reliability is improved, but data transfer latency and resource consumption increase
Solution Approach 1:
The patent segments data into data segments and check segments, storing them separately in different memory spaces. This allows parallel processing and eliminates the need for multiple sequential copying operations, reducing latency while maintaining data reliability through distributed storage of data and its verification information.
Solution Approach 2:
The patent introduces a storage acceleration module as an intermediary that directly accesses memory spaces and manages data segment assembly. This intermediary eliminates the need for traditional multi-step copying through the CPU, reducing resource consumption and transfer latency while ensuring data integrity.
2Reliability
If multiple copying processes are used to store data, then data is properly formatted and verified, but CPU resources and memory bandwidth are consumed
Solution Approach 1:
The storage acceleration module performs data assembly and verification autonomously by directly accessing memory spaces and assembling data segments according to assembly instructions. This self-service approach eliminates the need for CPU intervention in data copying and verification processes, significantly reducing CPU resource consumption while maintaining data integrity.
Solution Approach 2:
The patent replaces the traditional CPU-based mechanical copying process with a memory-direct assembly approach. The storage acceleration module generates assembly instructions that enable direct reconstruction of data from segments in memory, substituting CPU-intensive operations with efficient memory management operations.
3Reliability
If check information is generated and stored with data, then data reliability is ensured, but the number of copying processes increases
Solution Approach 1:
The patent merges data segments and check segments into a unified storage structure managed by the storage acceleration module. Both data and check information are stored as segments in the same memory space, allowing simultaneous processing and assembly operations that improve productivity while ensuring data reliability through integrated verification.
Solution Approach 2:
The patent performs preliminary segmentation of data and generation of check information before storage. Data is divided into segments and paired with check segments in advance, creating pre-assembled units that can be efficiently stored and retrieved without requiring additional processing during write operations, thus improving productivity.
4Ease of manufacture
If additional memory is allocated for temporary data storage, then data copying can proceed, but memory occupation demand increases
Solution Approach 1:
The patent implements a universal memory space where data segments, check segments, and assembled data share the same memory region. The storage acceleration module manages this multi-functional memory space, allowing different data types to coexist and be accessed simultaneously without requiring separate temporary storage areas, thus reducing total memory occupation.
Solution Approach 2:
The patent transitions from a sequential copying model requiring temporary storage to a parallel segment-based model. By organizing data in segments with associated check information in the same memory space, the system eliminates the need for additional temporary memory buffers, reducing memory occupation while maintaining copying capability.
Data Source
AI summary
The present disclosure provides a data processing system and method, a computer device and a storage medium. The method includes: reading first data from a first memory space of a target memory space and splitting the first data to obtain multiple segments of second data; determining check information corresponding to each segment of the second data, and storing the check information of each segment of the second data as third data in a second memory space of the target memory space; generating a data assembly instruction based on a data assembly order for the second data and the third data; and sending the data assembly instruction to a storage acceleration module, enabling the storage acceleration module to write the fourth data into the hard disk based on read data and the target data structure.