Ethernet Data Stream Recording via User-Space Polling
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
High-speed data acquisition systems face challenges with low CPU utilization, poor system compatibility, and complex deployment and packaging due to high data bandwidth requirements, particularly with traditional Ethernet-based systems that suffer from high packet loss and system load issues.
Innovation Solution
The method involves isolating CPU cores, uninstalling kernel network card drivers, creating hugepage memory pools, and initializing 10-gigabit network cards using the sysfs file system to enable user-space polling and disk recording, utilizing worker threads for efficient packet reception and processing, and leveraging NVMe SSDs for improved write performance.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If traditional PCAP library is used for packet capturing, then system compatibility is maintained, but packet loss rate increases and system load becomes high when bandwidth exceeds 1 Gbps
Solution Approach 1:
The patent extracts the packet capturing function from the traditional kernel-based PCAP library and implements it in user space through a custom driver. This separation allows the system to handle high bandwidth without being constrained by kernel interrupt processing limitations, thereby reducing packet loss while maintaining compatibility.
Solution Approach 2:
The patent replaces the traditional mechanical interrupt-driven packet capturing mechanism with a polling-based user-space implementation. This substitution eliminates the bottleneck of kernel interrupt handling and enables continuous high-speed data capture with minimal packet loss.
2Power
If dedicated data acquisition network card with hardware acceleration is used, then CPU load is significantly reduced and high-bandwidth data recording is achieved, but cost increases
Solution Approach 1:
The patent implements self-service by enabling the general-purpose network card to perform high-speed packet capturing through user-space polling without requiring dedicated hardware acceleration. The system leverages the existing network card capabilities combined with optimized software implementation, eliminating the need for expensive dedicated hardware while achieving similar performance benefits.
3Productivity
If PF_RING framework is used for high-speed network packet processing, then packet capture rate is significantly increased, but OS compatibility requirements increase and deployment complexity increases due to special driver module installation
Solution Approach 1:
The patent extracts the complex driver module installation requirement from the PF_RING framework and implements a simplified user-space solution. By moving packet processing to user space through polling, the system achieves high packet capture rates without requiring special kernel driver modules, thereby reducing deployment complexity while maintaining compatibility.
4Speed
If DPDK is used for packet processing, then transmission efficiency is optimized and bandwidth exceeds 10 Gbps, but development environment complexity increases and packaging and deployment becomes cumbersome
Solution Approach 1:
The patent adopts a simpler, more flexible approach compared to DPDK's complex development environment. Instead of requiring sophisticated build systems and dependency management, the implementation uses straightforward user-space polling code that can be deployed without complex packaging, making it more suitable for rapid deployment while maintaining high transmission efficiency.
Data Source
AI summary
The field of high-speed data acquisition and network data processing, and particularly relates to an Ethernet data stream recording method, an Ethernet data stream recording system, and an Ethernet data stream recording device for a high-speed data acquisition system. It is intended to solve problems such as a low utilization rate of CPU, poor system compatibility, difficulty in packaging and deployment and low reliability of system transmission of the traditional high-speed data acquisition system. The method of the present disclosure includes: isolating a preset number of CPU cores after a Linux operating system is booted; uninstalling a kernel network card driver of the operating system and creating a hugepage memory pool; for each 10-gigabit network card, allocating a corresponding data-receiving buffer pool and a corresponding lock-free FIFO buffer, and initializing a PCIE register of each 10-gigabit network card such that each 10-gigabit network card enters into an acquisition state; and continuously receiving packets acquired by each 10-gigabit network card in a driving manner of user space polling and performing disk recording. According to the present disclosure, the utilization rate of CPU, system compatibility and transmission reliability are improved and the difficulty in packaging and deployment is decreased.