Packet Flow Offload Architecture for Full-Speed Traffic Analysis
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Conventional data packet capture devices struggle to maintain full packet capture abilities at high speeds due to processing and storage limitations, often resorting to sampling which limits network traffic analysis.
Innovation Solution
A network interface module captures data packets in binary format, utilizing non-volatile memory for temporary storage and multiple arrays of processing elements for independent and asynchronous operations, including flow identification, aggregation, and writing to a database, enabling high-speed and low-latency packet capture.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If conventional data packet capture devices are used, then packet capture functionality is provided, but processing and storage capabilities cannot keep up with high-speed network traffic
Solution Approach 1:
The system segments the packet capture and processing workload across multiple independent processing elements (CPUs, FPGAs, or ASICs) that operate in parallel. Each processing element handles a portion of the packet stream, enabling the system to scale processing capacity linearly with the number of elements while maintaining high capture speeds.
Solution Approach 2:
A packet capture card serves as an intermediary device between the network interface and the processing elements. It performs initial packet capture, buffering, and preprocessing, offloading the burden from the main system and enabling high-speed capture without overwhelming the processing and storage subsystems.
2Device complexity
If sampling of data packets is performed to accommodate limited processing capacity, then processing load is reduced, but complete network traffic analysis is prevented
Solution Approach 1:
The system performs partial processing on all packets through selective copying based on filter criteria. Instead of sampling a fixed percentage, it processes every packet that matches specified filters (e.g., specific protocols, address patterns), ensuring complete analysis of relevant traffic while ignoring unrelated packets to maintain manageable processing loads.
Solution Approach 2:
Different processing paths and levels of analysis are applied to different packets based on their characteristics. Packets matching specific criteria receive full processing and detailed analysis, while others receive minimal handling or are discarded, optimizing the balance between processing load and analysis completeness for each packet type.
Data Source
AI summary
An embodiment may involve digital circuitry configured to: (i) receive a plurality of data packets, (ii) calculate, based on content at a pre-determined set of locations within the data packets, respective hash values for each of the data packets, and (iii) store, in a first memory, metadata containing the respective hash values; and a plurality of processors configured to: (i) read, from the first memory, the metadata, (ii) aggregate, based on the respective hash values, the metadata into flow statistics of flows defined by the data packets, and (iii) write, to a second memory, the flow statistics, wherein the flows are subsets of the data packets having common values in each of the pre-determined set of locations.