Packet Flow Offload Architecture for Full-Speed Traffic Capture
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Conventional data packet capture devices struggle to maintain full packet capture abilities at high speeds due to processing and storage limitations, often resorting to sampling which limits network traffic analysis.
Innovation Solution
A network interface module captures data packets in binary format, utilizing non-volatile memory for temporary storage and multiple arrays of processing elements for independent and asynchronous operations, including flow identification, aggregation, and filtering, with the ability to write data to a database.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Speed
If conventional data packet capture devices are used at high speeds, then processing and storage capabilities are exceeded, but full packet capture ability is lost
Solution Approach 1:
The system segments the packet capture and processing workload across multiple processing elements (CPUs, FPGAs, or other processors) that operate independently and asynchronously. Each processing element handles a portion of the packet stream, allowing the system to maintain full packet capture capability at high speeds by distributing the processing burden rather than relying on a single device's limited capabilities.
Solution Approach 2:
A buffer memory component serves as an intermediary between the high-speed packet capture interface and the processing elements. This buffer temporarily stores captured packets, decoupling the capture speed from processing speed and enabling the system to accept packets at line rate while processing occurs asynchronously at the capability limit of the processing elements.
2Device complexity
If sampling is used to accommodate processing limitations, then device complexity is reduced, but measurement precision is degraded
Solution Approach 1:
The system performs processing on all packets (excessive action relative to sampling) by utilizing multiple processing elements that collectively handle the full packet stream. This approach maintains measurement precision by analyzing every packet rather than relying on samples, while managing device complexity through parallel processing architecture and asynchronous operation.
3Productivity
If multiple arrays of processing elements are used for independent and asynchronous operations, then productivity is improved, but device complexity increases
Solution Approach 1:
The processing architecture is segmented into multiple independent processing elements that each handle specific portions of the packet stream. This segmentation enables parallel processing and asynchronous operation, dramatically improving throughput. The complexity is managed by making each processing element relatively simple and independent, with coordination handled through the buffer memory interface.
Solution Approach 2:
The processing elements are designed to be universal and multi-functional, capable of performing various packet processing tasks such as filtering, analysis, and protocol handling. This universality allows the same hardware architecture to handle diverse network traffic types and processing requirements, improving productivity without proportionally increasing complexity.
Data Source
AI summary
An embodiment may involve digital circuitry configured to: (i) receive a plurality of data packets, (ii) calculate, based on content at a pre-determined set of locations within the data packets, respective hash values for each of the data packets, and (iii) store, in a first memory, metadata containing the respective hash values; and a plurality of processors configured to: (i) read, from the first memory, the metadata, (ii) aggregate, based on the respective hash values, the metadata into flow statistics of flows defined by the data packets, and (iii) write, to a second memory, the flow statistics, wherein the flows are subsets of the data packets having common values in each of the pre-determined set of locations.