Asynchronous Packet Flow Processing via Segmented Arrays
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current data packet capture devices struggle to maintain full packet capture capabilities at high speeds due to processing and storage limitations, often resorting to sampling packets, which limits analysis and provides an incomplete view of network traffic.
Innovation Solution
A high-speed data packet capture system featuring a network interface module, non-volatile memory, and processing elements that capture, filter, and store data packets in binary format, allowing for independent and asynchronous operations to identify flows, aggregate data, and write relevant information to a database, enabling efficient processing and storage of packets.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If data packets are captured at high speed, then the volume of captured data increases, but the processing and storage capabilities of computing devices become insufficient
Solution Approach 1:
The system segments the captured data stream into discrete data packets and organizes them into structured formats with headers and payloads. This segmentation enables efficient processing by allowing the system to handle individual packets independently through multiple processing elements, rather than treating the entire data stream as a monolithic block.
Solution Approach 2:
The patent introduces an intermediary structured data format that sits between the raw captured packets and the final analysis results. This intermediary format includes standardized headers with metadata that facilitate efficient routing, filtering, and processing by subsequent system components, acting as a mediator that simplifies the transformation from raw high-speed data to analyzable information.
2Productivity
If packet sampling is used to accommodate limited processing capabilities, then processing load is reduced, but the completeness and accuracy of network traffic analysis deteriorates
Solution Approach 1:
The system performs preliminary actions by organizing and structuring data packets immediately upon capture, creating standardized formats with metadata headers before further processing. This preliminary organization enables subsequent filtering and analysis operations to work efficiently on pre-processed data, reducing the need for sampling while maintaining processing feasibility.
Solution Approach 2:
The patent changes parameters by transforming raw packet data into structured formats with specific metadata fields that can be efficiently queried and filtered. This parameter transformation allows the system to maintain complete packet capture while enabling fast processing through indexed access to packet characteristics, avoiding the need to sample packets.
3Measurement precision
If multiple processing operations are performed on captured packets, then analysis depth increases, but processing time and system complexity increase
Solution Approach 1:
The system segments processing operations into distinct stages, with each processing element handling specific tasks on structured packet data. This segmentation allows parallel processing of multiple packets through different processing elements simultaneously, increasing analysis depth without proportionally increasing total processing time.
Solution Approach 2:
The patent replaces traditional sequential mechanical processing with a parallel processing architecture where multiple processing elements operate simultaneously on structured packet data. This substitution of processing mechanics enables deep analysis of multiple packets in parallel, significantly reducing total processing time while maintaining comprehensive analysis depth.
Data Source
AI summary
An embodiment may involve a network interface configured to capture data packets into a binary format and a non-volatile memory configured to temporarily store the data packets received by way of the network interface. The embodiment may also involve a first array of processing elements each configured to independently and asynchronously: (i) read a chunk of data packets from the non-volatile memory, (ii) identify flows of data packets within the chunk, and (iii) generate flow representations for the flows. The embodiment may also involve a second array of processing elements configured to: (i) receive the flow representations from the first array of processing elements, (ii) identify and aggregate common flows across the flow representations into an aggregated flow representation, (iii) based on a filter specification, remove one or more of the flows from the aggregated flow representation, and (iv) write information from the aggregated flow representation to the database.