Network Packet Storage via Flow-Based Segmentation
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing network communication systems lack efficient methods for capturing, organizing, and analyzing network traffic, particularly in identifying and storing network packets of interest amidst high volumes and varieties of network traffic, which hinders anomaly detection and security monitoring.
Innovation Solution
Implementing programmable hardware processors, such as FPGAs, to capture and filter network traffic, using multi-stage storage with dynamically configurable filtering and purging, to identify and store network packets and flows of interest for efficient analysis.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If all network packets are stored for analysis, then complete network traffic analysis is achieved, but storage resources are consumed and analysis efficiency decreases
Solution Approach 1:
The patent extracts only the necessary network packets for analysis by implementing selective storage based on flow characteristics and analysis requirements. The system identifies and stores only relevant packets while discarding unnecessary ones, thereby reducing storage resource consumption while maintaining analysis effectiveness.
Solution Approach 2:
The patent applies different storage strategies to different network flows based on their specific characteristics. High-priority flows with security concerns are stored in detail, while low-priority flows are stored summarily or not at all. This localized quality approach optimizes storage resource allocation according to actual analysis needs.
2Stability of the object's composition
If network packets are stored in sequential order, then packet sequence integrity is maintained, but storage access efficiency decreases when packets need to be retrieved out of sequence
Solution Approach 1:
The patent segments the storage structure into multiple indexed regions organized by network flow identifiers. Each flow's packets are stored in separate segments with their own indexing, allowing independent access to any flow's packets without affecting others. This segmentation enables efficient random access while maintaining sequence integrity within each flow segment.
Solution Approach 2:
The patent introduces an indexing dimension beyond sequential ordering by creating multi-dimensional access paths through flow-based indexing and hashing structures. This allows packets to be accessed through multiple dimensions (flow ID, timestamp, packet number) rather than only sequential position, dramatically improving retrieval efficiency for out-of-sequence access while preserving sequential integrity for replay scenarios.
3Reliability
If network traffic is captured and stored in real-time, then complete traffic monitoring is achieved, but system performance and processing speed decrease
Solution Approach 1:
The patent implements partial action by selectively capturing and storing only the portion of network traffic that is necessary for security analysis. Instead of capturing all traffic indiscriminately, the system applies filtering rules to capture only suspicious or relevant flows, achieving adequate monitoring coverage with reduced processing overhead.
Solution Approach 2:
The patent performs preliminary filtering and classification of network packets before they are stored. By pre-processing packets to identify their flow characteristics and priority levels, the system prepares the data in advance for efficient storage and retrieval, reducing the processing burden during actual analysis operations and improving overall system throughput.
4Measurement precision
If detailed indexing of all network packets is implemented, then precise packet retrieval is enabled, but storage overhead and processing complexity increase
Solution Approach 1:
The patent implements dynamic indexing where the level of indexing detail adapts based on flow characteristics and analysis requirements. High-priority flows receive detailed multi-level indexing, while low-priority flows use simpler indexing or none at all. This dynamic approach maintains retrieval precision for critical packets while reducing overall indexing complexity and storage overhead.
Data Source
AI summary
Examples relate to organizing and storing network communications. In one example, a programmable hardware processor may: receive a first set of network packets; identify, for each network packet included in the first set, a network flow, each network flow including at least one related packet; store each network packet included in a subset of the first set in a first data storage device; for each network packet included in the subset, organize the network packet according to the network flow identified for the network packet; identify, from the network flows, a set of network flows that each have at least one characteristic of interest; and store, in a second data storage device, each network packet included in each network flow of the set of network flows.