Flow-Ordered Packet Capture Indexing for Disk I/O Reduction
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing packet capture processes are inefficient in searching and retrieving network traffic data due to high disk I/O strain and random access reads, which hinder scalability with increasing network speeds.
Innovation Solution
Organizing packet data by network flows and generating indexes to allow for quick retrieval and recombination of data, reducing disk I/O and improving search efficiency through flow-ordered FCAP files and parallel processing.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If packets are captured and stored using conventional methods with indexes built during capture, then packet capture functionality is achieved, but disk I/O operations become excessive and random access reads slow down retrieval speed
Solution Approach 1:
The patent applies preliminary action by organizing packets into flow-based buckets during the capture phase, pre-sorting them by flow characteristics before storage. This preliminary organization eliminates the need for random access reads during retrieval, as all packets for a given flow are stored contiguously. The indexing structure is also built during capture rather than during retrieval, preparing the data in advance for efficient access patterns.
Solution Approach 2:
The patent segments the packet data into flow-based buckets, where each bucket contains packets belonging to a specific network flow. This segmentation transforms the monolithic packet capture data into organized units that can be stored and retrieved efficiently. By segmenting data at the flow level rather than individual packet level, the system reduces the number of disk I/O operations needed for retrieval.
2Quantity of substance
If conventional packet capture processes are used with sustained disk writes, then packets can be archived, but the system disk I/O becomes strained and scalability is limited
Solution Approach 1:
The patent merges multiple packet streams into flow-based buckets during capture, combining related packets into contiguous storage units. This merging approach reduces the total number of separate write operations to disk, as entire buckets can be written as unified units rather than individual packets. The index structure is also merged with the data during storage, eliminating separate index write operations.
3Ease of operation
If indexes are built during packet capture to enable search functionality, then packet search capability is achieved, but the indexing process increases processing time and reduces capture speed
Solution Approach 1:
The patent maintains continuity of useful action by performing indexing operations concurrently with packet capture rather than sequentially. The flow-based bucket structure allows indexing to occur continuously during the capture process without interrupting the packet flow. This continuous operation ensures that both capture and indexing proceed simultaneously, maintaining high capture speeds while still building the search index.
Data Source
AI summary
A packet capture system may copy packets from an interface to a bucket. When the bucket is full of packets, a new bucket for incoming packets may be started, and the full bucket may be indexed. During the indexing, each packet may be sorted in the bucket by flow, and each flow may be indexed. Once indexing is complete, the packets are written to a flow ordered FCAP file and the indexes are written to disk. The flow ordered nature of the FCAP file combined with the indices and their associated search algorithms allow for rapid retrieval of captured flows.