Packet Retrieval via Flow ID and Metadata Indexing
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
High-speed packet-based communications networks face inefficiencies in packet retrieval due to the costly and time-consuming process of saving all traffic data for analysis, making it difficult to quickly retrieve packets meeting specified criteria.
Innovation Solution
A storage system with multiple repositories for raw packets, metadata, timestamp-indexed data, and data flow information, where packet search criteria specify a time range, allowing for efficient retrieval of relevant data by narrowing the search scope through data flow information, metadata, and timestamp indexing.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If all traffic data is saved for later analysis, then complete data availability is achieved, but retrieval time and processing overhead increase significantly
Solution Approach 1:
The system performs preliminary actions by creating and storing metadata about packet flows (including flow identifiers, time ranges, and packet counts) at the time of packet capture. This pre-processing enables rapid retrieval operations later without requiring full re-analysis of captured packets, thus resolving the contradiction between complete data availability and fast retrieval.
Solution Approach 2:
The patent introduces metadata as an intermediary layer between the raw captured packets and the retrieval process. This metadata acts as a mediator that enables efficient searching and filtering without requiring direct examination of the actual packet data, thereby maintaining data availability while dramatically reducing retrieval time.
2Reliability
If all captured packets are stored in a single repository, then data completeness is maintained, but search efficiency decreases
Solution Approach 1:
The system segments the storage structure by separating packet data from metadata about packet flows. Instead of storing all packets in a single repository, the patent divides the storage into packet repositories (for actual data) and metadata repositories (for flow information), enabling efficient searches through metadata without compromising data completeness.
Solution Approach 2:
The patent adds a new dimension to the storage architecture by introducing a metadata layer that organizes packet information separately from the packets themselves. This dimensional separation allows the system to maintain complete packet data while enabling efficient retrieval through the metadata dimension, which contains flow identifiers, time ranges, and other searchable attributes.
3Measurement precision
If metadata is stored for all packets, then retrieval precision is improved, but storage overhead increases
Solution Approach 1:
The system extracts only the essential metadata elements needed for retrieval operations (flow identifiers, time ranges, packet counts) from the complete packet data. This selective extraction provides sufficient retrieval precision while minimizing storage overhead by storing only the metadata necessary for efficient searching and filtering.
Data Source
Figure 1
Figure 2
Figure 3~4
AI summary
A storage system includes network monitoring device having NIC coupled to network and configured to capture raw data packets. The system further includes 4 data repositories. A first repository stores captured packets. A second repository stores captured packets' metadata. A third repository stores captured packets' and metadata's timestamp indexed data. A fourth repository stores captured packets' data flow. The storage system further includes a storage engine coupled to the repositories. The storage engine receives packet search criteria specifying at least a first time range. Data flow information associated with the search criteria is retrieved from the fourth repository. The retrieved data flow information is associated with a second, narrower, time range. Metadata information associated with the second time range is retrieved from the second repository using corresponding timestamp indexed data. Captured packets associated with the retrieved metadata are retrieved from the first repository using corresponding timestamp indexed data.