Packet Filtering via Hash Template Indexing
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing packet capture systems face challenges in efficiently filtering and retrieving specific packets from vast storage due to high network speeds and large data volumes, leading to lengthy processing times when searching for subsets of packets.
Innovation Solution
The implementation of a packet filter definition compilation into low-level code, generation of a hash template, and creation of a hash table to index packets based on byte offsets and counts, allowing for rapid lookup and retrieval of specific packets by generating skeleton packets and matching metadata.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Quantity of substance
If packet capture systems store all captured packets for logging and debugging purposes, then the system can provide comprehensive data for analysis, but the storage capacity is rapidly consumed and filtering through stored packets to identify specific subsets takes hours
Solution Approach 1:
The patent applies preliminary action by creating hash tables and metadata indexes during the packet capture process itself, before any filtering or analysis is needed. This pre-processing organizes packets into hash buckets based on their content, so when filtering is required, the system can immediately query the pre-built indexes rather than scanning through all stored packets. This resolves the contradiction by preparing the data structure in advance, making both storage efficient and retrieval fast.
Solution Approach 2:
The patent introduces metadata as an intermediary layer between the raw packet storage and the filtering/analysis operations. Instead of directly searching through stored packets, the system uses metadata indexes and hash tables that summarize packet characteristics. This intermediary structure enables rapid identification of packet subsets without examining every packet, thus reducing filtering time while maintaining comprehensive storage capability.
2Productivity
If packet capture systems process packets at high network speeds (tens of gigabits per second), then the system can capture all traffic for analysis, but the processing and storage requirements become overwhelming
Solution Approach 1:
The patent applies the extraction principle by selectively capturing and storing only the metadata and essential characteristics of packets at high speeds, rather than storing complete packet data for all traffic. The system extracts key identifying features (such as packet headers, protocol types, source/destination addresses) and stores these in compact metadata structures and hash tables, while the full packet data can be retained or discarded based on storage capacity. This enables the system to process tens of gigabits per second without overwhelming storage requirements.
3Ease of operation
If conventional filtering methods are used to search through stored packets, then the system can identify specific packet subsets, but the process takes hours even for modest packet rates
Solution Approach 1:
The patent applies the copying principle by creating compact metadata copies and hash table representations of the packet data during capture. These copies contain essential filtering information (packet characteristics, protocols, addresses, timestamps) in a highly organized format. When filtering is needed, the system queries these pre-created copies rather than examining the original packet streams, reducing lookup time from hours to seconds while maintaining full filtering capability.
Data Source
AI summary
An example embodiment may involve obtaining a packet filter definition that specifies characteristics of packets; compiling the packet filter definition to instructions of low-level code; generating a hash template based on a subset of the instructions, wherein the hash template includes pairs of byte offsets and byte counts that define locations within the packets at which the characteristics are disposed; based on application of the hash template to a plurality of stored packets, creating hash table entries in a hash table, wherein the hash table entries are respectively associated with subsets of the stored packets, wherein the subsets of the stored packets have respectively unique patterns of values within their bytes at the locations defined by the pairs of byte offsets and byte counts; updating the hash table entries to refer to metadata relating to their respective subset of the stored packets; and storing the hash table and the metadata.