Hash-Based Packet Flow Sampling for Network Monitoring
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing network packet communication systems face performance bottlenecks due to memory bandwidth and size requirements for flow tables in IPFIX processing, particularly in terms of memory resources and processing power needed for collecting and reporting packet flow statistics.
Innovation Solution
Implementing a hash-based packet flow sampling method that selects a subset of packet flows for further processing, eliminating the need for maintaining flow information in a state table, and using a mask to adjust the percentage of flows ignored or processed, thereby reducing memory and processing demands.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If IPFIX processing is implemented within the network router to collect and report packet flow statistics, then network traffic monitoring capability is improved, but memory bandwidth and size requirements for flow tables increase significantly
Solution Approach 1:
The patent extracts only the essential flow identification fields (source IP, destination IP, source port, destination port, protocol) from complete flow records and stores them in a compact flow table. By taking out only the necessary identifying information rather than storing complete flow statistics, the memory requirements are significantly reduced while maintaining the ability to monitor and report packet flow statistics.
Solution Approach 2:
The patent applies different data storage strategies to different types of flow information. Complete flow records with all statistics are stored only when flows match sampling criteria, while non-matching flows use a simplified representation. This local differentiation optimizes memory usage by applying detailed tracking only where needed rather than uniformly across all flows.
2Loss of information
If complete packet flow statistics are collected and stored for all packet flows, then comprehensive flow data availability is improved, but processing resources and memory demands increase
Solution Approach 1:
The patent implements partial action by collecting complete flow statistics only for a sampled subset of flows rather than all flows. The sampling mechanism selects representative flows based on sampling rate configurations, allowing the system to obtain sufficient monitoring data with reduced processing overhead. This partial collection approach maintains data availability for analysis while avoiding the resource costs of comprehensive collection.
Solution Approach 2:
The patent changes the sampling parameter dynamically based on network conditions and configuration. The sampling rate can be adjusted to balance between data comprehensiveness and resource consumption. By modifying this parameter, the system adapts to different operational requirements, collecting more detailed statistics when resources are abundant and using simplified sampling when resources are constrained.
3Loss of information
If flow table size is increased to store more packet flow records, then flow statistics completeness is improved, but memory bandwidth requirements increase
Solution Approach 1:
The patent segments flow records into two categories: sampled flows with complete statistics and non-sampled flows with minimal identification data. This segmentation allows the flow table to accommodate a larger number of flows without proportionally increasing memory bandwidth requirements, as only the segmented sampled portion requires full statistical tracking.
Data Source
Figure 1A
Figure 1B
Figure 2A
AI summary
Systems and methods are disclosed for hash-based selection of network packets for packet flow sampling in network communication systems. Input packets associated with packet flows within a network communication system are received by a hash-based sampler. The hash-based sampler then generates hash values for the input packets based upon fields within the input packets. These fields are selected to identify packet flows for the input packets. The hash values for the input packets are then compared to a mask. The mask is configured to determine a subset of packet flows for which to forward packets. Based upon this comparison, certain input packets are selected to be forwarded for further processing, and non-selected packets are discarded. The further processing can include processing the selected input packets to generate flow statistics data (e.g., IPFIX) for the selected input packets.