Network Interception Bandwidth Statistics Top-N Tracking
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing network intercept devices face significant memory footprint and CPU usage issues when monitoring large numbers of enterprise IP addresses due to the need to track extensive bandwidth usage statistics across numerous conversations, leading to inefficiencies and high costs.
Innovation Solution
Implementing a data structure that maintains statistics for a 'top' number of conversations using a map and vector approach, where new conversations replace decayed ones, and decay counters are topped up to minimize time bias, allowing for efficient bandwidth usage tracking without overwhelming processing and storage resources.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If a simple look-up table by (Source IP, Destination IP and Destination Port) is used to hold a counter for each conversation, then bandwidth usage statistics can be tracked for each conversation, but the memory footprint becomes prohibitively large and CPU time for look-ups and to add information becomes excessive
Solution Approach 1:
The patent extracts only the most significant portions of conversation data by implementing a hierarchical data structure that stores detailed statistics for top-N conversations per source IP while aggregating less significant conversations. This selective extraction reduces memory footprint while preserving essential bandwidth monitoring capabilities for the most active conversations.
Solution Approach 2:
The patent segments the conversation monitoring system into hierarchical levels: source IP level, conversation level, and aggregated level. By organizing data in this segmented manner with map<vector<pair<string, ConversationStats>>> structure, it enables efficient memory usage where only necessary conversation details are stored at granular levels while summary statistics are maintained at higher levels.
2Measurement precision
If a simple look-up table by (Source IP, Destination IP and Destination Port) is used to hold a counter for each conversation, then bandwidth usage statistics can be tracked for each conversation, but CPU time for look-ups and to add information becomes excessive
Solution Approach 1:
The patent segments the lookup process into hierarchical stages: first lookup by source IP, then by conversation identifier within that source's data structure. This segmentation enables the system to quickly eliminate entire groups of conversations that don't match, significantly reducing average CPU time for lookups compared to searching through all conversations in a flat structure.
Solution Approach 2:
The patent performs preliminary organization of conversation data by source IP and maintains sorted/indexed structures within each source's conversations. This preliminary arrangement enables efficient binary search or hash-based lookups when monitoring new packets, reducing the CPU time required for each lookup operation.
3Measurement precision
If statistics for all active conversations are maintained during a monitoring period, then comprehensive bandwidth statistics can be obtained, but processing and storage resources are overwhelmed
Solution Approach 1:
The patent implements partial monitoring by tracking detailed statistics for only the top-N most active conversations per source IP address, while aggregating statistics for less significant conversations. This partial action approach provides sufficient bandwidth monitoring capability for security analysis without overwhelming processing and storage resources with complete conversation-level data for all active conversations.
Solution Approach 2:
The patent changes the monitoring parameter from tracking all conversations to tracking only top-N conversations per source IP. By adjusting this parameter (N), the system can balance between comprehensive statistics and resource consumption, maintaining effective security monitoring while preventing resource overload.
Data Source
AI summary
Computer networks are monitored for usage and security purposes. A network intercept device (NID) monitors network traffic such as conversations between source and destination endpoints in the network. Bandwidth usage statistics may be maintained for each monitored endpoint (e.g. by source IP address). Rather than determining statistics for all active conversations during a particular monitoring period, which comprehensive statistics determining may tax NID processing and storage resources, statistics for a “top” number of conversations (N) may be determined. A lossy approach may be used to gather statistics over the period for up to N conversations of each monitored endpoint. When a new conversation commences during the period for which statistics are not being gathered, each of the monitored endpoint's conversations may be decayed. The new conversation is added for statistics gathering, replacing one of the N conversations, when the one conversation has sufficiently decayed to or below a threshold.