Network Flow Deduplication via Hash-Based Cluster Coordination
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
In distributed network monitoring environments, multiple network monitoring devices often receive duplicate network traffic, making it difficult to determine correlations and extract meaningful information from duplicative data, leading to challenges in troubleshooting and analysis.
Innovation Solution
Implementing a system where network monitoring computers (NMCs) in a cluster register and manage network flows using hashing and policy-based methods to identify responsible NMCs for processing specific flows, thereby eliminating duplicates and optimizing data processing.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If multiple network monitoring devices are deployed to monitor network traffic, then monitoring coverage and reliability are improved, but data duplication increases and processing efficiency deteriorates
Solution Approach 1:
The patent divides the network monitoring task into segments by assigning different network flows to different NMCs based on hashing of flow identifiers. Each NMC monitors specific flows rather than all traffic, eliminating duplicates while maintaining comprehensive coverage through coordinated segmentation of the monitoring workload
Solution Approach 2:
The patent introduces a flow manager as an intermediary component that coordinates between multiple NMCs. The flow manager receives flow identification information, determines responsible NMCs using hashing, and directs traffic accordingly, enabling efficient deduplication and load distribution across the monitoring cluster
2Measurement precision
If multiple network monitoring devices process all network traffic, then monitoring accuracy is improved, but resource consumption and system complexity increase
Solution Approach 1:
The patent implements local quality by having each NMC specialize in monitoring specific network flows determined by hashing. Each device optimizes its resources for its assigned flows rather than processing all traffic, reducing individual device complexity while maintaining overall monitoring accuracy through the collective capability of the cluster
Solution Approach 2:
The patent changes the parameter of flow assignment from static to dynamic based on hashing calculations. The system uses configurable hash functions and policies to determine flow distribution, allowing flexible adjustment of monitoring parameters while maintaining consistent deduplication behavior across the cluster
Data Source
AI summary
Embodiments are directed to monitoring flows of packets over a network. If a network monitoring computer (NMC) in a cluster of NMCs observes a new network flow, the NMC may perform a variety of actions to determine the NMC that is responsible for monitoring the new network flow. Network traffic associated with the new network flow may be buffered in a non-transitory processor readable media. The new network flow may be registered with the plurality of NMCs, providing an identifier that corresponds to one NMC. Registering may include, assigning the NMC a responsibility to monitor the new network flow. If the identifier corresponds to the NMC that observed the new network flow, the network traffic associated with the new network flow is processed using that NMC. If the identifier corresponds to another NMC, the buffered network traffic is forwarded to the other NMC.