Sequence-Based Flow Deduplication for Network Traffic Visibility
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Encryption in network communications limits the capabilities of security tools that require clear text data, complicating key management and increasing processing overhead.
Innovation Solution
Implementing software hooks at entry and exit points of encryption/decryption functions in worker nodes to capture clear text payloads, eliminating the need for complex key management and resource-intensive processing.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If encryption is used to protect sensitive data, then data security is improved, but the capabilities of security tools requiring clear text data are limited
Solution Approach 1:
The patent segments the network traffic handling process into separate streams: an encrypted stream for data transmission and a clear text stream for security analysis. By capturing packets at different points in the encryption/decryption process (entry/exit points of encryption functions), the system maintains both encryption security and security tool accessibility without requiring key management complexity
Solution Approach 2:
The patent introduces an intermediary mechanism (software hooks at entry/exit points of encryption functions) that captures clear text payloads without requiring actual decryption. This intermediary approach allows security tools to access clear text data while the main data flow remains encrypted, resolving the contradiction between security and analysis capability
2Loss of information
If traditional key-based decryption approaches are used, then clear text access is achieved, but key management complexity and processing overhead increase
Solution Approach 1:
The patent applies preliminary action by capturing packets at the entry points of encryption functions (before encryption occurs) or exit points (after decryption occurs). This eliminates the need for actual decryption operations and key management, as the clear text payloads are captured directly at these boundary points, reducing both complexity and processing overhead
Solution Approach 2:
The patent creates a copy of the packet data at the entry/exit points of encryption functions, separating the clear text payload capture from the encrypted data flow. This copying approach allows security tools to analyze clear text without interfering with the main encrypted communication, eliminating key management requirements while maintaining data accessibility
3Loss of information
If multiple network visibility nodes acquire traffic at numerous entry points, then traffic visibility is improved, but duplicate flows are processed increasing system load
Solution Approach 1:
The patent implements feedback mechanisms where visibility nodes send flow identification information (such as five-tuple data) back to a central controller or shared data structure. This feedback allows the system to recognize and eliminate duplicate flows, maintaining comprehensive visibility while reducing redundant processing through coordinated operation across multiple nodes
Data Source
Figure 1A
Figure 1B
Figure 2
AI summary
A system and method for flow-level deduplication of network traffic are disclosed. A network node receives a first plurality of packets from a first network endpoint. The first plurality of packets represent a flow of data being communicated between the first network endpoint and a second network endpoint. The network node further receives a second plurality of packets from the second network endpoint. The network node identifies a sequence identifier of each packet of the first and second pluralities of packets. The network node determines that the first and second pluralities of packets are all associated with the same flow, based on the sequence identifiers of the first and second pluralities of packets. In response to that determination, the network node deduplicates the flow by discarding the first plurality of packets or the second plurality of packets. The network node may be a traffic visibility node.