Flow-Level Traffic Deduplication for Clear Text Network Visibility
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Encryption techniques limit the capabilities of security tools that require clear text data, complicating key management and increasing processing overhead in network traffic visibility systems.
Innovation Solution
Implementing software hooks at entry and exit points of encryption/decryption functions in worker nodes to capture clear text payloads, eliminating the need for complex key management and resource-intensive processing.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If encryption is used to protect sensitive data, then data security is improved, but the capabilities of security tools requiring clear text data are limited
Solution Approach 1:
A visibility node is introduced as an intermediary component between the encrypted data flow and security tools. This visibility node captures clear text data at the application layer within the worker node, converts it to a visible format, and forwards it to security tools for analysis, while the original encrypted data continues its transmission. This mediator enables security tools to access clear text data without compromising the encryption protection of the main data flow.
2Loss of information
If keys-based approach is used for decryption, then clear text visibility is achieved, but processing overhead and key management complexity increase
Solution Approach 1:
The invention extracts the clear text data capture function from the traditional key-based decryption approach. Instead of decrypting data using keys to achieve visibility, the system extracts clear text data directly at the application layer within the worker node through software hooks, and forwards only this extracted clear text to the visibility node. This eliminates the need for key management and complex decryption processing while maintaining clear text visibility for security analysis.
3Loss of information
If separate proxies are deployed for each worker node, then clear text data capture is achieved, but device complexity and resource consumption increase
Solution Approach 1:
The invention merges the clear text data capture functionality directly into the worker node's application layer through software hooks, eliminating the need for separate proxy components. The visibility node consolidates the reception and distribution of clear text data from multiple worker nodes, creating a unified architecture that reduces overall system complexity while maintaining comprehensive data capture capabilities across all worker nodes.
Data Source
AI summary
A system and method for flow-level deduplication of network traffic are disclosed. A network node receives a first plurality of packets from a first network endpoint. The first plurality of packets represent a flow of data being communicated between the first network endpoint and a second network endpoint. The network node further receives a second plurality of packets from the second network endpoint. The network node identifies a sequence identifier of each packet of the first and second pluralities of packets. The network node determines that the first and second pluralities of packets are all associated with the same flow, based on the sequence identifiers of the first and second pluralities of packets. In response to that determination, the network node deduplicates the flow by discarding the first plurality of packets or the second plurality of packets. The network node may be a traffic visibility node.