Distributed Packet Deduplication for High-Volume Network Traffic
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing network visibility appliances struggle to handle the volume of traffic from multiple virtual machines due to the limitations of a single instance of a deduplication program, leading to duplicate data packets being forwarded to network tools, which overwhelms the system.
Innovation Solution
Implementing a distributed deduplication scheme by routing traffic among multiple instances of a deduplication program across a pool of network visibility appliances, ensuring that duplicate copies of data packets are directed to the same instance for elimination, using load balancing mechanisms to distribute traffic efficiently.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If a single instance of a deduplication program is used in a network visibility appliance, then the device complexity is reduced, but the productivity decreases when handling traffic from multiple virtual machines due to duplicate packets overwhelming the system
Solution Approach 1:
The deduplication program is divided into multiple instances, each capable of independently processing duplicate packets. This segmentation allows the system to handle higher traffic volumes from multiple virtual machines by distributing the deduplication workload across several parallel instances rather than overloading a single instance.
2Productivity
If multiple instances of a deduplication program are implemented across a pool of network visibility appliances, then the productivity increases for handling duplicate packets, but the device complexity increases due to distributed routing requirements
Solution Approach 1:
A load balancer acts as an intermediary component that receives traffic from virtual machines and intelligently distributes it to appropriate deduplication program instances across multiple network visibility appliances. This mediator manages the complexity of distributed routing by providing a centralized point that makes routing decisions based on current system state and packet characteristics.
Solution Approach 2:
The load balancer performs preliminary routing decisions before packets reach the deduplication instances, pre-distributing traffic based on hashing algorithms or other distribution strategies. This preliminary action ensures that duplicate packets are directed to the same instance while normal packets are distributed across multiple instances, reducing the complexity of real-time routing decisions.
3Measurement precision
If duplicate data packets are forwarded to network tools, then the measurement precision of network analysis is improved, but the loss of energy increases due to overwhelming the system with redundant data
Solution Approach 1:
The deduplication program extracts and removes duplicate packets from the traffic stream before forwarding packets to network tools for analysis. By taking out redundant data, the system maintains measurement precision for legitimate traffic while significantly reducing energy consumption and resource utilization, preventing network tools from being overwhelmed by duplicate packets.
Data Source
AI summary
Introduced here are network visibility appliances capable of implementing a distributed deduplication scheme by routing traffic amongst multiple instances of a deduplication program. Data traffic can be forwarded to a pool of multiple network visibility appliances that collectively ensure no duplicate copies of data packets exist in the data traffic. The network visibility appliances can route the traffic to different instances of the deduplication program so that duplicate copies of a data packet are guaranteed to arrive at the same instance of the deduplication program, regardless of which network visibility appliance(s) initially received the duplicate copies of the data packet.