Packet De-duplication via Hierarchical Header Field Checking
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Duplicate packets in network switch devices can hinder accurate analysis and overwhelm monitoring tools, leading to bandwidth issues due to their generation in packet-switching networks, especially when multiple copies of the same packet are created through tapping points in redundant network segments.
Innovation Solution
A method and system for identifying and removing duplicate packets by using a predetermined field-checking hierarchy, where fields with higher change rates are checked first, and packets are determined as duplicates if they match corresponding fields in earlier received packets within a prescribed time period, utilizing an integrated circuit in a packet switch device.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If span port copies every packet for monitoring, then monitoring coverage is improved, but duplicate packets are generated causing bandwidth consumption and tool overload
Solution Approach 1:
The patent extracts and removes duplicate packets from the packet stream before forwarding to monitoring tools. The de-duplication module identifies packets with identical payloads and header fields, keeping only the first occurrence and discarding subsequent duplicates, thereby reducing bandwidth consumption while maintaining monitoring coverage
Solution Approach 2:
The patent discards duplicate packets that have already been processed and monitored. By tracking seen packets through hashing and comparison mechanisms, the system recovers bandwidth resources by preventing redundant transmission of identical packet copies to monitoring tools
2Reliability
If multiple taps are deployed in redundant network segments, then monitoring redundancy is improved, but duplicate packets are generated overwhelming analysis tools
Solution Approach 1:
The patent extracts duplicate packets from the multi-tap packet stream by comparing payloads and header fields from different tap points. The de-duplication module identifies and removes redundant copies generated by multiple taps in redundant network segments, reducing packet volume while preserving monitoring redundancy
Solution Approach 2:
The patent uses copying mechanisms to create hash representations of packet payloads and header fields for efficient comparison. By copying and hashing packet identifiers from multiple taps, the system能够快速 identify duplicates without processing entire packet streams, managing packet volume from redundant taps
3Measurement precision
If packet de-duplication is performed using multiple header fields, then de-duplication accuracy is improved, but processing complexity increases
Solution Approach 1:
The patent segments the de-duplication process into hierarchical stages: first comparing packet payloads, then header fields only when payloads match. This segmentation allows the system to achieve high de-duplication accuracy by examining multiple fields while reducing processing complexity through staged comparison that avoids unnecessary full-packet analysis
Solution Approach 2:
The patent applies local quality by examining different header fields with different levels of scrutiny based on their reliability for duplicate identification. Critical fields like IP addresses and ports are examined with higher priority, while less reliable fields are used as supplementary checks, optimizing the balance between accuracy and processing complexity
Data Source
Figure 1~2
Figure 3
Figure 4
AI summary
A method of packet processing includes receiving a first packet that includes a header, the header having a plurality of fields, one of the plurality of fields being an identification field, determining an identification value for the identification field in the header of the first packet, determining whether the identification value of the first packet matches an identification value in a header of a second packet, and using another one of the fields in the header of the first packet to determine whether the first packet is a duplicate packet when the identification value of the first packet matches the identification value of the second packet.