FPGA Packet Deduplication via Hash Store
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Traditional software-based packet deduplication in networking is limited by CPU core count and PCIe bandwidth, leading to constrained throughput and potential for inaccurate results and redundant processing, especially in high-speed network environments.
Innovation Solution
Implementing packet deduplication logic in hardware using a field programmable gate array (FPGA) with a hash module to generate packet partial signatures, append unhashed packet information, and store entries in a high-bandwidth memory hash store, allowing for efficient duplicate packet detection and prevention from reaching downstream network nodes.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If packet deduplication is performed in software using CPU cores, then deduplication functionality is achieved, but throughput is constrained by the number of CPU cores and PCIe bandwidth
Solution Approach 1:
The patent replaces software-based CPU processing with hardware-based FPGA implementation. The FPGA uses parallel hash computation circuits and high-bandwidth memory to perform packet deduplication at line rate, eliminating the throughput constraints of CPU cores and PCIe bandwidth while reducing software processing complexity.
Solution Approach 2:
The patent creates a hardware copy of the deduplication function in the FPGA, which mirrors the software logic but executes in parallel hardware circuits. This hardware copy processes packets simultaneously across multiple logic elements, achieving throughput independent of CPU core count.
2Reliability
If software deduplication processes all packets through CPU, then accurate duplicate detection is achieved, but redundant processing occurs and false positives may be generated
Solution Approach 1:
The patent segments the packet data into multiple portions and computes hashes in parallel using separate FPGA logic elements. Each segment is processed independently and simultaneously, reducing overall processing time while maintaining detection accuracy through coordinated comparison of all segment hashes.
Solution Approach 2:
The patent performs preliminary hash computation on packet segments before full packet processing. By computing hashes of critical segments first and comparing them against the hash store early in the processing pipeline, the system can quickly eliminate duplicates without completing full packet processing, reducing processing time while maintaining accuracy.
3Productivity
If more CPU cores are added to increase throughput, then processing capacity increases, but PCIe bandwidth constraints and system complexity increase
Solution Approach 1:
The patent substitutes the mechanical addition of CPU cores with a parallel hardware architecture in the FPGA. Instead of adding more sequential processing units, the system uses spatial parallelism with multiple logic elements operating simultaneously, achieving increased processing capacity without adding PCIe bandwidth requirements or system complexity.
4Productivity
If duplicate packets are processed by downstream tools, then comprehensive analysis is performed, but redundant processing wastes resources and may cause denial of service
Solution Approach 1:
The patent performs preliminary duplicate detection in the FPGA before packets reach downstream analysis tools. By checking the hash store and identifying duplicates early in the processing pipeline, the system prevents duplicate packets from being forwarded to downstream tools, eliminating redundant processing and protecting against denial of service while improving resource efficiency.
Data Source
AI summary
Techniques to eliminate duplicate packets from a stream of packets. A method includes receiving a packet, hashing elements of the packet to obtain a packet partial signature, generating an action data value by appending, to the packet partial signature, unhashed packet information, accessing a hash store location in a hash store based on the action data value, determining, based on data stored at the hash store location, whether the packet is a duplicate of a previously-received packet, and based on whether the packet is a duplicate of a previously-received packet, one of allowing and preventing one of the packet and a copy of the packet to reach a downstream network node.