Network Interface Data Deduplication via Filter Engine
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
In distributed computing systems handling massive application data, duplicate messages between nodes consume unnecessary computing resources and bandwidth, as existing systems lack effective deduplication mechanisms.
Innovation Solution
A method and system for data deduplication that employs network interfaces with filter engines to identify and discard duplicate messages by extracting and matching keys, using cache structures and filter bits to determine message uniqueness before transmission or reception.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If duplicate messages are transmitted between computing nodes, then message delivery completeness is improved, but bandwidth consumption and resource waste increase
Solution Approach 1:
The patent applies preliminary action by implementing deduplication checks at the network interface layer before messages are fully transmitted or processed. The filter engine extracts keys from incoming messages and checks them against cached keys from the same computing node, discarding duplicates before they consume full network bandwidth and processing resources. This preliminary filtering prevents wasted transmission of redundant messages while ensuring unique messages are delivered.
2Loss of energy
If deduplication mechanisms are implemented at the network interface layer, then bandwidth efficiency is improved, but device complexity increases
Solution Approach 1:
The patent applies segmentation by dividing the deduplication function into separate modular components within the network interface: a filter engine for key extraction and comparison, a cache structure for storing keys, and a message routing component. This segmentation allows the deduplication functionality to be added to existing network interfaces without completely redesigning the system, reducing the impact of increased complexity while achieving bandwidth efficiency.
Solution Approach 2:
The patent introduces an intermediary deduplication layer at the network interface that sits between the application layer and the network transmission layer. This intermediary filter engine intercepts messages, performs key-based deduplication checks, and decides whether to allow message transmission. By placing this intermediary at the network interface rather than at the application layer or deep in the protocol stack, the system achieves deduplication with minimal impact on overall system complexity.
3Productivity
If messages are filtered and deduplicated before transmission, then processing efficiency is improved, but message delivery time may increase
Solution Approach 1:
The patent applies preliminary action by performing deduplication checks at the network interface layer before messages are transmitted over the network. The filter engine extracts keys and checks the cache quickly using efficient data structures, allowing most duplicate messages to be discarded in O(1) time. This preliminary filtering prevents the system from wasting time transmitting and then processing duplicate messages, improving overall processing efficiency while minimizing additional delivery time for unique messages.
Data Source
AI summary
A method for data deduplication during execution of an application on a plurality of computing nodes, including: generating, by a first processor in a first computing node executing the application, a first message to process application data owned by a second computing node executing the application; receiving, by a first network interface (NI) of the first computing node, the first message; extracting, by the first NI, a first key from the first message; determining, by the first NI, the first key is not a duplicate; and placing, by the first NI and in response to the first key not being a duplicate, the first message on a network connecting the first computing node to the second computing node.