RDMA Hash-Based Lookup for Bandwidth Reduction
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current remote direct memory access (RDMA) technologies face inefficiencies in data transmission and latency, particularly for medium and large messages, due to the need for extensive data transfer and CPU involvement.
Innovation Solution
Integration of cryptographic hash functions into the RDMA communication protocol allows network interface controllers to reduce data transmission by using hash values to match data locations in translation protection tables, enabling direct memory access without transferring data when matches are found, thus optimizing bandwidth and latency.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If data is transferred from source node to destination node in traditional RDMA operations, then data transmission is completed, but data transmission volume and latency increase for medium and large messages
Solution Approach 1:
The system pre-computes and stores cryptographic hash values of data blocks in translation protection tables at destination nodes before actual data transfer occurs. When an RDMA operation is initiated, the source node sends only the hash value instead of the entire data block, allowing the destination node to verify data presence or compute hashes locally, thereby avoiding unnecessary data transmission and reducing latency
Solution Approach 2:
Instead of transferring actual data blocks over the network, the system transfers cryptographic hash values (digital fingerprints) which serve as compact representations of the data. The destination node uses these hash values to determine whether data already exists locally or needs to be transferred, significantly reducing network bandwidth consumption while maintaining data integrity verification
2Loss of energy
If cryptographic hash lookup is performed in translation protection table, then data transfer is avoided when matches are found, but additional lookup operation complexity is introduced
Solution Approach 1:
The translation protection table acts as an intermediary data structure that stores pre-computed cryptographic hash values alongside data location information. This intermediary layer enables efficient hash-based lookup operations without requiring complex distributed file system protocols or metadata exchanges, simplifying the overall system architecture while reducing network bandwidth consumption through localized hash verification
Data Source
AI summary
A mechanism is provided in a data processing system for performing a remote direct memory access operation. Responsive to receiving in a network interface controller a hash value of data to be copied from a source address in a source node to a destination address in a destination node in the remote direct memory access operation, the network interface controller performs a lookup operation in a translation protection table in the network interface controller to match the hash value to a hash value for data existing in memory of the destination node. Responsive to the network interface controller finding a match in the translation protection table, the network interface controller completes the remote direct memory access operation without transferring the data from the source node to the destination node.