InfiniBand Queue Pair Fault Handling via Callback Re-queuing
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
High performance computing networks face inefficiencies in communication error fault handling, leading to terminated execution of applications due to unmanaged communication errors in InfiniBand environments, lacking responsive and accurate error notification mechanisms.
Innovation Solution
Implementing a method to detect InfiniBand queue pair errors, infer message status, and establish an IB Direct Connect queue pair connection to re-queue messages, allowing applications to handle errors via callback mechanisms, thereby bypassing conventional execution paths and providing efficient, accurate, and responsive fault handling.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If conventional InfiniBand error handling is used, then communication errors are detected, but application execution is terminated and fault handling is inefficient
Solution Approach 1:
Instead of terminating application execution when errors occur (conventional approach), the patent inverts the approach by allowing applications to continue executing and handling errors through callback mechanisms. The error handling flow is reversed from 'error → termination' to 'error → callback → continued execution', enabling fault tolerance while maintaining productivity.
Solution Approach 2:
The patent introduces callback mechanisms as intermediaries between error detection and application response. Rather than directly terminating execution, the system uses callbacks to notify applications of errors, allowing them to handle faults gracefully. This intermediary layer enables efficient error notification while preserving application continuity.
2Reliability
If message re-queuing is implemented after error detection, then communication reliability is improved, but system complexity increases
Solution Approach 1:
The patent extracts the error handling logic from the core communication path by using separate callback mechanisms. Message re-queuing is handled independently through callbacks rather than being embedded in the main message flow, reducing complexity in the primary communication path while maintaining reliability through the extracted error handling subsystem.
Solution Approach 2:
The system performs preliminary error detection and callback notification before attempting message re-queuing. By detecting errors and notifying applications in advance, the system prepares for potential re-queuing operations without immediately complicating the communication flow, thereby improving reliability while managing complexity through staged error handling.
3Speed
If direct application access to InfiniBand messaging service is enabled, then communication speed is improved, but error management becomes more difficult
Solution Approach 1:
The patent implements feedback mechanisms through callbacks that provide error information directly to applications. This feedback loop allows applications to access the high-speed InfiniBand messaging service directly while receiving timely error notifications, enabling them to manage errors effectively without sacrificing communication speed. The feedback ensures that direct access does not lead to unmanaged errors.
Data Source
AI summary
A computer-implemented method and system for inter-processor communications fault handling in high performance computing networks. The method includes detecting that an InfiniBand (IB) queue pair has transitioned into an error state based on an unsuccessful completion status that relates to unsuccessful delivery of a message from an initiator endpoint at a first server device to at least one target endpoint at a second server device. The initiator and target endpoints are associated with at least one application under execution. An embodiment includes inferring, when the unsuccessful completion status is indicated as flushed, that the message was in a send queue of the IB queue pair when the IB queue pair transitioned into the error state. An embodiment includes establishing an IB Direct Connect queue pair connection between the target and initiator endpoints. An embodiment includes re-queueing the message in the IB queue pair for dispatch to the target endpoint.