InfiniBand Non-Posted Operations for Data Consistency and Latency

Solution Overview

Problem

InfiniBand protocols face challenges in maintaining data consistency and reducing latency during two-phase commit operations due to the distinction between posted and non-posted operations, leading to potential data inconsistency and network bottlenecks.

Innovation Solution

Introducing new non-posted InfiniBand operations such as non-posted RDMA write, atomic, and send operations that allow for flexible ordering and execution, enabling the packet processing circuitry to write data to host memory while allowing subsequent operations to complete before acknowledging the non-posted write, and using flush operations to ensure data persistence without interrupting the CPU.

Engineering Contradictions & Design Principles

VSEngineering Contradiction Analysis

1Reliability

If posted operations are used to ensure sequential execution order, then data consistency is improved, but latency increases due to waiting for previous operations to complete

Engineering Contradiction:

Improvedata consistencyVSAvoidlatency

Core Design Contradiction:

ReliabilityVSLoss of time

Solution Approach 1:

The patent introduces dynamic operation modes (posted and non-posted) that can be selected based on consistency requirements. Non-posted operations allow the system to dynamically bypass sequential execution constraints when consistency is not required, while posted operations maintain strict ordering when needed. This dynamic adaptability resolves the contradiction by allowing the system to optimize for speed when consistency is less critical.

Inventive Principle:

Principle #15Dynamics

Solution Approach 2:

The patent segments operations into two distinct categories: posted operations that maintain sequential ordering and non-posted operations that do not. This segmentation allows different parts of the operation sequence to be handled differently - critical consistency-maintaining operations use posted mode while non-critical operations use non-posted mode for faster execution, thereby resolving the latency-consistency tradeoff.

Inventive Principle:

Principle #1Segmentation

2Reliability

If fencing techniques are used to ensure data consistency, then reliability is improved, but productivity decreases due to execution delays

Engineering Contradiction:

Improvedata consistencyVSAvoidexecution speed

Core Design Contradiction:

ReliabilityVSProductivity

Solution Approach 1:

Instead of applying fencing to all operations (excessive action), the patent applies fencing selectively only to posted operations where consistency is critical (partial action). Non-posted operations execute without fencing constraints, maintaining productivity. This selective application resolves the contradiction by providing consistency only where necessary.

Inventive Principle:

Principle #16Partial or excessive action

3Reliability

If strict sequential ordering is enforced for all operations, then data consistency is improved, but network throughput decreases due to bottlenecks

Engineering Contradiction:

Improvedata consistencyVSAvoidnetwork throughput

Core Design Contradiction:

ReliabilityVSProductivity

Solution Approach 1:

The patent introduces dynamic operation modes (posted and non-posted) that can be selected based on consistency requirements. Non-posted operations allow the system to dynamically bypass sequential execution constraints when consistency is not required, while posted operations maintain strict ordering when needed. This dynamic adaptability resolves the contradiction by allowing the system to optimize for speed when consistency is less critical.

Inventive Principle:

Principle #15Dynamics

Solution Approach 2:

The patent segments operations into two distinct categories: posted operations that maintain sequential ordering and non-posted operations that do not. This segmentation allows different parts of the operation sequence to be handled differently - critical consistency-maintaining operations use posted mode while non-critical operations use non-posted mode for faster execution, thereby resolving the latency-consistency tradeoff.

Inventive Principle:

Principle #1Segmentation

Data Source

PatentUS10552367B2Network data transactions using posted and non-posted operations

Publication Date: 2020.02.04 MELLANOX TECHNOLOGIES LTD(IL)

AI summary

Communication apparatus includes a host interface, configured to be coupled to a host processor having a host memory, and a network interface, which is configured to receive over a network from a sending node data packets conveying operations for execution in a sequential order on a predefined queue pair (QP), including at least a first packet conveying a posted write operation and a second packet conveying a non-posted write operation. Packet processing circuitry is configured to execute the posted write operation in accordance with the sequential order so as to write first data to the host memory prior to the execution of any subsequent operations in the sequential order, and to execute the non-posted write operation so as to write second data to the host memory while allowing one or more of the subsequent operations in the sequential order to be executed prior to completion of writing the second data.