Software Fault Detection for Parallel Hardware Reliability

Resolve Bottlenecks,
Find Innovative Solutions
Generate Solutions

Solution Overview

Problem

Current fault detection methods in computing hardware, particularly in parallel hardware, are inadequate for identifying faults in safety-critical applications like autonomous machines and data centers, as they often rely on redundant systems that are costly and less effective for detecting errors in parallel processors such as GPUs.

Innovation Solution

The implementation of a software-based fault detection system that uses a library of diagnostic tests to thoroughly examine hardware components like GPUs, scheduling tests to occur frequently, including vector operations, matrix multiplication, bit operations, and cache tests, to identify potential faults and trigger mitigating actions.

Engineering Contradictions & Design Principles

VSEngineering Contradiction Analysis

1Reliability

If redundant hardware systems are used to detect faults in parallel processors, then fault detection capability is improved, but system cost and hardware complexity increase significantly

Engineering Contradiction:
Improvefault detection capabilityVSAvoidhardware redundancy
Core Design Contradiction:
ReliabilityVSDevice complexity

Solution Approach 1:

The patent replaces hardware-based redundant systems with a software-based fault detection mechanism. The software system executes diagnostic routines on the parallel processor hardware, eliminating the need for duplicate hardware components while maintaining fault detection capability. This substitution of mechanical/hardware systems with software systems directly resolves the contradiction by improving reliability without increasing hardware complexity.

Inventive Principle:
Principle #28Mechanics substitution (Replace mechanical system)

Solution Approach 2:

The patent introduces a software intermediary layer between the parallel processor hardware and the fault detection function. This intermediary executes diagnostic routines that test hardware components without requiring the hardware itself to be redundant. The software mediator performs the detection function that would otherwise require duplicate hardware, thus resolving the contradiction between reliability and hardware complexity.

Inventive Principle:
Principle #24Intermediary (Mediator)

2Reliability

If conventional fault detection tests are performed periodically to satisfy safety standards, then compliance is achieved, but the detection frequency is insufficient for runtime errors within FTTI intervals

Engineering Contradiction:
Improvesafety complianceVSAvoiddetection interval
Core Design Contradiction:
ReliabilityVSLoss of time

Solution Approach 1:

The patent implements dynamic fault detection scheduling that adapts to system operational state. Rather than fixed periodic tests, the system dynamically determines when diagnostic routines should execute based on runtime conditions, allowing detection within FTTI intervals for critical errors while maintaining safety compliance. This dynamic adjustment resolves the contradiction between compliance and detection timing.

Inventive Principle:
Principle #15Dynamics

Solution Approach 2:

The patent performs preliminary diagnostic tests that can detect faults before they cause safety-critical failures. By executing detection routines proactively rather than reactively after failures occur, the system maintains safety compliance while reducing the effective detection interval for preventing harmful outcomes. This preliminary action approach addresses both compliance requirements and timely detection needs.

Inventive Principle:
Principle #10Preliminary action

3Adaptability or versatility

If built-in self-tests are repurposed to detect runtime faults, then test coverage is expanded, but detection effectiveness for runtime errors within FTTI is reduced

Engineering Contradiction:
Improvetest coverageVSAvoidruntime error detection
Core Design Contradiction:
Adaptability or versatilityVSReliability

Solution Approach 1:

The patent segments fault detection into distinct functional components: hardware self-tests for manufacturing defects and software-based diagnostic routines for runtime faults. This segmentation allows each component to be optimized for its specific purpose, with the software routines providing the necessary runtime detection capability within FTTI intervals while hardware tests maintain their original functionality. This resolves the contradiction between test coverage versatility and runtime error detection effectiveness.

Inventive Principle:
Principle #1Segmentation

Data Source

PatentUS20240402250A1Fault detection in parallel hardware
Publication Date: 2024.12.05 NVIDIA CORP
  • US20240402250A1 patent drawing
  • US20240402250A1 patent drawing
  • US20240402250A1 patent drawing

AI summary

In various examples, faults are detected based at least in part on result value(s) generated by hardware component(s) by performing one or more diagnostic tests in accordance with a diagnostic test pattern. The diagnostic test pattern may be used to perform an assessment of functionality of the hardware component(s) by causing the hardware component(s) to generate the result value(s), which may be used to identify one or more hardware faults (e.g., by comparing the result value(s) to expected value(s)).