Parallel Compute Diagnostics for Zero-Downtime Hardware Failure Detection
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing datacenter environments struggle with early detection of hardware degradations and failures, leading to potential catastrophic failures and service disruptions due to the inability to efficiently identify and replace faulty compute hardware, especially in mission-critical systems like large-scale e-commerce, financial, healthcare, and scientific data processing, and Autonomous Driving.
Innovation Solution
Implement a software test harness that runs on CPU cores in parallel with customer workloads, performing predetermined platform functionality tests and using comparison algorithms to detect discrepancies across cores and accelerators, with a daemon service periodically testing unutilized cores to identify and report errors.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If hardware is taken offline for validation testing, then detection accuracy improves, but system availability deteriorates
Solution Approach 1:
The system performs preliminary validation testing by executing diagnostic algorithms on standby or underutilized cores before they are needed for production workloads. This allows hardware to be validated in advance, ensuring detection accuracy while maintaining system availability when the validated hardware is activated.
Solution Approach 2:
The system creates software copies of validation test algorithms that can execute on multiple cores simultaneously. By copying the test harness to run on available cores during idle periods or on standby hardware, the system achieves thorough validation without taking production systems offline.
2Reliability
If comprehensive hardware testing is performed, then reliability improves, but testing time increases
Solution Approach 1:
The system implements periodic execution of validation algorithms at defined intervals rather than continuous testing. This allows comprehensive reliability checking while managing testing time through scheduled periodic validation cycles that balance thoroughness with operational efficiency.
Solution Approach 2:
The system performs validation on a subset of cores or hardware components at any given time rather than all components simultaneously. By distributing validation across multiple cores sequentially or in parallel batches, comprehensive testing is achieved without excessive testing time impacting overall system performance.
3Productivity
If parallel testing across multiple cores is implemented, then productivity improves, but device complexity increases
Solution Approach 1:
The system segments the validation testing process into independent test harness instances that can execute on different cores simultaneously. Each segment handles specific validation tasks, enabling parallel testing across multiple cores while managing complexity through modular, independent test modules.
Solution Approach 2:
The test harness is designed as a universal software component that can execute on any CPU core, accelerator, or XPU without modification. This multi-functional design enables parallel testing across diverse hardware types while reducing overall system complexity through a single standardized testing solution.
Data Source
AI summary
Methods, apparatus, and software for hardware reliability diagnostics and failure detection via parallel software computation and compare. Parallel testing is performed on hardware resources such as processor cores, accelerators, and Other Processing Units (XPUs) using test algorithms such as encryption/decryption. The results of the testing (the algorithm outputs) are compared to detect errant hardware. Comparison may be across cores (via execution of software-based algorithms), across accelerators/XPUs (via algorithms implement in hardware) or between cores and accelerators/XPUs. Techniques are disclosed to enable all cores to be tested while a platform is performing a workload, such as in a data center environment, wherein unused cores are used for testing, with workloads being migrated between cores between tests.