Multi-processor failure detection via shared memory timestamps
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing multi-processor systems face challenges in detecting processor failures, particularly when the master CPU fails, and current heartbeat algorithms can cause inefficiencies due to message-based communication, which may lead to undetectable failures and operational interruptions.
Innovation Solution
A method that involves retrieving and comparing timestamps from shared memory across CPUs to determine delta values, with additional time accounting for message transfer and clock synchronization, allowing for timely detection of failures and initiating error handling, including non-critical and critical interrupts to manage master and slave CPU roles.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If a master CPU monitors slave CPUs using message-based heartbeat algorithms, then CPU failure detection is implemented, but system efficiency decreases due to message interruptions and master CPU failure becomes undetectable
Solution Approach 1:
The system segments the failure detection function by having each CPU independently monitor its own heartbeat status and the heartbeats of other CPUs through shared memory timestamps, rather than relying on a single master CPU. This distributes the monitoring responsibility across all processors, eliminating the bottleneck and single point of failure while reducing message-based interruptions.
Solution Approach 2:
The patent introduces shared memory with timestamp registers as an intermediary mechanism for heartbeat communication. Instead of direct CPU-to-CPU messaging that causes interruptions, CPUs write timestamps to shared memory locations, and other CPUs read these timestamps to detect failures. This intermediary approach eliminates message interruptions while maintaining failure detection capability.
2Reliability
If message-based communication is used for heartbeat algorithms, then CPU functionality monitoring is achieved, but operational interruptions occur reducing system performance
Solution Approach 1:
The patent replaces the mechanical message-passing system with a memory-based timestamp comparison system. Instead of sending messages that interrupt CPU operations, each CPU simply writes its current timestamp to a shared memory location, and other CPUs read and compare these timestamps without causing interruptions. This substitution eliminates operational interruptions while maintaining monitoring capability.
3Ease of operation
If a master CPU architecture is used for failure detection, then centralized control is achieved, but the master CPU failure becomes undetectable creating a single point of failure
Solution Approach 1:
The patent inverts the traditional master-slave monitoring model by having each CPU monitor itself and other CPUs through shared memory timestamps, rather than having the master monitor slaves. This inversion eliminates the single point of failure problem while maintaining centralized error handling through the kernel module that receives notifications from any CPU detecting a failure.
Data Source
AI summary
An approach to detecting processor failure in a multi-processor environment is disclosed. The approach may include having each CPU in the system responsible for monitoring another CPU in the system. A CPUn reads a timestampn+1 created by CPUn+1 which CPUn is monitoring from a shared memory location. The CPUn reads its own timestampn and compares the two timestamps to calculate a delta value. If the delta value is above a threshold, the CPUn determines that CPUn+1 has failed and initiates error handling for the CPUs in the system. One CPU may be designated a master CPU, and be responsible for beginning the error handling process. In such embodiments, the CPUn may initiate error handling by notifying the master CPU that CPUn+1 has failed. If CPUn+1 is the master CPU, the CPUn may take additional steps to initiate error handling, and may broadcast a non-critical interrupt to all CPUs, triggering error handling.