Cluster Failover Crash Identification via Process Trace Analysis
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
High availability systems face challenges in quickly and accurately identifying the cause of node failures in information handling systems, leading to potential disruptions during the relocation and restart of applications, as existing methods often fail to isolate the crash-causing application effectively.
Innovation Solution
A cluster fallover method is introduced, where a shared storage system communicates with multiple information handling systems to dump and access state information, derive process traces, and determine the closest fit match among applications to identify the failed application, thereby isolating and managing the crash-causing application to minimize disruptions.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If existing failure identification methods are used, then the system can detect node failures, but the crash-causing application cannot be accurately identified, leading to service disruptions
Solution Approach 1:
The system performs preliminary actions by dumping process traces to shared storage before the failure fully impacts service. The HACM on the fallover node proactively accesses and analyzes these pre-dumped traces to identify the crash-causing application, enabling faster recovery without service disruption.
Solution Approach 2:
The patent introduces shared storage as an intermediary between the failed node and the fallover node. The process traces are dumped to this intermediary storage medium, allowing the HACM to analyze failure causes without direct access to the crashed processes, thus maintaining service continuity while achieving accurate identification.
2Measurement precision
If comprehensive process trace analysis is performed to identify the crash-causing application, then identification accuracy improves, but the time required for failure analysis increases
Solution Approach 1:
Process traces are dumped to shared storage in advance, before failure analysis is needed. This preliminary action ensures that when a failure occurs, the HACM can immediately access pre-prepared trace data without waiting for live process information, significantly reducing analysis time while maintaining comprehensive identification accuracy.
Solution Approach 2:
The system creates copies of process traces and stores them in shared storage. Instead of analyzing live processes during failure (which would be time-consuming), the HACM analyzes these pre-made copies, enabling rapid identification of the crash-causing application without time loss.
3Reliability
If the system relocates all applications during node failure, then service availability is maintained, but resource overhead and recovery complexity increase
Solution Approach 1:
The patent extracts and identifies the specific crash-causing application from the set of all running applications using process trace analysis. Instead of relocating all applications during failure, only the identified problematic application is targeted for relocation or termination, reducing resource overhead and simplifying recovery management while maintaining service availability.
Solution Approach 2:
The system applies different treatment to different applications based on their role in the failure. The crash-causing application identified through trace analysis receives targeted intervention (termination or relocation), while other healthy applications continue running without disruption. This localized approach reduces complexity compared to relocating all applications.
Data Source
AI summary
The method determines whether a particular node of a high availability cluster is functioning properly or is a failed node. The method dumps node process state information as a dump data for the failed or crashed node in a shared storage area of the high availability cluster. A high availability cluster manager identifies the dump data that corresponds to the failed node as the most recent dump data for that failed node. The high availability cluster manager interrogates the dump data using kernel debugger services to identify a process trace and thereby identify the crash-causing application for the failed node. The method determines if the dump data includes a process match for the failed node process. The high availability cluster manager may initiate a crash-causing application notification to administrators or other entities of the high availability cluster. The method provides relocation and restoration capability of applications from the failed node to a fallover node and returns those application resources to the user and other entities using the high availability cluster.