Investigation Node Local Analysis for Distributed System Failure
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
In parallel distributed processing systems, the time-consuming process of transferring and analyzing dump files from multiple nodes to a file server hinders efficient failure investigation, due to the large amount of data and the number of nodes involved.
Innovation Solution
An information processing apparatus is designed to allocate investigation nodes within the system, allowing them to acquire and analyze data locally, using investigation agents to determine failure by executing commands such as searching data patterns, tracking pointers, and checking field values, thereby reducing the reliance on file server data and enhancing processing efficiency.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If dump files are transferred from all nodes to a file server for analysis, then complete failure investigation data is obtained, but the time required for failure investigation increases significantly
Solution Approach 1:
The failure investigation process is segmented into multiple independent analysis tasks that can be executed in parallel. Different investigation nodes are assigned to analyze different aspects of the failure (e.g., stack trace analysis, memory dump analysis, log file analysis) simultaneously, rather than sequentially transferring all data to a single file server for centralized analysis.
Solution Approach 2:
The system transitions from a single-dimension centralized analysis model to a multi-dimensional distributed analysis model. Multiple investigation nodes operate in parallel across different dimensional spaces (different analysis types, different data subsets), enabling simultaneous processing and significantly reducing the overall investigation time while maintaining comprehensive coverage.
2Reliability
If dump files from multiple nodes are acquired and stored in a file server, then comprehensive failure data is collected, but the data transfer burden on the file server increases
Solution Approach 1:
The centralized file server burden is segmented and distributed to multiple investigation nodes. Each investigation node receives and processes a subset of the dump files locally, eliminating the need for a single file server to handle all data transfers. This distributes the I/O burden across multiple nodes and reduces the bottleneck at any single point.
Solution Approach 2:
Investigation nodes perform self-service by locally acquiring and analyzing dump files from failed nodes without requiring centralized file server mediation. Each investigation node independently executes analysis tasks on the data it receives, reducing dependency on the file server and distributing the processing workload across the system.
3Loss of information
If all memory dump data from failed nodes is transferred for analysis, then complete failure information is obtained, but the amount of data to be transferred and processed increases
Solution Approach 1:
The system extracts and transfers only the essential failure-related information from the complete memory dump, rather than transferring entire dump files. Investigation nodes extract critical data elements (such as stack traces, error codes, relevant register states) and transfer only these extracted portions for analysis, significantly reducing data transfer volume while preserving all necessary failure information.
Solution Approach 2:
The system performs partial action by analyzing only the portions of dump files that are relevant to the specific failure being investigated. Rather than processing complete dump files in their entirety, the system identifies and analyzes only the necessary subsets of data required to diagnose the failure, reducing the quantity of data processed while maintaining investigation completeness.
Data Source
AI summary
An information processing apparatus, includes a processor, and memory storing instructions for causing the processor to allocate a node in an information processing system including the information processing apparatus as an investigation node configured to perform an investigation of data stored in memory of a node of which an error is detected. The processor further instructs the investigation node to acquire data to be investigated from the node of which the error is detected, instructs the investigation node to perform an operation for determining whether a predetermined value in the acquired data is a normal value, and determines that a failure occurs in the node of which the error is detected when the predetermined value is not a normal value.