Automated Data Center Fault Management via Machine Learning
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Conventional fault management systems in data centers, such as S.M.A.R.T, are limited in predicting and responding to unexpected system errors, primarily monitor hard disk drives, and fail to address software-related faults, leading to unnecessary tasks and missed alarms, with a lack of proactive measures for new fault types.
Innovation Solution
An automated system and method for managing fault events in data centers, which includes a fault event analysis unit, a knowledge storage unit, a monitoring unit, and an action unit to analyze and respond to type-specific fault events, using data collection, machine learning, and artificial intelligence to generate and execute automated responses.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If conventional fault management systems like S.M.A.R.T. are used to monitor hard disk drives, then monitoring of hard disk attributes (read error rate, reallocated sectors) is achieved, but the system cannot detect or respond to software-related faults or unexpected system errors
Solution Approach 1:
The monitoring system is extended from exclusively monitoring hard disk drive attributes to universally monitoring multiple data sources including system logs, application logs, and performance metrics. The analysis engine is designed to handle various fault types (hardware failures, software errors, system crashes) through a unified approach, making the system multi-functional and adaptable to different fault scenarios.
Solution Approach 2:
The system dynamically adapts its monitoring and analysis capabilities based on the type of fault detected. Instead of using fixed thresholds for hard disk monitoring, the system employs machine learning models that continuously learn from new data patterns and adjust their detection criteria. This allows the system to respond appropriately to both expected hardware failures and unexpected software-related faults.
2Reliability
If threshold-based monitoring is used for hard disk drives, then prediction of hard disk failures is achieved, but 50% of hard disk failures occur without alarm notification
Solution Approach 1:
The system implements continuous feedback loops where detected faults and their outcomes are fed back into the machine learning models. This allows the system to learn from both successful predictions and missed detections (false negatives). The feedback mechanism enables the system to refine its prediction algorithms and reduce the rate of undetected failures over time by adjusting its detection criteria based on actual failure patterns.
Solution Approach 2:
Instead of waiting for threshold violations to trigger alerts, the system performs preliminary analysis of multiple data sources and trends to predict potential failures before they occur. The machine learning models analyze patterns in system logs, performance metrics, and error rates to identify early signs of both hardware and software failures, enabling proactive intervention before the actual failure occurs.
3Ease of operation
If manual fault management is used in data centers, then operation based on previous records is achieved, but automated rapid response and prediction of new fault types are impossible
Solution Approach 1:
The system implements self-service capabilities through automated fault detection, analysis, and response mechanisms. When a fault is detected, the system automatically correlates it with historical data, identifies the root cause, and executes predefined remediation actions without human intervention. This self-service approach enables rapid response to both common and novel fault types, significantly improving productivity while maintaining ease of operation through centralized automation.
Solution Approach 2:
The system performs preliminary analysis and preparation by continuously learning from historical fault data and building knowledge bases of fault patterns and remediation strategies. When a new fault occurs, the system has already prepared response protocols based on similar historical cases, enabling immediate automated action. This preliminary preparation eliminates the need for manual analysis and speeds up the response time for both known and novel fault types.
4Reliability
If monitoring is focused on internally generated data from hard disks, then prediction of hard disk errors is achieved, but system errors from software conflicts cannot be predicted or responded to
Solution Approach 1:
The monitoring system is designed with universal capabilities to collect and analyze multiple types of data sources simultaneously. In addition to hard disk metrics, the system monitors system logs, application logs, performance counters, and event data from various system components. This multi-functional approach enables the system to detect and respond to both hardware failures and software-related faults using a unified analysis framework.
Solution Approach 2:
The system segments its monitoring capabilities into specialized modules: one for hard disk monitoring, another for system log analysis, and additional modules for application-specific metrics. Each segment focuses on specific data types and fault patterns, but they all feed into a centralized machine learning engine that correlates data across segments. This segmentation allows deep specialization in each area while maintaining system-wide fault coverage through integrated analysis.
Data Source
AI summary
Disclosed are an automated management system and method for fault events of a data center. A fault event monitoring unit is configured to check statuses of servers to detect occurrence of a fault event, and if a fault event occurs, determine whether an automated action is possible. A fault event related action recommending unit is configured to, with respect to a fault event that is newly detected by the fault event monitoring unit or determined to be difficult to be resolved by an action stored in a fault event related action storage unit, search for and extract information related to the fault event determined to be difficult to be resolved from information accumulated in a fault event information knowledge storage unit, and based on the extracted information, generate and output a group of automated action candidates for the fault event determined to be difficult to be resolved. A fault event action unit is configured to automatically take an action against a fault event detected by the fault event monitoring unit by use of the action accumulated in the fault event related action storage.