HPC Log Anomaly Scoring for Early Error Prediction
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
High-performance computing (HPC) systems face increased frequency of errors and failures due to their complex structure and growing number of components, making it difficult to predict and prevent errors effectively.
Innovation Solution
A method and system utilizing a deep learning-based model, including a discriminator and generator, to analyze log data from HPC systems, tokenize and preprocess the data, and generate anomaly scores to predict future errors by distinguishing between normal and abnormal system states.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If the HPC system includes more components and becomes more complex to handle larger workloads, then processing capability and versatility are improved, but error frequency and system reliability deteriorate
Solution Approach 1:
The system performs preliminary actions by analyzing log data and generating anomaly scores to predict potential errors before they occur. The error prediction model continuously monitors system logs and identifies patterns that precede failures, enabling proactive error prevention through early warning signals.
Solution Approach 2:
The system implements feedback mechanisms by continuously collecting log data from system operations, processing it through the discriminator model, and using the generated anomaly scores to adjust system behavior. The feedback loop enables the system to learn from past errors and improve its error prediction accuracy over time.
2Device complexity
If traditional error monitoring methods are used, then system complexity is maintained at acceptable levels, but error prediction accuracy and detection capability deteriorate
Solution Approach 1:
The patent replaces traditional mechanical error monitoring methods with an AI-based deep learning system. The discriminator model and generator model substitute conventional threshold-based monitoring with intelligent anomaly detection that analyzes log patterns, enabling accurate error prediction without requiring proportional increases in system complexity.
Solution Approach 2:
The system changes parameters by transforming raw log data into anomaly scores through the discriminator model. This parameter transformation enables the system to detect subtle error patterns that would be invisible to traditional monitoring methods, significantly improving error prediction accuracy while maintaining manageable complexity through automated processing.
3Ease of manufacture
If log data is analyzed in raw format, then data processing simplicity is maintained, but anomaly detection precision and error identification capability deteriorate
Solution Approach 1:
The system applies segmentation by dividing raw log data into discrete tokens that can be individually processed by the discriminator model. This tokenization process transforms unstructured log text into structured units, enabling precise anomaly detection while maintaining processing simplicity through automated sequential analysis of tokenized data.
Solution Approach 2:
The patent introduces an intermediary processing layer that transforms raw log data into anomaly scores. The discriminator model acts as an intermediary between raw logs and error detection, converting unstructured text into quantifiable anomaly metrics that enable precise error identification without requiring complex manual analysis.
Data Source
AI summary
A method and device for predicting errors in a computing system are disclosed. The error prediction method includes: receiving log data generated by the computing system during operation of the computing system; tokenizing the log data into tokens; inputting the tokens to a discriminator model which generates scores of the respective tokens, each score corresponding to a probability that the corresponding token is an anomaly token; determining an anomaly score based on the scores; and determining a likelihood of future occurrence of an error in the computing system based on the anomaly score.