Log Data Preprocessing for Neural Network Error Prediction
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
As computing systems become more complex, predicting system errors with varying error lead times becomes increasingly challenging due to the large volume of log data generated by numerous components, which often includes redundant and repetitive information, making it difficult to accurately anticipate and respond to potential errors in a timely manner.
Innovation Solution
An apparatus and method utilizing a processor to preprocess log data by eliminating redundant pieces and grouping them into sequences, which are then used to train a multi-label deep neural network for system error prediction, generating reports on predicted error times, types, and probabilities, enabling timely and effective error handling.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If log data from multiple components is collected to track system activities, then the ability to analyze system errors is improved, but the volume of data increases making error prediction more difficult
Solution Approach 1:
The patent extracts and eliminates redundant log data pieces through preprocessing, removing duplicate entries and filtering out unnecessary information while preserving critical error indicators. This extraction process reduces the overall data volume while maintaining the essential information needed for accurate error prediction.
Solution Approach 2:
The patent segments the large volume of log data into manageable sequences of predetermined sizes. By dividing the continuous stream of logs into discrete sequences, the system can process and analyze smaller, more manageable data units that are easier for the neural network to process while still capturing temporal patterns of system errors.
2Productivity
If redundant log data pieces are eliminated through preprocessing, then the efficiency of error prediction is improved, but the complexity of data processing increases
Solution Approach 1:
The patent performs preliminary actions by eliminating redundant log data pieces and organizing them into sequences before feeding the data to the neural network. This preprocessing step, while adding initial processing complexity, significantly reduces the computational burden during the actual error prediction phase, improving overall system efficiency.
3Device complexity
If log data is grouped into sequences of predetermined size, then the neural network processing is simplified, but the loss of temporal information may occur
Solution Approach 1:
The patent segments log data into sequences of predetermined sizes, which simplifies the input structure for neural network processing. The segmentation is designed to maintain temporal relationships by preserving the sequential order of log entries within each sequence, thus reducing processing complexity while minimizing temporal information loss.
Data Source
AI summary
An apparatus includes a processor configured to execute instructions, and a memory storing the instructions, which when executed by the processor configure the processor to generate system error prediction data using an error prediction neural network provided with one of a plurality of log data sequences generated by pre-processing a plurality of log data pieces of component log data of a system. The system error prediction data comprises information of a plurality of system errors occurring at a plurality of respective timepoints.