ML Model Training Data Acquisition for Anomaly Detection
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Acquiring training data for a machine learning model to monitor a data processing system can introduce biases and disturbances, as the normal input/output functionality and memory usage during data acquisition may interfere with the system's operation, leading to inaccurate anomaly detection.
Innovation Solution
A dedicated hardware interface and a dummy function are used to minimize disturbances during data acquisition, with the training data temporarily stored in the ML model memory and transferred using general-purpose I/O ports during non-operational periods, ensuring unbiased data collection by discarding biased portions and restarting the system if necessary.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If training data is acquired using normal I/O functionality during system operation, then the ML model can be trained with real operational data, but the data acquisition process introduces network congestion and biases that distort the training data
Solution Approach 1:
The patent segments the data acquisition process into distinct phases: a data collection phase where training data is gathered during system operation, and a data transfer phase where the collected data is transferred to the training environment. This segmentation allows the system to separate the harmful data transfer operations from the data collection operations, preventing network congestion during critical measurement periods.
Solution Approach 2:
The patent implements preliminary action by collecting training data during system operation before the actual training process begins. The data is stored in memory and then transferred to the training environment in advance, allowing the ML model to be trained with real operational data without introducing congestion during the training phase.
2Reliability
If training data is acquired from the system being monitored, then the ML model learns accurate system behavior, but the acquisition process itself biases the system operation and corrupts the training data
Solution Approach 1:
The patent extracts the training data acquisition process from the normal system operation by implementing a dedicated data collection mode. During this mode, the system operates in a special state where data is collected for training purposes separately from normal operational data flow, allowing the ML model to learn accurate system behavior without the acquisition process corrupting the training data.
Solution Approach 2:
The patent introduces an intermediary mechanism (dummy function and dedicated data transfer interface) that mediates between the anomaly detector and the training environment. This intermediary allows training data to be collected and transferred without directly interfering with normal system operations, maintaining system stability while ensuring data representativeness.
3Measurement precision
If a dedicated hardware interface is implemented for data transfer, then data acquisition disturbances are minimized, but system complexity and cost increase
Solution Approach 1:
The patent applies universality by using general-purpose I/O ports for data transfer instead of implementing a dedicated hardware interface. The same I/O infrastructure used for normal system operations is repurposed for training data transfer, eliminating the need for additional specialized hardware while still achieving unbiased data collection through careful timing and protocol management.
Data Source
AI summary
A data processing system and a method are provided for acquiring data for training a machine learning (ML) model for use in self-monitoring the data processing system. The data processing system operates in a data acquisition mode to acquire training data for training the ML model. The training data is acquired from an anomaly detector of the data processing system while operating in the data acquisition mode. At least a portion of the training data is determined to be biased, and a portion of the training data is unbiased. The unbiased portion of the training data is transferred to a training environment external to the data processing system. The unbiased portion of the training data is acquired for training the ML model to function with the anomaly detector during a normal operating mode to determine when an anomaly is present in the data processing system.