Trained Function Accuracy Monitoring via Data Distribution Distance
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current software management systems lack efficient methods for detecting accuracy decreases in trained functions due to distribution drifts in incoming data, leading to potential failures in analyzing, monitoring, and controlling devices in industrial environments.
Innovation Solution
A computer-implemented method and system that receives input data, applies a trained function to generate output data, determines the distance of input data to a reference dataset, calculates accuracy using a regression model, and provides an alarm if the accuracy falls below a threshold, allowing for timely intervention and updating of the trained function.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If a trained function is used for analyzing, monitoring, operating and/or controlling devices, then productivity and automation are improved, but reliability deteriorates when distribution drift occurs in incoming data
Solution Approach 1:
The system performs preliminary actions by continuously monitoring the distribution of incoming data and comparing it against the training data distribution before the trained function is applied. This early detection of distribution drift allows the system to identify when the trained function's accuracy is degrading due to statistical changes in the data, enabling proactive intervention to maintain reliability while preserving automation productivity.
2Reliability
If manual monitoring of function accuracy is performed, then reliability is maintained, but device complexity and operational burden increase
Solution Approach 1:
The system implements self-service by automatically monitoring its own accuracy through a monitoring unit that continuously evaluates the distribution of incoming data and compares it against the training data distribution. This self-monitoring mechanism eliminates the need for complex manual monitoring systems while maintaining reliability, as the system detects and reports accuracy degradation autonomously based on statistical changes in the data distribution.
3Ease of operation
If the trained function is applied without accuracy monitoring, then ease of operation is improved, but loss of information about accuracy degradation occurs
Solution Approach 1:
The system implements feedback by continuously monitoring the distribution of incoming data and providing feedback signals about accuracy degradation to users or operational systems. The monitoring unit compares the distribution of incoming data with the training data distribution and generates feedback when statistical changes indicate declining accuracy. This feedback mechanism maintains ease of operation by automatically detecting accuracy issues without requiring manual intervention, while preventing loss of accuracy information through continuous automated reporting.
Data Source
AI summary
For improved provision of an alarm relating to an accuracy of a trained function, such as detecting an accuracy decrease of a trained function under a distribution drift of incoming data, the following computer-implemented method is suggested: receiving input data messages (140) relating to at least one variable of at least one device (142); applying a trained function (120) to the input data messages (140) to generate output data (152), the output data (152) being suitable for analyzing, monitoring, operating and/or controlling the respective device (142); determining at least one respective distance of the respective variable of a respective received input data message (140) to a reference data set, determining an accuracy value of the trained function (120) using the respective distance and a regression model (130); and if the determined accuracy value is smaller than an accuracy threshold: providing an alarm (150) relating to the determined accuracy value to a user, to the respective device (142) and/or an IT system connected to the respective device (142).