Unsupervised Drift Detection via Reconstruction Error Margins
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Machine learning model drift detection in unsupervised domains is challenging, particularly in edge environments like warehouses, where labeled data is hard to acquire and dangerous cornering events are difficult to generate, making it impractical to train predictive models for real-time event detection.
Innovation Solution
An autoencoder-based approach is employed for unsupervised event detection, using a training set of normative behavior to reconstruct sensor data, with reconstruction errors indicating potentially dangerous events, and a margin-based method for drift detection to assess model performance over time.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If supervised learning models are used for event detection, then detection accuracy is improved, but labeled data acquisition becomes infeasible due to volume and complexity
Solution Approach 1:
Instead of using supervised learning that requires labeled data, the patent inverts the approach by using unsupervised learning with autoencoders. The system trains on unlabeled normative data and detects events through reconstruction error, eliminating the need for expensive labeled data while maintaining detection capability
Solution Approach 2:
The patent introduces reconstruction error as an intermediary metric between raw sensor data and event detection. The autoencoder learns to reconstruct normative data, and the reconstruction error serves as a mediator that indicates anomalous events without requiring labeled training data
2Ease of manufacture
If unsupervised models are used for event detection, then labeled data requirement is eliminated, but drift detection capability deteriorates due to lack of labels
Solution Approach 1:
The patent implements feedback by continuously monitoring reconstruction error statistics and comparing them against baseline distributions. This feedback mechanism enables drift detection in unsupervised settings by identifying shifts in the distribution of reconstruction errors over time, even without labeled data
Solution Approach 2:
The system performs preliminary action by establishing baseline reconstruction error distributions during a training phase on normative data. These baselines are prepared in advance and used later for drift detection, enabling the system to identify performance degradation without requiring labels at deployment time
3Productivity
If more sensor data is collected for better event detection, then detection coverage is improved, but data processing complexity increases
Solution Approach 1:
The patent extracts only the essential features needed for event detection by training the autoencoder on raw sensor data and using reconstruction error as the detection signal. This extraction approach processes all available sensor data without requiring complex preprocessing or feature engineering pipelines
Data Source
AI summary
One example method includes receiving a stream of unlabeled data samples from a model, obtaining a first reconstruction error for the unlabeled data samples, obtaining a second reconstruction error for a set of normative data, defining a margin based on the first reconstruction error and the second reconstruction error, computing an initial proportion of samples from the set of normative data whose reconstruction errors fall within a range of reconstruction errors defined by the margin, computing a new proportion of unlabeled data samples that fall within the range of reconstruction errors defined by the margin, and signaling drift in the performance of the model when said new proportion differs from said initial proportion by more than a predefined tolerance threshold.