Edge Device Data Drift Detection via Multi-Model Ratio Analysis
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Drift detection in AI or ML models deployed on edge devices without network connectivity is challenging due to the difficulty in obtaining feedback, leading to undetected performance degradation, as data drift can cause models to incorrectly classify data, such as image detection systems mistaking deer for kangaroos.
Innovation Solution
A system comprising a processor and memory that verifies the accuracy of two models, computes ratios based on their performance, and determines performance degradation using these ratios to detect data drift, enabling alert generation for edge devices without network connectivity.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If drift detection is implemented in edge devices without network connectivity, then model performance can be monitored locally, but obtaining feedback for detection becomes difficult
Solution Approach 1:
The patent uses confidence scores as an intermediary to detect data drift. Instead of requiring external feedback, the system uses the confidence scores generated by the model itself as a mediator to infer whether data drift is occurring. When confidence scores drop below a threshold, it indicates potential data drift, allowing the system to monitor performance reliably without network connectivity.
2Measurement precision
If multiple models are used for drift detection, then detection accuracy improves, but computational complexity increases
Solution Approach 1:
The patent segments the drift detection task into two distinct models: a primary model for normal inference and a secondary model specifically trained to detect data drift. This segmentation allows each model to specialize in its function, improving detection accuracy while keeping the overall system manageable by dividing the complex detection task into separate, dedicated components.
Solution Approach 2:
The secondary model acts as an intermediary detector that monitors the primary model's confidence scores. This mediator approach enables accurate drift detection without requiring the primary model to handle detection logic, thereby improving measurement precision while maintaining clear separation of concerns and manageable system complexity.
3Speed
If confidence score thresholding is used to detect data drift, then detection speed improves, but false positives may increase
Solution Approach 1:
The patent implements dynamic threshold adjustment where the confidence score threshold is not fixed but adapts based on the secondary model's detections. The system dynamically modifies the threshold to balance detection speed and reliability, reducing false positives while maintaining fast detection response. This dynamic approach allows the system to adjust its sensitivity based on actual data conditions rather than using a static threshold.
Data Source
AI summary
One or more systems, devices, computer program products and/or computer-implemented methods provided herein relate to data drift detection in an edge device. A system can comprise a memory configured to store computer executable components; and a processor configured to execute the computer executable components stored in the memory, wherein the computer executable components can comprise a verification component that can verify accuracy of a first model and accuracy of a second model to detect data drift associated with an edge device that is deployed without network connectivity; a computation component that can compute at least a first ratio based on the accuracy of the first model and the accuracy of the second model; and an analysis component that can use the at least the first ratio to determine whether performance degradation of at least one of the first model or the second model is a function of the data drift.