Classifier Data Drift Management via Activation Entropy
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Machine learning models, such as classifiers, experience performance degradation due to data drift over time and variations across endpoints, leading to mismatches between training data distributions and new input data.
Innovation Solution
The implementation of systems and methods that adaptively manage data drift by using activity regularization for neural networks, which involves identifying low-confidence results through internal activity and activation entropy calculations, and triggering model updates or notifications to mitigate data drift.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Manufacturing precision
If a machine learning model is trained using traditional methods without activity regularization, then the model may achieve high accuracy on training data, but the model suffers from overfitting and poor generalizability to new data distributions
Solution Approach 1:
The patent applies activity regularization by modifying the training parameters of the neural network, specifically adding entropy regularization to the loss function. This changes the optimization parameters to simultaneously minimize classification error and maximize activation entropy, leading to better generalizability while maintaining accuracy on training data.
Solution Approach 2:
The system implements self-monitoring by calculating activation entropy during inference and comparing it to thresholds. When entropy indicates low confidence (data drift detected), the system triggers feedback loops to request model updates from back-end servers, creating a closed-loop system that adapts to changing data distributions.
2Reliability
If the system continuously monitors and updates the machine learning model to adapt to data drift, then the model maintains high accuracy, but the computational overhead and system complexity increase
Solution Approach 1:
The client device performs self-monitoring by calculating activation entropy locally during inference without requiring constant server intervention. The system autonomously detects data drift conditions and only contacts the back-end server when updates are needed, reducing overall system complexity while maintaining accuracy.
Solution Approach 2:
Instead of continuously monitoring and updating the model, the system performs partial monitoring by only triggering model updates when activation entropy thresholds indicate actual data drift. This selective approach reduces computational overhead while maintaining model accuracy when needed.
3Adaptability or versatility
If the system requests frequent model updates from the back-end server, then the model adapts quickly to data drift, but the communication overhead and update frequency increase
Solution Approach 1:
The system preemptively detects data drift conditions by monitoring activation entropy before classification accuracy significantly degrades. By detecting drift early through entropy thresholds, the system can request model updates proactively, preventing performance degradation rather than reacting after accuracy drops.
Solution Approach 2:
The system implements periodic monitoring of activation entropy during inference operations and only triggers model update requests when entropy values cross predefined thresholds. This periodic check-and-act approach balances adaptability with reduced communication overhead compared to continuous update requests.
Data Source
AI summary
The disclosed computer-implemented method for adaptively managing data drift in a classifier may include (i) receiving, at a computing device, an input sample of digital information having an unknown reputation and (ii) performing a security action that may include (A) identifying the input sample as benign or malicious based on a result obtained by classifying the input sample using a machine learning model trained using activity regularization, (B) calculating an internal activity of the machine learning model occurring during the classifying, (C) calculating an activation entropy of the machine learning model occurring during the classifying, (D) comparing a combination of the internal activity and the activation entropy to a threshold, and (E) when the combination of the internal activity and the activation entropy meets or exceeds the threshold, identifying the result as a low-confidence result. Various other methods, systems, and computer-readable media are also disclosed.