Neural Network Activation Logging for Prediction Confidence
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing methods for determining the confidence of deep learning neural network predictions are inaccurate and unreliable, as they rely on the network's output or input space, which can lead to improper application of the model, especially when generalizing to new data outside the training domain.
Innovation Solution
Treat the entire activation state of the neural network as a multivariate sample and create a measurement log during training to record the output distribution of all neurons, using this log to determine if a prediction falls within the training data domain by comparing the activation pattern of new data.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Extent of automation
If the network uses its own output to estimate confidence, then the confidence value is generated automatically, but the confidence becomes inaccurate and unreliable
Solution Approach 1:
The patent introduces an intermediary measurement log that records actual model performance during training. This log serves as a mediator between the model's predictions and the ground truth, allowing the confidence estimation to be based on actual performance data rather than the model's self-output, thereby resolving the circular reference problem while maintaining automation.
Solution Approach 2:
The patent implements feedback by using the measurement log of actual model performance to adjust and improve confidence estimation. The feedback loop compares predicted confidence with actual performance metrics, enabling continuous refinement of confidence values without requiring manual intervention.
2Reliability
If additional computational steps are added during training to improve confidence prediction, then confidence accuracy improves, but training time and computational cost increase
Solution Approach 1:
The patent applies preliminary action by collecting and storing performance measurements during the training process itself. The measurement log is built up gradually as training progresses, so that by the time inference is needed, the data is already ready for use, eliminating the need for additional post-training computational steps.
Solution Approach 2:
The training process itself serves the dual purpose of both learning model parameters and collecting confidence calibration data. The same training iterations that adjust model weights also accumulate the measurement log, making the system self-sufficient and eliminating separate computational overhead.
3Adaptability or versatility
If the model is trained on diverse data to improve generalization, then the model can handle more input types, but confidence estimation becomes less reliable for new data
Solution Approach 1:
The patent applies local quality by segmenting the confidence estimation into per-neuron or per-layer components. Instead of using a single global confidence value, the system analyzes activation patterns locally across different parts of the network, allowing it to detect when specific portions of the model are activated in ways that differ from training data patterns, even when the overall model handles diverse inputs.
Data Source
AI summary
The invention relates to a technique for improving confidence estimates associated with neural networks. The technique involves computing neuron activation statistics during training, evaluating neuron activations during inferencing and determining how the activations compare with the previously computed statistics (e.g. whether prediction activations are within the bounds of the training activation statistics). The comparison may be used to compute a confidence value for the neural network.