White-Box Temperature Scaling for DNN Data Drift
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Deep neural networks (DNNs) used in automated driving and other perception tasks face calibration challenges, leading to mis-calibration errors due to factors like data drift and class imbalance, affecting the reliability of uncertainty estimation and confidence scores.
Innovation Solution
The implementation of white-box temperature scaling (WB-TS) for object detection DNNs, which involves scaling logit vectors with a temperature value T, optimized using the validation dataset, to calibrate pre-non-maximum suppression (pre-NMS) detections and address data drift by measuring intersection-over-union (IoU) conditioned expected calibration error (ECE), enabling ongoing training and calibration for evolving datasets.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If DNNs are used for perception tasks in automated driving, then object detection capability is improved, but calibration accuracy deteriorates due to data drift and class imbalance
Solution Approach 1:
The patent applies preliminary action by performing temperature scaling calibration on the validation dataset before deploying the DNN to new environments. This pre-calibration step establishes a baseline temperature value that compensates for expected data drift, allowing the model to maintain better calibration accuracy when encountering evolving datasets in automated driving scenarios
Solution Approach 2:
The patent changes the temperature parameter of the DNN output to improve calibration. By optimizing the temperature value on the validation dataset and applying it to pre-NMS detections, the system adjusts the confidence score distribution to better reflect true detection accuracy, thereby resolving the calibration deterioration caused by data drift and class imbalance
2Speed
If confidence scores are used to determine uncertainty, then decision-making speed is improved, but reliability of uncertainty estimation deteriorates due to mis-calibration
Solution Approach 1:
The patent modifies the temperature parameter that scales confidence scores from object detection DNNs. By optimizing this temperature value on the validation dataset and applying it to pre-NMS detections, the system adjusts the confidence score distribution to better reflect true detection accuracy, thereby resolving the calibration deterioration caused by data drift and class imbalance
3Productivity
If DNNs are trained on static datasets, then training efficiency is improved, but adaptability to new environments deteriorates
Solution Approach 1:
The patent performs temperature scaling calibration on the validation dataset before deployment, establishing a baseline that accounts for potential environmental variations. This preliminary calibration step enables the model to adapt to new environments more effectively without requiring retraining, thus maintaining training efficiency while improving adaptability
Solution Approach 2:
The patent optimizes the temperature parameter on the validation dataset to better reflect the distribution of target environments. This parameter adjustment allows the DNN to adapt to new environments by correcting calibration drift, enabling ongoing learning and adaptation without retraining the entire model
Data Source
AI summary
A system and method to identify a data drift in a trained object detection deep neural network (DNN) includes receiving a dataset based on real world use, wherein the dataset includes scores associated with each class in an image, including a background (BG) class, measuring an intersection-over-union (IoU) conditioned expected calibration error (ECE) IoU-ECE by calculating an ECE under a white-box setting with detections from the dataset prior to non-maximum suppression (pre-NMS detections) that are conditioned on a specific IoU threshold, upon a determination of the IoU-ECE being greater than a preset first threshold, performing a white-box temperature scaling (WB-TS) calibration on the pre-NMS detections of the dataset to extract a temperature T, and identifying that the data drift has occurred upon a determination that temperature T exceeds a preset second threshold.