Selective Knowledge Distillation for Edge Device Accuracy
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Edge devices, such as cameras and drones, face challenges in supporting resource-hungry deep learning models due to limited resources, leading to inaccurate student model training when using knowledge distillation from teacher models with false positives and low-quality predictions.
Innovation Solution
Implementing selective knowledge distillation systems that determine when to apply distillation loss based on accuracy thresholds, allowing the student model to skip learning from inaccurate teacher outputs, thereby improving prediction accuracy and reducing resource demand.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If knowledge distillation is applied to all training examples from the teacher model, then the student model can learn from the teacher's knowledge, but the student model learns from false positives and inaccurate predictions which reduces accuracy
Solution Approach 1:
The patent converts the harmful effect of teacher model false positives into a beneficial filtering mechanism. By using the teacher model's confidence scores and predictions as a filter, the system selectively applies distillation loss only to high-confidence, accurate predictions, thereby eliminating the harmful influence of false positives while preserving the beneficial knowledge transfer from accurate teacher predictions.
Solution Approach 2:
The patent changes the parameter of distillation loss application from a constant (applied to all training examples) to a dynamic parameter that varies based on teacher model prediction quality. Specifically, the system adjusts whether to apply distillation loss on a per-example basis using an indicator variable that depends on teacher confidence scores and prediction accuracy, thereby optimizing the knowledge distillation process.
2Productivity
If the student model is trained on all teacher model outputs including low-quality predictions, then more training data is utilized, but the prediction accuracy decreases due to learning from incorrect examples
Solution Approach 1:
The patent applies local quality by making the distillation loss application quality-dependent. Instead of uniformly applying distillation to all training examples, the system evaluates the local quality of each teacher prediction using confidence scores and accuracy metrics, then selectively applies distillation loss only to high-quality local regions (individual predictions or image-level aggregates), thereby ensuring that only accurate knowledge is transferred to the student model.
3Reliability
If selective knowledge distillation is implemented with accuracy thresholds, then false positives are filtered out improving accuracy, but some valid training examples are excluded reducing training data size
Solution Approach 1:
The patent segments the training process into two distinct phases: first, the teacher model is trained on the complete training dataset to learn from all available data; second, the student model is trained using selective knowledge distillation that segments the teacher's predictions into high-confidence and low-confidence categories, applying distillation loss only to the high-confidence segment. This segmentation allows the student to learn from filtered, high-quality examples while the teacher benefits from comprehensive training data.
Data Source
AI summary
Methods, systems, and apparatus, including computer programs encoded on computer storage media, for updating a knowledge distillation training system. One of the methods includes: providing, to a teacher model in a knowledge distillation training system, first data representing an image to cause the teacher model to generate teacher output data that indicates whether the image depicts an object of interest; providing, to a student model in the knowledge distillation training system, second data representing the image to cause the student model to generate student output data that indicates whether the image depicts an object of interest; determining whether an accuracy of the teacher output data satisfies an accuracy threshold; and in response to determining that the accuracy of the teacher output data does not satisfy the accuracy threshold: determining to skip updating the student model; and updating the student model using the student output data and ground truth data.