Distilled Machine Learning Models for Lower Compute and Memory
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing cumbersome machine learning models are computationally intensive and require significant memory resources, making deployment challenging, while their outputs are often not significantly more accurate than those of simpler models.
Innovation Solution
Training a distilled machine learning model using a cumbersome model's parameters, employing a higher temperature during training to generate soft outputs, and adjusting parameters to match target soft outputs, resulting in a lighter model that maintains accuracy.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If a cumbersome machine learning model is used, then classification accuracy is improved, but computational complexity and memory requirements increase
Solution Approach 1:
The patent creates a distilled model that copies the essential classification capabilities of the cumbersome teacher model. The distilled model learns to reproduce the teacher model's soft outputs (probability distributions) across multiple temperature values, capturing both classification accuracy and calibration properties. This copying approach enables deployment of a smaller, faster model that maintains the accuracy benefits of the larger model.
Solution Approach 2:
The patent transforms the cumbersome model's parameters into a distilled form by training a smaller model to match the teacher model's outputs across multiple temperatures. The distillation process involves learning parameters that reproduce soft outputs at different temperature values, effectively compressing the knowledge from the cumbersome model into a more efficient representation with fewer parameters.
2Measurement precision
If a cumbersome machine learning model is used, then classification accuracy is improved, but memory requirements increase
Solution Approach 1:
The distilled model copies the classification knowledge from the cumbersome teacher model into a compact form. By learning to reproduce the teacher's soft outputs across multiple temperatures, the distilled model captures the essential decision boundaries and probability calibrations without requiring the original model's large parameter storage, thereby reducing memory requirements while maintaining accuracy.
3Productivity
If a distilled model is trained without multiple temperatures, then training speed is improved, but output calibration quality deteriorates
Solution Approach 1:
The patent segments the temperature parameter space into multiple discrete temperature values for training. Instead of using a single temperature, the distillation process trains the student model to match soft outputs at multiple temperature points (e.g., T=0.5, T=1.0, T=2.0). This segmentation allows the model to learn both sharp classifications (at low temperatures) and well-calibrated probabilities (at high temperatures), improving overall output calibration quality.
Solution Approach 2:
The patent applies partial action by selectively matching outputs at specific temperature values rather than all possible temperatures. The training process uses a finite set of temperature points, which is sufficient to capture the essential calibration properties without requiring exhaustive temperature coverage. This approach balances training complexity with calibration quality.
Data Source
AI summary
Methods, systems, and apparatus, including computer programs encoded on computer storage media, for training a distilled machine learning model. One of the methods includes training a cumbersome machine learning model, wherein the cumbersome machine learning model is configured to receive an input and generate a respective score for each of a plurality of classes; and training a distilled machine learning model on a plurality of training inputs, wherein the distilled machine learning model is also configured to receive inputs and generate scores for the plurality of classes, comprising: processing each training input using the cumbersome machine learning model to generate a cumbersome target soft output for the training input; and training the distilled machine learning model to, for each of the training inputs, generate a soft output that matches the cumbersome target soft output for the training input.