Temperature Parameter Calculation for Knowledge Distillation

Resolve Bottlenecks,
Find Innovative SolutionsGenerate Solutions

Solution Overview

Problem

Existing techniques for deep learning using knowledge distillation require manual tuning of the temperature parameter, which is time-consuming and may miss the optimal temperature due to discrete value settings, especially as the structure and size of models, problem types, and hyperparameters vary.

Innovation Solution

A learning device and method that automatically calculates the temperature parameter using estimation information from both the student and teacher models, optimizing the parameter based on loss calculations to enhance the accuracy of the student model.

Engineering Contradictions & Design Principles

VSEngineering Contradiction Analysis

1Measurement precision

If manual grid search is used to find optimal temperature, then temperature parameter can be tuned, but learning time increases significantly

Engineering Contradiction:

Improvetemperature parameter accuracyVSAvoidlearning time

Core Design Contradiction:

Measurement precisionVSLoss of time

Solution Approach 1:

The system performs self-service by automatically calculating the optimal temperature parameter through the temperature calculation unit, which computes temperature based on estimation information from both student and teacher models without requiring manual grid search intervention

Inventive Principle:

Principle #25Self-service

Solution Approach 2:

The patent replaces the mechanical manual grid search process with an automated computational system that calculates optimal temperature parameters through mathematical operations on estimation information, eliminating the need for iterative manual tuning

Inventive Principle:

Principle #28Mechanics substitution (Replace mechanical system)

2Ease of operation

If discrete temperature values are used in grid search, then parameter search is simplified, but optimal temperature may be omitted

Engineering Contradiction:

Improveparameter search simplicityVSAvoidtemperature parameter accuracy

Core Design Contradiction:

Ease of operationVSMeasurement precision

Solution Approach 1:

The patent changes the parameter approach by calculating continuous optimal temperature values based on estimation information from models, rather than testing discrete predetermined temperature values, thereby achieving both ease of operation and precision

Inventive Principle:

Principle #35Parameter changes

3Measurement precision

If knowledge distillation is applied, then student model accuracy can be improved, but requires additional temperature parameter tuning

Engineering Contradiction:

Improvestudent model accuracyVSAvoidparameter tuning complexity

Core Design Contradiction:

Measurement precisionVSDevice complexity

Solution Approach 1:

The patent implements feedback by using estimation information from both student and teacher models to calculate the optimal temperature parameter, creating a closed-loop system where the temperature is determined based on actual model performance rather than manual guesswork

Inventive Principle:

Principle #23Feedback

Data Source

PatentUS12223694B2Learning device, learning method and program

Publication Date: 2025.02.11 NEC CORP

AI summary

The learning device includes: a first estimation unit configured to perform estimation using a temperature parameter, based on a student model; a second estimation unit configured to perform estimation using the temperature parameter, based on a teacher model; and a temperature calculation unit configured to calculate the temperature parameter, based on estimation information generated by the first estimation unit and the second estimation unit.