Condenser Model for Heterogeneous Knowledge Distillation

Resolve Bottlenecks,
Find Innovative SolutionsGenerate Solutions

Solution Overview

Problem

Existing knowledge distillation methods are limited in their ability to transfer knowledge between heterogeneous neural networks and multi-domain data, particularly for models performing tasks like object recognition and speech recognition, as they cannot effectively handle different architectures and types of machine learning models.

Innovation Solution

A system and method for training a condenser model to learn a parameter mapping function between a pre-trained teacher model and a student model, using a third training dataset that includes both models' parameters and data, allowing for the generation of new student models that can perform tasks like object recognition and speech recognition, even with unlabelled or semi-supervised data.

Engineering Contradictions & Design Principles

VSEngineering Contradiction Analysis

1Adaptability or versatility

If traditional knowledge distillation methods are used, then knowledge can be transferred between homogeneous models, but they cannot handle heterogeneous neural networks with different architectures and multi-domain data

Engineering Contradiction:

Improveability to handle heterogeneous models and multi-domain dataVSAvoidcomplexity of handling different architectures

Core Design Contradiction:

Adaptability or versatilityVSDevice complexity

Solution Approach 1:

The condenser model is designed as a universal knowledge distillation framework that can handle heterogeneous neural networks with different architectures (CNN, RNN, Transformer, etc.) and multi-domain data types simultaneously. It achieves this by learning architecture-agnostic parameter mappings that work across diverse model types without requiring architecture-specific customization

Inventive Principle:

Principle #6Universality (Multi-functionality)

Solution Approach 2:

The invention transforms the knowledge distillation problem from architecture-specific feature matching to architecture-agnostic parameter space mapping. By changing the representation space from domain-specific features to universal model parameters, the system can distill knowledge between heterogeneous models with different architectures and data types

Inventive Principle:

Principle #35Parameter changes

2Reliability

If existing distillation methods are applied to heterogeneous models, then model performance can be maintained, but the methods cannot effectively transfer knowledge across different model types and architectures

Engineering Contradiction:

Improveeffectiveness of knowledge transferVSAvoidcompatibility with different model architectures

Core Design Contradiction:

ReliabilityVSAdaptability or versatility

Solution Approach 1:

The condenser model serves as an intermediary that bridges heterogeneous teacher and student models. It learns to map parameters between different architectures through a universal parameter space, enabling reliable knowledge transfer between models of different types (e.g., CNN to Transformer) without direct architecture-specific mappings

Inventive Principle:

Principle #24Intermediary (Mediator)

3Productivity

If conventional knowledge distillation is used, then training can be performed with labelled data, but it cannot effectively utilize unlabelled or semi-supervised data

Engineering Contradiction:

Improvetraining efficiency with various data typesVSAvoidunderutilization of unlabelled data

Core Design Contradiction:

ProductivityVSLoss of information

Solution Approach 1:

The system performs partial knowledge distillation by utilizing only the portion of data that is labelled, while still extracting useful parameter mappings from unlabelled data through the condenser model. This allows the system to make progress with incomplete supervision, effectively using whatever labelled data is available without discarding unlabelled data

Inventive Principle:

Principle #16Partial or excessive action

Data Source

PatentUS20230351203A1Method for knowledge distillation and model genertation

Publication Date: 2023.11.02 SAMSUNG ELECTRONICS CO LTD

AI summary

The present techniques generally relate to a system and method for knowledge distillation between machine learning, ML, models. In particular, the present application relates to a computer-implemented method for training a condenser model to learn how to transfer knowledge between a teacher model and a student model, and using this trained condenser model to more quickly generate new student models.