Layer-Wise AI Distillation for Low-Level Feature Retention
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing knowledge distillation methods for artificial intelligence models fail to accurately reflect low-level features during model compression, leading to performance degradation in applications requiring feature extraction.
Innovation Solution
A method involving an electronic device that compares and retrains a student network with a teacher network by aligning outputs of corresponding layers, using a discriminator to enhance low-level feature extraction, and reducing filter sizes to compress the model while maintaining performance.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Quantity of substance
If knowledge distillation is applied to compress the artificial intelligence model, then the capacity of the model is reduced, but the low-level feature extraction capability deteriorates
Solution Approach 1:
The patent segments the knowledge transfer process into multiple layers, comparing and transferring features from each layer of the teacher network to the corresponding layer of the student network. This layered segmentation allows the student network to learn both high-level and low-level features systematically, preventing degradation of low-level feature extraction capability while achieving model compression.
Solution Approach 2:
The patent applies different treatment to different layers of the network. Specifically, it preserves and emphasizes low-level feature layers while transferring high-level features, giving different importance to different parts of the model. This local quality approach ensures that low-level feature extraction capability is maintained while still achieving compression through knowledge distillation.
2Device complexity
If only last softmax information is used for knowledge distillation, then the training process is simple, but only high-level features are highlighted while low-level features are not highlighted
Solution Approach 1:
The patent extends the knowledge distillation process from a single dimension (last softmax output) to multiple dimensions by incorporating intermediate layer outputs from both the teacher and student networks. This dimensional expansion allows simultaneous supervision of multiple feature levels, enabling the student network to learn comprehensive features including low-level features that would otherwise be lost.
3Measurement precision
If additional processes are performed during training to compare outputs of layers, then the training complexity increases, but the low-level feature may not be accurately reflected if the number of layers or channels is different
Solution Approach 1:
The patent dynamically adjusts the number of comparison layers and feature channels based on the actual architecture of the teacher and student networks. By making the knowledge distillation process adaptive to different network configurations rather than fixed, it achieves accurate low-level feature transfer while managing training complexity through parameter optimization.
Data Source
AI summary
An electronic device is disclosed. The electronic device may comprise: a memory in which information on a first artificial intelligence model learned through first learning data and information on a second artificial intelligence model learned through the first learning data are stored; and a processor connected to the memory to control the electronic device, wherein the processor is configured to: input second learning data to each of the first artificial intelligence model and the second artificial intelligence model and relearns the second artificial intelligence model on the basis of an output of each of a plurality of first layers included in the first artificial intelligence model and an output of each of a plurality of second layers included in the second artificial intelligence model, each of the plurality of first layers includes a plurality of two-dimensional filters, and each of the plurality of second layers includes a plurality of filters obtained by reducing the size of each of the plurality of two-dimensional filters of a corresponding first layer.