Attention Map Distillation for Cross-Architecture Model Training
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
On-device models with limited memory and computing resources struggle to perform well in tasks like audio classification due to the inability to effectively transfer knowledge from large-scale models, especially when their architectures differ.
Innovation Solution
A method involving attention maps and weights is used to train student models by generating attention maps from teacher models, calculating loss functions, and updating parameters to align dimensions, allowing knowledge distillation across various architectures, and enabling efficient deployment and updates on user terminals.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Adaptability or versatility
If knowledge distillation is attempted between large-scale model and on-device model with different architectures, then architecture diversity is enabled, but important temporal information in audio classification is not transmitted well
Solution Approach 1:
The patent introduces an intermediary mechanism that processes attention maps through a series of operations including global average pooling, concatenation across time steps, and dimensionality transformation. This intermediary layer bridges the gap between teacher and student models with different architectures, enabling effective knowledge transfer while preserving temporal information through the structured processing of attention mechanisms.
Solution Approach 2:
The patent transforms attention maps by changing their dimensional parameters through operations like global average pooling (reducing spatial dimensions), concatenation (adding temporal dimensions), and linear transformations. These parameter changes enable the attention maps to be compatible between models of different architectures while maintaining the essential temporal information needed for audio classification.
2Use of energy by moving object
If on-device model uses limited memory and computing resources, then device portability is maintained, but performance is significantly poorer than large-scale model
Solution Approach 1:
The patent creates a compressed copy of the teacher model's knowledge through attention map distillation. Instead of transferring the entire large-scale model, it extracts and transmits only the essential attention patterns and temporal information through the processed attention maps, enabling the on-device model to achieve performance comparable to large-scale models while using limited computing resources.
Solution Approach 2:
The patent extracts the most important information from the teacher model by processing attention maps to identify and transmit only the critical temporal patterns and attention weights. This extraction process filters out redundant information, leaving only the essential knowledge needed for audio classification tasks, thereby enabling effective performance on resource-constrained devices.
3Measurement precision
If attention maps are processed through multiple transformation steps, then knowledge transfer accuracy is improved, but computational complexity increases
Solution Approach 1:
The patent segments the attention map processing into distinct functional steps: global average pooling to capture spatial patterns, concatenation to preserve temporal sequences, and dimensionality transformation to match target model requirements. This segmentation allows each step to be optimized independently and facilitates parallel computation, reducing overall computational complexity while maintaining high knowledge transfer accuracy.
Data Source
AI summary
Provided is a method for training a model, including generating a plurality of attention maps by inputting training data into a previously trained teacher model, generating a set of attention weights of the teacher model based on the plurality of attention maps, generating a set of attention weights of a student model by inputting the training data into the student model, calculating a value of a first loss function based on the set of attention weights of the teacher model and the set of attention weights of the student model, calculating a value of a second loss function according to an inference of the student model with respect to the training data, and training the student model based on the value of the first loss function and the value of the second loss function.