Transformer Layer Pruning via Internal Classification Feedback
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing methods for compressing transformer-based pre-trained language models (PLMs) face challenges in determining the optimal number of layers to remove for efficient model compression, particularly in limited computing resources, leading to performance degradation and high computational costs.
Innovation Solution
The proposed method involves adding an internal classification layer to each encoder layer, measuring performance, and pruning upper encoder layers based on a preset performance drop tolerance, with additional pruning and knowledge distillation techniques to improve compression efficiency.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Volume of moving object
If layer pruning is applied to compress the model, then model size is reduced, but determining the optimal number of layers to remove requires repeated training and verification which increases computational cost
Solution Approach 1:
The patent applies preliminary action by adding internal classification layers to encoder layers before the actual pruning process. This allows the system to measure information contribution in advance and identify which layers to remove without requiring repeated full model training and verification, thereby reducing computational cost while achieving model compression
2Volume of moving object
If upper encoder layers are removed to compress the model, then model size is reduced, but performance may degrade if important task-specific knowledge is lost
Solution Approach 1:
The patent implements feedback by using internal classification layers to measure the information contribution of each encoder layer. This measurement provides feedback on which layers contain important task-specific knowledge, allowing the system to make informed pruning decisions that maintain model performance while reducing size. The feedback mechanism ensures that layers critical for task performance are preserved
Solution Approach 2:
The patent applies parameter changes by modifying the model architecture dynamically - adding internal classification layers to measure information flow, then adjusting which encoder layers are retained or removed based on measured performance contributions. This parameter adjustment allows optimization of both model size and performance
3Measurement precision
If repeated training is performed to determine layers to prune, then optimal pruning points are identified, but training time and computational resources increase significantly
Solution Approach 1:
The patent performs preliminary measurement of layer information contribution by adding internal classification layers before the pruning decision. This preliminary action provides accurate measurement data about which layers are important, eliminating the need for repeated full model training to determine optimal pruning points, thereby reducing training time while maintaining pruning accuracy
Data Source
AI summary
Disclosed is a model compression method and system for understanding natural language through layer pruning. A model compression method may include adding an internal classification layer to each encoder layer of an input model; measuring performance for an output of the internal classification layer; determining an encoder layer in which the measured performance is lower than performance of the input model by a preset performance drop tolerance range or more; and pruning upper encoder layers of a final layer which is an upper layer of the determined encoder layer.