Feature-Transformation Layer Sampling for GPU-Friendly Training
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing acceleration methods for removing computational redundancy in neural networks are not hardware-friendly and struggle to achieve practical acceleration on platforms like general-purpose GPUs, despite their high theoretical acceleration rates.
Innovation Solution
A model training method that involves obtaining a target neural network with multiple feature transformation layers, sampling to obtain a submodel with fewer layers or smaller weight matrices, training this submodel, augmenting it to match the original model's size and complexity, and then training the augmented model to ensure performance.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If existing acceleration methods set parameters or features to 0 to remove computational redundancy, then theoretical acceleration rate is high, but hardware implementation difficulty increases and actual acceleration on GPU platforms is difficult to achieve
Solution Approach 1:
The model training process is segmented into distinct phases: initial full-model training, submodel creation by setting certain feature transformation layers to zero, submodel training, and gradual restoration of layers. This segmentation allows the system to benefit from both the theoretical acceleration of removing redundancy and the practical advantage of hardware-friendly structured operations that can be efficiently implemented on GPU platforms
Solution Approach 2:
The patent employs dynamic adjustment of model structure during training. The submodel starts with fewer feature transformation layers (set to zero), and these layers are gradually restored and trained in subsequent phases. This dynamic approach allows the model to benefit from reduced computational complexity during initial training while progressively recovering full model capacity, achieving both acceleration and hardware compatibility
2Productivity
If the quantity of feature transformation layers and weight matrix size are reduced to improve training efficiency, then training speed increases, but model performance may deteriorate
Solution Approach 1:
The patent performs preliminary training on a reduced submodel first, allowing the system to quickly learn from data with lower computational overhead. After the submodel is trained, the previously zeroed-out feature transformation layers are restored and trained in subsequent phases. This preliminary action on a simplified model enables faster initial learning while preserving the capability to achieve full model performance through progressive restoration
Solution Approach 2:
The training process is divided into periodic phases: Phase 1 trains the submodel with reduced layers for rapid learning, Phase 2 restores and trains the previously zeroed-out layers to recover full model capacity, and Phase 3 performs fine-tuning on the complete model. This periodic alternation between reduced and full model training ensures both training efficiency and model performance
Data Source
AI summary
A model training method includes performing sampling on the target model to obtain a submodel of the target model, and a quantity of feature transformation layers of the submodel is less than a quantity of feature transformation layers of the target model, and/or a size of a weight matrix of at least one of feature transformation layers of the submodel is less than a size of a weight matrix of a corresponding feature transformation layer of the target model; augmenting the submodel to obtain an augmented model; and training the augmented model to obtain a trained augmented model.