Dynamic Dropout Rate Adjustment for Neural Network Model Size Optimization
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current neural network training methods, such as dropout, do not effectively generate models of appropriate size based on the training mode, particularly with respect to the dropout rate, which affects the model's accuracy and output quality.
Innovation Solution
An information processing method that acquires the dropout rate during model training and generates models of specific sizes based on this rate, optimizing the hidden layer size to improve accuracy and naturalness of the model's output.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Ease of manufacture
If dropout is merely performed before a softmax layer, then the training process is simple, but the model size is not appropriately adjusted according to training mode
Solution Approach 1:
The patent applies parameter changes by dynamically adjusting the dropout rate based on the training mode. When text-to-text translation mode is detected, the dropout rate is set to a first value; when text-to-multiple translation mode is detected, it is set to a second value. This allows the model size to adapt appropriately to different training modes while maintaining training effectiveness.
2Measurement precision
If the model size is increased to improve accuracy, then the prediction performance improves, but the computational cost and training time increase
Solution Approach 1:
The patent implements dynamics by making the dropout rate adjustable based on training mode rather than fixed. The determination unit dynamically selects appropriate dropout rates (first value for text-to-text, second value for text-to-multiple modes), allowing the model to achieve optimal accuracy for each mode without unnecessarily increasing model size or training time.
3Device complexity
If the dropout rate is set to a fixed value, then the training configuration is simple, but the model performance varies across different translation modes
Solution Approach 1:
The patent applies segmentation by dividing the training modes into distinct categories (text-to-text translation mode and text-to-multiple translation mode) and assigning different dropout rate values to each category. The determination unit identifies the current training mode and selects the appropriate dropout rate, ensuring consistent and reliable performance for each specific translation mode while maintaining clear configuration management.
Data Source
AI summary
An information processing method according to the present application is an information processing method executed by a computer, the information processing method including: acquiring information indicating a dropout rate in training of a model; and generating the model having a size based on the dropout rate.