Multi-Subspace Attention Training for Higher Output Accuracy
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current attention networks in neural network models have a poor capability to capture feature information, leading to low accuracy in output results.
Innovation Solution
A neural network training method that involves mapping training samples to multiple subspaces using attention networks, calculating space difference and output similarity degrees, and adjusting model parameters until convergence is achieved to enhance the accuracy of the neural network model.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If a single attention network is used in the neural network model, then the model structure is simple, but the capability to capture feature information is poor and output accuracy is low
Solution Approach 1:
The patent divides the feature extraction process into multiple subspaces, each handled by a separate attention network. Instead of using one attention network to capture all features, the system segments the feature space into K distinct subspaces, with each attention network specializing in capturing features within its assigned subspace. This segmentation allows the model to capture diverse feature information more effectively while maintaining manageable complexity through modular architecture.
Solution Approach 2:
The patent introduces an additional dimension to the feature extraction process by creating multiple subspaces rather than operating in a single feature space. Each attention network operates in a different subspace, effectively adding a dimensional aspect to feature capture. This dimensional expansion enables the model to perceive and process feature information from multiple perspectives simultaneously, improving overall feature representation and output accuracy.
2Reliability
If multiple attention networks map to different subspaces, then the capability to capture distinct feature information is improved, but the calculation of space difference degree increases computational complexity
Solution Approach 1:
The patent implements a feedback mechanism where the space difference degree calculation results are used to guide the training process. The calculated space difference degree between subspaces provides feedback information that helps adjust the attention networks to maintain appropriate差异性 (difference) in their feature capture. This feedback loop ensures that each attention network continues to capture distinct feature information while the system learns to manage the computational complexity through iterative optimization.
Solution Approach 2:
The patent introduces the space difference degree as a new parameter to control and measure the差异性 between subspaces. By monitoring and adjusting this parameter during training, the system can maintain optimal feature capture capability across multiple attention networks. The parameter changes approach allows the model to dynamically adjust the relationship between subspaces, balancing feature distinctiveness with computational efficiency through learned parameter optimization.
Data Source
AI summary
A neural network training method, apparatus, a storage medium, and a computer device are provided. The method includes: obtaining a training sample set, each training sample including a standard label; inputting the each training sample into a neural network model including n attention networks, the n attention networks respectively mapping the each training sample to n subspaces, each of the n subspaces including a query vector sequence, a key vector sequence, and a value vector sequence; calculating a space difference degree between the n subspaces by using the neural network model; calculating an output similarity degree according to an output of the neural network model and the standard label corresponding to the each training sample; and adjusting a model parameter of the neural network model according to the space difference degree and the output similarity degree until a convergence condition is satisfied to obtain a target neural network model.