Random Feature Attention Approximation Error Reduction
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
The Random Feature Attention (RFA) mechanism, while improving computational efficiency, suffers from significant approximation errors due to its biased estimation, which affects the accuracy of output results in neural network models.
Innovation Solution
A method for feature extraction that determines target data to extract features, generates query, key, and value vectors, and computes key-value pair information based on sampled data from probability distributions specific to each query vector, enabling random mapping and improved feature information extraction.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If the Random Feature Attention (RFA) mechanism is used to linearize the similarity computation function, then computational efficiency is improved, but approximation error increases and output accuracy deteriorates
Solution Approach 1:
The patent segments the uniform sampling process into query-specific sampling operations. Instead of using a single set of random features for all queries (RFA approach), the method generates separate sampling distributions and samples for each query vector, allowing tailored feature extraction that reduces approximation error while maintaining computational efficiency.
Solution Approach 2:
The patent applies local quality by making the sampling distribution specific to each query vector rather than using a global uniform distribution. Each query vector has its own probability distribution derived from its characteristics, enabling localized optimization of feature extraction accuracy for different regions of the input space.
2Measurement precision
If different processing methods are applied to each query vector, then feature extraction accuracy is improved, but computational complexity increases
Solution Approach 1:
The patent performs preliminary action by pre-computing the sampling distribution parameters (mean and variance) based on each query vector's characteristics before the actual sampling process. This preprocessing step enables efficient sampling without requiring complex real-time computations during the main attention calculation phase.
Solution Approach 2:
The patent changes parameters by transforming the fixed uniform distribution parameters into query-dependent parameters (mean and variance calculated from query vector properties). This parameter transformation allows the system to adapt the sampling characteristics to each query while maintaining a consistent computational framework.
Data Source
AI summary
The disclosure relates to a method, an apparatus, a storage medium, an electronic device, a computer program product, and a computer program for feature extraction, device. The method includes: determining target data for a feature to be extracted, and determining, based on the target data, a plurality of query vectors, a plurality of key vectors, and a plurality of value vectors; determining a plurality of key-value pair information corresponding to each of the query vectors; and performing, for each of the query vectors, a random mapping based on the query vector and the plurality of data samples, to obtain a plurality of random query vectors, and determining feature information corresponding to the query vector based on the plurality of random query vectors and the plurality of key-value pair information.