Neural Network Accelerator Candidate Selection Preprocessor
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing neural network accelerators face inefficiencies in performing large-scale operations, particularly when using the attention mechanism, as they require a significant number of calculations that increase with the amount of stored information, leading to increased computational load.
Innovation Solution
An accelerator is designed with a preprocessor that selects candidate rows from a key matrix by allocating maximum and minimum pointers, calculating partial similarity values, and estimating scores, allowing for reduced calculations by omitting rows with expected small scores and treating weights below a certain threshold as zero.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If the attention mechanism is used to select information having high similarity with input data, then the accuracy of information retrieval is improved, but the computational load increases significantly
Solution Approach 1:
The patent applies preliminary action by pre-calculating and storing maximum pointers that indicate the maximum elements of each column in the key matrix before the actual attention computation. This allows the system to quickly identify candidate rows without performing full similarity calculations on all stored information, thereby reducing computational load while maintaining retrieval accuracy
Solution Approach 2:
The patent implements partial action by calculating similarity scores only for candidate rows identified through maximum pointers, rather than computing similarities for all rows in the key matrix. This selective computation approach significantly reduces the number of operations while still capturing the most relevant information for the attention mechanism
2Quantity of substance
If the amount of stored information increases to improve comprehensive analysis, then the coverage of information retrieval is improved, but the number of calculations increases
Solution Approach 1:
The system performs preliminary organization of stored information by maintaining maximum pointers that track the maximum elements in each column of the key matrix. This pre-processed structure enables efficient retrieval even as the amount of stored information grows, allowing the system to scale without linearly increasing computational complexity
Solution Approach 2:
The patent extracts only the necessary candidate rows from the large stored information by using maximum pointers to identify rows with potential high similarity. This extraction approach allows the system to work with a small subset of relevant data rather than processing the entire stored information set, maintaining calculation efficiency despite increased information volume
Data Source
AI summary
An accelerator includes a key matrix register configured to store a key matrix, a query vector register configured to store a query vector; and a preprocessor configured to calculate similarities between the query vector and the key matrix.