In-Memory k-NN Attention for Constant-Time NLP Retrieval
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing k-Nearest Neighbors (k-NN) algorithms face inefficiencies in processing large datasets due to computational complexity proportional to dataset size, leading to increased response times for retrieving k minimum or maximum values.
Innovation Solution
Utilizing an SRAM memory array with an in-memory processor and neural network feature extractor for natural language processing, performing operations in parallel across columns of the memory array, including similarity, SoftMax, and multiplication operations, with a marker vector guiding operations and bit line processors, allowing constant-time computation irrespective of vector count.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If traditional k-NN algorithms process large datasets using sequential computation, then classification accuracy is maintained, but computational complexity increases proportionally with dataset size, leading to slower response times
Solution Approach 1:
The patent segments the large dataset into multiple partitions stored in different memory banks, allowing parallel processing of multiple data subsets simultaneously. This segmentation enables the system to handle large datasets without proportionally increasing computational complexity, as each bank processes a portion of the data independently and concurrently.
Solution Approach 2:
The patent transitions from sequential one-dimensional processing to parallel multi-dimensional processing by utilizing multiple memory banks that operate simultaneously. This dimensional change allows the system to process k extreme values across the entire dataset in constant time O(k) rather than linear time O(n), effectively adding a parallelism dimension to the computational architecture.
2Adaptability or versatility
If the dataset size increases to improve classification coverage, then model generalization improves, but the time required to find k nearest neighbors increases linearly with dataset size
Solution Approach 1:
By dividing the large dataset into multiple manageable partitions stored in separate memory banks, the system can maintain comprehensive classification coverage while processing each partition in parallel. This segmentation allows the system to scale to larger datasets without linearly increasing query response time, as the parallel architecture processes multiple partitions simultaneously.
Solution Approach 2:
The patent performs preliminary organization of data into partitioned memory structures during system initialization, enabling efficient query processing later. This preliminary action of structuring data into parallel-accessible formats allows the system to quickly retrieve k nearest neighbors from large datasets without performing linear scans during actual classification queries.
3Measurement precision
If sorting algorithms are used to find k extreme values, then accurate results are obtained, but the computational complexity becomes O(n) for full sorting or O(k) for partial sorting, which is inefficient for large n
Solution Approach 1:
The patent segments the dataset into multiple partitions that can be processed in parallel to find local extreme values. By dividing the sorting task across multiple memory banks operating simultaneously, the system maintains accuracy in identifying k extreme values while reducing overall computational complexity from O(n) or O(k) to constant time O(k) through parallel execution.
Solution Approach 2:
The patent creates multiple copies of data partitions in separate memory banks, allowing simultaneous processing of identical or similar operations on different data subsets. This copying approach enables parallel comparison and selection of extreme values across partitions, achieving accurate results without sequential processing overhead.
Data Source
AI summary
A system for natural language processing includes a memory array and a processor. The memory array is divided into a similarity section storing a plurality of feature vectors, a SoftMax section in which to determine probabilities of occurrence of the feature vectors, a value section storing a plurality of modified feature vectors, and a marker section. The processor activates the array to perform parallel operations in each column indicated by the marker section: a similarity operation in the similarity section between a vector question and feature vectors stored in indicated columns; a SoftMax operation in the SoftMax section to determine an associated SoftMax probability value for indicated feature vectors; a multiplication operation in the value section to multiply the associated SoftMax value by modified feature vectors stored in indicated columns; and a vector sum in the value section to accumulate an attention vector of output of the multiplication operation.