Sparse Memory Access for Neural Network Accuracy
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current neural network architectures face challenges in efficiently processing sequential data due to high computational complexity and memory overhead, especially when interacting with external memory resources, which limits their accuracy and training efficiency.
Innovation Solution
An augmented neural network system is introduced, featuring a controller neural network and an external memory with a sparse memory access subsystem that generates sparse reading and writing weights, allowing efficient interaction with the external memory by determining the most similar locations to a read key and using interpolation for writing weights, thereby reducing computational resources and maintaining high accuracy.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If neural networks interact with external memory resources to improve processing of sequential data, then accuracy is improved, but computational complexity and memory overhead increase
Solution Approach 1:
The patent segments the memory access operations by dividing the external memory into multiple banks and using parallel read/write heads. This allows simultaneous access to multiple memory locations, reducing the computational complexity of sequential data processing while maintaining accuracy through parallel information retrieval.
Solution Approach 2:
The patent implements preliminary action by pre-computing and storing attention weights and memory states in the external memory before they are needed for the final computation. This allows the neural network to quickly retrieve pre-processed information during inference, reducing real-time computational complexity while preserving accuracy.
2Measurement precision
If neural networks use external memory to store and retrieve sequential data, then processing accuracy is improved, but memory overhead increases
Solution Approach 1:
The patent applies local quality by making the memory access pattern selective rather than uniform. The attention mechanism computes different access patterns for different parts of the sequence, allowing the system to focus memory resources on the most relevant data portions. This reduces overall memory overhead while maintaining accuracy by accessing only necessary memory locations with high precision.
3Measurement precision
If neural networks increase external memory size to improve sequence processing, then accuracy is improved, but training efficiency decreases
Solution Approach 1:
The patent implements dynamics by making the memory access pattern adaptive rather than static. The attention weights are dynamically computed based on the input sequence characteristics, allowing the system to efficiently access large external memory during inference without requiring proportional increases in training complexity. This dynamic adaptation maintains training efficiency while enabling accurate processing of long sequences.
Data Source
Figure 1
Figure 2
Figure 3
AI summary
Methods, systems, and apparatus, including computer programs encoded on computer storage media, for augmenting neural networks with an external memory. One of the systems includes a sparse memory access subsystem that is configured to perform operations comprising generating a sparse set of reading weights that includes a respective reading weight for each of the plurality of locations in the external memory using the read key, reading data from the plurality of locations in the external memory in accordance with the sparse set of reading weights, generating a set of writing weights that includes a respective writing weight for each of the plurality of locations in the external memory, and writing the write vector to the plurality of locations in the external memory in accordance with the writing weights.