Neural Network External Memory LRUA Access
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Conventional neural networks face limitations in storing and accessing long-term information efficiently, which hampers their ability to perform effectively in machine learning tasks that require flexible memory storage and incremental learning.
Innovation Solution
An augmented neural network system is introduced, which incorporates an external memory and a memory access subsystem that uses weighted values to read from and write to this memory, allowing for flexible storage and retrieval of information, enabling the network to leverage longer-term experiences for improved prediction accuracy and meta-learning capabilities.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Loss of information
If conventional neural networks are used, then the network structure remains simple, but the ability to store and access long-term information is limited
Solution Approach 1:
The system divides the neural network into distinct functional components: a controller neural network for processing and an external memory subsystem for storage. This segmentation allows the network to offload long-term information storage to the external memory, reducing information loss without requiring the controller network to be overly complex.
Solution Approach 2:
An external memory subsystem acts as an intermediary between the controller neural network and the stored information. This mediator enables the network to access long-term information efficiently through weighted read operations while keeping the controller network structure relatively simple.
2Quantity of substance
If external memory is added to neural network, then information storage capacity increases, but system complexity increases
Solution Approach 1:
The external memory subsystem is designed with multi-functionality, serving both as a storage medium and an active processing component through its weighted read operations. This universal design allows the system to handle both storage and selective retrieval functions without requiring separate dedicated components, thereby managing complexity while expanding capacity.
Solution Approach 2:
The system transitions from a purely internal network state to an externalized memory dimension. By moving information storage outside the traditional network architecture and introducing weighted access mechanisms, the system expands its functional dimensions while maintaining manageable complexity through modular design.
3Speed
If conventional memory access is used, then access speed is limited, but the ability to rapidly assimilate new data is improved
Solution Approach 1:
The system performs preliminary organization of information in external memory with associated weight values that indicate relevance and importance. This pre-organization enables rapid assimilation of new data by allowing the controller network to quickly access pertinent information through weighted read operations without requiring complex real-time processing.
Solution Approach 2:
The memory access mechanism uses parameter changes in the form of time-varying weight values to dynamically adjust access patterns. These parameter changes enable the system to rapidly adapt to new data by modifying read weights based on temporal relationships and relevance, achieving fast data assimilation without overly complex hardware.
Data Source
Figure 1
Figure 2
Figure 3
AI summary
Methods, systems, and apparatus, including computer programs encoded on computer storage media, for augmenting neural networks with an external memory. One of the methods includes a controller neural network that includes a Least Recently Used Access (LRUA) subsystem configured to: maintain a respective usage weight for each of a plurality of locations in the external memory, and for each of the plurality of time steps: generate a respective reading weight for each location using a read key, read data from the locations in accordance with the reading weights, generate a respective writing weight for each of the locations from a respective reading weight from a preceding time step and the respective usage weight for the location, write a write vector to the locations in accordance with the writing weights, and update the respective usage weight from the respective reading weight and the respective writing weight.