Augmented Recurrent Neural Network External Memory

Resolve Bottlenecks,
Find Innovative SolutionsGenerate Solutions

Solution Overview

Problem

Recurrent neural networks face limitations in implementing continuously differentiable analogues of traditional data structures like Stacks, Queues, and Double-Ended Queues, which restrict their generalization performance on sequence transduction tasks and memory management.

Innovation Solution

Augmenting recurrent neural networks with an external memory that allows for efficient constant-time push and pop operations, enabling the implementation of continuous stack, queue, and double-ended queue memories, and facilitating superior generalization performance by integrating these data structures into the network architecture.

Engineering Contradictions & Design Principles

VSEngineering Contradiction Analysis

1Adaptability or versatility

If recurrent neural networks use traditional internal state memory, then the network can process sequences, but the memory capacity is bounded and generalization performance is restricted

Engineering Contradiction:

Improvegeneralization performanceVSAvoidmemory capacity

Core Design Contradiction:

Adaptability or versatilityVSQuantity of substance

Solution Approach 1:

The patent introduces an external memory dimension separate from the traditional internal state. The network architecture includes an external memory matrix that stores information externally, allowing the system to access arbitrarily large memory capacities without increasing the dimensional constraints of the hidden state. This dimensional separation resolves the contradiction by providing unbounded memory capacity while maintaining the network's ability to generalize.

Inventive Principle:

Principle #17Another dimension (Dimensionality change)

Solution Approach 2:

The patent introduces memory control gates (read gate, write gate, forget gate) as intermediary components that mediate between the external memory and the neural network. These gates control information flow to and from the external memory, enabling the network to efficiently manage large memory capacities while maintaining selective access patterns that improve generalization performance.

Inventive Principle:

Principle #24Intermediary (Mediator)

2Productivity

If recurrent neural networks implement data structures like Stacks and Queues, then operational efficiency improves, but the implementation becomes discrete and non-differentiable

Engineering Contradiction:

Improveoperational efficiencyVSAvoiddifferentiability

Core Design Contradiction:

ProductivityVSEase of manufacture

Solution Approach 1:

The patent transforms discrete data structure operations into continuous parameter-based operations. Instead of discrete push/pop operations, the system uses continuous memory control gates with real-valued parameters that control information flow. The read, write, and forget gates use sigmoid and tanh functions to continuously regulate memory access, making the operations differentiable while maintaining efficiency comparable to traditional data structures.

Inventive Principle:

Principle #35Parameter changes

Solution Approach 2:

The patent replaces the mechanical discrete operations of traditional data structures with neural network-based continuous operations. The stack and queue operations are implemented through differentiable memory control mechanisms that use gradient-based optimization, substituting the non-differentiable mechanical operations with learnable continuous transformations.

Inventive Principle:

Principle #28Mechanics substitution (Replace mechanical system)

3Quantity of substance

If external memory is added to augment the network, then memory capacity becomes unbounded, but the device complexity increases

Engineering Contradiction:

Improvememory capacityVSAvoidnetwork architecture complexity

Core Design Contradiction:

Quantity of substanceVSDevice complexity

Solution Approach 1:

The external memory mechanism serves multiple functions simultaneously: it provides unbounded storage capacity, enables efficient read/write operations through control gates, and improves generalization performance. The same external memory structure supports different data structure implementations (stacks, queues, double-ended queues) without requiring separate mechanisms, reducing overall system complexity despite the added memory capacity.

Inventive Principle:

Principle #6Universality (Multi-functionality)

Data Source

PatentUS12099928B2Augmented recurrent neural network with external memory

Publication Date: 2024.09.24 GDM HOLDING LLC

AI summary

Methods, systems, and apparatus, including computer programs encoded on computer storage media, for augmenting neural networks with an external memory. One of the methods includes providing an output derived from the neural network output for the time step as a system output for the time step; maintaining a current state of the external memory; determining, from the neural network output for the time step, memory state parameters for the time step; updating the current state of the external memory using the memory state parameters for the time step; reading data from the external memory in accordance with the updated state of the external memory; and combining the data read from the external memory with a system input for the next time step to generate the neural network input for the next time step.