Attention Neural Networks With Episodic and Compressed Memory

Resolve Bottlenecks,
Find Innovative SolutionsGenerate Solutions

Solution Overview

Problem

Conventional neural networks face challenges in capturing long-term dependencies within sequences due to high computational and storage costs associated with maintaining extensive memory of previous network inputs.

Innovation Solution

Implementing a memory-augmented attention neural network that uses both episodic and compressed memory to store and update hidden states, allowing the network to attend to distant context with minimal computational overhead by compressing older representations.

Engineering Contradictions & Design Principles

VSEngineering Contradiction Analysis

1Reliability

If conventional neural networks maintain extensive memory of previous network inputs to capture long-term dependencies, then the ability to capture long-term dependencies is improved, but computational cost and storage cost increase

Engineering Contradiction:

Improveability to capture long-term dependenciesVSAvoidcomputational cost

Core Design Contradiction:

ReliabilityVSLoss of energy

Solution Approach 1:

The patent segments the memory into two distinct types: episodic memory for recent inputs and compressed memory for older inputs. This segmentation allows the system to maintain detailed representations for important recent information while using compressed representations for older information, thereby reducing overall computational and storage costs while preserving the ability to capture long-term dependencies.

Inventive Principle:

Principle #1Segmentation

Solution Approach 2:

The patent changes the parameter of memory representation by introducing compressed memory that stores compressed versions of hidden states for older inputs. This parameter change allows the system to reduce storage and computational requirements while still maintaining the ability to access and utilize long-term dependencies when needed.

Inventive Principle:

Principle #35Parameter changes

2Reliability

Engineering Contradiction:

Improveability to capture long-term dependenciesVSAvoidstorage cost

Core Design Contradiction:

ReliabilityVSQuantity of substance

Solution Approach 1:

The patent segments the memory into two distinct types: episodic memory for recent inputs and compressed memory for older inputs. This segmentation allows the system to maintain detailed representations for important recent information while using compressed representations for older information, thereby reducing overall storage costs while preserving the ability to capture long-term dependencies.

Inventive Principle:

Principle #1Segmentation

Solution Approach 2:

The patent creates compressed copies of hidden states for older inputs and stores them in compressed memory. These compressed representations serve as simplified copies that reduce storage requirements while still capturing the essential information needed for long-term dependency capture.

Inventive Principle:

Principle #26Copying

3Loss of information

If the neural network maintains detailed hidden states for all previous inputs, then information preservation is improved, but computational overhead increases

Engineering Contradiction:

Improveinformation preservationVSAvoidcomputational overhead

Core Design Contradiction:

Loss of informationVSLoss of energy

Solution Approach 1:

The patent segments the memory into episodic and compressed portions, applying different levels of detail preservation. Recent inputs are stored in detail in episodic memory, while older inputs are stored in compressed form. This segmentation allows the system to preserve necessary information while reducing computational overhead by avoiding detailed storage of all historical inputs.

Inventive Principle:

Principle #1Segmentation

Solution Approach 2:

The patent changes the parameter of information representation by introducing compressed memory that stores compressed versions of hidden states. This parameter change enables the system to maintain information preservation for recent inputs while using compressed representations for older inputs, thereby reducing computational overhead.

Inventive Principle:

Principle #35Parameter changes

4Quantity of substance

If the neural network uses compressed memory for older inputs, then storage efficiency is improved, but information loss may occur

Engineering Contradiction:

Improvestorage efficiencyVSAvoidinformation loss

Core Design Contradiction:

Quantity of substanceVSLoss of information

Solution Approach 1:

The patent segments the memory into episodic memory for recent inputs and compressed memory for older inputs. This segmentation ensures that information loss from compression is minimized by preserving detailed representations in episodic memory for recent inputs that are more likely to be relevant, while using compression for older inputs where some information loss is more acceptable.

Inventive Principle:

Principle #1Segmentation

Data Source

PatentUS12443851B2Augmenting attention-based neural networks to selectively attend to past inputs

Publication Date: 2025.10.14 GDM HOLDING LLC

AI summary

Methods, systems, and apparatus, including computer programs encoded on a computer storage medium, for performing a machine learning task on a network input that is a sequence to generate a network output. In one aspect, one of the methods includes, for each particular sequence of layer inputs: for each attention layer in the neural network: maintaining episodic memory data; maintaining compressed memory data; receiving a layer input to be processed by the attention layer; and applying an attention mechanism over (i) the compressed representation in the compressed memory data for the layer, (ii) the hidden states in the episodic memory data for the layer, and (iii) the respective hidden state at each of the plurality of input positions in the particular network input to generate a respective activation for each input position in the layer input.