Neural Network Hidden State Reversion via External Memory

Solution Overview

Problem

Conventional reinforcement learning systems require agents to relearn previously acquired information when resuming training for reoccurring tasks, leading to inefficiencies in performance and resource utilization.

Innovation Solution

The system employs an external memory to store context experience tuples, allowing agents to recall and reuse previously learned hidden states, thereby enabling the agent to revert to a previous action selection policy without relearning, and uses a memory interface subsystem to determine a remembered value of the hidden state based on current context data, which is combined with the current hidden state to generate action selection outputs.

Engineering Contradictions & Design Principles

VSEngineering Contradiction Analysis

1Reliability

If agents relearn previously acquired information when resuming training for reoccurring tasks, then the action selection policy can be updated with current information, but training time and computational resources are wasted

Engineering Contradiction:

Improveaction selection policy accuracyVSAvoidtraining time

Core Design Contradiction:

ReliabilityVSLoss of time

Solution Approach 1:

The system performs preliminary action by storing the hidden state (which encodes learned information) in external memory during initial training. When the same task context reoccurs, the pre-stored hidden state is retrieved and reused, avoiding the need to relearn the information and thus saving training time while maintaining policy accuracy.

Inventive Principle:

Principle #10Preliminary action

Solution Approach 2:

The system creates a copy of the hidden state (which represents the learned action selection policy) and stores it in external memory. When a reoccurring task is detected through context matching, this copied hidden state is retrieved and reused, eliminating redundant learning while preserving the effectiveness of the action selection policy.

Inventive Principle:

Principle #26Copying

2Reliability

If agents relearn previously acquired information when resuming training for reoccurring tasks, then the action selection policy can be updated with current information, but computational resources are consumed inefficiently

Engineering Contradiction:

Improveaction selection policy accuracyVSAvoidcomputational resources

Core Design Contradiction:

ReliabilityVSUse of energy by moving object

Solution Approach 1:

The system performs preliminary action by storing the hidden state (which encodes learned information) in external memory during initial training. When the same task context reoccurs, the pre-stored hidden state is retrieved and reused, avoiding the need to relearn the information and thus saving training time while maintaining policy accuracy.

Inventive Principle:

Principle #10Preliminary action

Solution Approach 2:

The system creates a copy of the hidden state (which represents the learned action selection policy) and stores it in external memory. When a reoccurring task is detected through context matching, this copied hidden state is retrieved and reused, eliminating redundant learning while preserving the effectiveness of the action selection policy.

Inventive Principle:

Principle #26Copying

3Productivity

If the system stores and retrieves hidden states from external memory, then previously learned information can be reused, but the device complexity increases

Engineering Contradiction:

Improvetraining efficiencyVSAvoidsystem architecture complexity

Core Design Contradiction:

ProductivityVSDevice complexity

Solution Approach 1:

The system introduces an intermediary component (external memory with key-value stores) that bridges the neural network and the task context. This intermediary stores hidden states indexed by context keys, enabling efficient retrieval of previously learned information without fundamentally altering the neural network's internal structure, thus balancing productivity improvement with manageable complexity increase.

Inventive Principle:

Principle #24Intermediary (Mediator)

Solution Approach 2:

The system segments the overall system into distinct functional components: the neural network for processing, the external memory for storage, and the memory interface subsystem for coordination. This segmentation allows each component to be optimized independently and facilitates the reuse of learned information while keeping the architecture organized and manageable.

Inventive Principle:

Principle #1Segmentation

Data Source

PatentUS11423300B1Selecting actions by reverting to previous learned action selection policies

Publication Date: 2022.08.23 GDM HOLDING LLC

AI summary

Methods, systems, and apparatus, including computer programs encoded on a computer storage medium, for generating a system output using a remembered value of a neural network hidden state. In one aspect, a system comprises an external memory that maintains context experience tuples respectively comprising: (i) a key embedding of context data, and (ii) a value of a hidden state of a neural network at the respective previous time step. The neural network is configured to receive a system input and a remembered value of the hidden state of the neural network and to generate a system output. The system comprises a memory interface subsystem that is configured to determine a key embedding for current context data, determine a remembered value of the hidden state of the neural network based on the key embedding, and provide the remembered value of the hidden state as an input to the neural network.