Actor-Critic Neural Networks for Continuous Action Policy Learning

Resolve Bottlenecks,
Find Innovative SolutionsGenerate Solutions

Solution Overview

Problem

Reinforcement learning systems struggle to effectively learn action selection policies in high-dimensional, continuous action spaces, particularly when the action space is intractable for discretization and when observations are either low-dimensional or high-dimensional pixel inputs.

Innovation Solution

A reinforcement learning system utilizing an actor neural network trained with a replay memory, critic neural network, and target neural networks to learn effective action selection policies, even in complex environments, by generating and updating experience tuples to refine parameter values.

Engineering Contradictions & Design Principles

VSEngineering Contradiction Analysis

1Adaptability or versatility

If reinforcement learning systems use traditional discretization methods for action selection, then the action space becomes tractable for exploration, but the system cannot effectively learn in high-dimensional continuous action spaces that require fine control

Engineering Contradiction:

Improvecapability to handle high-dimensional continuous action spacesVSAvoidcomplexity of action space representation

Core Design Contradiction:

Adaptability or versatilityVSDevice complexity

Solution Approach 1:

The patent replaces traditional discretization-based mechanical exploration methods with a neural network-based continuous action selection system. The actor neural network directly outputs continuous action values without requiring discretization of the action space, enabling effective learning in high-dimensional continuous domains while maintaining tractability through gradient-based optimization.

Inventive Principle:

Principle #28Mechanics substitution (Replace mechanical system)

Solution Approach 2:

The patent transforms the action selection mechanism by changing from discrete action indices to continuous action values parameterized by neural network outputs. The action space is represented as continuous parameters that can be directly optimized through gradient descent, allowing fine-grained control in high-dimensional spaces without the combinatorial explosion inherent in discretization.

Inventive Principle:

Principle #35Parameter changes

2Loss of information

If reinforcement learning systems train with high-dimensional pixel inputs, then the system can learn from rich visual observations, but the computational complexity and data requirements increase significantly

Engineering Contradiction:

Improveinformation retention from visual observationsVSAvoidcomputational complexity of processing

Core Design Contradiction:

Loss of informationVSDevice complexity

Solution Approach 1:

The patent segments the high-dimensional pixel input processing into distinct neural network components: convolutional layers for feature extraction from images, followed by fully connected layers for action prediction. This segmentation allows the system to process rich visual information efficiently by breaking down the complex transformation from high-dimensional pixels to continuous actions into manageable computational stages.

Inventive Principle:

Principle #1Segmentation

3Productivity

If reinforcement learning systems explore continuous action spaces without actor-critic architecture, then the exploration becomes intractable, but with actor-critic architecture the system can efficiently learn effective policies

Engineering Contradiction:

Improvelearning efficiency in continuous spacesVSAvoidarchitectural complexity of actor-critic system

Core Design Contradiction:

ProductivityVSDevice complexity

Solution Approach 1:

The patent segments the reinforcement learning system into two distinct but coordinated components: the actor neural network that selects actions and the critic neural network that evaluates them. This segmentation allows independent optimization of action selection and value estimation, making continuous space exploration tractable through efficient gradient computation while maintaining architectural clarity.

Inventive Principle:

Principle #1Segmentation

Solution Approach 2:

The patent implements feedback through the critic network evaluating actor actions and using this information to update the actor's policy. The critic provides continuous feedback signals (value estimates and advantages) that guide the actor's learning in continuous action spaces, enabling efficient policy improvement without exhaustive exploration.

Inventive Principle:

Principle #23Feedback

Data Source

PatentUS12561573B2Continuous control with deep reinforcement learning

Publication Date: 2026.02.24 GDM HOLDING LLC

AI summary

Methods, systems, and apparatus, including computer programs encoded on computer storage media, for training an actor neural network used to select actions to be performed by an agent interacting with an environment. One of the methods includes obtaining a minibatch of experience tuples; and updating current values of the parameters of the actor neural network, comprising: for each experience tuple in the minibatch: processing the training observation and the training action in the experience tuple using a critic neural network to determine a neural network output for the experience tuple, and determining a target neural network output for the experience tuple; updating current values of the parameters of the critic neural network using errors between the target neural network outputs and the neural network outputs; and updating the current values of the parameters of the actor neural network using the critic neural network.