Deep Reinforcement Learning Algorithm Parallel Execution

Resolve Bottlenecks,
Find Innovative SolutionsGenerate Solutions

Solution Overview

Problem

Existing deep reinforcement learning accelerators can only handle either training or inference at a time, leading to long delays and low hardware utilization due to the sequential nature of these processes.

Innovation Solution

An algorithm system and method for deep reinforcement learning that enables parallel execution of experience collection and network update processes, utilizing a processor with an inference processing module and a training processing module to read and execute corresponding programs from memory.

Engineering Contradictions & Design Principles

VSEngineering Contradiction Analysis

1Productivity

If the same computing resources are used for both training and inference, then hardware utilization is improved, but training and inference cannot be processed simultaneously causing long delays

Engineering Contradiction:

Improvehardware utilizationVSAvoidprocessing delay

Core Design Contradiction:

ProductivityVSLoss of time

Solution Approach 1:

The computing resource is segmented into two independent processing modules: an inference processing module for experience collection and a training processing module for network update. This segmentation allows both processes to execute simultaneously without resource conflicts, resolving the contradiction between hardware utilization and processing delay.

Inventive Principle:

Principle #1Segmentation

Solution Approach 2:

The patent transitions from sequential time-domain execution to parallel space-domain execution by introducing separate processing modules. This dimensional change enables experience collection and network update to occur simultaneously, eliminating the turn-based processing limitation and reducing overall execution time.

Inventive Principle:

Principle #17Another dimension (Dimensionality change)

2Device complexity

If experience collection and network update are executed sequentially, then resource requirements are reduced, but the overall execution time increases significantly

Engineering Contradiction:

Improveprocessing architectureVSAvoidalgorithm execution time

Core Design Contradiction:

Device complexityVSLoss of time

Solution Approach 1:

The processing architecture is segmented into independent inference and training processing modules that can operate simultaneously. This segmentation enables parallel execution of experience collection and network update, significantly reducing the overall algorithm execution time while maintaining manageable system complexity through modular design.

Inventive Principle:

Principle #1Segmentation

Solution Approach 2:

The patent implements continuous parallel execution where experience collection and network update proceed simultaneously without idle waiting periods. This continuity of useful action eliminates the sequential bottleneck and optimizes the time efficiency of the reinforcement learning algorithm.

Inventive Principle:

Principle #20Continuity of useful action

3Ease of operation

If only one process (training or inference) is executed at a time, then process management is simplified, but hardware utilization becomes inefficient

Engineering Contradiction:

Improveprocess managementVSAvoidhardware utilization

Core Design Contradiction:

Ease of operationVSProductivity

Solution Approach 1:

The system is segmented into dedicated inference and training processing modules with independent execution paths. This segmentation simplifies process management by assigning specific functions to specific modules while simultaneously improving hardware utilization through parallel processing of both experience collection and network update operations.

Inventive Principle:

Principle #1Segmentation

Data Source

PatentUS20250165793A1Algorithm system of deep reinforcement learning and algorithm method thereof

Publication Date: 2025.05.22 IND TECH RES INST

AI summary

An algorithm method for deep reinforcement learning includes initializing an environment and a model; executing an experience collection process and a network update process in parallel, and determining whether the experience collection process and the network update process have reached a termination condition; and continuing executing the experience collection process and the network update process in parallel in response to neither of the experience collection process and the network update processes has met the termination conditions; and stopping executing the experience collection process and the network update process in response to one of the experience collection processes and the network update process having met the termination conditions. The experience collection process includes obtaining a current state of the environment; calculating to determine the current action based on the current observation values according to a current policy of the model; and returning the current action to the environment.