AI Bot Training via Discrete Encoder and Ape-X
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing methods for training AI bots in computer games require modification of game programs, use of internal game states, and resource-intensive processes, leading to inefficiencies and instability in the training process.
Innovation Solution
A method divided into three phases: collecting demonstration data to train a discrete encoder, converting professional players' demonstrations into state and action sets, and using deep reinforcement learning with a reward function based on these sets to train the AI bot, all while utilizing an Ape-X distributed training framework and a variant of Deep Q Network.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If conventional AI training methods use huge demonstration materials to train Neural Network behavior, then training completeness can be improved, but training time increases and training efficiency deteriorates
Solution Approach 1:
The patent applies preliminary action by pre-processing demonstration data into structured state sets and state-action sets before training. The discrete encoder converts raw demonstration materials into organized representations in advance, allowing the reinforcement learning agent to learn from pre-structured data rather than raw data, significantly reducing training time while maintaining training completeness.
Solution Approach 2:
The patent introduces an intermediary component - the discrete encoder - that mediates between raw demonstration materials and the reinforcement learning training process. This encoder transforms unstructured demonstration data into structured state representations, enabling efficient learning without requiring the agent to process huge amounts of raw demonstration materials directly.
2Measurement precision
If AI training process runs local games, online game clients, or simulators inside the AI training agent, then training accuracy can be improved, but system complexity and resource requirements increase
Solution Approach 1:
The patent extracts the essential training functionality from complex game clients and simulators. Instead of running full game environments inside the training agent, it extracts only the necessary state observations and action feedback mechanisms, decoupling the training core from the game runtime environment. This reduces system complexity while maintaining training accuracy through direct access to game state data.
Solution Approach 2:
The patent segments the training system into independent components: a discrete encoder for state representation, a reinforcement learning agent for decision-making, and separate data collection modules. This segmentation allows each component to operate independently with optimized resource requirements, reducing overall system complexity while maintaining high training accuracy through specialized function allocation.
3Productivity
If Ape-X distributed training framework uses multiple actors to generate data in parallel, then data generation speed can be improved, but coordination complexity and communication overhead increase
Solution Approach 1:
The patent merges the experience replay functionality into a centralized shared buffer that all actors access simultaneously. Instead of each actor maintaining separate replay buffers requiring complex synchronization, the unified experience pool allows parallel data generation while simplifying coordination. The learner module centrally manages experience sampling and model updates, reducing communication overhead while maintaining high data generation speed.
Data Source
AI summary
The invention discloses a pure end-to-end deep reinforcement learning for training car racing game AI bot that uses only the velocity information extracted from screen for both training and testing phases without using any internal state from game environment, such as the car facing angle. The learned AI bot can play better than the average performance of human players. In addition, the reward function is designed to consist only the velocity value, and use Ape-X distributed training framework combined with a variant of Deep Q Network to solve the sparse training signal problem caused by the reward function of an original design. Moreover, limit learner rate method is designed that improves the training efficiency and training performance. The AI bot trained in this way can achieve performance beyond the average human level and reach a level close to professional players.