AI Bot Training via Screen Velocity Observations
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current methods for training AI bots in car racing games require modification of game programs to obtain internal states, leading to integration challenges, resource inefficiencies, and slower sample generation due to insufficient computing power, resulting in overfitting and longer convergence times.
Innovation Solution
A method that decouples the learning environment from the AI training agent, using an Ape-X distributed training framework combined with a variant of Deep Q Network, allowing the AI bot to train using only velocity data from screen observations without modifying the game, and controlling learning speed and data sampling to prevent overfitting.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If the AI training agent runs local games or simulators to obtain internal states, then the training can proceed, but the integration effort increases and the system becomes more complex
Solution Approach 1:
The system is divided into two independent components: the game server that runs the playing environment and the AI training agent that performs learning. This segmentation allows each component to function independently without requiring integration, thus reducing system complexity while maintaining training capability.
Solution Approach 2:
A state observation mechanism is introduced as an intermediary that extracts necessary game states from the visual output without requiring direct access to internal game variables. This mediator enables the AI agent to obtain training data through screen capture and image processing rather than direct game program integration.
2Ease of operation
If the learning environment and AI training agent are coupled, then the training process is straightforward, but the system stability decreases and resource utilization is poor
Solution Approach 1:
By separating the learning environment (game server) from the AI training agent into independent processes, the system achieves both operational simplicity and stability. Each component can run independently without affecting the other, eliminating crash propagation while maintaining ease of deployment.
Solution Approach 2:
The system transitions from a tightly coupled single-process architecture to a distributed multi-process architecture operating across different computational dimensions. This dimensional shift allows independent resource allocation and failure isolation, improving stability while preserving operational simplicity through standardized interfaces.
3Productivity
If more computing power is allocated to actors for parallel game playing, then sample generation speed increases, but resource utilization becomes unbalanced and training time increases
Solution Approach 1:
The system implements continuous parallel execution of multiple actor processes that continuously generate training samples without idle periods. While actors continuously play games and generate data, the learner process continuously processes available samples, ensuring that computational resources are continuously utilized productively without waiting for batch completions, thus reducing overall convergence time.
Solution Approach 2:
The system dynamically balances the workload between actors and learner based on available computing resources and data generation rates. Multiple actors can be launched with varying computational demands, and the learner adaptively processes samples as they become available, creating a dynamic system that optimizes resource utilization and minimizes training time.
Data Source
AI summary
The invention discloses a pure end-to-end deep reinforcement learning for training car racing game AI bot that uses only the velocity information extracted from screen for both training and testing phases without using any internal state from game environment, such as the car facing angle. The learned AI bot can play better than the average performance of human players. In addition, the reward function is designed to consist only the velocity value, and use Ape-X distributed training framework combined with a variant of Deep Q Network to solve the sparse training signal problem caused by the reward function of an original design. Moreover, limit learner rate method is designed that improves the training efficiency and training performance. The AI bot trained in this way can achieve performance beyond the average human level and reach a level close to professional players.