Segmented Replay Buffer for Distributed RL Training
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
In the context of real-time strategy games like StarCraft, the sparsity of rewards during gameplay leads to insufficient training of models and unsatisfactory improvement in robot performance, as current partial observations are insufficient to infer future states and rewards, hindering correct macro action selection.
Innovation Solution
A method is introduced where game data from multiple terminal devices is stored in segment replay buffers and sampled based on priority, with a loss function used to train the model, synchronizing the model across devices to obtain Q-values for actions, and exploration degrees are calculated to determine actions, with target reward values calculated based on scores from other devices, enabling effective training and updating of the model.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Quantity of substance
If game data is collected from multiple terminal devices, then the quantity of training data is improved, but the system complexity increases due to distributed data collection and management
Solution Approach 1:
The replay buffer is segmented into multiple segment replay buffers, with each terminal device having its own dedicated segment. This segmentation allows distributed data collection from multiple devices while maintaining independent data management for each device, thus increasing training data quantity without proportionally increasing system complexity.
2Productivity
If priority-based sampling is implemented, then the training efficiency is improved by focusing on important transitions, but the computational complexity increases due to priority calculation and dynamic buffer management
Solution Approach 1:
Priorities for experience transitions are pre-calculated and stored in the replay buffer before sampling occurs. This preliminary calculation of priorities allows the sampling process to efficiently select important transitions without performing complex real-time priority calculations during training, thus improving training efficiency while controlling computational complexity.
3Stability of the object's composition
If the replay buffer is divided into segments for different terminal devices, then the data organization is improved for distributed training, but the buffer management complexity increases
Solution Approach 1:
The replay buffer is divided into multiple segment replay buffers, with each segment dedicated to a specific terminal device. This segmentation improves data organization by maintaining device-specific data structures that are stable and predictable, while the modular nature of segmented buffers actually simplifies management compared to a monolithic buffer, as each segment can be independently managed.
4Stability of the object's composition
If model synchronization across terminal devices is performed, then the training consistency is improved, but the communication overhead and time consumption increase
Solution Approach 1:
The trained model parameters are copied from the central server to multiple terminal devices for local execution and data collection. This copying approach allows training consistency to be maintained across devices without requiring continuous real-time synchronization, thus reducing communication overhead and time loss while preserving training consistency.
Data Source
AI summary
The present disclosure describes techniques for training a model and improving game performance using the model. The disclosed techniques comprise obtaining a plurality of sets of game data from a plurality of terminal devices, the plurality of sets of game data being stored in a plurality of segment replay buffers; sampling the plurality of sets of game data based on a predetermined priority and obtaining a sampling result; inputting the sampling result into a loss function and training a model by using a calculation result of the loss function; and determining a quality (Q) value of each action performed by a virtual object in a game run on a terminal device among the plurality of terminal devices by using the model.