Distributed Vehicle Control Learning Under Memory Constraints
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Resource-constrained control modules in vehicles, such as those in hybrid and electric vehicles, are unable to perform reinforcement learning effectively due to memory and computational limitations, limiting their ability to optimize engine control and energy management.
Innovation Solution
A distributed reinforcement learning system is implemented, where a control module executes a portion of the reinforcement learning and a separate learning module, which can be onboard or remote, updates the policy at a slower rate, balancing memory and computational demands, and uses exploratory control to improve engine control and energy management.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Adaptability or versatility
If reinforcement learning is implemented in control modules, then learning capability and adaptive control are improved, but memory and computational resources are exceeded
Solution Approach 1:
The system divides the reinforcement learning functionality into two separate modules: a control module that executes control actions and a learning module that performs the computationally intensive learning tasks. This segmentation allows the learning module to have sufficient memory resources while the control module maintains real-time control capabilities with minimal resource consumption.
Solution Approach 2:
A communication interface acts as an intermediary between the control module and learning module, transmitting only essential state information and control actions. This intermediary mechanism reduces the memory burden on the control module while enabling the learning module to access necessary data for training.
2Adaptability or versatility
If reinforcement learning is implemented in control modules, then adaptive control is improved, but computational power is exceeded
Solution Approach 1:
The system separates control execution from learning computation by creating distinct control module and learning module. The control module handles real-time control decisions with minimal computational requirements, while the learning module performs intensive reinforcement learning computations during non-critical periods or with access to greater computational resources.
Solution Approach 2:
The learning module updates the policy periodically rather than continuously, allowing computational-intensive learning operations to occur at scheduled intervals rather than requiring constant computational power. This periodic update approach maintains adaptive control while managing computational resource usage.
3Productivity
If policy updates are performed frequently, then learning speed is improved, but control stability is reduced
Solution Approach 1:
The system implements periodic policy updates where the learning module trains on accumulated experience data and updates the control policy at scheduled intervals. This approach allows frequent learning (high productivity) while maintaining control stability by avoiding continuous policy changes that could destabilize the controlled system.
Solution Approach 2:
The learning module performs preliminary learning computations on accumulated data before applying policy updates to the control module. This preliminary action allows the system to process multiple data points and stabilize learning results before implementing changes, balancing learning speed with control stability.
Data Source
AI summary
A distributed learning system of a vehicle includes: a control module configured to: control a plant of the vehicle using a policy; send signals to a learning module including information on an impact of the control on the plant; and selectively control the plant using exploratory control; and the learning module, where the learning module is separate from the control module and is configured to selectively update the policy based on (a) the signals from the control module, (b) state parameters resulting from the control of the plant using the policy, and (c) performance feedback determined based on the control of the plant using the policy and the selective control of the plant using exploratory control, where the control module is configured to receive the exploratory control from the learning module.