MDP Decomposition for Computational Complexity Reduction
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Markov Decision Processes (MDPs) with a large number of states and actions become computationally intensive, leading to the 'curse of dimensionality' issue, which makes them difficult to process and requires significant computing resources.
Innovation Solution
The method involves decomposing a large MDP into smaller sub-MDPs, each with a portion of the state variables and the same set of actions, and then aggregating the policies of these sub-MDPs based on expected rewards to create a resultant policy.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If a large MDP with complete state variables is processed, then the solution accuracy is improved, but the computational complexity and processing time increase significantly
Solution Approach 1:
The patent divides the complete MDP into multiple sub-MDPs by partitioning the state variables into disjoint sets. Each sub-MDP processes a subset of state variables independently, reducing the computational complexity from processing the entire state space to processing smaller, manageable portions. The results are then aggregated to form the final policy for the original MDP.
2Measurement precision
If a large MDP with complete state variables is processed, then the solution accuracy is improved, but the processing time increases significantly
Solution Approach 1:
The patent segments the MDP into sub-MDPs that can be processed in parallel or sequentially with reduced time requirements. By dividing the computational workload across multiple smaller problems rather than solving one large problem, the overall processing time is significantly reduced while maintaining solution accuracy through proper aggregation of results.
Solution Approach 2:
The patent applies partial action by processing only the necessary subset of state variables for each sub-MDP rather than processing all state variables simultaneously. This selective processing approach reduces the time required to reach a solution while still achieving accurate results through the aggregation of partial policies.
3Measurement precision
If a large MDP with complete state variables is processed, then the solution accuracy is improved, but the computing resources required increase significantly
Solution Approach 1:
The patent segments the computational workload into smaller sub-MDPs that require fewer computing resources each. By distributing the processing across multiple independent sub-problems rather than concentrating all computation in a single large MDP, the overall computing resource consumption is reduced while maintaining solution accuracy through proper aggregation.
Solution Approach 2:
The patent uses partial action by allocating computing resources to process only the necessary subset of state variables for each sub-MDP. This selective resource allocation reduces total computing resource consumption while still achieving accurate solutions through the aggregation of results from partial processing.
Data Source
AI summary
A method for Markov Decision Process (“MDP”) decomposition includes receiving data elements for a problem that include finite state data for a set of state variables and a finite set of actions. A portion of the state data corresponding to state variables represents states. The method incudes creating two or more sub-MDPs. Each sub-MDP includes a portion of the set of state variables, the set of actions and a same reward function. The method includes executing each sub-MDP. Results include a policy and an expected reward from the reward function. The policy of the sub-MDP maps states of the sub-MDP to actions. The method includes aggregating, based on the expected rewards of the results, the actions of the policies of the sub-MDPs to create a resultant policy with resultant actions and generating, using state entries for the set of state variables, results to the problem based on the resultant policy.