Autonomous Vehicle Policy Selection via Markov Decision Process

Resolve Bottlenecks,
Find Innovative SolutionsGenerate Solutions

Solution Overview

Problem

Autonomous vehicles face challenges in navigating through traffic scenarios due to the complexity of reasoning through unexpected road events, exceeding the capabilities of scripted rules and predefined knowledge bases, and require socially acceptable driving behavior.

Innovation Solution

A system and method utilizing a Markov state model and Markov Decision Process to generate and select driving policies that optimize navigation by considering vehicle states, environmental information, and expected reward values, ensuring safe and smooth progression towards a destination without penalties.

Engineering Contradictions & Design Principles

VSEngineering Contradiction Analysis

1Adaptability or versatility

If scripted rules and predefined knowledge bases are used for autonomous driving decisions, then the system is simple to implement, but it cannot handle unexpected road events and complex traffic scenarios

Engineering Contradiction:

Improveability to handle unexpected road eventsVSAvoiddecision-making system complexity

Core Design Contradiction:

Adaptability or versatilityVSDevice complexity

Solution Approach 1:

The patent implements a dynamic decision-making system using Markov Decision Processes that continuously adapts to changing traffic conditions. The system transitions from static scripted rules to dynamic policy selection based on current state, allowing the autonomous vehicle to handle unexpected events by recalculating optimal actions in real-time based on environmental feedback and reward functions.

Inventive Principle:

Principle #15Dynamics

Solution Approach 2:

The patent introduces an intermediary layer between simple sensor input and actuator output in the form of a Markov Decision Process framework. This intermediary includes state representation, policy selection, and reward evaluation components that bridge the gap between basic rule-based systems and complex adaptive behavior, enabling sophisticated decision-making without requiring complete reengineering of the control system.

Inventive Principle:

Principle #24Intermediary (Mediator)

2Reliability

If multiple driving policies are generated and evaluated using Markov Decision Process, then navigation effectiveness is optimized, but computational processing time increases

Engineering Contradiction:

Improvenavigation safety and effectivenessVSAvoidpolicy selection time

Core Design Contradiction:

ReliabilityVSLoss of time

Solution Approach 1:

The patent applies preliminary action by pre-defining the Markov state space, transition models, and reward functions before actual navigation. The system prepares multiple candidate policies in advance based on predicted traffic conditions and evaluates them offline when possible, so that during real-time operation, the vehicle can quickly select from pre-evaluated policies rather than computing everything from scratch.

Inventive Principle:

Principle #10Preliminary action

Solution Approach 2:

The system generates multiple driving policies beyond what is strictly necessary, evaluating more options than needed to ensure optimal safety and effectiveness. By computing and comparing several candidate policies with different risk profiles and navigation strategies, the system can select the most appropriate one while having backups ready, trading some computational excess for increased reliability in critical safety decisions.

Inventive Principle:

Principle #16Partial or excessive action

3Adaptability or versatility

If the system continuously updates vehicle state and determines updated policies, then driving behavior adapts to changing conditions, but computational load increases

Engineering Contradiction:

Improveresponse to changing traffic conditionsVSAvoidprocessor energy consumption

Core Design Contradiction:

Adaptability or versatilityVSUse of energy by moving object

Solution Approach 1:

The patent implements periodic action by updating the vehicle state and re-evaluating policies at discrete time intervals rather than continuously. The Markov Decision Process is executed periodically based on significant state changes or time thresholds, allowing the system to adapt to changing conditions while avoiding the excessive computational load of continuous real-time recalculation. This periodic evaluation maintains adaptability while conserving processor energy.

Inventive Principle:

Principle #19Periodic action

Data Source

PatentUS10474149B2Autonomous behavior control using policy triggering and execution

Publication Date: 2019.11.12 GM GLOBAL TECHNOLOGY OPERATIONS LLC

AI summary

An autonomous vehicle, a system and method of operating the autonomous vehicle. An environmental sensor obtains one or more parameters of external agents of the vehicle. A processor of the vehicle obtains a route having a destination at the autonomous vehicle, builds a Markov state model of the route that includes a plurality of states for the autonomous vehicle and one or more parameters of the external agents, generates a plurality of driving policies for navigating the route, selects a policy for navigating the route from the plurality of driving policies using a Markov Decision Process, and executes the selected policy at the autonomous vehicle to navigate the vehicle along the route towards the destination.