Vehicle Action Prediction With Command-Conditioned LSTM Branches
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing deep-learning based autonomous driving methods, particularly 'CNN-only' architectures, lack memory of scene dynamics as they depend solely on current frames without representing past frames, leading to poor performance in multimodal target distributions and lacking high-level control over network behavior.
Innovation Solution
A system that incorporates a command-conditioned Recurrent Neural Network (RNN) with multiple LSTM branches, where control commands are used to select the appropriate branch for processing visual features and vehicle position, enabling the network to model scene dynamics and account for high-level goals, such as passenger destinations, thereby providing a high-level control over network behavior.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If CNN-only architecture is used to extract visual information from current frames, then the network can process visual data efficiently, but the network lacks memory of scene dynamics and cannot represent past frames
Solution Approach 1:
The patent combines CNN for visual feature extraction with LSTM for temporal memory, merging the strengths of both architectures. The CNN processes current frame visual information while the LSTM maintains memory of past frames and scene dynamics, resolving the contradiction between processing efficiency and information retention.
Solution Approach 2:
The LSTM network acts as an intermediary between the CNN visual encoder and the decision-making components. It receives visual features from CNN, processes them through recurrent layers that maintain temporal context, and outputs actions that incorporate both current visual information and historical scene dynamics.
2Measurement precision
If regression task with mean squared error loss is used for continuous prediction, then the network can output continuous action values, but the loss performs poorly when the target distribution is multimodal
Solution Approach 1:
The patent changes the loss function parameter from mean squared error (MSE) to cross-entropy loss, and changes the output representation from continuous values to discrete bins. This parameter transformation allows the network to effectively handle multimodal target distributions by treating the prediction as a classification problem over binned action spaces.
3Reliability
If discrete bins are used to cast the regression problem into classification task, then the network can handle multimodal distributions, but the solution is based only on sensor data and lacks high-level control of network behavior
Solution Approach 1:
The patent segments the single neural network into multiple specialized LSTM branches, each trained to handle specific control commands or action types. This segmentation allows the network to maintain specialized expertise for different driving scenarios while being controlled by high-level command signals, combining the benefits of specialized processing with flexible control.
Data Source
AI summary
A system for obtaining a prediction an action (at) of a vehicle (V), including a camera for acquiring a sequence of images (Ft) a convolutional neural network visual encoder which obtains a corresponding visual features vector (vt), one or more sensor that obtains a position of the vehicle (st) at the same time step (st), a Recurrent Neural Network configured to receive the visual features vector (vt) and position of the vehicle (st) at the time step (t) and to generate a prediction of the action (at) of the vehicle (V). The system comprising a command conditioned switch configured upon reception of a control command (ci) to select a corresponding branch of the Recurrent Neural Network. The system is configured to operate the selected corresponding branch to process the visual features vector (vt) and position of the vehicle (st) at the time step (t) to obtain the prediction of the action (at) of the vehicle (V).