Fusing LIDAR and Camera Data for Object Action Classification
Solution Overview
Problem
Autonomous vehicles face challenges in accurately classifying the actions of perceived objects, such as pedestrians, using existing sensor data, which is crucial for making appropriate driving decisions but lacks precision and historical context.
Innovation Solution
A method involving server computing devices that receive and sequence LIDAR sensor data frames with object bounding boxes and action labels, project them into camera images to create fused data, and train a deep neural network model to improve action classification precision, using historical labels to predict future object behaviors.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If LIDAR sensor data alone is used for action classification, then the system is simpler, but the precision of action classification deteriorates
Solution Approach 1:
The patent combines LIDAR sensor data with camera images to create fused sensor data. The LIDAR data provides precise depth and spatial information while camera images provide visual context, together enabling more accurate action classification than either sensor alone.
Solution Approach 2:
The patent introduces a deep neural network model as an intermediary that processes and fuses LIDAR sensor data with camera images. This model learns to combine features from both data sources to improve action classification precision.
2Reliability
If historical action labels are used for prediction, then the context awareness improves, but the computational complexity increases
Solution Approach 1:
The patent uses historical action labels from previous time steps as input to predict current object actions. This temporal context allows the model to understand object behavior patterns and improve prediction reliability by considering past actions.
Solution Approach 2:
The model incorporates feedback from historical action labels and previous predictions to continuously improve action classification. The sequential processing allows the model to learn from past predictions and adjust current classifications accordingly.
Data Source
AI summary
Aspects of the disclosure relate to training and using a model for identifying actions of objects. For instance, LIDAR sensor data frames including an object bounding box corresponding to an object as well as an action label for the bounding box may be received. Each sensor frame is associated with a timestamp and is sequenced with respect to other sensor frames. Each given sensor data frame may be projected into a camera image of the object based on the timestamp associated with the given sensor data frame in order to provide fused data. The model may be trained using the fused data such that the model is configured to, in response to receiving fused data, the model outputs an action label for each object bounding box of the fused data. This output may then be used to control a vehicle in an autonomous driving mode.