Camera-Radar Object Recognition for Adverse-Weather Activity Classification
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current object and activity recognition systems in vehicles face challenges in accurately classifying objects and activities, especially in non-ideal weather conditions, as they rely solely on radar data for object location and velocity, and image data for object type, which is unreliable in adverse conditions.
Innovation Solution
A system and method that combines data from a camera and a radar sensor, using a neural network to analyze both image and radar data, converting radar data to a time-frequency image, and simultaneously classifying objects and activities performed by them.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If radar data is used for object location and velocity determination, then measurement accuracy is improved, but object classification capability deteriorates
Solution Approach 1:
The patent combines radar data and image data into a unified analysis framework. The neural network simultaneously processes both data types to perform joint object classification and activity recognition, merging the complementary strengths of radar (velocity, location) and camera (visual classification) to resolve the contradiction between measurement precision and classification capability
2Measurement precision
If image data is used for object classification, then classification accuracy is improved in ideal conditions, but reliability deteriorates in non-ideal weather conditions
Solution Approach 1:
The patent creates a composite data representation by fusing radar and image data. The neural network processes both data types together, creating a robust classification system that maintains reliability in adverse weather by combining the weather-resistant properties of radar data with the classification strength of image data
3Reliability
If only radar data is used, then robustness in various weather conditions is improved, but velocity determination capability deteriorates
Solution Approach 1:
The system merges radar data processing with image data processing in a unified neural network framework. The radar data provides robust weather-resistant detection while the image data contributes additional features for improved velocity and activity determination, combining both strengths to resolve the contradiction
4Device complexity
If only image data is used, then object classification is simplified in ideal conditions, but velocity determination capability deteriorates
Solution Approach 1:
The patent merges image data processing with radar data processing in a unified neural network. This combination maintains the simplicity of image-based classification while adding radar-derived velocity information, resolving the contradiction between system simplicity and velocity measurement capability
Data Source
AI summary
A system for performing object and activity recognition based on data from a camera and a radar sensor. The system includes a camera, a radar sensor, and an electronic processor. The electronic processor is configured to receive an image from the camera and determine a portion of the image including an object. The electronic processor is also configured to receive radar data from the radar sensor and determine radar data from the radar sensor associated with the object in the image from the camera. The electronic processor is also configured to convert the radar data associated with the object to a time-frequency image and analyze the time-frequency image and the image of the object to classify the object and an activity being performed by the object.