Vehicle-Cloud Multimodal Data Mining for Scalable Target Recognition
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current autonomous driving data mining methods require large amounts of training data, labor, and resources for model testing, and the designed rules are not extensible, limiting the scalability and precision of target recognition.
Innovation Solution
A device-cloud collaborative data mining method using a large image-text multimodal model, where the cloud encodes target text and delivers a task configuration file to a vehicle, which encodes image data to determine target pictures based on similarity, utilizing compressed and tuned picture encoders for efficient data mining.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If a large image-text multimodal model is deployed for data mining, then measurement precision and mining precision are improved, but device complexity and resource requirements increase
Solution Approach 1:
The patent segments the large multimodal model into separate text encoder and picture encoder modules that can be independently deployed and managed. This allows the system to leverage the precision benefits of large models while reducing the operational complexity by handling encoders as modular components rather than a monolithic system.
Solution Approach 2:
The patent uses compressed and tuned picture encoder copies that replicate the functionality of the full large model picture encoder but with reduced complexity. These compressed encoders are deployed on vehicles while the text encoder remains on the cloud, enabling precise data mining without requiring the entire large model to be deployed on resource-constrained devices.
2Ease of operation
If compressed and tuned picture encoders are used on vehicles, then ease of operation and resource efficiency are improved, but manufacturing precision and model accuracy may deteriorate
Solution Approach 1:
The patent applies parameter changes through compression and tuning operations on the picture encoder. The encoder is compressed to reduce its size and computational requirements for vehicle deployment, while tuning optimizes its parameters to maintain accuracy. This allows the model to run efficiently on vehicle hardware without significant loss of mining precision.
Solution Approach 2:
The compression and tuning of the picture encoder is performed in advance on the cloud before deployment to vehicles. This preliminary action ensures that the compressed model is pre-optimized for accuracy and performance, so when deployed on vehicles with limited resources, it can operate efficiently without sacrificing mining precision.
3Measurement precision
If task-specific rules and models are designed for each requirement, then measurement precision for specific tasks is improved, but adaptability and extensibility deteriorate
Solution Approach 1:
The patent implements universality by using a general-purpose large image-text multimodal model that can handle multiple different data mining tasks. Instead of designing separate rules and models for each task, the system uses a unified multimodal model with text and picture encoders that can be configured through task configuration files to perform various mining operations, thereby improving extensibility while maintaining precision through the model's inherent capabilities.
4Measurement precision
If large amounts of training data are collected and manual annotation is performed, then measurement precision is improved, but loss of time and labor resources increase
Solution Approach 1:
The system enables self-service through automated annotation capabilities where the large multimodal model performs annotation tasks autonomously without requiring extensive manual intervention. The model can process and annotate data automatically, significantly reducing the time and labor resources needed while maintaining high precision through its advanced image-text understanding capabilities.
Data Source
Figure 1
Figure 2
Figure 3
AI summary
A device-cloud collaborative data mining method includes: A cloud determines a target text and a task configuration file based on a service requirement; encodes the target text by using a text encoder to obtain a text feature; and places the text feature in the task configuration file and delivers the task configuration file to a vehicle. The vehicle encodes image data by using a first picture encoder to obtain an image feature; calculates a value of a similarity between the text feature and the image feature; determines a target picture based on the value of the similarity and the task configuration file; and uploads the target picture to the cloud. The first picture encoder is obtained by compressing and tuning a second picture encoder, and the text encoder and the second picture encoder are two modules of a large image-text multimodal model. This application is applied to an autonomous driving shadow mode, and data mining of any target text of interest can be completed by using a large model, without designing and developing a detection rule for each type of target of interest. In this way, a picture encoder on the vehicle may be continuously updated and optimized, to improve model iteration efficiency.