Vehicle ML Update Verification Using Log-Based Output Comparison
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Automated driving control systems using machine learning models face challenges in verifying the normalcy of updates, as small changes in input can significantly affect inference results, and variations in vehicle sensors can lead to inconsistent performance across different vehicles.
Innovation Solution
A method and system for verifying updates to automated driving control apparatuses by comparing log data before and after upgrades, using a computer to acquire and analyze verification numbers such as the number of recognized objects or driving decisions within a predetermined period or distance, to determine if the update is normal.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If a machine learning model is upgraded to improve automated driving control performance, then control performance is improved, but the inference result may change greatly due to small changes in input, leading to inconsistent performance across different vehicles
Solution Approach 1:
The patent performs preliminary verification of the upgraded machine learning model by comparing inference results before and after the upgrade using log data from actual vehicles. This preliminary action identifies potential inconsistencies in inference results before widespread deployment, allowing for adjustments to be made while maintaining the performance improvements of the upgraded model.
Solution Approach 2:
The patent implements a feedback mechanism where inference results from the upgraded machine learning model are compared against baseline results from before the upgrade. This feedback loop uses actual vehicle log data to detect significant changes in inference behavior, enabling continuous monitoring and verification of model consistency across different vehicles after the upgrade is deployed.
2Reliability
If offline verification using an experimental vehicle is performed to verify the upgraded machine learning model, then model performance is verified, but the verification process is time-consuming and may not identify vehicle-specific issues
Solution Approach 1:
The patent makes the verification system universal by using log data from actual production vehicles in the field rather than requiring dedicated experimental vehicles. The same verification methodology can be applied to any vehicle that has the upgraded model installed, allowing parallel verification across multiple vehicles simultaneously, thus reducing total verification time while capturing vehicle-specific issues.
Solution Approach 2:
The patent enables the vehicles themselves to participate in the verification process by collecting and providing their own log data for comparison. This self-service approach eliminates the need for separate experimental vehicle testing facilities and allows continuous verification as vehicles operate normally, significantly reducing verification time while maintaining comprehensive coverage.
3Productivity
If the upgraded machine learning model is distributed to all vehicles after offline verification, then control performance improvement is achieved, but vehicle-specific issues may not be detected early
Solution Approach 1:
The patent implements continuous feedback monitoring after the upgraded model is distributed to vehicles. By comparing inference results from the upgraded model against baseline data from before the upgrade, the system can detect vehicle-specific issues that arise in real-world conditions. This feedback mechanism allows for early identification of problems while the model is still in use, enabling targeted updates or rollbacks for affected vehicles.
Data Source
Figure 1
Figure 2
Figure 3
AI summary
An update verification method acquires a first verification number and a second verification number. The first verification number is either a number of objects around a vehicle (1) recognized by a machine learning model after upgrade, or a number of driving decisions in a travel plan generated by the machine learning model after the upgrade. The second verification number is either a number of objects around the vehicle (1) recognized by the machine learning model before the upgrade, or a number of driving decisions in the travel plan generated by the machine learning model before the upgrade. Then, the update verification method verifies whether an update of a control apparatus (200) is normal by comparing the first verification number with the second verification number.