Vehicle ML Update Verification Using Recognition Count Comparison
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Machine learning models used in vehicle control systems for automated driving can exhibit significant changes in inference results due to small input variations and differences in vehicle sensor states, leading to inconsistent performance after updates, making it difficult to verify if the updates are normal.
Innovation Solution
A method and system for verifying updates to control apparatuses using machine learning models, which involves acquiring and comparing verification numbers from log data before and after the update, specifically the number of recognized objects or driving decisions within a predetermined period or distance, to determine if the update is normal.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If a machine learning model is upgraded to improve automated driving control performance, then the control performance is improved, but the inference result may change greatly due to small changes in input or vehicle sensor states, making it difficult to verify if the update is normal
Solution Approach 1:
The patent creates a virtual copy of the machine learning model's inference environment by simulating sensor inputs and capturing inference results in a controlled virtual space. This virtual copy allows verification without affecting the actual vehicle operation, enabling accurate measurement of model performance changes while maintaining reliability.
Solution Approach 2:
The patent introduces a verification system as an intermediary between the upgraded machine learning model and the verification process. This intermediary captures inference results, compares them against baseline data, and determines whether updates are normal, thereby solving the verification accuracy problem while preserving control performance improvements.
2Reliability
If offline verification and evaluation using an experimental vehicle is performed before distributing the upgraded machine learning model, then the model performance is verified, but the verification process is time-consuming and delays the update deployment
Solution Approach 1:
The patent performs preliminary verification actions by capturing and storing baseline inference results before the model upgrade, and then automatically comparing post-upgrade results against this baseline. This preliminary preparation enables rapid verification without time-consuming manual testing, thus reducing update deployment time while maintaining verification reliability.
Solution Approach 2:
The verification system performs self-service by automatically capturing inference results, comparing them with baseline data, and determining update normality without requiring manual intervention. This automation eliminates the time-consuming nature of traditional offline verification while ensuring thorough model performance checking.
3Measurement precision
If the machine learning model is made more sensitive to detect subtle changes in surrounding situation, then the recognition accuracy is improved, but the inference result changes greatly due to small input variations, reducing the stability of the update verification
Solution Approach 1:
The patent implements feedback by continuously monitoring inference results and comparing them against baseline data from before the model upgrade. This feedback mechanism identifies whether changes in inference results are within normal ranges or indicate abnormal updates, thereby maintaining verification stability even when the model is highly sensitive to input variations.
Solution Approach 2:
The patent changes the verification parameter from direct inference result comparison to statistical analysis of inference result distributions. By analyzing patterns and distributions rather than individual results, the system maintains stability in verification while allowing the model to remain sensitive to important changes in surrounding situations.
Data Source
AI summary
An update verification method acquires a first verification number and a second verification number. The first verification number is either a number of objects around a vehicle recognized by a machine learning model after upgrade, or a number of driving decisions in a travel plan generated by the machine learning model after the upgrade. The second verification number is either a number of objects around the vehicle recognized by the machine learning model before the upgrade, or a number of driving decisions in the travel plan generated by the machine learning model before the upgrade. Then, the update verification method verifies whether an update of a control apparatus is normal by comparing the first verification number with the second verification number.