HMI Automation Testing with Machine Learning Detection
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current automation testing methods for Human Machine Interface (HMI) screens are inefficient due to the need for constant updates and the challenge of detecting minute defects, often resulting in false positive or false negative errors, especially with techniques like pixel-to-pixel image comparison or template matching.
Innovation Solution
A method using a trained machine learning model to detect UI elements and their graphical attributes, with iterative training based on learning scores, and incorporating OCR for text detection, to generate validity scores and classify errors, facilitating autocalibration and reducing false errors through tensor flow records.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If traditional pixel-to-pixel image comparison or template matching is used for HMI testing, then the testing process is simpler to implement, but the detection accuracy is low resulting in false positive or false negative errors
Solution Approach 1:
The patent replaces traditional mechanical image comparison methods with machine learning-based object detection models. The system uses trained neural networks to automatically identify and classify UI elements, replacing the manual pixel-to-pixel comparison approach with intelligent algorithmic detection that achieves higher accuracy without requiring complex manual configuration.
Solution Approach 2:
The system transforms the testing approach by changing from fixed template matching parameters to dynamic machine learning model parameters. The model learns optimal detection parameters automatically during training, adapting to different UI element characteristics rather than relying on predetermined thresholds and templates.
2Adaptability or versatility
If constant updates are made to HMI screens during program development, then the user experience is improved, but the testing effort increases exhaustively
Solution Approach 1:
The machine learning model performs self-service by automatically adapting to HMI screen changes through continuous learning. The system captures screenshots during testing, uses them as training data to update the model, and improves its detection capabilities autonomously without requiring manual reconfiguration or extensive retesting for each UI update.
Solution Approach 2:
The system performs preliminary action by pre-training the machine learning model with a diverse set of HMI screen images before actual testing begins. This preliminary training phase enables the model to quickly adapt to new UI elements and layouts, reducing the time required for subsequent testing iterations.
3Measurement precision
If manual testing of HMI screens is performed to detect minute defects, then the detection thoroughness is improved, but the productivity decreases
Solution Approach 1:
The patent replaces manual visual inspection with automated machine learning-based detection. The system uses trained object detection models to identify UI elements and their attributes automatically, achieving both the thoroughness of manual inspection and the speed of automation, thereby increasing testing productivity without sacrificing detection quality.
Solution Approach 2:
The machine learning model acts as an intermediary between the HMI screen and the testing system. It processes images, identifies UI elements, extracts attributes, and provides structured data to the testing framework, enabling automated thorough inspection at high speed without requiring direct human involvement in each detection task.
Data Source
Figure 1
Figure 2
Figure 3
AI summary
A method of performing Human Machine Interface (HMI) automation testing is disclosed. The method includes receiving at least one input image and detecting one or more of User Interface (UI) elements and corresponding graphical attributes of each of the one or more of UI elements in the at least one image using a trained machine learning model. The graphical attributes comprise at least one of an appearance and a placement of each of the one or more of UI elements. Further, the method includes analyzing each of the graphical attributes detected by the trained machine learning model with reference attributes stored in and generating an output image annotated with a validity score for each of the User Interface (UI) elements appearing on the HMI screen. The trained machine learning model is created by iterative training with a set of training images based on a learning score.