Learning Model Contour Evaluation for Incomplete Elongated Masks
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing learning models for image processing in appearance inspection struggle to accurately evaluate performance when dealing with elongated subjects, as incomplete masks can occur, leading to difficulties in assessing their effectiveness using metrics like mIoU.
Innovation Solution
A method and system that evaluates learning models by comparing first and second contour information, calculating inter-contour distances, and using mean values and standard deviations to assess model performance, allowing for appropriate evaluation even with incomplete masks.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If segmentation processing is performed using a learning model to identify pixels belonging to a subject, then the subject can be detected and classified, but the mask may stick out from or become discontinuous for elongated subjects, making it difficult to appropriately evaluate the learning model's performance
Solution Approach 1:
The patent segments the evaluation into two independent parts: contour information extraction and performance evaluation. The contour information is extracted separately from the mask, allowing the evaluation to focus on contour accuracy rather than being affected by mask completeness issues. This segmentation enables appropriate evaluation even when masks are incomplete or discontinuous for elongated subjects.
Solution Approach 2:
The patent extracts contour information from the mask and uses it as a separate evaluation criterion. By taking out the contour information from the mask and using it independently, the evaluation can assess the learning model's performance based on contour accuracy rather than being compromised by mask stick-out or discontinuity problems.
2Productivity
If mean Intersection over Union (mIoU) is used as the evaluation metric, then the overall area overlap can be measured, but the metric is unlikely to largely degrade even when part of an elongated object is missing, making it insufficient for evaluating contour accuracy
Solution Approach 1:
The patent segments the evaluation into two independent parts: contour information extraction and performance evaluation. The contour information is extracted separately from the mask, allowing the evaluation to focus on contour accuracy rather than being affected by mask completeness issues. This segmentation enables appropriate evaluation even when masks are incomplete or discontinuous for elongated subjects.
Solution Approach 2:
The patent introduces local quality evaluation by measuring the distance between contours at specific locations. Instead of using a global metric like mIoU that averages over the entire area, the patent evaluates local contour accuracy by calculating distances at corresponding points between the predicted and actual contours, providing precise measurement of contour quality.
3Measurement precision
If contour information is extracted and evaluated separately from the mask, then the learning model can be appropriately evaluated based on contour accuracy, but additional processing steps are required to extract and compare contour information
Solution Approach 1:
The patent performs preliminary action by extracting contour information from the mask before the evaluation process. The contour information is extracted in advance and stored, so that during evaluation, the system can directly compare the extracted contour information with the ground truth contour information without performing complex real-time processing. This preliminary extraction simplifies the overall evaluation process.
Solution Approach 2:
The patent creates a copy of the contour information from the predicted mask and compares it with the ground truth contour information. By copying the contour information and storing it for later comparison, the evaluation process becomes more straightforward, as the system only needs to compare the copied contour information with the reference rather than performing complex real-time analysis.
Data Source
Figure 1
Figure 2
Figure 3(a)~3(d)
AI summary
Provided are a method, a learning model evaluation system, and a program that can appropriately evaluate a learning model for identifying a subject to be detected. A method includes: by a computer, inputting a subject image in which at least one subject to be detected is imaged to a learning model and acquiring first contour information indicating a contour of the at least one subject to be detected in the subject image; acquiring second contour information indicating a contour of the at least one subject to be detected and serving as a reference for evaluating the first contour information; acquiring a condition to be satisfied by the first contour information; and generating, based on the first contour information, the second contour information, and the condition, evaluation information in which performance of the learning model is evaluated.