Contour-Based Learning Model Evaluation for Incomplete Segmentation Masks
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing learning models for image processing in appearance inspection struggle to accurately evaluate the performance when dealing with subjects of elongated shapes, as incomplete masks can occur, leading to difficulties in assessing their effectiveness using metrics like mean Intersection over Union (mIoU).
Innovation Solution
A method and system that utilize first and second contour information to evaluate the learning model's performance by calculating inter-contour distance information and generating evaluation information based on mean value and standard deviation, allowing for a more comprehensive assessment of the model's accuracy.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If segmentation processing is performed using a learning model to identify pixels of subjects, then classification results can be obtained, but incomplete masks may occur for elongated subjects causing inaccurate performance evaluation
Solution Approach 1:
The evaluation is segmented into multiple components: contour overlap evaluation (comparing contours of predicted masks with ground truth), discontinuity evaluation (detecting breaks in mask contours), and sticking-out evaluation (identifying regions where masks extend beyond ground truth). This multi-faceted segmentation of evaluation metrics allows comprehensive assessment beyond simple area overlap, enabling accurate performance measurement even when masks are incomplete for elongated subjects.
2Ease of manufacture
If mIoU metric is used to evaluate learning model performance based on area overlap, then evaluation can be performed, but it fails to detect incomplete masks that still have sufficient overlap with subjects
Solution Approach 1:
The evaluation transitions from a two-dimensional area overlap metric (mIoU) to incorporate contour-based evaluation in the spatial domain. By comparing contours of predicted masks with ground truth contours and evaluating discontinuities along contour paths, the system adds dimensional depth to the evaluation, detecting incomplete masks that would otherwise be missed by area-based metrics alone.
Solution Approach 2:
Contour information serves as an intermediary between the raw mask pixels and the final performance evaluation. Instead of directly comparing pixel areas, the system extracts and compares contour lines of masks and ground truth, using contour overlap rates and discontinuity detections as intermediate metrics that provide more sensitive feedback on mask completeness while maintaining evaluation simplicity.
3Measurement precision
If contour-based evaluation is performed to detect incomplete masks, then performance evaluation accuracy is improved, but evaluation complexity increases
Solution Approach 1:
The system performs preliminary extraction of contour information from both predicted masks and ground truth before the actual evaluation. By pre-processing and storing contour data (contour coordinates, contour lines) in advance, the system avoids repeated complex calculations during evaluation, reducing computational complexity while maintaining high measurement precision through contour-based metrics.
Data Source
AI summary
A method, a learning model evaluation system, and a program can appropriately evaluate a learning model for identifying a subject to be detected. At least one embodiment of the method includes: by a computer, inputting a subject image in which at least one subject to be detected is imaged to a learning model and acquiring first contour information indicating a contour of the at least one subject to be detected in the subject image; acquiring second contour information indicating a contour of the at least one subject to be detected and serving as a reference for evaluating the first contour information; acquiring a condition to be satisfied by the first contour information; and generating, based on the first contour information, the second contour information, and the condition, evaluation information in which performance of the learning model is evaluated.