Confusion Matrix Review for Defect Classification Retraining
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Domain experts face challenges in evaluating and improving defect recognition algorithms due to their complexity, as they lack direct interaction with the 'black box' nature of these algorithms, making it difficult to assess and correct misclassifications during ongoing operations.
Innovation Solution
A method and system that visually presents miniature images of sample data within a confusion matrix, allowing domain experts to interactively adjust classifications by assigning images to different segments based on criteria, thereby providing training data for the algorithm.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If a defect recognition algorithm is used to classify samples automatically, then productivity is improved, but the reliability of classification assessment deteriorates because domain experts cannot easily evaluate the 'black box' algorithm
Solution Approach 1:
The patent introduces an intermediary visualization layer between the black box algorithm and domain experts. This layer includes confusion matrix visualizations, class distribution graphs, and sample image presentations that translate algorithmic outputs into interpretable formats, enabling experts to assess classification quality without needing to understand the underlying algorithm complexity
Solution Approach 2:
The system implements feedback mechanisms by allowing domain experts to review classified samples, provide corrections, and vote on classification quality. These feedback loops are used to continuously improve the algorithm through retraining with corrected labels, creating a closed-loop system that improves reliability over time
2Reliability
If domain experts manually evaluate each sample classification, then reliability of assessment is improved, but productivity deteriorates due to the time-consuming nature of manual review
Solution Approach 1:
Instead of requiring experts to review every single classification, the system applies partial action by selectively presenting only the most problematic cases (e.g., low-confidence predictions, misclassified samples, or samples from specific classes) for expert review. This reduces the evaluation workload while maintaining reliable assessment of critical areas
Solution Approach 2:
The evaluation process is segmented into different levels: automated algorithmic classification handles the bulk of processing, while human experts focus only on specific segments of the data (e.g., boundary cases, rare defects, or high-value samples). This segmentation allows experts to maintain high reliability without reviewing all samples individually
3Productivity
If classification algorithms are trained without domain expert involvement, then productivity is improved, but reliability deteriorates because the algorithm lacks domain-specific knowledge
Solution Approach 1:
The system enables domain experts to participate in the training process through self-service mechanisms such as voting systems, correction interfaces, and feedback forms. Experts can review algorithmic training data, correct misclassifications, and provide feedback that automatically updates the training set, allowing them to contribute domain knowledge without manual intervention in every training step
Solution Approach 2:
The training process incorporates continuous feedback loops where domain experts review algorithm performance on validation sets, provide corrections to misclassified samples, and vote on class assignments. This feedback is fed back into the training data, allowing the algorithm to learn from expert knowledge and improve domain-specific accuracy over time
Data Source
AI summary
A method for checking samples for defects is provided, in which image data of the samples are recorded and classified into predeterminable defect categories by a defect detection algorithm, and the samples classified into a defect category are represented in a multi-dimensional confusion matrix as a classification result of the defect detection algorithm, characterized in that miniature images which reproduce the image data are assigned according to the classified defect categories of the image data to segments of the confusion matrix which represent the defect categories, and these miniature images are displayed visually, the miniature image is assigned by an interaction with a user or a software robot to a different segment from the assigned segment of the confusion matrix, and is either provided as training image data for the defect detection algorithm or is output as training image data for the defect detection algorithm.