Weak-to-Strong Labeling with Explainable ANN Feedback
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Obtaining high-quality labels for machine learning models, particularly for identifying structural defects in civil engineering structures, is time-intensive and costly, as it relies heavily on manual annotation by subject matter experts.
Innovation Solution
A method that uses a trained artificial neural network (ANN) to generate weak labels, extracts explanatory features through an explainability technique, and prompts user interaction to refine these labels into stronger annotations, iteratively improving the model's performance using a weak supervision learning scheme.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If manual annotation by subject matter experts is used to obtain high-quality labels, then labeling accuracy is improved, but time consumption and cost increase
Solution Approach 1:
The system performs preliminary automated labeling using a pre-trained ML model to generate initial labels before user review. This preliminary action creates a draft annotation that users can then refine, significantly reducing the time users need to spend on each label while maintaining high accuracy through their expert review of the pre-generated labels.
Solution Approach 2:
The patent introduces an automated ML model as an intermediary between raw data and final labels. The model generates preliminary labels that serve as an intermediate product, which then undergoes user review and refinement. This intermediary step reduces the direct time users would otherwise spend on manual annotation while preserving label quality through their expert validation.
2Measurement precision
If manual annotation by subject matter experts is used to obtain high-quality labels, then labeling accuracy is improved, but cost increases
Solution Approach 1:
The system performs preliminary automated labeling using a pre-trained ML model to generate initial labels before user review. This preliminary action creates a draft annotation that users can then refine, significantly reducing the time users need to spend on each label while maintaining high accuracy through their expert review of the pre-generated labels.
Solution Approach 2:
The patent introduces an automated ML model as an intermediary between raw data and final labels. The model generates preliminary labels that serve as an intermediate product, which then undergoes user review and refinement. This intermediary step reduces the direct time users would otherwise spend on manual annotation while preserving label quality through their expert validation.
3Loss of time
If automated ML model is used to generate labels, then time consumption is reduced, but labeling accuracy deteriorates
Solution Approach 1:
The patent merges automated ML model labeling with manual expert annotation into a hybrid system. The automated model provides speed and consistency, while human experts provide accuracy and domain knowledge. The combination of these two approaches in a sequential workflow (automated first, then manual review) achieves both time efficiency and high labeling accuracy.
Solution Approach 2:
The system implements feedback loops where user corrections and validations of automated labels are fed back into the training process. This feedback mechanism allows the ML model to learn from expert annotations, continuously improving its accuracy over time while maintaining the time efficiency of automated generation.
4Measurement precision
If user interaction is required to refine labels, then labeling accuracy is improved, but device complexity increases
Solution Approach 1:
The system implements self-service features where the automated ML model performs the initial heavy lifting of label generation, requiring minimal user intervention. Users only need to review and refine labels when necessary, rather than creating all labels manually. This self-service approach maintains high accuracy through selective user involvement while keeping the system relatively simple to operate.
Data Source
AI summary
A method for generating strong labels for examples labelled with weak labels leverages an artificial neural network, or ANN, which is assumed to have been trained on a training set of examples labelled according to weak labels (e.g., classes of structural defects in images of civil engineering structures). The method processes each example of a set of test examples by performing the following operations. The trained ANN is first executed on each example to infer a weak label. Then, the method extracts explanatory features from the ANN as executed on the example. The method generates a strong label (e.g., a region boundary of the structural defect), based on the extracted explanatory features. The method subsequently prompts a user to react to one or each of the inferred weak label and the generated strong label. The response obtained is then interpreted by the method to obtain a further weak label.