Digital Pathology Consensus Labeling to Resolve Annotation Conflicts
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing techniques for generating ground-truth labels in digital pathology images are challenged by inconsistent and conflicting annotations from multiple pathologists, leading to inaccurate training of machine-learning models, which affects their classification accuracy.
Innovation Solution
An annotation-processing system determines consensus locations and labels by resolving annotation-location and annotation-label conflicts using statistical values and weighted values based on annotator performance, generating accurate ground-truth labels for training machine-learning models.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Quantity of substance
If multiple pathologists provide annotations for training images, then the quantity of training data increases, but annotation conflicts and inconsistencies increase
Solution Approach 1:
The patent introduces an intermediary consensus determination mechanism that mediates between multiple pathologists' annotations. The system processes conflicting annotations through statistical analysis and consensus algorithms to generate unified ground-truth labels, thereby maintaining reliability while accepting multiple annotators' input.
Solution Approach 2:
The system implements feedback loops where annotation conflicts are detected, analyzed, and resolved through iterative consensus determination. The process continuously refines annotations by comparing multiple pathologists' inputs and adjusting labels based on agreement levels, ensuring consistent output despite diverse initial annotations.
2Adaptability or versatility
If multiple experts annotate various training images, then the coverage of training data improves, but annotation conflicts propagate across images
Solution Approach 1:
The patent segments the annotation process into independent image-level processing units. Each training image is processed separately through the consensus determination system, allowing conflicts to be resolved locally without propagating errors across the entire dataset. This segmentation maintains overall data coverage while ensuring local accuracy.
Solution Approach 2:
The system dynamically adjusts annotation parameters based on consensus levels. When conflicts arise, the system modifies labeling parameters and re-evaluates annotations until consensus is reached, thereby maintaining accuracy across diverse training images with varying levels of annotator agreement.
3Adaptability or versatility
If conflicting ground-truth labels are used for training, then the diversity of training scenarios increases, but machine-learning model performance decreases
Solution Approach 1:
The patent converts the harmful effect of annotation conflicts into a beneficial process for improving model training. By systematically resolving conflicts through consensus determination, the system transforms diverse and conflicting annotations into refined, high-quality ground-truth labels that enhance both training scenario diversity and classification accuracy.
Data Source
AI summary
The present disclosure relates to systems and methods for determining a consensus location and label for a set of annotations associated with an object or region within an image. An annotation-processing system can access a plurality of annotations associated with the image depicting at least part of a biological sample. The annotation-processing system can determine a consensus location for a set of annotations that are positioned in different locations within a region of the image. At the determined consensus location, a consensus label can be determined for the set of annotations that identify different targeted types of biological structures. The consensus labels across different locations can be used to generate ground-truth labels for the image. The ground-truth labels can be used to train a machine-learning model configured to predict different types of biological structures in digital pathology images.