Fundus Image Annotation Quality Control Using Consistency Screening
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current methods for data annotation on fundus images, particularly in medical imaging, lack accuracy and consistency, necessitating improved quality control to enhance the training of machine learning models for diabetic retinopathy staging and other eye diseases.
Innovation Solution
A quality control method involving standardization, preliminary filtering, and consistency evaluations using self-consistency and gold-standard consistency thresholds to refine annotation results, ensuring accuracy through multiple doctor annotations and arbitration when necessary.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If multiple annotation doctors annotate the same fundus images, then annotation reliability improves, but annotation time and processing complexity increase
Solution Approach 1:
The patent applies preliminary action by performing image quality assessment and filtering before the annotation process. Low-quality images are identified and excluded in advance using automated quality metrics, ensuring that only high-quality images are annotated by multiple doctors. This preliminary filtering step prevents wasting annotation time on images that would not contribute meaningfully to the dataset quality.
Solution Approach 2:
The patent introduces an intermediary quality control mechanism that mediates between multiple annotators. A quality control system calculates agreement metrics (such as inter-rater reliability) and uses these as intermediaries to objectively evaluate annotation consistency. This intermediary system resolves conflicts between different annotators' interpretations and identifies cases requiring re-annotation, thereby managing the complexity of multi-annotator workflows efficiently.
2Reliability
If multiple annotation doctors annotate the same fundus images, then annotation consistency improves, but system complexity increases
Solution Approach 1:
The patent segments the annotation quality control process into distinct modular components: image quality assessment, annotation execution, agreement calculation, and conflict resolution. Each module performs a specific function and can be independently managed. This segmentation reduces system complexity by breaking down the complex multi-annotator process into manageable, standardized steps that can be automated and tracked separately.
Solution Approach 2:
The patent changes parameters related to annotation quality control by introducing quantitative metrics such as inter-rater reliability coefficients and agreement thresholds. Instead of qualitative assessments, the system uses measurable parameters to evaluate consistency. This parameter-based approach transforms subjective consistency evaluation into objective, computable metrics, thereby managing system complexity through standardized quantitative measures.
3Measurement precision
If image quality is strictly filtered before annotation, then annotation accuracy improves, but data quantity decreases
Solution Approach 1:
The patent applies local quality principles by assessing and filtering images based on specific quality dimensions (such as focus, illumination, artifact presence) rather than applying a single global quality threshold. Images are evaluated on their local quality characteristics, and only those meeting minimum standards in critical areas are included. This approach preserves more images in the dataset compared to strict global filtering while still ensuring sufficient local quality for accurate annotation.
Solution Approach 2:
The patent implements partial filtering by applying quality assessment to only the most critical image attributes that directly impact annotation accuracy, rather than filtering on all possible quality dimensions. This partial action approach maintains a larger dataset by being selective about which quality aspects trigger filtering, thereby balancing annotation accuracy requirements with data quantity preservation.
Data Source
Figure 1
Figure 2
Figure 3~4
AI summary
The present disclosure describes a quality control method for data annotation on a fundus image, including acquiring a plurality of fundus images; performing standardization processing on the fundus images so as to obtain a plurality of standardized fundus images; performing preliminary filtering on quality of the standardized fundus images to acquire a plurality of qualified fundus images; preparing a target fundus image set (200); a plurality of first annotation doctors (A) respectively annotating the images of the target fundus image set (200), to acquire a plurality of groups of doctor annotation results (130); calculating, on the basis of the doctor annotation results (130), self-consistency and gold-standard consistency of the corresponding first annotation doctors (A), to acquire the doctor annotation results (130) of the first annotation doctors (A) satisfying a preset condition as target annotation results (140); and gathering a plurality of groups of target annotation results (140) to acquire a final annotation result (150). According to the present disclosure, it is able to provide a quality control method and a quality control system for data annotation on fundus images with high accuracy.