Crowdsourced Image Annotation Segregation and Grouping
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
The challenge in crowdsourcing for image annotation is the inefficiency and unreliability of human annotations due to diverse annotator quality, leading to noisy data, and the complexity of evaluating annotation reliability in generating high-quality training databases for machine learning models.
Innovation Solution
A method that segregates candidate annotations into groups based on overlap, selects groups with a sufficient number of annotations, and builds a final annotation as a combination of overlapping regions, using a Jaccard index and concentricity criteria, to create a reliable and accurate ground truth for image annotation, facilitating the generation of a training database for machine learning.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If crowdsourcing is used to annotate images, then the cost and time required for annotation is reduced, but the quality and reliability of annotations deteriorates due to diverse annotator expertise
Solution Approach 1:
The patent segments annotations into groups based on spatial overlap and concentricity relationships. By dividing the set of candidate annotations into distinct groups (overlapping group, concentric group, non-overlapping group), the system can process and evaluate annotations in a structured manner, enabling quality filtering while maintaining high throughput from crowdsourced data
Solution Approach 2:
The patent introduces an intermediary evaluation mechanism that uses geometric relationships (Jaccard index for overlapping, concentricity checks) as mediators between candidate annotations and final ground truth. This intermediary layer filters and selects reliable annotations without requiring direct human verification, thus maintaining crowdworker productivity while ensuring annotation quality
2Adaptability or versatility
If multiple annotators work on the same data to aggregate annotations, then the coverage and diversity of annotations is improved, but the complexity of the aggregation model increases
Solution Approach 1:
The patent segments the complex aggregation problem into three distinct groups based on geometric relationships between annotations. This segmentation simplifies the evaluation process by providing clear, computationally simple criteria for each group (overlapping threshold, concentricity check, non-overlapping condition) rather than requiring a single complex Bayesian model
Solution Approach 2:
The patent changes the parameters for annotation aggregation from complex probabilistic models to simple geometric metrics. By using the Jaccard index for overlapping measurements and concentricity ratios, the system achieves adaptable annotation aggregation with minimal computational complexity, making it suitable for large-scale crowdsourced data processing
3Measurement precision
If manual annotation is performed to ensure high quality, then the accuracy of training data is improved, but the time and resource consumption increases
Solution Approach 1:
The patent enables the annotation system to self-evaluate and self-filter by using geometric relationships between annotations as automatic validation criteria. The overlapping and concentricity checks provide self-service quality control that eliminates the need for time-consuming manual verification while maintaining high training data accuracy
Solution Approach 2:
The patent replaces manual mechanical evaluation with automated computational geometry checks. By substituting human verification with algorithmic assessment of annotation relationships (Jaccard index calculations, concentricity determinations), the system achieves rapid, consistent quality filtering that scales efficiently without time loss
Data Source
AI summary
The present invention relates to a method for processing a plurality of candidate annotations of a given instance of an image, each candidate annotation being defined as a closed shape matching the instance, characterized in that the method comprises performing, by a processing unit (21) of a server (2), steps of:(a) segregating said candidate annotations into a set of separate groups of at least overlapping candidate annotations;(b) selecting a subset of said groups as a function of the number of candidate annotations in each group;(c) building a final annotation of the given instance of said image as a combination of regions of the candidate annotations of said selected groups where at least a second predetermined number of the candidate annotations of said selected groups overlap.