Entropy-Based Sample Selection for Annotation Efficiency
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing annotation techniques require extensive human effort and time for training classifiers, as they focus on increasing annotation accuracy rather than minimizing the number of samples that need human annotation, making it inefficient for large datasets.
Innovation Solution
A computer-implemented method that selects a subset of samples for human annotation based on minimizing the entropy mean difference, using these human-annotated samples to configure an annotation process, and subsequently verifying machine-annotated samples to optimize the annotation process and reduce human effort.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If all digital objects are manually annotated by human annotation, then annotation accuracy is improved, but time consumption and cost increase significantly
Solution Approach 1:
The patent segments the annotation task into two parts: human annotation for a selected subset of digital objects (training set) and machine annotation for the remaining digital objects. This segmentation allows the system to leverage both human accuracy and machine efficiency, resolving the contradiction between annotation accuracy and time consumption.
Solution Approach 2:
The patent performs preliminary human annotation on a carefully selected subset of digital objects to create training data before proceeding with machine annotation. This preliminary action enables the machine learning model to learn from high-quality human-annotated examples and subsequently annotate the remaining digital objects automatically, significantly reducing overall time consumption while maintaining accuracy.
2Loss of time
If a small number of digital objects are selected for human annotation, then time consumption is reduced, but annotation accuracy of the classifier deteriorates
Solution Approach 1:
The patent changes the parameter of sample selection from random or uniform sampling to entropy-based sampling. By selecting digital objects with entropy values closest to the average entropy of the entire dataset, the system ensures that the training set represents the overall data distribution, maintaining classifier accuracy while using fewer human-annotated samples.
Solution Approach 2:
The patent replaces traditional mechanical sampling methods (random sampling, stratified sampling) with an entropy-based selection mechanism. This substitution enables automatic identification of representative samples based on their information content, ensuring that a small subset of human-annotated digital objects can effectively train the classifier for the entire dataset.
3Measurement precision
If existing techniques focus on increasing annotation accuracy of the classifier, then classification performance is improved, but the number of digital objects requiring human annotation is not minimized
Solution Approach 1:
The patent inverts the conventional approach by not focusing on increasing the number of human-annotated samples to improve accuracy, but rather on minimizing human annotation through intelligent selection. The system selects the most representative digital objects for human annotation based on entropy, allowing the majority of digital objects to be annotated by the machine, thereby improving productivity while maintaining classification performance.
Data Source
AI summary
A method for annotating a batch of original samples is provided. A first subset of original samples, selected from the batch and determined by minimizing an entropy-mean difference between the first subset and the batch, is used for human annotation to yield human-annotated samples. The human-annotated samples are used as training data to configure an annotation process for annotating an input sample to yield an annotated output sample, and a check process for verifying annotation accuracy of the annotated output sample. Remaining original samples in the hatch are processed by the annotation process to yield machine-annotated samples, whose accuracy is verified by the check process. In one embodiment, part of the original samples corresponding to erroneous machine-annotated samples are selected for human annotation. Resultant additional human-annotated samples are used to update the two processes. The remaining original samples not yet annotated are then processed by the two processes.