Image Annotation QA with Full-Coverage and Sample Inspection

Resolve Bottlenecks,
Find Innovative SolutionsGenerate Solutions

Solution Overview

Problem

Existing annotation quality assurance processes for image datasets in AI training only inspect a small sample size, missing critical errors that significantly impact AI model performance and user trust.

Innovation Solution

Implement a hybrid quality assurance process combining a quicker inspection of the entire dataset for obvious errors and a more detailed inspection of a sample, using thumbnail and full-resolution images to identify and correct critical errors.

Engineering Contradictions & Design Principles

VSEngineering Contradiction Analysis

1Measurement precision

If a detailed review of a small sample size (2-3%) is performed, then the inspection thoroughness is improved, but the coverage of the entire dataset deteriorates

Engineering Contradiction:

Improveinspection thoroughnessVSAvoiddataset coverage

Core Design Contradiction:

Measurement precisionVSQuantity of substance

Solution Approach 1:

The quality assurance process is segmented into two distinct inspection workflows: a quick inspection workflow that covers 100% of the dataset for obvious errors, and a detailed inspection workflow that covers 2-3% of the dataset for comprehensive error detection. This segmentation allows each workflow to be optimized for its specific purpose without compromise

Inventive Principle:

Principle #1Segmentation

Solution Approach 2:

The quick inspection workflow performs a superficial but comprehensive review of all images to catch obvious errors, while the detailed inspection workflow performs a thorough but partial review of a sample. The combination ensures both complete coverage and deep inspection

Inventive Principle:

Principle #16Partial or excessive action

2Quantity of substance

If a quick inspection of the entire dataset is performed, then the dataset coverage is improved, but the inspection thoroughness deteriorates

Engineering Contradiction:

Improvedataset coverageVSAvoidinspection thoroughness

Core Design Contradiction:

Quantity of substanceVSMeasurement precision

Solution Approach 1:

Inventive Principle:

Principle #1Segmentation

Solution Approach 2:

The two inspection workflows operate in parallel or sequence to provide continuous quality assurance coverage. The quick inspection continuously monitors all images for obvious errors, while the detailed inspection periodically performs deep dives into samples, ensuring uninterrupted quality monitoring

Inventive Principle:

Principle #20Continuity of useful action

3Reliability

If only a detailed review of a small sample is conducted, then critical errors are more likely to be detected, but the time and cost required increase

Engineering Contradiction:

Improveerror detection capabilityVSAvoidinspection time

Core Design Contradiction:

ReliabilityVSLoss of time

Solution Approach 1:

The detailed inspection workflow extracts and focuses only on a representative sample (2-3%) of the dataset that is most likely to contain critical errors. This sampling approach concentrates inspection resources on high-value targets while avoiding unnecessary review of obviously correct images

Inventive Principle:

Principle #2Taking out (Extraction)

Solution Approach 2:

Different inspection depths are applied to different portions of the dataset: obvious errors are detected across all images through quick inspection, while critical errors are detected through deep inspection of sampled images. Each region of the dataset receives the appropriate level of inspection quality

Inventive Principle:

Principle #3Local quality

4Productivity

If only a quick inspection is performed, then the inspection speed is improved, but the ability to detect critical errors deteriorates

Engineering Contradiction:

Improveinspection speedVSAvoiderror detection capability

Core Design Contradiction:

ProductivityVSReliability

Solution Approach 1:

The detailed inspection of sampled images acts as an intermediary that validates and enhances the findings of the quick inspection. By examining a representative sample in depth, the system confirms that the quick inspection parameters are appropriately detecting errors and identifies any critical errors that may have been missed

Inventive Principle:

Principle #24Intermediary (Mediator)

Data Source

PatentUS12573016B2Methods and non-transitory computer readable storage medium for image annotation quality assurance

Publication Date: 2026.03.10 WOVEN BY TOYOTA INC

AI summary

A method for inspecting image annotation quality in an annotated image dataset, the method including: obtaining an annotated image dataset; implementing a first inspection of X % of the annotated image dataset, the first inspection including at least one of first categories of annotation inspections; implementing a second inspection of a sample of the annotated image dataset, the second inspection including at least one of second categories of annotation inspections; and identifying errors in the annotated image dataset using the first inspection and the second inspection.