Multimodal UI for AI Training Data Annotation

Resolve Bottlenecks,
Find Innovative SolutionsGenerate Solutions

Solution Overview

Problem

Training AI models to associate image information with corresponding text information is impractical due to the scarcity of annotated data, making it difficult to identify objects, operations, and their interrelations in instructional videos.

Innovation Solution

A user interface device and method that allows users to interactively generate training data by associating image information with text information, enabling the creation of annotated image information that can be used to train AI models, by displaying images and text descriptions of objects and operations, and receiving user inputs to link them.

Engineering Contradictions & Design Principles

VSEngineering Contradiction Analysis

1Loss of time

If AI models are trained to identify objects, operations, and processes in instructional videos, then the user can quickly access relevant information without watching the entire video, but the training data annotated with all these aspects does not exist making the task impossible

Engineering Contradiction:

Improvetime to access relevant informationVSAvoidavailability of annotated training data

Core Design Contradiction:

Loss of timeVSEase of manufacture

Solution Approach 1:

The system performs preliminary action by pre-processing instructional videos to automatically extract and annotate objects, operations, and processes before user queries are submitted. This preliminary annotation creates the training data structure needed for rapid information retrieval without requiring manual pre-annotation of all possible video content.

Inventive Principle:

Principle #10Preliminary action

Solution Approach 2:

An intermediary processing layer is introduced between the raw instructional video and the AI model training data. This intermediary automatically generates annotations by analyzing video content, bridging the gap between unannotated video data and the structured training data required for effective AI model training.

Inventive Principle:

Principle #24Intermediary (Mediator)

2Reliability

If manual annotation of instructional videos is performed to create training data, then annotated data can be generated, but the process is tedious and time-consuming

Engineering Contradiction:

Improvequality of annotated training dataVSAvoidtime to create annotated data

Core Design Contradiction:

ReliabilityVSLoss of time

Solution Approach 1:

The system implements self-service by enabling the instructional video content to automatically annotate itself. The video's own visual and auditory information is processed to extract objects, operations, and processes, eliminating the need for external manual annotation while maintaining high data quality and reliability.

Inventive Principle:

Principle #25Self-service

Solution Approach 2:

Manual mechanical annotation processes are replaced with automated computational analysis. AI algorithms and computer vision techniques substitute for human annotators, automatically identifying and labeling objects, operations, and processes in the video content, thereby eliminating the tedious and time-consuming manual workflow.

Inventive Principle:

Principle #28Mechanics substitution (Replace mechanical system)

3Adaptability or versatility

If comprehensive annotation of all objects, operations, and processes is attempted, then complete training data can be created, but the complexity of annotating all aspects makes the task impractical

Engineering Contradiction:

Improvecompleteness of training data coverageVSAvoidcomplexity of annotation process

Core Design Contradiction:

Adaptability or versatilityVSDevice complexity

Solution Approach 1:

The annotation task is segmented into distinct, manageable components: object detection, operation identification, and process recognition. Each component is handled by specialized processing modules that analyze specific aspects of the video independently, reducing overall complexity while maintaining comprehensive coverage of all training data elements.

Inventive Principle:

Principle #1Segmentation

Solution Approach 2:

The system applies partial action by focusing annotation efforts on the most critical and frequently queried aspects of instructional videos. Rather than attempting to annotate every possible detail with equal rigor, the system prioritizes objects, operations, and processes that provide the greatest value for user information retrieval, achieving practical completeness without overwhelming complexity.

Inventive Principle:

Principle #16Partial or excessive action

Data Source

PatentUS11694021B2Apparatus for generating annotated image information using multimodal input data, apparatus for training an artificial intelligence model using annotated image information, and methods thereof

Publication Date: 2023.07.04 SAMSUNG ELECTRONICS CO LTD

AI summary

A method for providing a user interface (UI) for generating training data for an artificial intelligence (AI) model may include providing, for display via the UI, image information that depicts an object, a set of operations of the object, and a process associated with the set of operations. The method may include providing, for display via the UI, text information that describes the object, the set of operations of the object, and the process associated with the set of operations. The method may include receiving, via the UI, a user input that associates respective image information of the image information with corresponding text information of the text information. The method may include generating association information that associates the respective image information with the corresponding text information, based on the user input. The method may include generating discourse and semantic information from the text information associated to the image information.