Unified Question Generation System for Multimodal Data

Resolve Bottlenecks,
Find Innovative SolutionsGenerate Solutions

Solution Overview

Problem

Current question generation systems struggle to generate multilingual and multimodal questions across languages and modalities, particularly failing when faced with combinations of tables and passages due to lack of training data, especially in low-resource languages like Chinese, and often refer only to passages or titles instead of table contents.

Innovation Solution

A unified question generation system that employs a unified training strategy across modalities, using a transformer model like mT5 to generate questions in a language from one modality (e.g., tables or passages) without requiring specific training data, enabling cross-lingual and cross-modal transfer by leveraging multilingual and multimodal datasets.

Engineering Contradictions & Design Principles

VSEngineering Contradiction Analysis

1Adaptability or versatility

If current question generation systems are used, then they can generate questions from passages, but they fail to generate questions from tables or combinations of tables and passages, especially in low-resource languages

Engineering Contradiction:

Improvecapability to generate questions across different modalities and languagesVSAvoidperformance when generating questions from tables or multimodal data

Core Design Contradiction:

Adaptability or versatilityVSReliability

Solution Approach 1:

The patent combines multiple question generation models (table-based, passage-based, and multimodal models) into a unified system that can handle different data modalities. The fusion component integrates outputs from various models to generate comprehensive questions from tables, passages, or their combinations, resolving the limitation of individual models being modality-specific.

Inventive Principle:

Principle #5Merging (Combining)

Solution Approach 2:

The system creates a universal question generation framework that can process multiple data types (tables, passages, images) and multiple languages through a single unified architecture. The multilingual capability and modality-agnostic design allow the system to function across diverse scenarios without requiring separate specialized models for each case.

Inventive Principle:

Principle #6Universality (Multi-functionality)

2Adaptability or versatility

If systems are trained with modality-specific data, then they can generate questions for that specific modality, but they cannot generalize to other modalities or languages without additional training data

Engineering Contradiction:

Improvecross-lingual and cross-modal transfer capabilityVSAvoidtraining data requirements for each language-modality combination

Core Design Contradiction:

Adaptability or versatilityVSQuantity of substance

Solution Approach 1:

The system performs preliminary training on available data for each modality-language combination separately, then uses the fusion component to generalize across modalities and languages. This preliminary action allows the system to learn from limited data in each specific case while building transferable knowledge that can be applied to unseen language-modality combinations without requiring extensive training data for each combination.

Inventive Principle:

Principle #10Preliminary action

Solution Approach 2:

The fusion component acts as an intermediary that bridges different modality-specific models and languages. It receives inputs from various specialized models and produces unified question outputs, enabling cross-lingual and cross-modal transfer without requiring direct training data for each target combination. The intermediary facilitates knowledge transfer between different data domains.

Inventive Principle:

Principle #24Intermediary (Mediator)

3Ease of operation

If question generation systems refer only to passages or titles, then they can maintain simplicity, but they fail to utilize table contents for question generation

Engineering Contradiction:

Improvesimplicity of question generation processVSAvoidfailure to utilize table contents

Core Design Contradiction:

Ease of operationVSLoss of information

Solution Approach 1:

The system segments the information source into distinct components (table contents, passage text, titles) and processes each through appropriate specialized models. The table-based model specifically extracts information from table structures, the passage-based model processes textual content, and these segmented processing results are then fused together. This segmentation allows the system to maintain operational simplicity while fully utilizing information from all source types including table contents.

Inventive Principle:

Principle #1Segmentation

Data Source

PatentUS20240386218A1Systems and methods to build oneqg: a unified question generation system across modalities

Publication Date: 2024.11.21 INTERNATIONAL BUSINESS MACHINE CORPORATION

AI summary

One or more systems, devices, computer program products and/or computer-implemented methods of use provided herein relate to building a unified question generation system across languages and modalities. The computer-implemented system can comprise a memory that can store computer executable components. The computer-implemented system can further comprise a processor that can execute the computer executable components stored in the memory, wherein the computer executable components can comprise a training component that can train a unified question generation model to generate questions in a language from a first modality in the language using training data comprising one or more second modalities in the language different from the first modality, wherein the first modality and the one or more second modalities can include at least one of one or more tables, one or more passages, or a combination of the one or more tables and the one or more passages.