Multimodal Feature Mixing for Image Classification

Resolve Bottlenecks,
Find Innovative SolutionsGenerate Solutions

Solution Overview

Problem

Existing image classification methods fail to effectively extract multimodal features by independently comparing feature amounts between image and text data, lacking integration of both modalities.

Innovation Solution

An image processing apparatus that includes an image understanding unit to vectorize image data, a text understanding unit to vectorize text data, and a feature amount mixing unit to project and mix both feature amounts onto the same vector space, generating a mixed feature amount.

Engineering Contradictions & Design Principles

VSEngineering Contradiction Analysis

1Measurement precision

If image feature amounts and text feature amounts are compared independently, then the processing is simple and fast, but multimodal feature amounts cannot be extracted and classification accuracy is limited

Engineering Contradiction:

Improveclassification accuracyVSAvoidfeature amount processing complexity

Core Design Contradiction:

Measurement precisionVSDevice complexity

Solution Approach 1:

The patent merges image feature amounts and text feature amounts by projecting both onto a unified vector space and computing their cosine similarity. This combination enables the extraction of multimodal feature amounts, improving classification accuracy by integrating information from both image and text modalities rather than processing them separately.

Inventive Principle:

Principle #5Merging (Combining)

Solution Approach 2:

The patent introduces a vector space projection mechanism as an intermediary that transforms both image and text features into a common representation space. This intermediary structure enables meaningful comparison and integration of multimodal features while maintaining computational efficiency through the use of cosine similarity metrics.

Inventive Principle:

Principle #24Intermediary (Mediator)

2Reliability

If multiple feature amounts are extracted and integrated, then multimodal information is captured, but the processing time and computational resources increase

Engineering Contradiction:

Improvefeature amount extraction reliabilityVSAvoidprocessing time

Core Design Contradiction:

ReliabilityVSLoss of time

Solution Approach 1:

The patent changes the parameter representation by converting both image and text features into vector space representations with normalized dimensions. This parameter transformation enables efficient comparison through cosine similarity, reducing computational complexity while maintaining the reliability of multimodal feature extraction.

Inventive Principle:

Principle #35Parameter changes

Data Source

PatentUS20250005913A1Image processing apparatus, image processing method, and program

Publication Date: 2025.01.02 NT T INC

AI summary

Therefore, an image processing apparatus according to the present disclosure is an image processing apparatus for extracting a feature amount of image data, the image processing apparatus including: an image understanding unit 41 that vectorizes an image pattern of the image data to extract an image feature amount; a text understanding unit 43 that vectorizes a text pattern of attached text data attached to the image data to extract a text feature amount; and a feature amount mixing unit 44 that generates a mixed feature amount as the feature amount by projecting the image feature amount extracted by the image understanding unit 41 and the text feature amount extracted by the text understanding unit 43 onto the same vector space and mixing the image feature amount and the text feature amount.