Cross-Modal Learning with Segment Embedding for Sparse Pair Data
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing cross-modal task estimation methods suffer from accuracy deterioration when insufficient multimodal pair data is available, and tasks cannot be estimated from single-modal data due to limited data collection and modal defects.
Innovation Solution
A learning device that extracts encoding features with a time series direction from both monomodal and multimodal data, embeds segment information to identify modal types, connects these features in a time series, and calculates model parameters using embedded features to estimate cross-modal tasks.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If multimodal pair data is used for cross-modal task estimation, then task estimation accuracy is improved, but data collection difficulty increases and system complexity increases
Solution Approach 1:
The patent segments the input data into different modal components (image modal, audio modal, text modal) and processes them through separate extraction units before integrating them. This segmentation allows the system to handle multimodal data systematically while maintaining the ability to process individual modal data independently, thus reducing overall system complexity while preserving accuracy.
Solution Approach 2:
The patent creates a universal processing framework where the same neural network model can handle both multimodal pair data and single modal data through a unified architecture. The model accepts various input types (multimodal or monomodal) and produces consistent cross-modal task estimation outputs, eliminating the need for separate processing systems for different data types.
2Measurement precision
If only multimodal pair data is processed, then cross-modal task estimation is accurate, but the system cannot handle modal defects or limited data availability
Solution Approach 1:
The patent implements a dynamic processing mechanism where the system automatically adapts its behavior based on the availability and quality of input data. When multimodal data is available, it processes both modalities; when modal defects occur or data is limited, it dynamically switches to processing only the available modal data, maintaining task estimation capability across varying data conditions.
Solution Approach 2:
The patent introduces an intermediary processing layer that handles the integration and alignment of different modal data. This intermediary mechanism mediates between the raw input data and the final task estimation, allowing it to gracefully handle missing or defective modal data by using available modal information through the intermediary's coordination and selection capabilities.
3Measurement precision
If a large amount of multimodal pair data is collected, then learning accuracy is improved, but data collection becomes difficult and time-consuming
Solution Approach 1:
The patent enables the system to achieve effective learning results using partial multimodal data rather than requiring complete and extensive multimodal pair data. The neural network can be trained and perform accurately even when only a subset of available modal data is used, significantly reducing the data collection time and effort required while maintaining acceptable learning accuracy.
Data Source
AI summary
A learning device includes processing circuitry configured to extract an encoding feature having a time series direction on a basis of input data of one or both of monomodal data that is data of a single modal or multimodal pair data including a plurality of different modals, embed segment information that is information for identifying a type of the modal of the input data in the encoding feature on a basis of a predetermined condition, connect, on a basis of input condition of a segment-embedded feature in which the segment information is embedded, a plurality of segment-embedded features in the time series direction as a modal-connected feature, and calculate a model parameter using an estimated vector of a cross-modal task estimated on a basis of one or both of the segment-embedded feature or the modal-connected feature and correct data.