Multimodal Defect Classification with Missing and Unpaired Data
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing multimodal fusion techniques struggle with datasets containing missing and unpaired modalities, leading to inefficiencies and noise introduction when classifying defects in display panels, as they often discard or impute incomplete data, which is wasteful and unreliable.
Innovation Solution
A method involving modality-specific encoders, such as Vision Transformer and Feature-Tokenizer Transformer, impute missing embeddings from paired modalities, and compute Shapley-based explanations to generate confidence values for unimodal and bimodal predictions, ensuring robustness and accuracy in defect classification.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If existing multimodal fusion techniques are used to handle datasets with missing and unpaired modalities, then the classification process can proceed, but the techniques introduce noise and inefficiency by discarding or imputing incomplete data
Solution Approach 1:
The system uses self-supervised learning where the model learns to predict missing modalities from available data without requiring external supervision or manual imputation. The modality-specific encoders and cross-attention mechanisms enable the system to serve itself by generating pseudo-labels and confidence scores from the training data, eliminating the need to discard incomplete samples while maintaining classification accuracy
Solution Approach 2:
The system dynamically adjusts the fusion strategy based on data quality parameters. By computing confidence scores for each prediction and using cross-attention mechanisms, the system adapts its behavior to handle different data completeness scenarios, switching between relying on single-modality predictions and multi-modality fusion based on the reliability of available data
2Reliability
If complete and paired modalities are required for training, then classification accuracy may be maintained, but large portions of available data must be discarded
Solution Approach 1:
The system converts the previously harmful incomplete or unpaired data into a beneficial training resource. By using self-supervised learning, the model treats missing modalities as prediction targets rather than reasons to discard samples. The cross-attention mechanisms and modality-specific encoders transform incomplete data into valuable training signals that improve both classification accuracy and data utilization
Solution Approach 2:
The system creates a universal training framework that handles multiple data scenarios simultaneously - complete paired data, incomplete data with missing modalities, and unpaired data. The same architecture with modality-specific encoders and cross-attention mechanisms works across all these scenarios, making the system multi-functional and eliminating the need for separate processing pipelines
3Adaptability or versatility
If multimodal fusion is performed with available data, then more information can be utilized, but the system becomes vulnerable to missing modalities during testing
Solution Approach 1:
The system prepares for potential missing modalities during testing by training with self-supervised learning that explicitly handles incomplete data. The model learns to generate confidence scores and relies on unimodal encoders as backups, cushioning against the harm of missing modalities before they occur in the testing phase
Solution Approach 2:
The system dynamically adjusts its fusion strategy based on the availability and quality of data. The cross-attention mechanisms and confidence score computation enable the system to flexibly switch between full multimodal fusion and single-modality prediction, adapting to the specific conditions of each test sample rather than following a rigid fusion approach
Data Source
AI summary
A method may include providing a data set including rows of data. The rows of data may include at least one row of unpaired modality including a first modality, and at least one row of paired modality may include both the first modality and a second modality. The method may further include imputing, by a modality-specific encoder, the at least one row of unpaired modality by interpolating embeddings from the second modality of the paired modality; training, in a latent space, the modality-specific encoder based on the imputation for unimodal prediction and bimodal prediction; and generating a confidence value for the unimodal prediction and the bimodal prediction.