Masked Feature Embedding for Robust Multimodal Sensor Fusion
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Machine learning models that rely on multimodal data, such as image and audio data from different sensors, experience performance drops when one data source becomes temporarily unavailable due to interference or other issues.
Innovation Solution
An apparatus and method that involve obtaining and transforming data samples into feature embeddings using machine learning models, generating a global representation by masking one of the embeddings, and training a third machine learning model based on this representation to create a compressed and robust feature embedding, enabling the system to handle missing data effectively.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If multimodal data from multiple sensors is used to improve prediction accuracy, then prediction accuracy is improved, but the system becomes vulnerable to performance drops when sensor data becomes unavailable
Solution Approach 1:
The system performs preliminary actions by training the third machine learning model in advance using masked feature embeddings that simulate missing sensor data. This preparation enables the model to learn robust representations that can handle data unavailability during actual operation, resolving the contradiction between using multimodal data for accuracy and maintaining reliability when sensors fail.
Solution Approach 2:
The patent applies beforehand cushioning by introducing masking operations during training that deliberately remove or hide certain feature embeddings. This creates a protective mechanism that cushions the system against future data loss, allowing the model to maintain stable performance even when actual sensor data becomes unavailable during deployment.
2Reliability
If feature embeddings are masked to simulate missing data during training, then robustness to data loss is improved, but the training process becomes more complex
Solution Approach 1:
The system applies parameter changes by dynamically modifying the input feature embeddings through masking operations during training. By controlling which embeddings are masked and to what extent, the system can adjust the training complexity while still achieving robustness. The masking mechanism allows flexible parameter adjustment to balance training complexity against the desired level of robustness.
3Adaptability or versatility
If a third machine learning model is trained on masked representations to handle missing data, then the model's ability to handle missing data is improved, but computational resources and training time increase
Solution Approach 1:
The patent applies partial action by masking only certain feature embeddings rather than all of them during training. This selective masking approach allows the third machine learning model to learn handling of missing data for specific sensor modalities while still receiving complete information from other modalities, reducing the overall training complexity and time compared to completely random or extensive masking.
Data Source
Figure 1
Figure 2
Figure 3
AI summary
Apparatus comprising means for: obtaining a first data sample and a second data sample; transforming the first data sample into a first feature embedding using a first machine learning model; transforming the second data sample into a second feature embedding using a second machine learning model; and generating a first global representation by masking at least one of: the first feature embedding or the second feature embedding. The apparatus further comprising means for: transforming the first global representation into a third feature embedding using a third machine learning model; and training at least the third machine learning model based on the third feature embedding.