CNN Shared Feature Space for Multi-Modal Sensor Data
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing machine learning approaches for neural networks face challenges due to a shortage of training data, high labeling costs, and the need for text labels for speech and vision data, which limits the use of sensor data from IoT devices and medical imaging modalities like PET and MRI, which are not mapped in the same feature space.
Innovation Solution
A computer-implemented method using convolutional neural networks (CNNs) processes audio-visual data sets with assigned weights to generate a shared feature space, allowing direct learning from multiple sensory inputs without manual labeling, and applies a contrastive loss function to simulate co-occurrence and long-term depression, enabling classification and probability outputs.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If manual labeling is used for training data, then data quality and accuracy are improved, but labeling cost and time consumption increase
Solution Approach 1:
The system performs self-labeling by automatically generating labels from sensor data through unsupervised learning and contrastive loss functions, eliminating the need for human annotators to manually label training data while maintaining data quality
Solution Approach 2:
The patent introduces an intermediary labeling mechanism that uses unsupervised learning algorithms and contrastive loss functions as mediators between raw sensor data and labeled training data, automatically generating labels without human intervention
2Measurement precision
If text labels are required for speech and vision data, then learning accuracy is improved, but data processing complexity and storage requirements increase
Solution Approach 1:
The patent extracts the labeling information directly from the sensor data itself through unsupervised learning, removing the need for separate text labels by extracting meaningful patterns and relationships directly from the raw audio-visual data
Solution Approach 2:
The system creates a universal labeling mechanism that works across multiple modalities (audio, visual, sensor data) simultaneously, allowing the same unsupervised learning framework to generate labels for different data types without requiring modality-specific text annotations
3Loss of information
If PET and MRI images are collected separately, then diagnostic information completeness is improved, but data mapping and integration difficulty increase
Solution Approach 1:
The patent merges PET and MRI image data into a unified neural network framework with shared feature spaces, combining multiple imaging modalities into a single integrated system that processes both types of images simultaneously through shared layers and unified loss functions
Solution Approach 2:
The system changes the parameter space by transforming PET and MRI images into a common feature representation through learned mapping functions, converting data from different modalities into a unified parameter space where they can be effectively integrated and compared
Data Source
AI summary
A computer-implemented method includes receiving first inputs associated with a first modality and second inputs associated with a second modality; processing the received first and second inputs with convolutional neural networks (CNN), wherein a set of first weights are used to handle the first inputs and a second set of weights are used to handle the second inputs; determining a loss for each of the first and the second inputs based on a loss function that applies the first set of weights, the second set of weights, and a presence of a co-occurrence; generating a shared feature space as an output of the CNNs, wherein a distance between cells associated with the first inputs and the second inputs in the shared feature space is determined based on the loss associated with each of the first inputs and the second inputs; and based on the shared feature space, providing an output.