fMRI EEG Image Reconstruction via Dimension-Specific Attention
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing neural network models are unable to effectively extract information from EEG and fMRI data, resulting in low image quality when inferring objects seen by individuals.
Innovation Solution
A method that involves extracting feature maps from fMRI and EEG data, converting these feature maps using attention models based on their respective temporal and spatial dimensions, and then generating an image based on the converted feature maps.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Manufacturing precision
If existing neural network models are used to extract information from EEG and fMRI data, then the processing is simple, but the information extraction effectiveness is insufficient resulting in low image quality
Solution Approach 1:
The patent divides the processing of EEG and fMRI data into separate feature extraction pathways. EEG data undergoes spatial dimension processing while fMRI data undergoes temporal dimension processing. This segmentation allows each modality to be optimized independently with appropriate attention mechanisms, improving overall image quality without requiring a monolithic complex model.
Solution Approach 2:
The patent introduces dimension-specific attention mechanisms: spatial attention for EEG data (exploiting electrode spatial relationships) and temporal attention for fMRI data (exploiting time-series relationships). By operating in different dimensional spaces appropriate to each data type, the model extracts more effective features, improving image reconstruction quality.
2Measurement precision
If correlation information across time and space is extracted by combining fMRI and EEG data, then the inference accuracy improves, but the processing complexity increases
Solution Approach 1:
The patent merges EEG and fMRI feature extraction pathways at the feature map level, combining spatial features from EEG with temporal features from fMRI. This fusion allows the model to leverage complementary information from both modalities, improving inference accuracy about visual content while maintaining modular processing that controls complexity.
Solution Approach 2:
The patent uses feature maps as intermediary representations between raw EEG/fMRI data and final image reconstruction. These feature maps serve as standardized intermediaries that capture modality-specific characteristics (spatial for EEG, temporal for fMRI) while enabling subsequent integration and reconstruction processes.
Data Source
AI summary
In an illustrative embodiment, a method is disclosed for generating an image from fMRI data and EEG data. The method includes extracting a first feature map of the fMRI data, the first feature map being multidimensional data having a temporal dimension related to a sample collection time of the fMRI data, and converting the first feature map based on the temporal dimension of the first feature map and a first attention model. The method further includes extracting a second feature map of the EEG data, the second feature map being multidimensional data having a spatial dimension related to an electrode for collecting the EEG data, and converting the second feature map based on the spatial dimension of the second feature map and a second attention model. The method further includes generating an image based on the converted first feature map and the converted second feature map.