Medical Visual Question Answering via Decoupled Multi-Modal Encoding
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Conventional visual question answering (VQA) systems in the healthcare domain face limitations due to small training datasets and the loss of information when concatenating image and textual data, leading to inaccurate answers.
Innovation Solution
A VQA system that combines medical imaging data and medical records into a joint representation using a decoupled encoder and decoder, allowing for accurate object detection and semantic analysis without requiring large-scale image-question-answering data, by employing neural networks for image and text embedding and a multi-modal encoder to generate a contextual representation.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If conventional VQA systems use small training datasets, then the system complexity is reduced, but the answer accuracy deteriorates
Solution Approach 1:
The patent introduces a multi-modal encoder as an intermediary component that processes both image and text data separately before generating answers. This mediator architecture allows the system to leverage medical records and imaging data independently, reducing dependency on large-scale image-question-answering training data while maintaining high answer accuracy through the integrated representation learning capability.
2Loss of information
If image and textual data are concatenated, then the data processing is simplified, but information loss occurs
Solution Approach 1:
The patent segments the data processing into separate modules: an image encoder that processes medical images independently and a text encoder that processes medical records independently. Each encoder extracts features from its respective modality without concatenation, preserving information integrity. The multi-modal encoder then integrates these segmented representations, achieving both information retention and manageable processing complexity through modular architecture.
3Quantity of substance
If a decoupled encoder and decoder architecture is used, then the training data requirement is reduced, but the system complexity increases
Solution Approach 1:
The multi-modal encoder in the patent serves multiple functions: it processes both image and text data independently, extracts domain-specific features from each modality, and generates integrated representations for answer generation. This universal component handles diverse data types through a unified architecture, reducing the need for separate specialized models and thereby reducing training data requirements while managing system complexity through functional consolidation.
Data Source
AI summary
Aspects of the invention include a computer-implemented method including extracting a domain-specific object feature from a first image data, wherein the feature describes an object in the first image data. A domain-specific semantic meaning of text data is determined. The object feature is mapped to a portion of the text data, wherein the portion of the text data describes the object. A joint representation of the object and the portion of the text data is created. A second image data and a query directed towards an object in the second image data is received. An answer to the query is generated based on the joint representation.