Multimodal Learning Model With Explainable Image Classification
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing deep learning models for image classification, such as those used in diagnosing skin lesions, provide accurate results but lack sufficient explainability, making it difficult for users to understand the basis for their determinations.
Innovation Solution
A learning apparatus and program that utilizes a multimodal model combining convolutional neural networks for image processing and neural networks for feature values, along with methods like LIME and Grad-CAM, to calculate and display the degree of contribution of image and feature values to the determination, enhancing explainability.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If deep learning models are used for image classification, then determination accuracy is improved, but explainability deteriorates
Solution Approach 1:
The patent introduces explanation data as an intermediary element that mediates between the deep learning model's determination result and the user's understanding. The explanation data (including contribution degrees and basis images) acts as a bridge that translates the model's internal reasoning into human-interpretable information, thereby maintaining high determination accuracy while improving explainability.
Solution Approach 2:
The patent segments the explanation information into distinct components: contribution degree data (quantifying the importance of each input) and basis image data (visualizing the regions in input images that influenced the determination). This segmentation allows the explanation system to provide structured, multi-faceted interpretability without compromising the model's accurate classification performance.
2Loss of information
If explanation information is provided for deep learning determinations, then interpretability is improved, but system complexity increases
Solution Approach 1:
The deep learning model generates its own explanation information (contribution degrees and basis images) as a byproduct of its determination process. The same model weights and activation values used for accurate classification are reused to create explanations, allowing the system to serve its own interpretability needs without requiring separate explanation-generation components, thereby limiting the increase in system complexity.
Solution Approach 2:
The determination result generation unit serves multiple functions: it produces both the classification determination and the explanation information (contribution degrees and basis images) using the same model parameters. This multi-functionality allows a single component to address both accuracy and interpretability requirements, reducing the need for additional specialized modules.
Data Source
Figure 1
Figure 2
Figure 3
AI summary
A learning apparatus includes: a storage unit which stores a learning model that is trained by setting, as an input, a training image set and a training feature value set that is related to a subject of the training image set and that is obtained by quantifying a predetermined interpretable feature, and by setting, as an output, results of a determination on the training image set and the training feature value set; a determination unit which outputs, by using the learning model stored in the storage unit, results of a determination on an target image and a first feature value that is related to a subject of the target image and that is obtained by quantifying the predetermined interpretable feature; and an explanation output unit which outputs a degree of contribution of the target image and a degree of contribution of the first feature value, for the result of the determination on the target image and the first feature value by the learning model.