Thyroid Nodule Classification Using Adaptive US-IRT Feature Fusion
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing methods for classifying thyroid nodules using deep learning are limited to single-modal US images, lacking comprehensive feature expression and fusion of intra- and inter-modal information, and fail to account for individual differences in patient cases.
Innovation Solution
A multi-modal method combining ultrasound (US) and infrared thermal (IRT) images using hybrid intra-modal encoders (HIMEs) for local feature extraction and adaptive cross-modal encoders (ACMEs) for personalized feature fusion, incorporating transformer-based global modeling and adaptive modality-weight generation.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If single-modal US images are used for classification, then the device complexity is reduced, but the classification accuracy and comprehensive feature expression are insufficient
Solution Approach 1:
The patent combines ultrasound (US) and infrared thermal (IRT) imaging modalities into a unified multi-modal classification system. The US image processing branch and IRT image processing branch are merged at the feature fusion layer, where extracted features from both modalities are integrated to improve classification accuracy while managing system complexity through structured fusion mechanisms.
Solution Approach 2:
The patent employs a multi-functional deep learning architecture that processes both US and IRT images through shared and modality-specific neural network branches. This universal framework enables the system to handle multiple imaging modalities simultaneously, extracting complementary features from each modality to enhance overall diagnostic performance.
2Loss of information
If traditional feature extraction methods are used, then the model complexity is reduced, but the feature expression capability and information interaction between modalities are insufficient
Solution Approach 1:
The patent introduces a feature fusion layer as an intermediary mechanism between the US and IRT image processing branches. This fusion layer acts as a mediator that receives features from both modalities, enables information interaction and integration, and produces fused features that capture complementary information from both imaging modalities for improved classification.
Solution Approach 2:
The patent creates composite feature representations by fusing features from US and IRT modalities. The fused features combine the morphological information from US images with the thermal distribution information from IRT images, creating a composite feature set that leverages the strengths of both modalities for more accurate thyroid nodule classification.
3Adaptability or versatility
If fixed feature fusion methods are used, then the system simplicity is maintained, but the adaptability to individual patient differences is reduced
Solution Approach 1:
The patent implements dynamic and adaptive feature fusion mechanisms that can adjust fusion strategies based on individual patient cases. The system adaptively weights and integrates features from different modalities according to the specific characteristics of each thyroid nodule case, enabling personalized diagnosis while managing complexity through learnable adaptation parameters.
Data Source
AI summary
The present disclosure provides a multi-modal method for classifying a thyroid nodule based on ultrasound (US) and infrared thermal (IRT) images. Based on ultrasound and infrared thermal images and in combination with a multi-modal learning method, the present disclosure provides an adaptive multi-modal hybrid (AmmH) model which is composed of three parts: an intra-modal hybrid encoder (HIME), an adaptive cross-modal encoder (ACME), and a multilayer perceptron (MLP) head. The HIME is capable of modeling a global feature while extracting a local feature. The ACME is capable of customizing personalized modality-weights according to different cases and performing information interaction and fusion of inter-modal features. The MLP head classifies a fused feature obtained. The method enables the AmmH model to automatically classify a thyroid nodule of a subject based on ultrasound and infrared thermal images of the subject, providing a doctor with an objective and accurate classification result to assist diagnosis.