Multi-level classification method and system based on multi-modal data fusion
By employing multimodal data fusion and multi-level classification methods, this study addresses the shortcomings in the scalability and applicability of existing multimodal data classification methods in complex scenarios. It achieves progressive optimization of classification from coarse-grained to fine-grained, thereby improving classification accuracy and applicability.
Patent Information
- Application Number
- PCT/CN2025/099692
- Authority / Receiving Office
- WO · WO
- Patent Type
- Applications
- Current Assignee / Owner
- Priority Date
- 2025-05-08
- Filing Date
- 2025-06-06
- Publication Date
- 2026-11-12
AI Technical Summary
Existing multimodal data classification methods struggle to effectively combine multimodal data with hierarchical structures, resulting in poor scalability and applicability of classification systems in complex scenarios, and an inability to meet the needs of hierarchical classification.
This paper adopts a multi-level classification method based on multimodal data fusion, including preprocessing, feature extraction and vectorization of image, text and tabular data, and uses self-attention mechanism for feature fusion. The multi-level classification model is then constructed and optimized step by step.
It enables more accurate and detailed classification in complex multi-level scenarios, meeting the needs of practical applications for fine classification and improving classification accuracy and applicability.
Smart Images

Figure CN2025099692_12112026_PF_FP_ABST
Abstract
Description
A multi-level classification method and system based on multimodal data fusion Technical Field
[0001] This invention relates to the field of big data analysis and intelligent information processing, specifically to a multi-level classification method and system based on multimodal data fusion. Background Technology
[0002] Data classification in complex scenarios is one of the core challenges in many application fields. For example, in commercial, industrial, and medical research, the complexity of massive amounts of data and the diversity of application scenarios often make traditional single-modal and single-level classification methods difficult to apply. Different data modalities often contain independent feature information, which, when analyzed individually, cannot fully demonstrate their potential in specific problems. Traditional multimodal classification models usually ignore the hierarchical relationships between categories, treating all categories as equal and discrete individuals, making it difficult to effectively utilize the complementarity and hierarchical structure information between modalities. Furthermore, different modalities contribute differently to each level of classification, with some modalities providing crucial information for specific levels. Therefore, research combining multimodal data fusion and multi-level classification techniques has gradually attracted attention. By jointly modeling and fusing multimodal features, the correlation information between modalities can be fully explored in classification tasks, while multi-level classification techniques can progressively optimize classification results, gradually improving classification accuracy from coarse-grained to fine-grained. This combination of multimodal fusion and multi-level classification provides a new approach to solving data classification problems in complex scenarios, such as achieving fine classification of goods in e-commerce, completing accurate disease typing in the medical field, or optimizing abnormal behavior detection in traffic monitoring.
[0003] Existing technical solutions, such as [Application Publication No.: CN112685565A, Invention Title: Text Classification Method and Related Device Based on Multimodal Information Fusion], propose text classification schemes based on image and text modality fusion features. However, their feature modality extraction is singular, only including image and text modalities. Furthermore, the text classification module is simple and fails to optimize classification for hierarchical data structures, only handling simple classification scenarios at a single level. Meanwhile, another technical solution [Application Publication No.: CN117056863A, Invention Title: A Big Data Processing Method Based on Multimodal Data Fusion] only proposes a multimodal data fusion model. Although it uses multimodal data, it fails to propose technical solutions for complex hierarchical classification scenarios based on multimodal data.
[0004] Most existing multimodal data classification methods are designed for specific single classification tasks. These models typically perform a one-time classification of the data, dividing the results into coarse-grained categories, without integrating multimodal data with a hierarchical structure. This makes it difficult for the classification system to effectively adapt to hierarchical requirements, resulting in poor scalability and applicability, and making it unsuitable for complex classification tasks. For example, existing techniques may perform well in single-level scenarios, but they are significantly inadequate in real-world scenarios requiring hierarchical classification (such as from food classification to specific brand and ingredient classification). Summary of the Invention
[0005] The purpose of this invention is to address the shortcomings of single classification of multimodal data by proposing a multi-level classification method and system based on multimodal data fusion, which aims to more comprehensively handle complex classification tasks in multimodal data scenarios through a hierarchical classification model.
[0006] The objective of this invention is achieved through the following technical solution: a multi-level classification method based on multimodal data fusion, comprising the following steps:
[0007] Acquire image, text, and table data to be classified, and preprocess the image, text, and table data to obtain preprocessed image data, text data, and table features;
[0008] The preprocessed image data is input into a pre-trained deep residual network model for feature extraction and structured vector representation to obtain vectorized image features; the preprocessed text data is input into a pre-trained language model for feature extraction and structured vector representation to obtain vectorized text features.
[0009] Based on the self-attention mechanism, image features, text features, and table features are fused to obtain fused features;
[0010] Construct a hierarchical tree structure, build a multi-level classification model based on the hierarchical tree, and input the fused features into the multi-level classification model for hierarchical classification training;
[0011] The test set portion of the multimodal fusion data is input into the trained multi-level classification model for testing and evaluation, and the classification results are output.
[0012] Furthermore, the preprocessing of the image, text, and table data includes: image enhancement, resizing, and normalization of the image data; removal of stop words, word segmentation, and truncation and padding of the text data; and handling of missing values, outliers, and standardization of the table data.
[0013] Furthermore, the preprocessed image data is input into a pre-trained deep residual network model for feature extraction and structured vector representation to obtain vectorized image features, including:
[0014] The ResNet50 model is selected. The preprocessed image data is input into the ResNet50 model. The global average pooling layer of the ResNet50 model outputs a 1×2048 feature vector. The feature vector contains the high-dimensional feature information of the image to represent the visual content of the image.
[0015] Furthermore, the step of inputting the preprocessed text data into a pre-trained language model for feature extraction and structured vector representation to obtain vectorized text features includes:
[0016] The BERT model is selected. The preprocessed text data is input into the BERT model. The token of each word unit is obtained through the BERT model. Each token is mapped to a vector representation in a high-dimensional space. The pooling layer in the BERT model is used to average and summarize all word embeddings to obtain a 768-dimensional numerical feature vector representation of the entire text.
[0017] Furthermore, the self-attention mechanism-based feature fusion of image features, text features, and table features yields fused features including:
[0018] Image features, text features, and table features are concatenated to obtain a concatenated feature vector. Different weights are automatically assigned to different features based on a self-attention mechanism. According to the weights, a preliminary fused feature is obtained by weighting and then input into a fully connected layer for dimensionality compression to obtain the final fused feature.
[0019] Furthermore, the step of constructing a multi-level classification model based on the hierarchical tree and inputting the fused features into the multi-level classification model for hierarchical classification training includes:
[0020] The multi-level classification model includes constructing a local classifier for each parent node, setting logistic regression as the local base classifier for each parent node; and representing the hierarchical labels as a true label matrix, with the first column being the parent node label and the second column being the corresponding child node label.
[0021] The dataset is split using the StratifiedShuffleSplit stratified sampler to ensure a balanced ratio of training and testing samples across different categories and levels. The training set is input into the logistic regression classifier corresponding to each parent node and trained from top to bottom. The corresponding column in the label matrix is used as the training objective. The regularization strength of the logistic regression classifier for each parent node is optimized using grid search to obtain the optimal local classifier trained for each parent node.
[0022] Furthermore, the step of inputting the test set portion of the multimodal fusion data into the trained multi-level classification model for testing and evaluation, and outputting classification results, includes:
[0023] The split test set is input into the trained multi-level classification model. Based on each local optimal logistic regression classifier, the predicted label of the node is output to obtain the predicted label matrix. The node predicted label is output as the classification result.
[0024] The classification results are evaluated using hierarchical classification metrics, namely hierarchical precision, hierarchical recall, and hierarchical F1 score, and the evaluation results are output.
[0025] This invention also provides a multi-level classification system based on multimodal data fusion, comprising:
[0026] A multimodal data preprocessing module is used for cleaning image, text, and tabular data;
[0027] The multimodal feature extraction and vectorization module is used to extract features and represent structured vectors for images and text, respectively.
[0028] The self-attention mechanism multimodal feature fusion module is used to fuse image features, text features, and table features;
[0029] The multi-level classification framework construction and training module is used to construct a multi-level classification framework and perform hierarchical classification training.
[0030] The model testing and evaluation module is used to test and evaluate the results of multi-level classification and display the output.
[0031] The present invention also provides an electronic device, including a memory and a processor, wherein the memory is coupled to the processor; wherein the memory is used to store program data, and the processor is used to execute the program data to implement the multi-level classification method based on multimodal data fusion.
[0032] The present invention also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the multi-level classification method based on multimodal data fusion.
[0033] The beneficial effects of this invention are as follows: Addressing the limitation that current multimodal fusion data classification is only applicable to simple classification tasks at a single level, this invention, by combining feature fusion of multimodal data with a hierarchical classification model that is progressively optimized, can provide a more accurate and refined classification scheme for complex multi-level classification scenarios, based on fully mining and fusing information from various modalities. It gradually refines the classification results based on coarse-grained classification, thereby meeting the needs for fine-grained classification in practical applications. Attached Figure Description
[0034] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.
[0035] Figure 1 is a system flowchart of the present invention;
[0036] Figure 2 is a schematic diagram of image data feature extraction and vectorization according to an embodiment of the present invention;
[0037] Figure 3 is a schematic diagram of text data feature extraction and vectorization according to an embodiment of the present invention;
[0038] Figure 4 is a schematic diagram of multimodal feature fusion using the self-attention mechanism in an embodiment of the present invention;
[0039] Figure 5 is a schematic diagram of a multi-level structure tree for fever-related diseases according to an embodiment of the present invention. Detailed Implementation
[0040] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numerals in different drawings denote the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the present invention. Rather, they are merely examples of apparatuses and methods consistent with some aspects of the invention as detailed in the appended claims.
[0041] As shown in Figure 1, this embodiment of the invention provides a multi-level classification system based on multimodal data fusion, including:
[0042] The multimodal data preprocessing module is used for cleaning image, text, and tabular data.
[0043] The multimodal feature extraction and vectorization module is used to extract features and perform structured vector representations for images, text, and tables, respectively.
[0044] The self-attention mechanism multimodal feature fusion module is used to fuse image features, text features, and table features.
[0045] The multi-level classification framework construction and training module is used to build multi-level classification frameworks and perform hierarchical classification training and testing.
[0046] The model testing and evaluation module is used to test and evaluate the results of multi-level classification and display the output.
[0047] Based on the system described above, this embodiment of the invention also provides a multi-level classification method based on multimodal data fusion, comprising the following steps:
[0048] Step 1: Preprocess image, text, and tabular data using the multimodal data preprocessing module.
[0049] This invention takes the diagnosis of fever with rash as a specific example. The preprocessing process specifically includes image enhancement, size adjustment and normalization of image data, removal of stop words, word segmentation and truncation and padding of text data, and processing of missing values, outliers and standardization of tabular data.
[0050] The table data for the cases of fever of unknown origin with rash contains a total of 144 indicators, including the patient's basic clinical characteristics such as gender and age, physiological characteristics such as respiration, heart rate, pulse, systolic blood pressure, diastolic blood pressure and body temperature, past medical history such as hypertension, diabetes, hepatitis B, heart disease, kidney disease and malignant tumors, 22 blood routine tests, 7 urine routine tests, 5 electrolyte tests, 16 liver function tests, 6 thyroid function tests, 6 coagulation function tests, 7 kidney function tests, 6 blood glucose and lipid tests, 6 APS-related antibody tests, 4 anti-human globulin tests, 5 immunoglobulin tests, 3 urine protein quantification tests, 7 cytokine tests, 5 myocardial enzyme profile tests, 7 tumor marker tests, 14 lymphocyte subset tests, 5 cerebrospinal fluid tests, 3 cerebrospinal fluid glucocorticoid assays, 5 serum protein electrophoresis tests, and 5 culture tests.
[0051] The preprocessing of the tabular data includes data cleaning such as handling missing values, outliers, and standardization. Specifically, this includes removing outliers from all variables, eliminating variables with more than half of their values missing, imputing missing values for categorical variables using the mode, imputing missing values for continuous variables using the mean, encoding categorical variables using LabelEncoder, and standardizing all variables using z-score.
[0052] The text data in the fever of unknown origin with rash case is the electronic medical record record of the patient's rash text. Wherein, the preprocessing of the text data includes removing stop words, performing word segmentation, and truncating the rash text electronic medical record record. The stop word removal step includes removing common words that do not contribute significantly to semantics in the text classification or modeling process when processing the rash text data, such as "de", "le", "shi", etc., so as to reduce the influence of noise on subsequent processing. The word segmentation step includes using a BERT (Bidirectional Encoder Representations from Transformers) tokenizer to segment the rash text of each patient, splitting the original text into a plurality of lexical units (tokens), so as to convert the text into a form that can be processed by the BERT model. The truncation step includes performing padding or truncation operations on the rash text to ensure that all input texts have the same length. For texts with insufficient length, a padding operation is used to add a special padding token at the end, and for excessively long texts, the maximum length of the text is maintained through a truncation operation, thereby ensuring that the model can accept fixed-length inputs during processing.
[0053] The image data of the fever of unknown origin with rash case is the patient's medical CT image. Wherein, the preprocessing of the image data includes performing image resampling, window width and window level adjustment, data augmentation, and size adjustment operations on the CT image. The resampling step includes adopting a B-spline interpolation algorithm to resample all image data to a uniform 1×1×1 cm 3 isotropic voxel size, so as to ensure that the slice thickness and in-plane resolution of the image are consistent. The window width and window level adjustment step includes setting the image window level to 0Hu and the window width to 400Hu according to the image grayscale value of the region of interest, and simultaneously using voxel value discretization to reduce image noise. The data augmentation step includes performing data augmentation operations such as rotation (90 degrees, 180 degrees) and mirroring (left-right, up-down) on the image to increase the diversity of samples. The image size adjustment step includes selecting the middlemost frame of the CT image, taking one layer each above and below, for a total of three grayscale images as the input of the three RGB channels, and using bilinear interpolation to scale all image data to a size of 224×224 to meet the input size requirement of the model.
[0054] Step 2: Perform vectorization on images, texts, and tables respectively based on the multimodal feature extraction and vectorization module.
[0055] In the implementation case of fever of unknown origin with rash of the present invention, the steps include performing feature vectorization on the preprocessed image data, and performing feature vectorization on the preprocessed text data. Since the table data is already a structured vector, no further processing is required.
[0056] In the image data feature extraction and vectorization process, preprocessed image data (a 224×224 three-channel RGB image) is input into the ResNet50 model. For CT images, the extracted three-layer grayscale image is used as the three-channel data. The ResNet50 is a deep residual network model with 50 layers and uses a residual learning strategy to avoid the vanishing gradient problem. The network includes an initial convolutional layer, a max-pooling layer, four residual block groups, and a global average pooling layer. The model has been pre-trained on the ImageNet dataset and possesses powerful image feature extraction capabilities. After the input image undergoes forward propagation, ResNet50 generates a global feature representation. Specifically, the global average pooling layer outputs a 2048×1×1 feature vector (i.e., a 2048-dimensional feature vector representation), which contains high-dimensional feature information of the image, representing the main visual content of the image, as shown in Figure 2.
[0057] In the text data feature extraction and vectorization process, the preprocessed text data is input into a pre-trained BERT model. The BERT model is a pre-trained language model based on the Transformer (Self-Attention Transformer) architecture, which models contextual information in the text through a bidirectional encoder. The segmented electronic medical record diagnosis text of the rash is input into the BERT model to obtain the embedding representation (token) of each word unit. Each token is mapped to a vector representation in a high-dimensional space. The pooling layer in the BERT model (using the output of the CLS token) is used to average and summarize all word embeddings to obtain a 768-dimensional numerical feature vector representation of the entire text. The output 768-dimensional numerical feature vector reflects the semantic information learned by the BERT model for each patient's electronic medical record diagnosis text of the rash, as shown in Figure 3.
[0058] Step 3: The self-attention mechanism-based multimodal feature fusion module fuses image, text, and table features using a self-attention mechanism.
[0059] In the implementation example of fever with rash of the present invention, the method includes fusing vectorized image features, text features and preprocessed table features to obtain multimodal fused features.
[0060] The self-attention mechanism feature fusion method described above primarily enhances the representation of effective features and suppresses the influence of irrelevant features by calculating the correlation between input features. This avoids overlooking the unique attributes and correlations between different modalities that might be missed by simply concatenating multimodal features. Specifically, self-attention is a special type of attention mechanism where each part of the input is calculated with other parts of the input to learn internal correlations. Self-attention captures the relationships between features and assigns more weight to effective features. The fusion process of image, text, and table features in this invention is shown in Figure 4, which specifically includes feature concatenation, calculating the weighted preliminary fused features using the self-attention mechanism, and compressing the dimensionality of the preliminary fused features using a fully connected layer to obtain the final fused features.
[0061] As shown in Figure 4, in the feature concatenation process, the previously obtained image features, text features, and table features are concatenated. The 2048-dimensional feature vector obtained after image processing using the ResNet50 model is represented as follows: Each of them A feature representing an image; text features are represented by a 768-dimensional feature vector obtained through the BERT model. Each of them A feature representing the text; the table feature is represented by a 144-dimensional feature vector obtained after preprocessing. Each of them This represents a feature of the table. The image feature, text feature, and table feature are concatenated to obtain an m+n+p dimensional feature vector x. v,s,t :
[0062] As shown in Figure 4, in the self-attention mechanism, the concatenated feature vector x v,s,t A self-attention mechanism is used to automatically assign different weights to different features. For each feature, a set of learned weight matrices is used to calculate Q (Query), K (Key), and V (Value) vectors, where V is the input vector representing the actual information of the feature; Q is the vector obtained from the input and weights W. q The product of the input and weights W represents the query vector of the features; K is the product of the input and weights W. k The product of these features represents their relationship with other features. Specifically, in the self-attention mechanism, Query, Key, and Value are identical, all originating from the same input feature vector x. v,s,t Q = K = V = x v,s,t .
[0063] The self-attention weight Att is calculated using Q and K, and the formula is as follows:
[0064] Among them, K T It is the transpose of the key, d k This is the dimension of the Key vector. In this step, the dot product of the Query and Key is first calculated, and then divided by the scaling factor. Finally, the result is normalized to a probability distribution using the Softmax function to obtain the attention weights of each part to the other parts. The final output is x. e The value can be obtained by multiplying the Attention weights Att by the Value vector: x e =Att·V.
[0065] In the fusion feature compression, the output x e The input is fed into a fully connected layer to obtain the final fused feature x. f : x f =W·x e +b.
[0066] Where W and b are the parameters of the fully connected layer. In the implementation example of fever with rash of the present invention, in order to obtain a highly abstract multimodal feature representation and remove unnecessary redundant information, the output dimension of the fully connected layer is set to 512, and finally a 512-dimensional fused feature x is obtained. f This is used for subsequent multi-level classification input.
[0067] Step 4: Construct a hierarchical structure tree based on the multi-level classification framework construction and training module, build a multi-level classification framework based on the hierarchical tree, and input training data to train the model.
[0068] In the implementation example of fever with rash of unknown cause in this invention, the process includes constructing a hierarchical structure tree of fever etiology of unknown cause and a multi-level classification model, and inputting fused data for training.
[0069] The etiology hierarchy structure of the fever-as-itself disease includes three levels, as shown in Figure 5. The first level is the root node; the second level is divided into four parent nodes, including infectious diseases, non-infectious inflammatory diseases (NIID), neoplastic diseases, and other diseases; the third level is subdivided into multiple child nodes, among which infectious diseases are subdivided into four child nodes: bacterial infection, viral infection, fungal infection, and parasitic infection; NIID is subdivided into two child nodes: autoinflammatory diseases and autoimmune diseases; neoplastic diseases are subdivided into three child nodes: lymphoma, hematologic diseases, and malignant tumors; and other diseases include one child node: drug fever.
[0070] The multi-level classification model based on a hierarchical tree includes constructing a local classifier for each parent node (as shown in Figure 5). For each parent node i, the local base classifier is set to logistic regression, denoted as C. i (x)=σ(w i ·x f +b i ), where x f It is the input feature vector, i.e., the obtained fused features, where σ is the activation function of logistic regression, and w i and b i These are the weights and biases of logistic regression; the hierarchical labels are represented as a true label matrix, with the first column being the parent node label and the second column being the label of its corresponding child node, denoted by Y = [P i C i,j ] indicates that P i Let C be the label of the i-th parent node. i,j Parent node P i The corresponding label of the j-th child node.
[0071] In the model classification training, the StratifiedShuffleSplit stratified sampler is used to split the dataset to ensure that the ratio of training and test samples for different categories and levels remains balanced. Then, the training set is input into each parent node P. i The corresponding logistic regression classifier C i Training is performed from top to bottom, using the corresponding columns in the label matrix Y as training targets. The regularization strength C of the logistic regression classifier for each parent node is optimized using a grid search to obtain the optimal local classifier trained on each parent node.
[0072] Step 5: Based on the model testing and evaluation module, input the test set portion of the multimodal fusion data into the pre-trained multi-level classification framework, perform testing and evaluation, and output the classification results.
[0073] In an embodiment of the present invention, fever with rash is to be investigated, the split test set is input into a multi-level classification model for testing, and the multi-level label of each sample is output.
[0074] The model testing involves inputting a test set split using the StratifiedShuffleSplit stratified sampler into a trained multi-level classification model, and performing a test based on each locally optimal logistic regression classifier. Output the predicted label for this node, and finally obtain the predicted label matrix. in This represents the predicted label of the i-th parent node. This indicates that it corresponds to the parent node P. iThe predicted label of the j-th child node is obtained, and the node's predicted label is finally output as the classification result.
[0075] The model evaluation uses hierarchical classification metrics, including hierarchical precision, hierarchical recall, and hierarchical F1 score (hF), to assess the prediction results of the multi-level disease etiology diagnosis model. The specific calculation method for each evaluation metric is as follows, where α... i It is the set of predicted subclasses and their parent classes for the test sample, β. i It is the set of the true subclasses and their parent classes of the test samples:
[0076] (1) Hierarchical accuracy (hP):
[0077] Here, hP represents the proportion of nodes correctly predicted by the model out of the total number of nodes. For each parent node and its corresponding child node, if both are correctly predicted, it is considered correct. This metric can evaluate the accuracy of a multi-level disease etiology diagnosis model at each level.
[0078] (2) Hierarchical recall (hR):
[0079] Here, hR represents the proportion of the number of correctly predicted positive cases out of the total number of positive cases. For each parent node and its corresponding child node, if both are correctly predicted, they are considered positive cases. This metric can evaluate the recall rate of a multi-level disease etiology diagnosis model at each level.
[0080] (3) Level F1 score (hF):
[0081] Here, hF represents the harmonic mean of the hierarchical precision and hierarchical recall of the model. This metric comprehensively considers the model's accuracy and recall, and can comprehensively evaluate the performance of a multi-level disease etiology diagnostic model. Finally, the above metrics are used to evaluate the classification performance of the multi-level classification model.
[0082] This invention deeply fuses multimodal data (such as images, text, and tables), utilizes a self-attention mechanism to model the relationships between different modalities, enhancing feature expressiveness, and combines a multi-level structure to progressively optimize classification performance, achieving a coarse-to-fine classification process. The proposed multi-level classification framework can adapt to complex classification tasks in different fields and diverse application scenarios. Whether in medical diagnosis, product recommendation, or other complex environments, this invention can provide accurate and efficient classification results.
[0083] This invention also provides an electronic device, including a memory and a processor, wherein the memory is coupled to the processor; wherein the memory is used to store program data, and the processor is used to execute the program data to implement the multi-level classification method based on multimodal data fusion.
[0084] This invention also provides a computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements the multi-level classification method based on multimodal data fusion.
[0085] The computer-readable storage medium can be an internal storage unit of any data processing device described in any of the foregoing embodiments, such as a hard disk or memory. The computer-readable storage medium can also be any data processing device, such as a plug-in hard disk, smart media card (SMC), SD card, flash card, etc., equipped on the device. Furthermore, the computer-readable storage medium can include both internal storage units of any data processing device and external storage devices. The computer-readable storage medium is used to store the computer program and other programs and data required by the data processing device, and can also be used to temporarily store data that has been output or will be output.
[0086] Other embodiments of this application will readily occur to those skilled in the art upon consideration of the specification and practice of the disclosure herein. This application is intended to cover any variations, uses, or adaptations of this application that follow the general principles of this application and include common knowledge or customary techniques in the art not disclosed herein. The specification and embodiments are to be considered exemplary only.
[0087] It should be understood that this application is not limited to the precise structure described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from its scope.
Claims
1. A multi-level classification method based on multimodal data fusion, characterized in that, Includes the following steps: Acquire image, text, and table data to be classified, and preprocess the image, text, and table data to obtain preprocessed image data, text data, and table features; The preprocessed image data is input into a pre-trained deep residual network model for feature extraction and structured vector representation to obtain vectorized image features. The preprocessed text data is input into a pre-trained language model for feature extraction and structured vector representation to obtain vectorized text features. Based on the self-attention mechanism, image features, text features, and table features are fused to obtain fused features; Construct a hierarchical tree structure, build a multi-level classification model based on the hierarchical tree, and input the fused features into the multi-level classification model for hierarchical classification training; The test set portion of the multimodal fusion data is input into the trained multi-level classification model for testing and evaluation, and the classification results are output.
2. The multi-level classification method based on multimodal data fusion according to claim 1, characterized in that, The preprocessing of the image, text, and table data includes: image enhancement, resizing, and normalization of image data; removal of stop words, word segmentation, and truncation and padding of text data; and handling of missing values, outliers, and standardization of table data.
3. The multi-level classification method based on multimodal data fusion according to claim 1, characterized in that, The process involves inputting the preprocessed image data into a pre-trained deep residual network model for feature extraction and structured vector representation, resulting in vectorized image features including: The ResNet50 model is selected. The preprocessed image data is input into the ResNet50 model. The global average pooling layer of the ResNet50 model outputs a 1×2048 feature vector. The feature vector contains the high-dimensional feature information of the image to represent the visual content of the image.
4. The multi-level classification method based on multimodal data fusion according to claim 1, characterized in that, The step of inputting the preprocessed text data into a pre-trained language model for feature extraction and structured vector representation to obtain vectorized text features includes: The BERT model is selected. The preprocessed text data is input into the BERT model. The token of each word unit is obtained through the BERT model. Each token is mapped to a vector representation in a high-dimensional space. The pooling layer in the BERT model is used to average and summarize all word embeddings to obtain a 768-dimensional numerical feature vector representation of the entire text.
5. A multi-level classification method based on multimodal data fusion according to claim 1, characterized in that, The self-attention mechanism-based feature fusion of image features, text features, and table features yields fused features including: Image features, text features, and table features are concatenated to obtain a concatenated feature vector. Based on a self-attention mechanism, different weights are automatically assigned to different features. According to the weights, a preliminary fused feature is obtained and input into a fully connected layer for dimensionality compression to obtain the final fused feature.
6. The multi-level classification method based on multimodal data fusion according to claim 1, characterized in that, The step of constructing a multi-level classification model based on the hierarchical tree and inputting the fused features into the multi-level classification model for hierarchical classification training includes: The multi-level classification model includes constructing a local classifier for each parent node, setting the local base classifier to logistic regression for each parent node, and representing the hierarchical labels as a true label matrix, with the first column being the parent node label and the second column being its corresponding child node label. The dataset is split using the StratifiedShuffleSplit stratified sampler to ensure a balanced ratio of training and testing samples across different categories and levels. The training set is input into the logistic regression classifier corresponding to each parent node and trained from top to bottom. The corresponding column in the label matrix is used as the training objective. The regularization strength of the logistic regression classifier for each parent node is optimized using grid search to obtain the optimal local classifier trained for each parent node.
7. A multi-level classification method based on multimodal data fusion according to claim 6, characterized in that, The step of inputting the test set portion of the multimodal fusion data into the trained multi-level classification model for testing and evaluation, and outputting classification results, includes: The split test set is input into the trained multi-level classification model. Based on each local optimal logistic regression classifier, the predicted label of the node is output to obtain the predicted label matrix. The node predicted label is output as the classification result. The classification results are evaluated using hierarchical classification metrics, namely hierarchical precision, hierarchical recall, and hierarchical F1 score, and the evaluation results are output.
8. A multi-level classification system based on multimodal data fusion, characterized in that, include: A multimodal data preprocessing module is used for cleaning image, text, and tabular data; The multimodal feature extraction and vectorization module is used to extract features and represent structured vectors for images and text, respectively. The self-attention mechanism multimodal feature fusion module is used to fuse image features, text features, and table features; The multi-level classification framework construction and training module is used to construct a multi-level classification framework and perform hierarchical classification training. The model testing and evaluation module is used to test and evaluate the results of multi-level classification and display the output.
9. An electronic device comprising a memory and a processor, characterized in that, The memory is coupled to the processor; wherein the memory is used to store program data, and the processor is used to execute the program data to implement a multi-level classification method based on multimodal data fusion as described in any one of claims 1-7.
10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the program is executed by the processor, it implements a multi-level classification method based on multimodal data fusion as described in any one of claims 1-7.
