Mental illness category recognition method and apparatus, and computer device and storage medium
By using multimodal feature fusion and information-based expression modes, the problem of insufficient generalization ability of existing mental illness assessment models has been solved, enabling effective identification and personalized diagnosis of various mental illnesses, while reducing resource requirements and management difficulties.
Patent Information
- Authority / Receiving Office
- WO · WO
- Patent Type
- Applications
- Current Assignee / Owner
- SHANGHAI MENTAL HEALTH CENT (SHANGHAI PSYCHOLOGICAL COUNSELLING TRAINING CENT)
- Filing Date
- 2024-12-17
- Publication Date
- 2026-05-21
AI Technical Summary
Existing mental illness assessment models have weak generalization ability, making it difficult to identify multiple mental illnesses simultaneously. Furthermore, training multiple assessment models requires a large amount of data and computing resources, increasing the difficulty of management.
Multimodal feature fusion technology is employed to acquire patients' multimodal data, extract and fuse features using the FHF model, construct an information-based representation pattern using the Apriori algorithm, identify the patient's mental illness category, and perform preprocessing using a non-parametric Bayesian model to dynamically adjust model parameters.
It enables the effective identification of various mental illnesses without training multiple models, reducing data and computing resource requirements, simplifying management, and providing personalized diagnostic reports and treatment plans.
Smart Images
Figure CN2024139950_21052026_PF_FP_ABST
Abstract
Description
Methods, devices, computer equipment, and storage media for identifying mental illness categories
[0001] Cross-references to related applications
[0002] This application claims priority to Chinese Patent Application No. 202411634901.6, filed on November 15, 2024, entitled “Method, Apparatus, Computer Equipment and Storage Medium for Identifying Mental Illness Categories”, the entire contents of which are incorporated herein by reference. Technical Field
[0003] This application relates to the interdisciplinary field of mental illness recognition and artificial intelligence, and in particular to a method, device, computer equipment, and storage medium for identifying mental illness categories. Background Technology
[0004] Mental illness is a type of disease that seriously affects people's physical and mental health. In the past, the diagnosis and assessment of mental illness mainly relied on doctors' subjective judgment and experience, lacking objective and accurate assessment methods.
[0005] With the development of information technology, some technologies utilize artificial intelligence to construct intelligent assessment models and build corresponding assessment systems based on these models. These assessment models can be, for example, models for depression, anxiety, and bipolar disorder, and the assessment systems can be systems for these conditions. These models can generate specific assessment results based on multi-dimensional patient data. For instance, a depression assessment model might include information indicating whether a patient has depression, and information indicating the degree of depression when the patient does have depression. However, these assessment models are typically only suitable for identifying or assessing a specific type of mental illness, exhibiting weak generalization ability and not applicable to identifying or assessing other mental illnesses. For example, a depression severity assessment model is trained using multimodal data from patients with depression to identify the degree of depression. This model primarily learns and identifies characteristics of depression. If the patient's condition is not depression but another mental illness, such as anxiety, the model cannot accurately identify whether the patient has another mental illness and its corresponding severity.
[0006] Because the generalization ability of the above assessment models is weak, if the assessment system is to be able to assess multiple mental illnesses, it is usually necessary to train different assessment models for different mental illnesses and integrate them into the assessment system. However, the inventors realized that training multiple assessment models for different mental illnesses requires a huge amount of training data and computing resources, and will also increase the difficulty of managing the assessment models in the later stage. Summary of the Invention
[0007] This application provides a method for identifying mental illness categories according to a first aspect. In some embodiments, the method includes:
[0008] Acquire multimodal data of the target patient;
[0009] Feature extraction is performed on various modal data to obtain feature vectors for each modal data, and multimodal fusion features are generated based on the feature vectors of various modal data;
[0010] The multimodal fusion features are matched with pre-constructed informational expression patterns of pathological manifestations of various mental illnesses to determine the target informational expression pattern; and
[0011] The target patient's mental illness category is determined based on the target information expression pattern.
[0012] In some embodiments, feature extraction is performed on various modal data to obtain feature vectors for each modal data, and multimodal fusion features are generated based on the feature vectors of various modal data, including:
[0013] Input various modal data into the multimodal feature fusion model to obtain the multimodal fusion features output by the multimodal feature fusion model;
[0014] The multimodal feature fusion model includes an input layer, multiple functional module layers for processing different modal data, and a feature fusion layer. The input layer transmits various types of modal data to their respective functional module layers. Each functional module layer uses its corresponding feature extraction network to extract feature vectors from the modal data input to it. The feature fusion layer assigns different weights to the feature vectors extracted by each functional module layer through a weighting mechanism or an attention mechanism, and generates multimodal fusion features based on the feature vectors extracted by each functional module layer and their corresponding weights.
[0015] In some embodiments, the multimodal data of the target patient is obtained by preprocessing the original multimodal data of the target patient using a non-parametric Bayesian model.
[0016] In some embodiments, the process of constructing an information-based expression model of the pathological manifestations of these various mental illnesses includes:
[0017] Acquire multimodal sample data from multiple patients;
[0018] Feature extraction was performed on the multimodal sample data of each patient to obtain the multimodal features of each patient;
[0019] Clustering was performed based on the multimodal characteristics of each patient to divide the multiple patients into multiple pathological clusters; and
[0020] The Apriori algorithm is used to perform association rule analysis on each pathological cluster, and an information-based expression model is constructed based on the analysis results to obtain information-based expression models of the pathological manifestations of various specified mental illnesses.
[0021] In some embodiments, the operation of performing association rule analysis on each pathological cluster using the Apriori algorithm and constructing an information-based representation pattern based on the analysis results includes:
[0022] For each pathological cluster, the multimodal features of each patient within that pathological cluster are converted into a feature itemset;
[0023] The Apriori algorithm is used to scan the feature itemsets of all patients within this pathological cluster, identify frequent itemsets whose support thresholds meet preset conditions, and mine association rules that reach preset confidence thresholds from these frequent itemsets; and
[0024] Based on the discovered association rules, an information expression pattern corresponding to the pathological cluster is constructed. The information expression pattern corresponding to the pathological cluster includes the common symptoms of patients within the cluster and the association rules between symptoms.
[0025] In some embodiments, after determining the target patient's mental illness category based on the target information expression pattern, the method further includes:
[0026] A diagnostic report is generated based on the target information expression pattern. The diagnostic report includes the detected case manifestations and recommended treatment plans.
[0027] In some embodiments, the method further includes:
[0028] The parameters and information expression patterns of the multimodal feature fusion model are updated based on feedback data from doctors and new multimodal data from patients.
[0029] This application provides a mental illness category identification device according to a second aspect. In some embodiments, the device includes:
[0030] The data acquisition module is used to acquire multimodal data of the target patient;
[0031] The feature fusion module is used to extract features from various modal data, obtain feature vectors for various modal data, and generate multimodal fusion features based on the feature vectors of various modal data;
[0032] The pattern matching module is used to match the multimodal fusion features with pre-constructed informational expression patterns of pathological manifestations of various mental illnesses to determine the target informational expression pattern; and
[0033] The classification module is used to determine the mental illness category of the target patient based on the target information expression pattern.
[0034] According to a third aspect, this application provides a computer device including a memory, one or more processors, and computer-readable instructions stored in the memory and executable on the one or more processors, wherein the one or more processors, when executing the computer-readable instructions, implement the mental illness category identification method provided in any of the above embodiments.
[0035] According to a fourth aspect, this application provides a computer-readable storage medium having computer-readable instructions stored thereon, which, when executed by one or more processors, implement the mental illness category identification method provided in any of the above embodiments. Attached Figure Description
[0036] Figure 1 is a flowchart illustrating a method for identifying mental illness categories provided in this application according to one or more embodiments;
[0037] Figure 2 is a schematic diagram of the structure of the multimodal feature fusion model provided in this application according to one or more embodiments;
[0038] Figure 3 is a flowchart illustrating the construction of an information-based expression model of the pathological manifestations of mental illness according to one or more embodiments of this application;
[0039] Figure 4 is a structural block diagram of a mental illness category determination device provided in this application according to one or more embodiments;
[0040] Figure 5 is an internal structural diagram of a computer device provided in this application according to one or more embodiments. Detailed Implementation
[0041] To make the objectives, technical solutions, and advantages of this application clearer, the embodiments of this application will be described in further detail below with reference to the accompanying drawings. It should be understood that the described embodiments are merely some embodiments of this application, and not all embodiments. All other embodiments obtained by those skilled in the art based on the embodiments of this application without creative effort are within the scope of protection of this application.
[0042] In the following description, when referring to the accompanying drawings, the same numbers in different drawings denote the same or similar elements unless otherwise indicated. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this application. Rather, they are merely examples of apparatuses and methods consistent with some aspects of this application as detailed in the appended claims.
[0043] In the description of this application, it should be understood that the terms "first," "second," "third," etc., are used only to distinguish similar objects and are not necessarily used to describe a specific order or sequence, nor should they be construed as indicating or implying relative importance. Those skilled in the art can understand the specific meaning of the above terms in this application according to the specific circumstances. Furthermore, in the description of this application, unless otherwise stated, "multiple" refers to two or more. "And / or" describes the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, or B existing alone. The character " / " generally indicates that the preceding and following related objects have an "or" relationship.
[0044] To address the shortcomings or defects of related technologies, this application provides a method for identifying mental illness categories. This method can effectively identify the category of a patient's mental illness even when the patient's mental illness is uncertain. At the same time, this method does not require pre-training multiple assessment models, reducing the data, manpower, and computing resources required during the model training phase, and also reducing the management difficulty of the trained model.
[0045] In some embodiments, the method includes the steps shown in FIG1, and the method is described in detail below.
[0046] S110: Acquire multimodal data of the target patient.
[0047] In this embodiment, the method can be applied to an intelligent assessment system, which can be a personal computer, server, or other computer device. It is important to note that the intelligent assessment system is not limited to a specific mental illness; it supports the assessment of various mental illnesses, such as depression, anxiety, and substance addiction. By collecting data from different modalities of the patient and combining it with artificial intelligence algorithms, the system can dynamically identify the category of the patient's mental illness, thus making it suitable for the auxiliary assessment of various mental and psychological disorders.
[0048] The target patients mentioned above refer to those whose mental illness category is to be identified. This system can collect multimodal data from target patients using various information acquisition devices. These devices can include advanced equipment such as recording devices, real-time cameras, 64-channel EEG recorders, near-infrared devices, and multi-channel biosignal recording systems. These devices can comprehensively collect multimodal data from patients, including speech, facial expressions, and biosignals. For example, recording devices can clearly record the patient's tone, speed, volume, and the coherence and logic of their speech; real-time cameras can capture subtle changes in facial expressions, such as frowning, smiling, and eye contact, which can be used to analyze the tension of facial muscles and the duration of expressions; 64-channel EEG recorders can acquire electrical activity signals from the patient's brain, including the frequency, amplitude, and phase of brain waves, reflecting the functional state and neural activity patterns of the brain; near-infrared devices and multi-channel biosignal recording systems can monitor changes in physiological indicators such as heart rate, blood pressure, respiratory rate, and skin conductance, as well as the fluctuations of these indicators under different circumstances.
[0049] Accordingly, multimodal data can include patient voice data, facial expression data, biosignal data, and other data. Furthermore, multimodal data can also include data entered by the patient through input devices (such as a mouse, keyboard, touchpad, etc.) in a user interface, such as information filled in on a questionnaire. It should also be noted that before the system collects the patient's multimodal data, the patient must be informed and their authorization and consent obtained before the data collection operation is performed through the information collection device. After the data collection operation is completed, attention must be paid to data security and privacy protection to ensure that the patient's information is not leaked.
[0050] S120: Extract features from various modal data to obtain feature vectors for each modal data, and generate multimodal fusion features based on the feature vectors of each modal data.
[0051] Different methods can be used to extract features from data of different modalities. For example, ASR (auto-speech recognition) technology can be used to process language data, and convolutional neural networks can be used to process image data.
[0052] Feature extraction from various modal data yields multimodal features, which are feature vectors for each modality. For example, speech features (such as speech rate, tone, volume, speech coherence, and emotional tone) can be extracted from speech data; facial expression features (such as facial muscle tension, eyebrow movement types, frequency and duration of various expressions, and frequency and duration of eye contact) can be extracted from facial expression data; and physiological signal features (such as frequency, amplitude, and phase information of brain waves, regional differences in brain electrical activity, heart rate and heart rate variability, blood pressure level, respiratory rate, and skin conductance level) can be extracted from physiological signal data. This allows us to obtain the multimodal features of the target patient (including speech features, facial expression features, and physiological signal features).
[0053] After obtaining the multimodal features of the target patient, they can be fused using feature fusion techniques (such as directly concatenating the feature vectors into a single feature vector) to obtain a comprehensive feature vector, i.e., multimodal fusion feature.
[0054] S130: Match the multimodal fusion feature with the pre-constructed information expression patterns of the pathological manifestations of various mental illnesses to determine the target information expression pattern.
[0055] The information-based expression patterns are used to describe the pathological manifestations of corresponding mental illnesses. In this embodiment, the system pre-constructs corresponding information-based expression patterns for the pathological manifestations of various specified mental illnesses using clustering algorithms and association rule mining techniques. These information-based expression patterns can be used to assist in diagnosing the category of the mental illness suffered by the patient.
[0056] S140: Determine the target patient's mental illness category based on the target information expression pattern.
[0057] When using the system to assist in the diagnosis of target patients, the type of mental illness of the target patient can be identified through a pre-built information expression pattern, which can provide doctors with diagnostic suggestions and treatment plans.
[0058] In some embodiments, feature extraction is performed on various types of modal data to obtain feature vectors for various types of modal data, and multimodal fusion features are generated based on the feature vectors of various types of modal data, including: inputting various types of modal data into a multimodal feature fusion model and obtaining the multimodal fusion features output by the multimodal feature fusion model.
[0059] Please refer to Figure 2. The multimodal feature fusion model includes an input layer, multiple functional module layers for processing different modal data, and a feature fusion layer. The input layer is used to transmit various types of modal data to the corresponding functional module layers. Each functional module layer is used to extract feature vectors from the modal data input to this layer using the feature extraction network corresponding to this layer. The feature fusion layer is used to assign different weights to the feature vectors extracted by each functional module layer through a weighting mechanism or an attention mechanism, and to generate multimodal fusion features based on the feature vectors extracted by each functional module layer and their corresponding weights.
[0060] In this embodiment, the multimodal feature fusion model is specifically the FHF (Function & Hierarchy Framework) model. FHF is a framework model for multimodal information fusion. It integrates data from different modalities, such as speech, facial expressions, and physiological signals, through a hierarchical structure and functional modules. In this model, data from different modalities are processed at independent levels and weighted and fused according to preset functional rules to ultimately form a comprehensive feature vector.
[0061] In this embodiment, the input layer of the FHF model receives multimodal data, such as speech data, facial expression data, and physiological signal data. The input layer is connected to various feature extraction layers, each processing different modalities. The input layer feeds the corresponding modal data into these layers, resulting in different feature extraction networks for each layer. These networks extract modal features from the data, and may include, for example, convolutional neural networks (for extracting features from facial expression data) or recurrent neural networks (for extracting features from speech or physiological signal data). Each feature extraction layer is connected to an intermediate fusion layer, which then inputs its extracted modal features to combine the features from different modalities.
[0062] The feature fusion layer assigns different weights to different modal features through a weighting or attention mechanism, and then combines the weights of each modal feature into a comprehensive feature vector. Furthermore, a deep neural network can be used to perform a non-linear mapping on the combined feature vector to obtain a multimodal fusion feature. Finally, the output layer outputs this multimodal fusion feature.
[0063] Traditional feature fusion methods primarily involve directly concatenating features from different modalities into a single vector. However, this approach is prone to information redundancy and noise when processing multimodal data. Early or late fusion strategies either lose some information or limit the interaction between different modalities. This embodiment employs the FHF model for feature fusion. The FHF model uses a hierarchical structure, ensuring that each modality's data is fully processed within its own layer, avoiding information loss that may occur during direct fusion. Furthermore, weighting and attention mechanisms assign weights to each modality's features, allowing the model to dynamically adjust the weights of each modality to ensure that important features function effectively in different contexts, resulting in greater flexibility. Additionally, nonlinear mapping of the composite feature vector helps capture the complex relationships between multimodal data, thus adapting to the diagnosis of different mental illnesses. It is also worth noting that compared to simple feature concatenation, the FHF model better handles situations where some modal data is missing and reduces the impact of noise and redundant information, exhibiting stronger fault tolerance.
[0064] The training process of the FHF model is described below. The training process involves joint learning of multimodal data and mainly includes the following steps:
[0065] (1) Data preprocessing: Standardize and denoise the data of each modality to ensure the quality and consistency of the data.
[0066] (2) Single-modal feature learning: The model is trained separately using data from each modality in order to extract key features. For example, convolutional neural networks are used to extract facial expression features, and recurrent neural networks are used to analyze time series data of speech and physiological signals.
[0067] (3) Multimodal feature fusion: Input the features of each modality into the feature fusion layer of the model, and perform weighted combination of features according to the weight or attention mechanism.
[0068] (4) Model optimization and update: The output layer results are evaluated using a loss function (such as cross-entropy loss), and the model parameters are continuously optimized through the backpropagation algorithm.
[0069] (5) Validation and testing: After training the model using the training set, use the validation set and test set to evaluate the model's performance to prevent overfitting.
[0070] Furthermore, in some embodiments, the multimodal data of the target patient is obtained after preprocessing the original multimodal data of the target patient using a non-parametric Bayesian model. That is, the system collects the original multimodal data of the target patient, and then preprocesses the original multimodal data of the target patient using a non-parametric Bayesian model to obtain the aforementioned multimodal data of the target patient.
[0071] Non-parametric Bayesian models can automatically adjust their parameters based on data distribution, thus better adapting to different data characteristics. In this embodiment, using a non-parametric Bayesian model to process raw multimodal data can remove noise and outliers, resulting in a higher-quality dataset. Furthermore, this model can smooth and complete time-series signal data (such as physiological signal data) based on data distribution, facilitating subsequent feature extraction and analysis of such data.
[0072] In some embodiments, the process of constructing an information-based expression model of the pathological manifestations of these various mental illnesses, as shown in Figure 3, includes:
[0073] S210: Acquire multimodal sample data from multiple patients.
[0074] Multimodal sample data refers to the multimodal data of patients used to construct an information-based expression model of the pathological manifestations of different mental illnesses. This includes data of various modalities, such as patients' voice data, facial expression data, and physiological signal data.
[0075] S220: Extract features from the multimodal sample data of each patient to obtain the multimodal features of each patient.
[0076] The feature extraction process in this step can be described using the feature extraction methods described in the above embodiments, and will not be repeated here.
[0077] S230: Clustering is performed based on the multimodal characteristics of each patient to divide the multiple patients into multiple pathological clusters.
[0078] This embodiment can use clustering algorithms such as K-means, KNN (K-Nearest Neighbor), and T-SNE (t-Distributed Stochastic Neighbor Embedding) to perform clustering operations.
[0079] Taking the K-means algorithm as an example, K-means clustering can group data points (each data point represents a patient) into K clusters. In practice, it can group patients with similar symptoms into the same group. For example, by analyzing patients' symptom data, patients can be divided into several different pathological clusters, each cluster representing a specific mental illness or pathological state.
[0080] S240: The Apriori algorithm is used to perform association rule analysis on each pathological cluster and an information-based expression model is constructed based on the analysis results to obtain information-based expression models of the pathological manifestations of various specified mental illnesses.
[0081] The Apriori algorithm is an association rule learning algorithm used to mine frequent itemsets and discover association rules within them. These rules can be represented in the form of "if...then..." to describe patterns in the data.
[0082] This embodiment employs the Apriori algorithm to analyze the pathological manifestations of mental illnesses, aiming to discover the correlations between different symptoms. Specifically, the Apriori algorithm can be applied to the symptom data within each pathological cluster to identify association rules between symptoms. The clustering results and association rules are then combined to construct an information-based representation pattern corresponding to each pathological cluster. This information-based representation pattern can be used to determine the interactions between the pathological features and symptoms of different mental illnesses.
[0083] In some embodiments, the operation of performing association rule analysis on each pathological cluster using the Apriori algorithm and constructing an information-based representation pattern based on the analysis results includes:
[0084] For each pathological cluster, the multimodal features of each patient within that pathological cluster are converted into a feature itemset;
[0085] The Apriori algorithm is used to scan the feature itemsets of all patients in the pathological cluster, determine the frequent itemsets whose support thresholds meet the preset conditions, and mine association rules that reach the preset confidence thresholds from the frequent itemsets.
[0086] Based on the discovered association rules, an information expression pattern corresponding to the pathological cluster is constructed. The information expression pattern corresponding to the pathological cluster includes the common symptoms of patients within the cluster and the association rules between symptoms.
[0087] In this embodiment, the multimodal features of each patient can be converted into itemsets (i.e., feature itemsets). For example, a patient's multimodal features can be represented as {slow speech rate, reduced smiling, increased heart rate}. Each itemset corresponds to a support threshold, which is used to evaluate the reliability of the corresponding itemset.
[0088] After completing the itemset transformation operation, a frequent itemset mining operation is required. Specifically, the Apriori algorithm scans these itemsets multiple times and finds frequent itemsets that meet a certain support threshold. Then, it mines association rules that meet the confidence threshold from these frequent itemsets. For example: Rule: {Slow speech rate, increased heart rate} → {Anxiety disorder}; Meaning: Patients with slow speech rate and increased heart rate are more likely to exhibit anxiety disorder.
[0089] In this embodiment, the information-based representation pattern refers to the combination of association rules and frequent features constructed using the Apriori algorithm, used to describe the pathological manifestations of different mental illnesses. Its data structure can be as follows:
[0090] A combination of pathological features, such as frequent frowning and rapid breathing.
[0091] Corresponding mental illness categories: such as depression or anxiety.
[0092] For example, the following are information expression patterns corresponding to several common mental illnesses.
[0093] (1) Depression:
[0094] Pattern: {Slow speech, reduced smiling, sleep deprivation} → Depression
[0095] Explanation: Patients exhibiting the above characteristics are more likely to have depression.
[0096] (2) Anxiety disorder
[0097] Pattern: {Increased breathing rate, increased heart rate, faster speech} → Anxiety disorder
[0098] Explanation: Anxiety disorder patients often experience an increase in respiratory rate and heart rate.
[0099] (3) Bipolar disorder
[0100] Pattern: {Frequent mood swings, significant changes in speech rate} → Bipolar disorder
[0101] Explanation: Patients with bipolar disorder often experience fluctuations between elevated and depressed moods.
[0102] In some embodiments, after determining the target patient's mental illness category based on the target information expression pattern, the method further includes: generating a diagnostic report based on the target information expression pattern, the diagnostic report including detected case manifestations and recommended treatment plans.
[0103] This embodiment can also provide targeted diagnostic reports for the target patients, improve the personalization of diagnosis, and help patients obtain better treatment.
[0104] In some embodiments, the method further includes updating the parameters and information expression mode of the multimodal feature fusion model based on physician feedback data and new patient multimodal data.
[0105] In this embodiment, the system will be regularly maintained and updated to ensure the accuracy and reliability of the multimodal feature fusion model and the information expression patterns of various mental illnesses.
[0106] It should be noted that, regarding the steps included in the mental illness category identification method provided in any of the above embodiments, unless otherwise explicitly stated herein, there is no strict order restriction on the execution of these steps; these steps can be executed in other orders. Moreover, at least some of these steps may include multiple sub-steps or multiple stages. These sub-steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these sub-steps or stages is not necessarily sequential, but can be performed alternately or in turn with other steps or at least a portion of the sub-steps or stages of other steps.
[0107] Based on the same inventive concept, this application also provides a mental illness category identification device. In some embodiments, as shown in FIG4, the device includes the following modules:
[0108] Data acquisition module 110 is used to acquire multimodal data of the target patient;
[0109] The feature fusion module 120 is used to extract features from various modal data, obtain feature vectors of various modal data, and generate multimodal fusion features based on the feature vectors of various modal data;
[0110] The pattern matching module 130 is used to match the multimodal fusion features with the pre-constructed information expression patterns of the pathological manifestations of various mental illnesses to determine the target information expression pattern.
[0111] Classification module 140 is used to determine the mental illness category of the target patient based on the target information expression pattern.
[0112] In some embodiments, the feature fusion module 120 is specifically used to: input various modal data into a multimodal feature fusion model, and obtain the multimodal fusion features output by the multimodal feature fusion model. The multimodal feature fusion model includes an input layer, multiple functional module layers for processing different modal data, and a feature fusion layer; the input layer is used to transmit various modal data to their respective functional module layers; each functional module layer is used to extract feature vectors from the modal data input to it using its corresponding feature extraction network; the feature fusion layer is used to assign different weights to the feature vectors extracted by each functional module layer through a weighting mechanism or an attention mechanism, and generate multimodal fusion features based on the feature vectors extracted by each functional module layer and their corresponding weights.
[0113] In some embodiments, the apparatus further includes a preprocessing module (not shown in the figures). The preprocessing module is used to preprocess the raw multimodal data of the target patient using a non-parametric Bayesian model to obtain the multimodal data of the target patient.
[0114] In some embodiments, the apparatus further includes an expression pattern construction module (not shown in the figure).
[0115] The expression pattern construction module includes:
[0116] The sample data acquisition submodule is used to acquire multimodal sample data from multiple patients;
[0117] The feature extraction submodule is used to extract features from the multimodal sample data of each patient to obtain the multimodal features of each patient;
[0118] The clustering analysis submodule is used to cluster patients based on their multimodal characteristics to divide the multiple patients into multiple pathological clusters.
[0119] The expression pattern construction submodule is used to perform association rule analysis on each pathological cluster using the Apriori algorithm and construct an information-based expression pattern based on the analysis results, thereby obtaining information-based expression patterns of the pathological manifestations of various specified mental illnesses.
[0120] In some embodiments, the expression pattern construction submodule uses the Apriori algorithm to perform association rule analysis on each pathological cluster and constructs an information-based expression pattern based on the analysis results. This includes: for each pathological cluster, converting the multimodal features of each patient within the pathological cluster into feature itemsets; using the Apriori algorithm to scan the feature itemsets of all patients within the pathological cluster, determining the frequent itemsets whose support thresholds meet preset conditions, and mining association rules from the frequent itemsets that reach a preset confidence threshold; and constructing an information-based expression pattern corresponding to the pathological cluster based on the mined association rules. The information-based expression pattern corresponding to the pathological cluster includes common symptoms of patients within the cluster and association rules between symptoms.
[0121] In some embodiments, the apparatus further includes a report generation module (not shown). The report generation module is used to generate a diagnostic report based on the target information expression pattern after determining the target patient's mental illness category based on the target information expression pattern. The diagnostic report includes the detected case manifestations and recommended treatment plans.
[0122] In some embodiments, the device further includes an update module (not shown in the figure), which is used to update the parameters and information expression mode of the multimodal feature fusion model based on doctor feedback data and new patient multimodal data.
[0123] Specific limitations regarding the mental illness category identification device can be found in the limitations of the mental illness category identification method described above, and will not be repeated here. Each module in the aforementioned mental illness category identification device can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in or independent of the processor in a computer device in hardware form, or stored in the memory of a computer device in software form, so that the processor can call and execute the operations corresponding to each module.
[0124] This application also provides a computer device, which in some embodiments includes a memory, one or more processors, and computer-readable instructions stored in the memory and executable on the one or more processors. When the one or more processors execute the computer-readable instructions, they can implement the steps of the mental illness category identification method provided in any of the above embodiments.
[0125] Furthermore, in some embodiments, the internal structure of the computer device may be as shown in Figure 5. The computer device includes a processor, memory, network interface, and database connected via a system bus. The processor provides computing and control capabilities. The memory includes a non-volatile storage medium and internal memory. The non-volatile storage medium stores an operating system, computer-readable instructions, and a database. The internal memory provides an environment for the operation of the operating system and computer-readable instructions in the non-volatile storage medium. The database stores data such as the informational expression patterns of pathological manifestations of various mental illnesses; the specific stored data may also refer to the limitations defined in the above method embodiments. The network interface of the computer device is used for communication with external terminals via a network connection. When the computer-readable instructions are executed by the processor, a method for identifying mental illness categories is implemented.
[0126] Those skilled in the art will understand that the structure shown in Figure 5 is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer device to which the present application is applied. Specific computer devices may include more or fewer components than those shown in the figure, or may combine certain components, or may have different component arrangements.
[0127] This application also provides a computer-readable storage medium, in some embodiments of which computer-readable instructions are stored on the computer-readable storage medium, and when executed by a processor, the computer-readable instructions implement the steps of the mental illness category identification method provided in any of the above embodiments.
[0128] In the above embodiments of this application, the descriptions of each embodiment have different focuses. For parts not described in detail in a certain embodiment, please refer to the relevant descriptions of other embodiments.
[0129] Those skilled in the art will understand that implementing all or part of the processes in the above method embodiments can be accomplished by instructing related hardware through computer-readable instructions. These computer-readable instructions can be stored in a non-volatile computer-readable storage medium. When executed, these computer-readable instructions can include the processes of the embodiments of the above methods. Any references to memory, storage, databases, or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory may include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory. Volatile memory may include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in a variety of forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), dual data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), Synchlink, DRAM (SLDRAM), memory bus, direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM), etc.
[0130] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.
[0131] The embodiments described above are merely illustrative of several implementation methods of this application, and while the descriptions are relatively specific and detailed, they should not be construed as limiting the scope of the invention patent. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this patent application should be determined by the appended claims.
Claims
1. A method for identifying mental illness categories, characterized in that, The method includes: Acquire multimodal data of the target patient; Feature extraction is performed on various modal data to obtain feature vectors for each modal data, and multimodal fusion features are generated based on the feature vectors of various modal data; The multimodal fusion features are matched with pre-constructed informational expression patterns of pathological manifestations of various mental illnesses to determine the target informational expression pattern; and The mental illness category of the target patient is determined based on the target information expression pattern.
2. The method as described in claim 1, characterized in that, Feature extraction is performed on various modal data to obtain feature vectors for each modality. Multimodal fusion features are then generated based on these feature vectors, including: Various modal data are input into a multimodal feature fusion model to obtain the multimodal fusion features output by the multimodal feature fusion model; The multimodal feature fusion model includes an input layer, multiple functional module layers for processing different modal data, and a feature fusion layer. The input layer transmits various types of modal data to their respective functional module layers. Each functional module layer uses its corresponding feature extraction network to extract feature vectors from the modal data input to it. The feature fusion layer assigns different weights to the feature vectors extracted by each functional module layer through a weighting mechanism or an attention mechanism, and generates multimodal fusion features based on the feature vectors extracted by each functional module layer and their corresponding weights.
3. The method as described in claim 1 or 2, characterized in that, The multimodal data of the target patient was obtained by preprocessing the original multimodal data of the target patient using a non-parametric Bayesian model.
4. The method as described in claim 1 or 2, characterized in that, The process of constructing an information-based expression model for the pathological manifestations of the various mental illnesses includes: Acquire multimodal sample data from multiple patients; Feature extraction was performed on the multimodal sample data of each patient to obtain the multimodal features of each patient; Clustering is performed based on the multimodal characteristics of each patient to divide the multiple patients into multiple pathological clusters; and The Apriori algorithm is used to perform association rule analysis on each pathological cluster, and an information-based expression model is constructed based on the analysis results to obtain information-based expression models of the pathological manifestations of various specified mental illnesses.
5. The method as described in claim 4, characterized in that, The operation of performing association rule analysis on each pathological cluster using the Apriori algorithm and constructing an information-based representation pattern based on the analysis results includes: For each pathological cluster, the multimodal features of each patient within that pathological cluster are converted into a feature itemset; The Apriori algorithm is used to scan the feature itemsets of all patients within the pathological cluster, identify frequent itemsets whose support thresholds meet preset conditions, and mine association rules that reach preset confidence thresholds from these frequent itemsets; and Based on the discovered association rules, an information expression pattern corresponding to the pathological cluster is constructed. The information expression pattern corresponding to the pathological cluster includes the common symptoms of patients within the cluster and the association rules between symptoms.
6. The method as described in claim 1, characterized in that, After determining the target patient's mental illness category based on the target information expression pattern, the method further includes: A diagnostic report is generated based on the target information expression pattern. The diagnostic report includes the detected case manifestations and recommended treatment plans.
7. The method as described in claim 2, characterized in that, The method further includes: The parameters of the multimodal feature fusion model and the information expression mode are updated based on feedback data from doctors and new multimodal data from patients.
8. A mental illness category identification device, characterized in that, The device includes: The data acquisition module is used to acquire multimodal data of the target patient; The feature fusion module is used to extract features from various modal data, obtain feature vectors for various modal data, and generate multimodal fusion features based on the feature vectors of various modal data; The pattern matching module is used to match the multimodal fusion features with pre-constructed informational expression patterns of pathological manifestations of various mental illnesses to determine the target informational expression pattern; and The classification module is used to determine the mental illness category of the target patient based on the target information expression pattern.
9. A computer device, comprising a memory, a processor, and computer-readable instructions stored in the memory and executable on the processor, characterized in that, When the processor executes the computer-readable instructions, it implements the steps of the method according to any one of claims 1 to 7.
10. A computer-readable storage medium storing computer-readable instructions thereon, characterized in that, When the computer-readable instructions are executed by a processor, they implement the steps of the method according to any one of claims 1 to 7.