Multi-Label Text Classification via BERT and Gradient Boosting
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current methods for multi-label classification of text data, particularly in medical contexts, face challenges in accurately assigning multiple labels to sentences due to the complexity of medical texts and the scarcity of annotated data, leading to inefficiencies in data processing and classification accuracy.
Innovation Solution
The proposed method employs a neural network-based classification engine that utilizes Bidirectional Encoder Representations from Transformers (BERT) and Bio-BERT for digital representation generation, combined with a light gradient boosting machine (LGBM) and decision trees, to perform multi-label classification. This approach includes soft labeling using a generative model and weighted majority votes from multiple labeling functions to generate accurate output probabilities based on text feature scores and probabilities from multiple digital representations.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If traditional classification methods are used, then the system is simpler to implement, but classification accuracy and precision deteriorate due to the complexity of medical texts and data scarcity
Solution Approach 1:
The patent segments the classification task into multiple independent classification heads, each targeting a specific label (e.g., population, intervention, outcome). Each head processes the same input features but produces separate probability distributions for its designated labels. This segmentation allows the system to handle complex multi-label classification by breaking it down into manageable components, improving accuracy without overwhelming system complexity.
Solution Approach 2:
The patent transforms the classification problem by adding a dimensional aspect through the use of multiple classification heads and the integration of text feature scores with neural network probabilities. Instead of a single flat classification layer, the system operates in a multi-dimensional probability space where each label has its own probability distribution, enabling more nuanced and accurate multi-label classification of medical texts.
2Measurement precision
If multiple digital representations and classification engines are used, then classification precision improves, but processing time and computational resources increase
Solution Approach 1:
The patent merges multiple classification outcomes by integrating probabilities from different classification heads and combining them with text feature scores through a weighted approach. Instead of running separate independent classification processes, the system combines multiple probability estimates and feature scores into a unified output, achieving improved precision while optimizing processing efficiency through consolidated computation.
Solution Approach 2:
The patent performs preliminary processing by generating text feature scores (such as TF-IDF and QIEF) before feeding the input through the neural network. This preliminary extraction of relevant features prepares the data in advance, allowing the classification engine to focus computational resources on the core classification task rather than feature extraction during inference, thereby reducing overall processing time while maintaining high precision.
3Reliability
If BERT and Bio-BERT are used for digital representation, then understanding of medical text complexity improves, but computational requirements and model size increase
Solution Approach 1:
The patent applies local quality by using Bio-BERT specifically for medical text representation where domain expertise is critical, while potentially using standard BERT or other lighter models for non-medical portions of the text. This selective application of heavy computational models only where needed (in medical contexts) maintains high text understanding accuracy for specialized domains while reducing overall computational energy consumption compared to applying Bio-BERT uniformly to all text.
Data Source
AI summary
There are provided methods and systems for multi-label classification of a sentence. An example method includes obtaining the sentence and generating a first digital representation corresponding to the words of the sentence. The method also includes performing a first classification of the sentence using a classification engine receiving as input the first digital representation. The first classification generates a first set of probabilities each associated with one of the possible labels for the sentence. The classification engine may include a neural network. The method further includes generating an output probability for each given label of the possible labels, which output probability is generated based on a first probability associated with the given label. The first probability is from the first set of probabilities. Moreover, the method includes outputting the output probability for each of the possible labels.