Keyword-Guided Message Classification With Sentence BERT Clustering
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Conventional machine learning methods require significant data and processing resources to create user-specific classifiers for message categorization, making them impractical for individual users, especially in classifying business communications.
Innovation Solution
A combination of keyword search and Sentence BERT (Bidirectional Encoder Representations from Transformers) is used to classify messages into projects without the need for user-specific training data, employing similarity metrics and clustering to correct false positives and negatives.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If conventional machine learning methods are used to create user-specific classifiers for message categorization, then classification accuracy is improved, but data requirements and processing resources increase significantly
Solution Approach 1:
The patent applies universality by using a single pre-trained BERT model that can classify messages for multiple different users and projects without requiring separate training data for each user. The model is trained on general corpus data and then adapted through knowledge distillation to perform user-specific classification, eliminating the need for each user to provide their own training data while maintaining high classification accuracy.
Solution Approach 2:
The patent applies preliminary action by pre-training the BERT model on a large corpus of text data before deployment. This pre-training phase captures general language patterns and semantics, allowing the model to be later adapted to specific user needs through knowledge distillation without requiring extensive user-specific training data. The preliminary training on general data enables the model to function effectively with minimal user-specific data.
2Measurement precision
If conventional machine learning methods are used to create user-specific classifiers, then classification accuracy is improved, but processing resources and training time increase significantly
Solution Approach 1:
The patent applies the extraction principle by separating the heavy training process from the deployment phase. The BERT model is trained once on general corpus data, and then through knowledge distillation, the essential classification knowledge is extracted and transferred to a smaller, more efficient model. This extracted knowledge can be deployed with minimal processing resources while maintaining the classification accuracy benefits of the larger pre-trained model.
Solution Approach 2:
The patent applies parameter changes by transforming the model architecture and training parameters during knowledge distillation. The large pre-trained BERT model (with billions of parameters) is used for training, but the final deployed model has significantly fewer parameters. The knowledge distillation process changes the parameter configuration to achieve a balance between accuracy and computational efficiency, allowing deployment on devices with limited processing resources.
3Adaptability or versatility
If LSTM or Transformer neural networks are used to classify documents, then classification capability is improved, but training data requirements and processing resources increase
Solution Approach 1:
The patent applies the copying principle by using knowledge distillation to copy the classification knowledge from a large pre-trained BERT model to a smaller, more efficient model. Instead of directly deploying the complex Transformer model that requires extensive training resources, the essential knowledge is copied into a streamlined model that maintains classification capability while reducing computational complexity and training resource requirements.
Solution Approach 2:
The patent applies preliminary action by performing the computationally intensive pre-training of the BERT model in advance, before deployment. This preliminary training phase captures the classification capabilities needed, and then through knowledge distillation, this pre-acquired knowledge is transferred to a more efficient model. This eliminates the need for complex training processes at deployment time, reducing device complexity while maintaining classification versatility.
Data Source
AI summary
Electronic communications and a keyword are provided to a machine learning algorithm. Similarity measures are received from the machine learning algorithm. The similarity measures indicate a similarity between the communications and the keyword. The communications are clustered as a function of the similarity measures. False positive communications are removed from a first cluster as a function of a sum of distances between the false positive communication and communications in the first cluster that include the keyword and a sum of distances between the false positive communication and communications in the first cluster that do not include the keyword. False negatives are added to the first cluster as a function of a sum of distances between the false negative and communications in the first cluster that include the keyword and the sum of distances between the false negative and communications in the cluster that do not include the keyword.