Text Classification Using Perplexity Features from Causal Language Models
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Moderate-sized Language Models like GPT-Neo-2.7B face challenges in achieving accurate text classification due to limitations in generating expected class labels in zero-shot settings and the need for costly hardware resources in few-shot settings, while large models like GPT-3 pose privacy concerns.
Innovation Solution
A method and system using perplexity and log-likelihood based features from label-specific augmentations, combined with a supervised machine learning classifier, to enhance text classification accuracy without requiring extensive hardware or fine-tuning, by deriving features from moderate-sized Language Models.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Quantity of substance
If moderate-sized Language Models like GPT-Neo-2.7B are used for text classification in zero-shot setting, then hardware resource requirements are reduced, but text classification accuracy deteriorates
Solution Approach 1:
The patent segments the text classification task into multiple components: generating label-specific augmentations for each class label, computing perplexity scores for each augmentation, and aggregating these scores to make classification decisions. This segmentation allows moderate-sized LMs to achieve better classification accuracy by breaking down the complex zero-shot classification task into manageable steps that leverage the model's strengths in language understanding while working around its limitations in direct classification.
Solution Approach 2:
The patent introduces label-specific augmentations as intermediary elements between the input text and the classification decision. These augmentations are generated by the LM and serve as intermediate representations that capture the relationship between the input text and each potential class label. By using these augmentations as mediators, the system improves classification accuracy without requiring the LM to directly output class labels, thus working around the LM's limitation in zero-shot classification.
2Measurement precision
If large Language Models like GPT-3 are used for text classification, then text classification accuracy is improved, but data privacy deteriorates due to API usage requirements
Solution Approach 1:
The patent employs moderate-sized Language Models that can be deployed locally as disposable alternatives to large commercial models. Instead of relying on expensive, centralized APIs that require data sharing, the system uses smaller models that can run on-premises, eliminating the need to send sensitive data to external services. This approach sacrifices some of the raw power of large models but gains data privacy and control.
Solution Approach 2:
The patent extracts and utilizes only the essential capabilities needed for text classification from large Language Models, rather than relying on the full power of massive models. By extracting the specific functionality of generating label-specific augmentations and computing perplexity scores, the system achieves adequate classification accuracy with smaller, privately-deployable models, thus removing the need to share data with external API providers.
3Measurement precision
If few-shot learning is used with moderate-sized Language Models, then text classification accuracy is improved, but hardware resource requirements and fine-tuning time increase
Solution Approach 1:
The patent performs preliminary actions by pre-computing label-specific augmentations and their corresponding perplexity scores before the actual classification task. This preliminary processing allows the system to quickly make classification decisions without requiring time-consuming fine-tuning during deployment. The augmentations are generated based on the model's pre-trained knowledge, enabling accurate few-shot classification without extensive adaptation time.
Solution Approach 2:
The patent changes the operational parameters of the Language Model by adjusting the perplexity computation approach and using label-specific augmentations rather than traditional fine-tuning methods. This parameter change allows the system to achieve few-shot classification accuracy without requiring extensive fine-tuning time, as the model leverages its pre-trained language understanding capabilities in a novel way tailored for classification tasks.
Data Source
Figure 1A
Figure 1B
Figure 2A
AI summary
State of art techniques using moderate sized Language Models (LMs) for text classification need fine-tuning or in-context learning. A method and system providing a two-step classification using moderate-sized (#params ≤ 2.7B) causal LM (Gen AI) is disclosed. Firstly, for a text instance to be classified, a set of perplexity and log-likelihood based features are obtained from an LM. Further, a light-weight classifier is trained in the second step to predict the final label. The system enables a new way of exploiting the available labelled instances, in addition to the existing ways like fine-tuning LMs or in-context learning. It neither needs any parameter updates in LMs like fine-tuning nor it is restricted by the number of training examples to be provided in the prompt like in-context learning. The key advantages of the disclosed system are explainability through most suitable key phrases and its applicability in resource poor environment.