Soft-Prompt Dataset Reconstruction for Low-Resource Text Classification
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing data augmentation methods for text classification in low-resource settings often produce grammatically and semantically incorrect texts, limiting the performance of machine learning models.
Innovation Solution
A method involving a trained multilayer perceptron (MLP) network to generate soft prompts from hidden representations of a pre-trained classifier, which are used to tune a frozen language model for reconstructing original sentences, thereby generating new text for augmentation.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Quantity of substance
If simple transformations (cropping, padding, flipping, shifting) are applied to textual data for augmentation, then the data volume increases, but the text becomes grammatically and semantically incorrect and misrepresents the actual text distribution
Solution Approach 1:
The patent replaces mechanical transformation methods (cropping, padding, flipping) with a language model-based generation system. The frozen language model generates new text instances that are semantically and grammatically correct, substituting the mechanical approach with an intelligent generation approach that understands language structure and meaning.
Solution Approach 2:
The patent changes the approach from transforming existing text instances to generating new text instances with different parameters. By using the language model to generate new sentences with varied wording, structure, and expressions while maintaining the same meaning, the system achieves data augmentation without compromising text quality.
2Quantity of substance
If rule-based data augmentation methods (synonym replacement, word deletion, word changing) are used for textual data, then the data volume increases, but the text distribution deviates from the actual distribution and quality deteriorates
Solution Approach 1:
The patent replaces rule-based augmentation methods with a language model-based generation system. Instead of applying rigid rules like synonym replacement or word deletion, the system uses a frozen language model to generate new text instances that naturally follow the actual text distribution and maintain grammatical correctness.
Solution Approach 2:
The frozen language model leverages its pre-trained knowledge and understanding of language to autonomously generate high-quality augmented text instances. The model uses its internalized language patterns and distributions to create text that naturally adheres to the actual text distribution without requiring external rules or constraints.
3Reliability
If pre-trained language models are used to generate new text for data augmentation, then the text quality and model performance improve, but the computational complexity and training requirements increase
Solution Approach 1:
The patent uses a frozen (pre-trained) language model that has already been pre-trained on large corpora. This preliminary training action is performed beforehand, allowing the model to be used for data augmentation without requiring additional training. The pre-trained model's knowledge and language understanding are leveraged directly, reducing the complexity of the current system.
Solution Approach 2:
The patent uses the frozen language model's pre-trained parameters and knowledge as a copy of general language understanding. Instead of training a new model from scratch, the system copies and utilizes the pre-trained model's capabilities for the specific task of text augmentation, significantly reducing computational complexity and training requirements.
Data Source
AI summary
A computer-implemented method comprising: receiving a source dataset comprising a plurality of textual data instances and corresponding labels in two or more classes; training a machine learning classifier on the source dataset; performing inference by the trained machine learning classifier over a subset of the data instances in the source dataset, to extract a hidden representation for each of said data instances in said subset; applying a trained multilayer perceptron (MLP) network to the extracted hidden representations, to generate a set of corresponding soft prompts; and feeding the generated set of soft-prompts as prompts for a trained language model, to tune the trained language model to reconstruct the data instances in the subset.