Soft Prompt Translation for Small LM In-Context Learning
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
In-context learning performance is compromised as model size reduces, with smaller language models like T5-large achieving only around 5% accuracy on symbol removal tasks compared to large models like GPT-3 175B, which achieves over 60% accuracy.
Innovation Solution
An in-context learning framework that encodes demonstration examples into fixed-length soft prompts using a prompt translation model and generates task outputs based on augmented contexts, meta-training the model to adapt the prompt translation model while keeping the backbone LM frozen, thereby improving performance without altering the LM parameters.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Device complexity
If model size is reduced, then computational complexity and resource requirements decrease, but in-context learning ability deteriorates
Solution Approach 1:
The patent introduces an auxiliary model as an intermediary component that works alongside the small-scale language model. This auxiliary model processes demonstration examples and generates enhanced context representations, enabling the small model to achieve in-context learning capabilities comparable to large models without increasing its own size or computational complexity
Solution Approach 2:
The patent modifies the input context representation by transforming demonstration examples into enhanced formats through the auxiliary model. By changing how context information is encoded and presented to the small language model, the system compensates for the model's limited parameters and achieves better in-context learning performance
2Quantity of substance
If model size is reduced, then resource requirements and training costs decrease, but performance on in-context learning tasks deteriorates
Solution Approach 1:
The patent divides the in-context learning system into two functional segments: a small-scale language model for efficient inference and an auxiliary model for context enhancement. This segmentation allows each component to be optimized independently, with the small model requiring fewer resources while the auxiliary model compensates for performance limitations
Solution Approach 2:
The auxiliary model serves as a mediator that bridges the gap between limited model size and required task accuracy. It processes demonstration examples and generates enhanced context that the small model can effectively utilize, maintaining high accuracy while keeping resource requirements low
Data Source
AI summary
Embodiments described herein provide a data driven framework that (i) translates demonstration examples to a fixed-length soft prompt—a sequence of soft tokens; and (ii) learns a global (not generated from demonstrations) soft prompt. The framework then combines the global prompt, the translated prompts and the original context to create an augmented context which is given as final input for the backbone LM to use.