Prompt Demonstration Selection via Embedding Similarity
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing machine learning systems require extensive computational resources and are time-consuming when training models for tasks like statement classification and paraphrasing, often resulting in biases and out-of-domain failures.
Innovation Solution
The system generates embeddings for sample statements and a target statement, compares them to determine similarity scores, and selects demonstrations based on these scores to create prompts for a machine learning model, allowing it to perform tasks without retraining or specifying tasks in the prompt.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If machine learning models are trained or augmented to include supervised generative models with large amounts of training data, then performance in classification and generative tasks is improved, but computational cost and training time increase significantly
Solution Approach 1:
The system pre-generates a library of demonstration samples with ground-truth labels and paraphrases before inference. During inference, it selectively retrieves relevant demonstrations based on similarity matching, avoiding the need to reprocess all training data. This preliminary preparation reduces computational cost during actual model usage while maintaining performance.
Solution Approach 2:
The system extracts only the most relevant demonstration samples from the reference data based on similarity scores, rather than using all available training data. This selective extraction reduces the amount of data processed during inference, lowering computational cost while maintaining model performance through targeted, relevant examples.
2Reliability
If all sample statements of reference data are included in prompts, then model performance may be improved, but computational resource consumption increases
Solution Approach 1:
The system assigns different weights to different demonstration samples based on their similarity to the target statement. Samples with higher similarity scores are given more weight in the prompt construction, while less similar samples receive lower weight or are excluded. This local quality differentiation ensures computational resources are focused on the most relevant information.
Solution Approach 2:
The system dynamically adjusts the number and selection of demonstration samples included in prompts based on similarity scores. By changing the parameter of demonstration sample selection from fixed (all samples) to dynamic (similarity-based selection), the system reduces computational resource consumption while maintaining performance through adaptive prompt construction.
3Reliability
If complex heuristics or rules are implemented to improve model performance, then accuracy may increase, but device complexity and computational resource utilization increase
Solution Approach 1:
The system introduces an embedding-based similarity scoring mechanism as an intermediary between the target statement and demonstration samples. This intermediary automatically ranks demonstrations by relevance, replacing the need for complex manual heuristics or rules. The embedding space serves as a mediator that simplifies the selection process while maintaining high accuracy through semantic similarity matching.
Data Source
AI summary
Techniques are described for a machine learning system configured to generate respective sample embeddings for a plurality of sample statements. The machine learning system may further be configured to generate a statement embedding for a statement. The machine learning system may further be configured to determine, based on the sample embedding and the statement embedding, respective similarity scores for the sample embeddings. The machine learning system may further be configured to select, based on the respective similarity scores for the sample embeddings, one or more sample statements from the plurality of sample statements. The machine learning system may further be configured to generate a prompt including the one or more sample statements, the statement, and at least one of respective ground-truth information or respective paraphrases for the selected one or more sample statements. The machine learning system may further be configured to provide the prompt to a machine learning model.