Human-AI Prompt Engineering with Similarity-Guided Feedback
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing large language models (LLMs) are highly dependent on prompt choices, leading to unsatisfactory results due to small changes in prompts, necessitating improved systems and techniques for efficient and effective prompt implementation.
Innovation Solution
A method involving a computing system that applies candidate prompts to LLMs, calculates a pairwise similarity matrix of responses, and determines when to involve human feedback based on response similarity, optimizing prompt generation through human-AI collaboration.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If multiple candidate prompts are applied to LLM to improve performance, then prompt accuracy is improved, but computational time and resources increase
Solution Approach 1:
The system applies partial action by evaluating multiple candidate prompts but selecting only the top-k most promising ones for further evaluation, rather than exhaustively testing all possible prompts. This reduces computational time while maintaining prompt accuracy by focusing resources on the most likely successful candidates.
Solution Approach 2:
The system performs preliminary filtering of candidate prompts using automated evaluation metrics before presenting them to users for feedback. This preliminary action reduces the number of prompts that require expensive computational resources or human review, thereby reducing overall computational time while maintaining accuracy.
2Measurement precision
If human feedback is integrated into prompt optimization, then prompt quality is improved, but system complexity increases
Solution Approach 1:
The system implements feedback loops where user responses to candidate prompts are collected and used to refine future prompt generation. This feedback mechanism improves prompt quality by incorporating human judgment while the automated processing of feedback keeps system complexity manageable.
Solution Approach 2:
The system uses automated evaluation metrics and algorithms as intermediaries between the LLM and human users. These intermediaries pre-process and filter prompts before human review, reducing the burden on users and simplifying the overall system architecture while maintaining prompt quality.
3Measurement precision
If computational resources are increased for prompt optimization, then prompt generation accuracy is improved, but resource efficiency deteriorates
Solution Approach 1:
The system applies partial action by using computational resources to evaluate only a subset of candidate prompts (top-k) rather than all possible prompts. This improves prompt generation accuracy for the selected prompts while maintaining resource efficiency by avoiding exhaustive evaluation of less promising candidates.
Solution Approach 2:
The system dynamically adjusts parameters such as the number of candidate prompts evaluated, the threshold for presenting prompts to users, and the level of automated vs. manual review based on available computational resources. This allows the system to maintain prompt generation accuracy while adapting resource efficiency to available capacity.
Data Source
AI summary
Embodiments herein describe techniques for optimizing prompts describing tasks for a large language model (LLM), to enable effective and efficient operations to optimize prompt generation and selection for various tasks in LLMs through human-AI collaboration. In an embodiment, a computing system applies a set of candidate prompts to the LLM, evaluates responses to the candidate prompts received from the LLM and calculates a pairwise similarity matrix of responses based on the received responses to the candidate prompts. The computing system evaluates the pairwise similarity matrix to determine whether or not to present a prompt to receive a user feedback input. The described techniques can enable enhanced processing speed, reducing an overall computer system time typically required for implementing optimized prompt generation, and enhancing performance of the computing system executing the LLM.