LLM Prompt Engineering With Feedback-Based Example Selection
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Traditional large language models (LLMs) face challenges such as hallucinations, high training costs, and inflexibility in handling complex or new questions, necessitating improved prompt engineering and feedback mechanisms.
Innovation Solution
Implementing multi-stage prompt engineering and iterative, feedback-based generative techniques to enhance LLM performance, using dual-purpose prompt examples and LLM-agent ensembles for flexible handling of various question complexities.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Adaptability or versatility
If traditional large language models are used for question-answering, then they can generate creative responses, but they hallucinate data and produce inaccurate outputs
Solution Approach 1:
The patent introduces an intermediary verification mechanism between the LLM's creative generation and the final output. A separate verification model or fact-checking module acts as a mediator to validate the generated content against the knowledge base, allowing the system to maintain both creative response generation and factual accuracy.
Solution Approach 2:
The patent replaces the purely probabilistic generation mechanism of traditional LLMs with a hybrid system that incorporates deterministic verification steps. Instead of relying solely on statistical patterns, the system substitutes in a verification layer that checks facts against grounded evidence, reducing hallucinations while preserving generative capabilities.
2Reliability
If extractive LLMs are used to constrain answers to supporting evidence, then hallucinations are prevented, but training costs and computational requirements become prohibitively expensive
Solution Approach 1:
The patent segments the question-answering process into distinct modules: an extractive component that grounds answers in evidence, a generative component that formulates natural language responses, and a verification component that ensures accuracy. This segmentation allows each component to be optimized independently, reducing overall computational costs while maintaining reliability.
Solution Approach 2:
The patent applies partial extraction by selecting only the most relevant supporting evidence portions needed to answer each question, rather than processing entire documents or datasets. This selective approach maintains grounding in evidence while significantly reducing computational requirements and training costs.
3Measurement precision
If specific prompts are tailored for each question scenario, then answer accuracy improves, but prompt engineering complexity and time requirements increase
Solution Approach 1:
The patent develops universal prompt templates that can handle multiple question types and scenarios through parameterization. Instead of creating unique prompts for each scenario, the system uses a single flexible template structure that adapts to different questions by inserting relevant parameters, significantly reducing prompt engineering complexity while maintaining accuracy.
Solution Approach 2:
The patent performs preliminary analysis of the question to identify key parameters and select appropriate evidence sources before generating the prompt. This preliminary action allows the system to configure the universal template correctly without requiring complex manual prompt engineering for each specific scenario.
4Device complexity
If a single prompt is used for all questions, then system simplicity is maintained, but the system fails to handle complex or unforeseen circumstances
Solution Approach 1:
The patent implements dynamic prompt selection where the system automatically adjusts the prompt structure and parameters based on the complexity and type of the input question. For simple questions, a basic prompt is used; for complex questions, the system dynamically enhances the prompt with additional context, constraints, or verification steps, maintaining simplicity when possible while adapting to complexity when needed.
Data Source
AI summary
Various embodiments of the present disclosure provide prompt engineering and iterative, feedback-based generative techniques that improve traditional LLM technology, including extractive LLM techniques. The techniques may include generating, using a machine learning classification model, a resolution capability classification for an input data object that comprises an input question and an input document; generating using a large language model (LLM), an initial predictive output for the input data object based on an initial generative model prompt for the input data object; identifying a classification model output divergence based on a comparison between the resolution capability classification and the initial predictive output; and in response to the classification model output divergence: generating an augmented generative model prompt by modifying the initial generative model prompt with a representation of the resolution capability classification, and generating, using the LLM, an updated predictive output based on the augmented generative model prompt.