LLM Evaluation Question Generation With Feedback-Refined Prompts
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing methods for evaluating generative AI models are inefficient and require significant time for evaluators to generate precise and effective question data, lacking a systematic approach for generating questions to assess the performance of these models.
Innovation Solution
A method involving a service server and user terminal that inputs prompts into a large language model, receives user feedback, and iteratively refines prompts to generate a plurality of final evaluation questions, improving reliability and accuracy through feedback-based refinement.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If evaluators directly generate precise and efficient question data to evaluate the generative AI model, then the evaluation quality and reliability are improved, but the working time and resource consumption increase significantly
Solution Approach 1:
The patent applies preliminary action by pre-generating candidate evaluation questions using the generative AI model itself before the actual evaluation process. The system creates multiple candidate questions in advance, which are then refined through feedback loops, allowing evaluators to work with pre-prepared materials rather than generating questions from scratch during evaluation, thus reducing working time while maintaining quality
Solution Approach 2:
The patent implements feedback mechanisms where evaluation results and performance data are fed back into the system to refine and improve future question generation. The generative AI model uses evaluation feedback to adjust its output, creating a continuous improvement loop that enhances evaluation reliability over time while reducing the need for manual question generation
2Reliability
If evaluators manually create comprehensive question lists to assess the generative AI model, then the coverage and thoroughness of evaluation are improved, but the complexity and resource requirements increase
Solution Approach 1:
The patent applies self-service by enabling the generative AI model to automatically generate its own evaluation questions without requiring external evaluators to manually create comprehensive question lists. The model serves itself by producing candidate questions, which are then processed through automated feedback loops, significantly reducing the complexity and resource requirements while maintaining evaluation thoroughness
Solution Approach 2:
The patent utilizes parameter changes by adjusting the generation parameters of the large language model to optimize question output. By modifying parameters such as temperature, top-k sampling, and prompt structures, the system efficiently generates diverse and comprehensive evaluation questions, reducing the need for complex manual evaluation frameworks
3Quantity of substance
If a large number of evaluation questions are generated to comprehensively assess the generative AI model, then the evaluation coverage is improved, but the time and computational resources required increase
Solution Approach 1:
The patent applies partial or excessive action by generating a large number of candidate evaluation questions beyond what is strictly necessary, then using feedback mechanisms to filter and refine them. The system generates excessive candidate questions initially to ensure comprehensive coverage, then efficiently reduces the set through automated evaluation, achieving both high coverage and maintained productivity
Solution Approach 2:
The patent implements segmentation by dividing the evaluation question generation process into distinct stages: initial candidate generation, feedback-based refinement, and final selection. This segmented approach allows the system to generate large numbers of questions efficiently in the first stage, then process them through specialized refinement stages, maintaining overall productivity while achieving comprehensive evaluation coverage
Data Source
AI summary
The present invention relates to a method for generating a question to evaluate a system using a deep learning-based generative model, which inputs, by a service server, a first prompt including a purpose of a question and information related to a system using a deep learning-based generative model to be evaluated into a large language model, transmits, by a user terminal, feedback information, which is input by a user with respect to a sample question output from the large language model, to the service server, and inputs, by the service server, a second prompt reflecting a feedback according to the feedback information into the large language model or requests the large language model to generate a plurality of final evaluation questions through a prompt input from the large language model.