LLM Multi-Stage Prompt Refinement for Accurate Math Reasoning
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Large Language Models (LLMs) often provide inaccurate answers, particularly in arithmetic reasoning tasks due to difficulties in analyzing natural language prompts involving different units or numerical relationships, leading to errors such as hallucinations.
Innovation Solution
A multi-stage processing approach is employed where a series of contextual sub-questions are generated and applied to the LLM in reverse order, using a taxonomical dictionary to refine the natural language prompt, ensuring accurate context is added before obtaining a final answer.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If a LLM directly answers a natural language prompt involving arithmetic reasoning, then the response is generated quickly, but the accuracy is poor due to hallucinations and incorrect analysis of numerical relationships
Solution Approach 1:
The patent segments the original natural language prompt into multiple contextual sub-questions that need to be answered. Each sub-question focuses on a specific aspect of the arithmetic reasoning task, such as identifying numerical values, determining operations, or comparing results. This segmentation allows the LLM to process complex arithmetic problems in manageable steps, improving accuracy while maintaining a systematic approach to handling the complexity.
Solution Approach 2:
The patent applies preliminary action by generating and answering contextual sub-questions before providing the final answer to the original prompt. The system first identifies and resolves intermediate computational steps, such as converting units, calculating intermediate values, or comparing numerical relationships, before synthesizing the final response. This preliminary processing ensures that all necessary computational steps are correctly executed before the final answer is generated.
2Reliability
If the LLM processes complex arithmetic reasoning tasks, then the potential for accurate answers increases, but the likelihood of hallucinations and factual errors increases
Solution Approach 1:
The patent implements feedback by using the answers to contextual sub-questions to refine and inform the final answer to the original prompt. The system continuously checks whether the intermediate results are consistent with the original question's requirements and adjusts the final response accordingly. This feedback mechanism helps prevent hallucinations by ensuring that each computational step is verified against the context established by previous steps.
Solution Approach 2:
The patent introduces contextual sub-questions as intermediaries between the original prompt and the final answer. These sub-questions act as mediators that break down the complex reasoning task into verifiable intermediate steps. By answering these intermediary questions first, the system creates a chain of logical reasoning that reduces the likelihood of factual errors, as each intermediate answer can be independently verified for consistency with the overall task.
3Reliability
If the LLM generates contextual sub-questions and processes them in reverse order, then the accuracy of the final answer improves, but the time and computational resources required increase
Solution Approach 1:
The patent applies inversion by processing the contextual sub-questions in reverse order of their generation. Instead of answering sub-questions sequentially from first to last, the system answers them from last to first, using the information gained from later sub-questions to inform earlier ones. This reverse processing allows the LLM to incorporate contextual information more effectively, improving the accuracy of the final answer while systematically managing the computational workload.
Data Source
AI summary
Example implementations include a method, apparatus, and computer-readable medium configured for receiving, at an interface between a user and a large language model, an original natural language prompt including a plurality of facts from the user. The implementations further include generating a series of contextual sub-questions based on the original natural language prompt using the large language model. Additionally, the implementations further include providing the contextual sub-questions to the large language model to obtain contextual answers. Additionally, the implementations further include applying the contextual sub-questions against the original natural language prompt with the contextual answers as a refined natural language prompt to the large language model in a reverse order of the series. Additionally, the implementations further include outputting, to the user, a final answer from the large language model to a terminal state of the refined natural language prompt.