LLM Math Reasoning with Consensus-Checked Expressions
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Large Language Models (LLMs) face challenges in accurately solving mathematical problems, often providing incorrect answers with false confidence, and existing prompting techniques are inadequate for multi-step, deliberate reasoning tasks.
Innovation Solution
The approach involves transforming the initial query into a template query with variable inputs, generating multiple prompts for different solutions, evaluating these solutions with randomized values, and achieving a statistically significant consensus to ensure accuracy.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If LLMs are used to solve mathematical problems directly, then the process is simple and fast, but the accuracy is low and incorrect answers are provided with false confidence
Solution Approach 1:
The solution process is segmented into distinct phases: template query creation, multiple prompt generation, analytical expression extraction, numerical evaluation with randomized values, and consensus determination. Each phase handles a specific aspect of the problem-solving task, improving overall reliability through structured decomposition of the mathematical reasoning process
Solution Approach 2:
An intermediary verification system is introduced that uses numerical evaluation with randomized values to validate the analytical expressions generated by the LLM. This intermediary layer acts as a mediator between the LLM's symbolic reasoning and the final numerical answer, filtering out incorrect solutions before they are presented as final results
2Reliability
If multiple prompts and evaluations are performed to achieve consensus, then the accuracy improves, but the time and computational resources increase
Solution Approach 1:
The system performs preliminary actions by generating multiple diverse prompts and their corresponding analytical expressions before numerical evaluation. This preliminary generation of multiple solution paths allows the consensus mechanism to work more efficiently, as the validation phase only needs to evaluate pre-generated expressions rather than generating and validating solutions iteratively
Solution Approach 2:
The system changes parameters by using randomized numerical values for evaluation instead of fixed test cases. This parameter variation allows the same analytical expression to be validated across multiple different input scenarios, providing more robust verification without requiring proportional increases in processing time
3Adaptability or versatility
If template queries with variables are used instead of direct input, then the generality and adaptability improve, but the complexity of query transformation increases
Solution Approach 1:
The template query structure serves multiple functions: it acts as a standardized input format for the LLM, a framework for generating diverse prompts, and a basis for creating analytical expressions with variables. This universal template approach increases adaptability across different mathematical problem types while the automation of template creation reduces the perceived complexity
Data Source
AI summary
Disclosed are techniques for an AI system with a large language mode (LLM) with improved accuracy and reliability in solving mathematical problems. An initial query is transformed into a template query by replacing the original input values with variables. Multiple prompts are sent to the LLM, each being different from one another, and contextually related to the template query. Multiple results are responsively received from the LLM, each result including an analytical expression to solve the mathematical problem. Each of the expressions is evaluated using a numerical evaluation tool with variables of the expression being assigned a common set of randomly sampled values. A consensus is achieved when the evaluated expressions satisfy a consensus condition, such as when all outputs match consistently over N experiments or trials. After the consensus condition is reached, the original inputs are evaluated with one or more of the expressions, and the solution is output.