Foundation Model Reasoning Trees Without Fine-Tuning Data
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing large language models (LLMs) face challenges in complex reasoning tasks due to the reliance on extensive training and extra data preparation, particularly in Supervised Fine-Tuning (SFT) methods, which require significant efforts and resources.
Innovation Solution
A data-free search-based approach is employed to enhance LLM reasoning ability by generating multiple reasoning trees through query augmentation and using a Process-supervised Reward Model (PRM) to score and select optimal reasoning paths, enabling robust reasoning without additional data collection or fine-tuning overhead.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If Supervised Fine-Tuning (SFT) is used to enhance LLM reasoning ability, then reasoning performance is improved, but data preparation effort and training complexity increase significantly
Solution Approach 1:
The patent applies preliminary action by pre-computing and storing reasoning trees during an offline phase, so that during the online inference phase, the model can directly query and retrieve pre-generated reasoning paths without performing complex training or data preparation. This separates the heavy computational work into a preliminary offline stage, leaving the online stage simple and efficient.
Solution Approach 2:
The patent uses copying by generating multiple candidate reasoning trees and selecting from them during inference, rather than training the model from scratch or requiring extensive fine-tuning data. The model copies existing reasoning patterns from pre-generated trees to solve new problems, avoiding the need for complex supervised fine-tuning procedures.
2Reliability
If Supervised Fine-Tuning (SFT) is used to enhance LLM reasoning ability, then reasoning performance is improved, but data collection requirements increase
Solution Approach 1:
The patent applies self-service by enabling the system to generate its own reasoning trees internally using a foundation model, without requiring external data collection efforts. The system serves itself by utilizing the foundation model's capabilities to create training data on-demand, eliminating the need for manual data gathering and preparation.
Solution Approach 2:
The patent performs preliminary action by pre-generating reasoning trees during an offline phase, so that during online inference, no additional data collection is required. The heavy data generation work is completed in advance, making the online system data-independent and simplifying deployment.
3Reliability
If extensive training is performed to improve LLM reasoning, then reasoning ability increases, but computational resources and time consumption increase
Solution Approach 1:
The patent applies preliminary action by performing heavy computational work during an offline phase to generate and store reasoning trees, so that during online inference, the system only needs to perform lightweight queries and retrievals. This shifts the computational burden from the online phase to the offline phase, reducing real-time resource requirements.
Solution Approach 2:
The patent uses copying by retrieving pre-computed reasoning trees during inference rather than performing new complex computations. The system copies existing reasoning patterns from stored trees to answer new questions, significantly reducing the computational resources needed during online service compared to training or generating reasoning from scratch.
4Ease of manufacture
If data-free approach is used to enhance reasoning, then data collection effort is reduced, but reasoning accuracy may be compromised
Solution Approach 1:
The patent applies feedback by using a reward model to evaluate and score generated reasoning trees, then using this feedback to guide the selection of the best reasoning path. The reward model provides continuous feedback during the reasoning process, allowing the system to iteratively improve the accuracy of selected reasoning paths without requiring additional training data.
Solution Approach 2:
The patent uses copying by retrieving pre-computed reasoning trees that were generated using the foundation model's internal knowledge, rather than requiring external training data. The system copies proven reasoning patterns from the foundation model to solve new problems, maintaining high accuracy while avoiding data collection requirements.
Data Source
AI summary
A computerized method has the steps of: generating one or more queries from an input question; generating one or more outputs; and outputting an answer based on the one or more outputs. Said generating the one or more outputs has the steps of: for each query, forming a reasoning tree with the query being a root node and a current node, generating one or more candidates as leaf nodes of the current node, by inputting a reasoning path from the root node to the current node into a foundation model, searching the reasoning tree using an artificial intelligence model to select a leaf node as the current node, and repeating said generating the one or more candidate nodes and said searching the reasoning tree until a termination condition is met, and using an updated reasoning path from the root node to the current node as the output for the query.