LLM-Guided Rule Synthesis for Accurate Symbolic Question Answering
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing methods struggle to effectively translate natural language questions into symbolic reasoning rules for automated reasoning on content such as digital images, time series, or sensor data, leading to inaccuracies in answer derivation.
Innovation Solution
A method using a large language model (LLM) to generate rules in answer set programming syntax, guided by preprompts and prompts, ensuring the rules are syntactically correct and lead to accurate answers through automated reasoning on provided content.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Extent of automation
If a large language model is used to translate natural language questions into symbolic reasoning rules, then the automation of reasoning is improved, but the accuracy and syntactic correctness of the generated rules deteriorate
Solution Approach 1:
The system implements feedback loops where the generated rules are validated against the original question-answer pairs and content. The large language model receives feedback about whether its generated rules successfully derive the correct answers, allowing it to iteratively improve the accuracy and syntactic correctness of its rule generation while maintaining high automation.
Solution Approach 2:
The patent introduces an intermediary validation mechanism between the large language model and the final rule set. This intermediary layer checks the syntactic correctness and logical validity of generated rules before they are used for automated reasoning, ensuring accuracy while preserving the automation benefits of using an LLM.
2Ease of operation
If rules are generated without structured guidance, then the ease of operation is improved, but the reliability of automated reasoning deteriorates
Solution Approach 1:
The system performs preliminary actions by providing the large language model with structured templates, syntax rules, and formatting guidelines before it generates the actual reasoning rules. This preliminary guidance ensures that the generated rules are reliable and syntactically correct while maintaining ease of operation through automated generation.
Solution Approach 2:
The patent changes key parameters of the rule generation process by introducing constraints on syntax, structure, and format that the large language model must adhere to. These parameter changes (syntax requirements, structural templates) ensure reliability of the generated rules while the automated nature maintains ease of operation.
Data Source
AI summary
A device and a computer implemented method for determining rules for determining an answer to a question about content, in particular content representing a digital image, a time series, for example of sensor data, audio, synthetic data, or data that can be represented as objects and their relations, by automated reasoning on the question, and the content. The method includes providing a set of rules for determining the answer to the question; providing the question, the answer, and the content; determining a preprompt for a large language model explaining the set of rules; determining a prompt for the large language model instructing the large language model to add at least one rule to the set of rules such that the answer to the question about the content is derivable by automated reasoning on the question, the content, and the set of rules including the at least one rule.