LLM Rule Generation for Reliable Automated Reasoning
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing methods struggle to effectively translate natural language questions into symbolic reasoning rules for automated reasoning on diverse content types such as digital images, time series, and sensor data, leading to inefficiencies and potential errors in deriving answers.
Innovation Solution
A method and device utilizing a large language model (LLM) to generate answer set programs by translating problem specifications into rules, ensuring compliance with answer set programming (ASP) syntax, and incorporating automated reasoning to refine and validate these rules.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Adaptability or versatility
If a large language model is used to translate natural language questions into symbolic reasoning rules, then the adaptability to diverse content types is improved, but the reliability of the generated rules deteriorates due to potential syntax errors and semantic inaccuracies
Solution Approach 1:
The system implements feedback loops where the generated rules are validated against the original question-answer pairs and content. The automated reasoning engine executes the generated rules and compares the derived answers with expected answers, providing feedback to identify and correct syntax errors and semantic inaccuracies in the rules generated by the large language model.
Solution Approach 2:
An intermediary validation and correction module is introduced between the large language model and the automated reasoning engine. This intermediary component checks the generated rules for syntax compliance, validates semantic correctness, and performs corrections before the rules are executed, thereby improving reliability without compromising adaptability.
2Reliability
If automated reasoning is implemented to validate generated rules, then the reliability of the reasoning process is improved, but the complexity of the system increases
Solution Approach 1:
The system is segmented into distinct functional modules: a large language model for rule generation, a validation module for syntax checking, a correction module for fixing errors, and an automated reasoning engine for execution. This segmentation allows each component to be optimized independently and simplifies the overall system architecture by clearly defining the responsibilities of each module.
Solution Approach 2:
The system incorporates self-service capabilities where the automated reasoning engine validates its own generated rules and performs self-correction. The system automatically identifies syntax errors, retrieves correct rule formulations, and updates the rule set without requiring external intervention, thereby managing complexity through autonomous operation.
3Manufacturing precision
If syntax validation is performed on generated rules, then the manufacturing precision of the rules is improved, but the productivity of the rule generation process decreases
Solution Approach 1:
The system performs preliminary syntax validation and error detection immediately after rule generation, before the rules are executed. By catching and correcting syntax errors early in the process, the system avoids time-consuming rework later and maintains high productivity while ensuring precision.
Solution Approach 2:
The validation process is optimized to skip redundant checks and focus only on critical syntax elements that are most likely to cause errors. The system rapidly validates essential rule structures without performing exhaustive checks on every aspect, thereby maintaining high precision while minimizing the impact on generation productivity.
Data Source
Figure 1~2
Figure 3
AI summary
A device and a computer implemented method for determining rules for determining an answer to a question about content, in particular content representing a digital image, a time series, for example of sensor data, audio, synthetic data, or data that can be represented as objects and their relations, by automated reasoning on the question, and the content, wherein the method comprises providing (202) a set of rules for determining the answer to the question, providing (208) the question, and the answer, and the content, determining (204) a preprompt for a large language model explaining the set of rules, determining (210) a prompt for the large language model instructing the large language model to add at least one rule to the set of rules such that the answer to the question about the content is derivable by automated reasoning on the question, and the content, and the set of rules including the at least one rule, prompting (206) the large language model with the preprompt, prompting (212) the large language model with the prompt, receiving (214) a response of the large language model to the prompt, wherein the response comprises the at least one rule.