Rule-Map LLM Integration for Transparent Automated Review
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current rule-based systems lack robustness and transparency in automated decision-making due to the unreliability of Large Language Models (LLMs) and the need for manual data-driven checks, which hinders full automation and user control.
Innovation Solution
A hybrid approach combining rule-based systems with semantic correlation-based LLMs, using a rule map to guide LLM queries through a visually representable decision tree, ensuring transparent and controlled evaluations.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Extent of automation
If LLM is used to conduct rule-based review, then automation capability is improved, but reliability deteriorates due to hallucination and lack of control
Solution Approach 1:
The patent segments the rule-based review process into distinct components: rule definitions, fact extraction, and rule application. Each component is handled separately with clear boundaries, allowing the LLM to operate within constrained segments while maintaining overall system reliability through structured integration of these segments.
Solution Approach 2:
The patent introduces an intermediary rule engine that mediates between the LLM and the final decision-making process. The LLM extracts facts and suggestions, but the rule engine independently evaluates these against predefined rules, acting as a mediator that ensures reliability while preserving automation benefits.
2Reliability
If manual data-driven checks are performed, then reliability is improved, but productivity deteriorates due to time-consuming processes
Solution Approach 1:
The patent applies preliminary action by pre-defining all rules, decision criteria, and evaluation frameworks before the actual review process. This preparation work is done once and reused across multiple cases, enabling fast automated processing while maintaining reliability through consistent application of pre-validated rules.
Solution Approach 2:
The system enables self-service automation where the LLM independently extracts facts, identifies applicable rules, and generates preliminary assessments without manual intervention. The rule engine then automatically applies rules to these outputs, creating a self-running pipeline that maintains reliability through systematic rule application while achieving high productivity.
3Adaptability or versatility
If LLM behavior is left uncontrolled, then adaptability is improved, but ease of operation deteriorates due to lack of user control
Solution Approach 1:
The patent implements dynamics by allowing the system to adapt between different operational modes. The LLM can be configured with varying degrees of autonomy and different prompt strategies depending on the specific review task, while the rule engine maintains consistent rule application. This dynamic configuration enables both adaptability and user control.
Solution Approach 2:
The patent utilizes parameter changes by allowing users to adjust LLM configuration parameters such as temperature, max tokens, and system prompts without changing the underlying rule structure. These parameter changes enable the LLM to adapt to different review contexts while the fixed rule parameters ensure consistent, controllable, and predictable rule application.
Data Source
Figure 1
Figure 2~3
Figure 4
AI summary
Method for rule-based integration of a Large Language Model (LLM) with comprehensive rule trees (rulemaps), whose internal nodes represent logical links and whose leaves contain prompts for passing to the LLM, wherein each prompt is composed of a system prompt and one or more user prompts, wherein user prompts comprise a specific scenario and a query definition that determines the form of the result and are used in the rule tree at the next internal node to the prompt in the logical link, wherein the system prompt defines the context from globally specified settings and information that are predefined for the specific rulemap, wherein the prompts thus generated are passed sequentially to the LLM to generate a logical response that is suitable for evaluation in the rule tree, wherein the rule tree is traversed from the leaves to the root.to produce a logical final result.