SLM Policy Generation with Verified Rationales and Hierarchical Reasoning
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing large language models (LLM) face challenges in operating efficiently on devices with limited computational resources, particularly in making short-term decisions and adapting to dynamic environments, due to high computational requirements and complexity in distilling complex tasks.
Innovation Solution
A policy generating apparatus and method for small language models (SLM) that constructs and verifies a rationale dataset, learns a reasoning policy and a planning policy through an embodied knowledge graph, and enables zero-shot deployment in new environments, using a two-step hierarchical structure and contrastive learning to distill LLM reasoning ability into SLM.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Adaptability or versatility
If LLM is used for reasoning and planning in commercial devices, then decision-making capability is improved, but computational resource requirements increase beyond device capacity
Solution Approach 1:
The patent segments the monolithic LLM into two distinct policies: a reasoning policy that handles high-level decision-making and a planning policy that handles executable action sequences. This segmentation allows the reasoning policy to operate with reduced computational overhead while the planning policy handles the complexity of action generation, making the system suitable for resource-constrained commercial devices.
Solution Approach 2:
The patent extracts the reasoning capability from the full LLM and encapsulates it in a dedicated reasoning policy that uses a simplified state representation and decision framework. This extraction allows the device to benefit from LLM-level reasoning without requiring the full computational resources of the original model.
2Quantity of substance
If direct end-to-end distillation of LLM into SLM is performed, then model size is reduced for resource efficiency, but performance on complex specific tasks deteriorates
Solution Approach 1:
Instead of direct end-to-end distillation, the patent segments the distillation process into two separate policies with distinct functions. The reasoning policy handles complex decision-making while the planning policy handles action generation. This segmented approach preserves task-specific performance by allowing each policy to specialize in its function rather than requiring a single small model to handle all complexities.
Solution Approach 2:
The patent introduces a rationale dataset as an intermediary that bridges the gap between LLM knowledge and SLM implementation. This rationale dataset contains task-specific reasoning patterns and domain knowledge that guide the reasoning policy, enabling the small model to achieve LLM-level performance on specific tasks without requiring the full LLM architecture.
3Adaptability or versatility
If LLM relies on continuing to use large model architecture, then reasoning ability is maintained, but short-term decision-making efficiency decreases in resource-constrained environments
Solution Approach 1:
The patent extracts only the essential reasoning components from the LLM architecture and implements them in a lightweight reasoning policy. This extraction removes unnecessary computational overhead while preserving the core reasoning ability needed for short-term decision-making in resource-constrained environments.
Solution Approach 2:
The patent changes the parameter representation from full LLM token sequences to a simplified state space representation that includes only the most relevant features for decision-making. This parameter change reduces computational complexity while maintaining reasoning effectiveness for short-term decisions.
Data Source
AI summary
It is about the policy generating apparatus and method for SLM, the policy generating method for SLM may comprise receiving an expert dataset, generating a rationale dataset based on the expert dataset and a pre-stored initial rationale set, verifying the rationale dataset through a self-verification function, learning a reasoning policy through an embodied knowledge graph based on the verified rationale dataset, learning a planning policy based on a rationale set of the learned reasoning policy and a planning policy reconstruction loss and generating an SLM policy including a final reasoning policy and a planning policy based on the learned reasoning policy and the planning policy.