LLM Self-Generated Demonstrations for Consistent Reasoning Prompts
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Large language models (LLMs) face challenges in zero-shot and few-shot settings due to sensitivity to the choice of examples, requiring significant effort and expertise, and often produce spurious reasoning steps or underperform without labeled examples.
Innovation Solution
The consistency-based self-adaptive prompting (COSP) method selects and builds a set of examples from LLM zero-shot outputs using predetermined criteria that combine consistency, diversity, and repetition, enhancing performance by generating and selecting suitable question-reasoning pairs without ground-truth labels.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Adaptability or versatility
If zero-shot chain of thought is used without template rationale, then task-agnostic operation is achieved, but spurious reasoning steps are produced and performance deteriorates
Solution Approach 1:
The system performs preliminary action by generating multiple candidate reasoning paths before final output selection. Multiple LLM inferences are executed with different reasoning trajectories, and these candidates are then evaluated through self-consistency checking and voting mechanisms to select the most reliable reasoning path, thereby eliminating spurious reasoning while maintaining task-agnostic operation
Solution Approach 2:
The system implements feedback through self-consistency evaluation where the LLM's own outputs are used to verify reasoning correctness. Multiple reasoning paths are generated and compared against each other through voting mechanisms, creating an internal feedback loop that identifies and corrects spurious reasoning steps without requiring external labeled data
2Reliability
If few-shot demonstrations are used to improve reasoning accuracy, then performance improves, but significant manual effort and expertise are required for example selection
Solution Approach 1:
The system applies self-service by enabling the LLM to generate its own few-shot demonstrations automatically without external human intervention. The model creates reasoning examples from its own capabilities, performing self-pretraining where it generates and selects demonstrations autonomously, thereby improving reasoning accuracy while eliminating the complexity of manual example selection
Solution Approach 2:
The system changes parameters by dynamically adjusting the number of shots and selecting demonstrations based on self-consistency metrics rather than fixed manual configurations. The demonstration selection is adapted automatically based on the specific task and model performance, transforming the static few-shot approach into a dynamic self-adaptive process that reduces complexity
3Reliability
If more demonstrations are provided to improve performance, then reasoning accuracy improves, but computational costs and memory usage increase
Solution Approach 1:
The system applies partial action by generating a limited number of reasoning paths (e.g., 5-10 candidates) rather than exhaustively exploring all possible reasoning trajectories. This partial sampling through beam search or temperature sampling provides sufficient diversity for accurate voting while avoiding the excessive computational cost of generating all possible reasoning paths
Solution Approach 2:
The system changes parameters by dynamically adjusting the number of demonstrations based on task difficulty and model confidence. For simpler tasks or high-confidence predictions, fewer demonstrations are used, reducing computational cost. For more challenging tasks, the system automatically increases the number of demonstrations, optimizing the balance between accuracy and computational efficiency
Data Source
AI summary
Aspects of the disclosure are directed to automatically selecting examples in a prompt for an LLM to demonstrate how to perform tasks. Aspects of the disclosure can select and build a set of examples from LLM zero-shot outputs via predetermined criteria that can combine consistency, diversity, and repetition. In the zero-shot setting for three different LLMs, using only LLM predictions, aspects of the disclosure can improve performance up to 15% compared to zero-shot baselines and can match or exceed few-shot base-lines for a range of reasoning tasks.