Few-Shot Task-Oriented Dialogue via Function-Calling LLMs
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing TOD models face challenges in scalability and require large amounts of labeled data for training, especially when dealing with complex domains, and there is a reliance on excessively large language models that are difficult to control.
Innovation Solution
A few-shot learning approach using a moderately sized LLM (less than 10 billion parameters) with a novel architecture that standardizes tasks into a function-calling paradigm, incorporating domain-specific knowledge and fine-tuning strategies for efficient multitask training.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If multi-task training on three NLP tasks (NLU, DST, NLG) is used in traditional TOD models, then task performance is improved, but data annotation costs and system complexity increase significantly
Solution Approach 1:
The patent merges three separate NLP tasks (NLU, DST, NLG) into a single unified LLM-based system. Instead of training separate models for each task with distinct annotation requirements, the invention uses one LLM to handle all three tasks through prompt engineering and few-shot learning, thereby reducing system complexity while maintaining task performance.
Solution Approach 2:
The LLM serves multiple functions simultaneously - it performs natural language understanding, dialogue state tracking, and natural language generation within a single model framework. This multi-functionality eliminates the need for separate specialized models for each task, reducing both annotation costs and system complexity while preserving reliability.
2Measurement precision
If meticulously labeled training data is used for multi-task training, then model accuracy is improved, but data annotation costs increase significantly
Solution Approach 1:
The LLM performs self-service by generating its own training data through few-shot learning and in-context examples. Instead of requiring extensive manually annotated datasets for each task, the model uses a small set of examples to learn task patterns and generates appropriate responses autonomously, significantly reducing annotation costs while maintaining accuracy.
Solution Approach 2:
The invention changes the parameter of training data quantity from large-scale meticulously labeled datasets to small-scale few-shot examples. By altering the data regime and using prompt engineering techniques, the system achieves high model accuracy without incurring significant annotation costs.
3Productivity
If pre-trained large language models with hundreds of billions of parameters are used, then task execution capability is improved, but model control difficulty and resource requirements increase
Solution Approach 1:
Instead of relying on expensive, massive pre-trained LLMs with hundreds of billions of parameters, the invention uses smaller, more manageable LLMs (e.g., 7B or 13B parameters) that are sufficient for TOD tasks. These smaller models are easier to control, deploy, and manage while still achieving high task execution capability through effective prompt engineering and few-shot learning.
4Adaptability or versatility
If domain-specific knowledge is embedded into prompts for TOD tasks, then adaptability to new domains is improved, but prompt complexity and training requirements increase
Solution Approach 1:
The patent segments domain-specific knowledge into structured function calls with defined schemas. Instead of embedding unstructured domain knowledge into complex prompts, the system divides knowledge into discrete functions with specific parameters and types. This segmentation makes prompts more manageable and easier to adapt to new domains by simply adding new function definitions rather than rewriting entire prompt structures.
Data Source
AI summary
The invention proposes a novel method for developing a TOD model by representing input data as function calls and utilizing LLM as the foundational model to train multi-tasks via finetuning LLM with instruction. Furthermore, to implement the proposed model on LLMs with moderate sizes (fewer than 10 billion parameters), the invention also presents a finetuning LLM method to enhance the capability of these LLMs in terms of handling function-calling tasks. Finally, an effective training strategy with customized loss functions for each specific task is presented to optimize the training process. Experimental results on the standard MultiWOZ 2.2 dataset demonstrate the superior performance of the proposed method compared to existing approaches in this field of research.