LLM Agent Test Dataset Generation With Conversation Graph Grounding

Resolve Bottlenecks,
Find Innovative SolutionsGenerate Solutions

Solution Overview

Problem

Evaluating large language model agents for realistic conversations and procedures is challenging due to the difficulty in obtaining high-quality test datasets and the tendency of LLMs to hallucinate content not grounded in input procedures, which affects the effectiveness and veracity of deployed virtual AI agents.

Innovation Solution

A method is provided to generate test datasets by extracting APIs, generating flowgraphs and conversation graphs, inserting noise, and sampling paths to create synthetic conversations that mimic real-world interactions, thereby reducing hallucinations and improving evaluation resilience.

Engineering Contradictions & Design Principles

VSEngineering Contradiction Analysis

1Productivity

If LLMs are used to generate test datasets, then productivity is improved, but measurement precision deteriorates due to hallucinations

Engineering Contradiction:

Improvetest dataset generation efficiencyVSAvoidevaluation accuracy

Core Design Contradiction:

ProductivityVSMeasurement precision

Solution Approach 1:

The patent introduces an intermediary validation mechanism where generated test datasets are verified against the original conversation graph and procedure definitions. This intermediary step filters out hallucinated content while preserving the efficiency of AI-generated datasets, resolving the contradiction between productivity and measurement precision.

Inventive Principle:

Principle #24Intermediary (Mediator)

Solution Approach 2:

The system implements feedback loops where the evaluation results are used to refine the test dataset generation process. By continuously comparing generated datasets against ground truth procedures and incorporating evaluation feedback, the system improves measurement precision while maintaining high productivity through automated iterative refinement.

Inventive Principle:

Principle #23Feedback

2Adaptability or versatility

If diverse test scenarios are generated, then adaptability is improved, but device complexity increases

Engineering Contradiction:

Improvetest scenario coverageVSAvoidgeneration system complexity

Core Design Contradiction:

Adaptability or versatilityVSDevice complexity

Solution Approach 1:

The patent segments the test dataset generation process into distinct modular components: conversation graph generation, procedure extraction, API identification, and validation modules. This segmentation allows diverse test scenarios to be generated through systematic combination of modular elements, improving adaptability while managing complexity through organized structure.

Inventive Principle:

Principle #1Segmentation

Solution Approach 2:

The system employs universal templates and structures for generating test datasets that can be applied across different domains and agent types. By using multi-functional generation rules that work across various scenarios, the system achieves broad adaptability without proportionally increasing system complexity.

Inventive Principle:

Principle #6Universality (Multi-functionality)

Data Source

PatentUS20260050539A1Generating test datasets for evaluating virtual agents

Publication Date: 2026.02.19 ZENDESK INC

AI summary

A method of generating a set of test datasets for evaluating large language model agents, the method including: extracting, using a large language model, application programming interfaces (APIs) associated with procedures for one or more target intents; generating, using the large language model, a flowgraph based on the APIs and the procedures for the one or more target intents; generating, using the large language model, a conversation graph based on the flowgraph; generating, using the large language model, conversations based on at least the conversation graph, the APIs, and a series of sampled paths from the conversation graph; and extracting the set of test datasets from the conversations.