Synthetic Query Data Generation with Catalog-Aware LLM Prompts
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Generative AI models used for generating query language queries often experience hallucination, leading to incorrect or misleading results, especially when dealing with new data sources or customer queries, and existing synthetic data may not align with the user's query or be sufficient in quantity.
Innovation Solution
A method is employed to generate synthetic data by using dataset pairs and predicted catalog information to create prompts for generative AI models, resulting in augmented query pairs that are refined through negative feedback and stored as synthetic data if they meet criteria, ensuring alignment with user queries and database structure.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Adaptability or versatility
If a generative AI model is used to generate query language queries, then query generation capability is improved, but hallucination occurs leading to incorrect or misleading results
Solution Approach 1:
The system performs preliminary actions by generating synthetic training data before the actual query generation task. Dataset pairs of natural language queries and query language queries are created in advance and used to train the generative AI model, preparing it with relevant knowledge and patterns to reduce hallucination during actual operation.
Solution Approach 2:
The system implements feedback mechanisms where generated queries are evaluated for correctness and quality. The evaluation results are fed back to improve the model, creating a closed-loop system that continuously learns from its own performance and corrects hallucination issues through iterative refinement.
2Ease of manufacture
If existing synthetic data is used for training, then model training is simplified, but the data may not align with user queries or be sufficient in quantity
Solution Approach 1:
The system expands the training data in another dimension by generating synthetic data across multiple dimensions - different query types, different domains, and varying complexity levels. This multi-dimensional approach ensures sufficient quantity and diversity of training data without manually creating each example.
Solution Approach 2:
The system employs self-service by using the generative AI model itself to create the training data it will later consume. The model generates synthetic dataset pairs that are then used for its own training and refinement, creating a self-sustaining data generation cycle that eliminates manual data preparation bottlenecks.
Data Source
AI summary
Systems, methods, devices, and computer readable storage media described herein provide techniques for generating synthetic data for use in query generation. In an aspect, a pair comprising a natural language (NL) query and a query language (QL) query and predicted catalog information are used to prompt a large language model (LLM) to generate an augmented pair that is a variation of the pair. Synthetic data is generated comprising the augmented pair. In another aspect, an indication of feedback for a QL query generated by a LLM is received and a corrected pair is generated based on the indication and a corresponding NL query, the corrected pair comprises a corrected QL query and the NL query. The corrected QL query is a syntactically valid conversion of the NL query. The corrected pair is determined to satisfy criteria of a data store and is stored as synthetic data of the data store.