Text-to-SQL Fine-Tuning Data Generation for LLM Accuracy
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing large language models (LLMs) struggle with accuracy in generating database statements due to the time and cost-intensive process of manually curating high-quality training samples for supervised finetuning, particularly in addressing specific use cases like complex JOIN queries or SQL aggregate functions, leading to erroneous patterns and inconsistent performance.
Innovation Solution
A multistep inference framework generates a finetuning corpus by leveraging a custom chain of prompts and self-planning techniques to automatically produce high-quality Text-to-SQL data samples, focusing on syntactic diversity and requiring minimal human annotation, using a modular pipeline that can be tailored to specific issues and LLMs.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If manual curation of high-quality training samples is performed, then LLM accuracy improves, but time consumption and cost increase
Solution Approach 1:
The system employs an LLM to automatically generate its own training data by synthesizing natural language questions and corresponding SQL queries based on database schemas. This self-service approach eliminates the need for manual curation of training samples while maintaining high data quality, directly resolving the contradiction between improving LLM accuracy through manual curation and reducing time consumption.
Solution Approach 2:
The invention introduces an intermediary automated data generation pipeline that uses the LLM itself to create training samples. This intermediary process transforms the manual curation task into an automated synthesis process, where the LLM generates diverse SQL queries and natural language questions based on provided database schemas, thereby improving accuracy without increasing time consumption or cost.
2Reliability
If manual curation of high-quality training samples is performed, then LLM accuracy improves, but cost increases
Solution Approach 1:
The system enables the LLM to self-generate training data by synthesizing natural language questions and SQL queries autonomously based on database schemas. This eliminates the need for expensive manual curation by skilled developers while maintaining high data quality, directly addressing the contradiction between improving LLM accuracy and reducing cost.
Solution Approach 2:
The invention creates synthetic copies of training data through automated synthesis rather than manual creation. The LLM generates multiple variations of natural language questions and corresponding SQL queries by copying and adapting patterns from the database schema, providing high-quality training samples at minimal cost compared to manual expert curation.
3Reliability
If a large number of training samples are collected manually, then LLM accuracy improves, but productivity decreases
Solution Approach 1:
The LLM autonomously generates large volumes of training samples by synthesizing natural language questions and SQL queries based on database schemas without requiring manual intervention. This self-service capability dramatically improves data generation productivity while ensuring the samples are sufficient in number and quality to improve LLM accuracy.
Solution Approach 2:
The system performs preliminary action by generating diverse training samples ahead of time using automated synthesis. The LLM creates comprehensive datasets including various SQL query types (SELECT, INSERT, UPDATE, DELETE, JOIN operations) and natural language questions based on provided database schemas, enabling efficient model training without time-consuming manual data collection.
Data Source
AI summary
Here is an innovative way to generate a finetuning corpus that maximizes the accuracy of a target large language model (LLM) that generates a database statement. From a natural language request, the target LLM infers an incorrect database statement that, based on a first database schema, could not satisfy a technical requirement. Based on the natural language request, a correct database statement is generated that, based on a second database schema, could satisfy the technical requirement. For the second database schema, a restatement of the natural language request is generated. In inputs during finetuning, the target LLM accepts: the correct database statement, the incorrect database statement, and the restatement of the natural language request.