Multi-LLM SQL Query Verification Without Database Execution
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing methods for verifying SQL queries are inefficient, particularly in large databases, and often fail to detect logical mistakes, requiring database execution which is not feasible due to data privacy constraints, and achieve limited accuracy improvements.
Innovation Solution
A method using multiple large language models (LLMs) to predict and simulate SQL query outputs without executing them, identifying mismatches, and iteratively refining the queries to ensure intended results, employing a pre-execution verification approach with ensemble-based verification and improvement pipelines.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If SQL queries are executed in the database for verification, then verification accuracy is improved, but data privacy constraints are violated and efficiency deteriorates
Solution Approach 1:
The patent creates a synthetic copy of the database schema and uses it to generate simulated execution results for verification purposes. Instead of executing queries on the actual database, the system generates synthetic data that mimics the database structure and produces expected results, allowing verification without accessing sensitive data.
Solution Approach 2:
The patent introduces an intermediary verification layer that acts as a mediator between the SQL query and the actual database. This intermediary generates simulated execution results and compares them with expected outcomes, preventing direct access to the actual database while maintaining verification capability.
2Ease of operation
If simple prompting or Chain-of-Thought prompting is used to correct SQL queries, then ease of operation is improved, but detection of logical mistakes deteriorates
Solution Approach 1:
The patent implements a feedback mechanism where the system generates multiple candidate corrections for SQL queries, evaluates each candidate against the synthetic execution results, and provides feedback on which corrections are most effective. This iterative feedback process improves logical mistake detection while maintaining ease of operation.
Solution Approach 2:
The patent applies partial correction by focusing on specific problematic portions of the SQL query rather than rewriting the entire query. It identifies logical mistakes in specific clauses and applies targeted corrections, reducing operational complexity while improving detection accuracy.
3Loss of time
If pre-execution verification with self-consistency techniques is used, then verification before execution is achieved, but accuracy improvement is limited and cost increases
Solution Approach 1:
The patent performs preliminary verification actions by generating synthetic execution results before actual query execution. This preliminary action allows the system to identify potential errors and correct them beforehand, preventing wasted execution time on incorrect queries.
Solution Approach 2:
The patent changes the verification parameter from relying solely on self-consistency checks to using synthetic execution results with configurable confidence thresholds. By adjusting parameters such as the number of synthetic executions and threshold values, the system achieves better accuracy improvement without excessive cost.
Data Source
AI summary
A method and system for verifying a structured query language (SQL) query are provided. The method includes: receiving a first request to retrieve first data that is accessible via a database; identifying an intention of the first request; generating, based on the first request, a first SQL query to retrieve the first data from the database; predicting, based on the generated first SQL query, an output of the generated first SQL query; determining whether the predicted output matches the identified intention of the first request; generating, based on the determining of whether the predicted output matches the identified intention of the first request, a second SQL query to retrieve the first data that is accessible via the database, when the predicted output of the first SQL query does not match the identified intention of the first request.