LLM-Generated SPL Query Correction Using Schema Verification
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
The widespread adoption of large language models (LLMs) in technology industries faces challenges due to the need for individuals to write software code in programming languages they are unfamiliar with, exacerbated by the numerous programming languages in use and rapid introduction of new ones, particularly in complex environments like Splunk's Search Processing Language (SPL).
Innovation Solution
A pipelined search query generation engine utilizing retrieval augmented generation (RAG) and a networked environment with logic modules like the RAG pipeline, explanation pipeline, and QA pipeline, which processes natural language prompts to generate, explain, or answer SPL queries, leveraging LLMs to automate code generation and execution.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Ease of operation
If LLMs are used to generate code in complex programming languages, then ease of operation is improved for users lacking programming expertise, but code accuracy and reliability deteriorate due to potential hallucinations and syntax errors
Solution Approach 1:
The system implements a feedback mechanism where the generated code is validated against a schema that defines expected structure and syntax rules. The schema validator receives the generated code, checks it against the schema requirements, and provides feedback to correct any deviations, ensuring the code meets accuracy requirements while maintaining ease of use.
Solution Approach 2:
A schema acts as an intermediary between the LLM's natural language generation capabilities and the requirements for syntactically correct code. The schema translates high-level semantic requirements into specific structural constraints, mediating between the simplicity of natural language input and the precision required for executable code.
2Reliability
If schema verification is implemented to reduce hallucinations, then code accuracy is improved, but device complexity increases
Solution Approach 1:
The verification system is segmented into distinct components: a schema definition module that stores validation rules, a validator module that executes checks against the schema, and an error correction module that generates fixes. This segmentation allows each component to be independently optimized and maintained, reducing overall system complexity while ensuring thorough verification.
Solution Approach 2:
The schema uses parameter-based validation where specific attributes of the generated code are checked against expected values and patterns. By changing the representation of validation rules into parameter constraints, the system achieves thorough verification without complex validation logic, simplifying the overall device complexity.
Data Source
AI summary
Disclosed herein are systems and methods for improving the auto-generation of pipelined search query statements by a large language model (LLM) through a post processing. In some examples, such a method includes operations of receiving, by a post processing engine, a response to an auto-generated prompt from the LLM that includes programming code generated by the LLM, performing a post processing of a response from the LLM that includes the programming code generated by the LLM including performing an error correction process when a term of the programming code generated by the LLM is inconsistent with terms of a schema of the user, and generating a graphical user interface (GUI) that displays the response to the auto-generated prompt when the terms of the programming code generated by the LLM including any replacement terms are consistent with the terms of the schema.