Vector Database Schema Embedding for Accurate NL2KQL Translation

Resolve Bottlenecks,
Find Innovative SolutionsGenerate Solutions

Solution Overview

Problem

Existing natural language to Kusto Query Language (NL2KQL) translation systems are inefficient due to inaccuracies, resource-intensity, and lack of operability across diverse Kusto databases with unstructured logs, leading to challenges in generating syntactically correct and contextually relevant queries.

Innovation Solution

A system utilizing AI embedding and vector databases to generate schemas and embeddings of security logs, performing similarity searches to identify relevant fields and values, and refining queries with Large Language Models (LLMs) for accurate KQL generation.

Engineering Contradictions & Design Principles

VSEngineering Contradiction Analysis

1Ease of operation

If standard large language models are used for NL2KQL translation, then ease of operation is improved, but translation accuracy deteriorates due to inherent ambiguity of natural language

Engineering Contradiction:

Improveease of query constructionVSAvoidtranslation accuracy

Core Design Contradiction:

Ease of operationVSMeasurement precision

Solution Approach 1:

The patent introduces an intermediary vector database layer between natural language queries and KQL generation. The system embeds log data into vectors, stores them in a vector database, and uses similarity search to bridge the gap between ambiguous natural language and precise KQL queries, thereby improving translation accuracy while maintaining ease of operation

Inventive Principle:

Principle #24Intermediary (Mediator)

Solution Approach 2:

The system performs preliminary embedding of log data into vectors and stores them in the vector database before actual query translation. This pre-processing enables fast similarity searches during query translation, improving both accuracy and efficiency by having the data ready in a structured format

Inventive Principle:

Principle #10Preliminary action

2Ease of operation

If existing NL2KQL tools are used, then ease of operation is improved, but adaptability deteriorates as they only work on particular Kusto databases with limited data schema

Engineering Contradiction:

Improveease of query generationVSAvoidoperability across diverse Kusto databases

Core Design Contradiction:

Ease of operationVSAdaptability or versatility

Solution Approach 1:

The patent creates a universal NL2KQL translation system that works across diverse Kusto databases with different data schemas. By using vector embeddings and similarity search, the system can adapt to any log structure without requiring database-specific knowledge, enabling one tool to serve multiple database types and scenarios

Inventive Principle:

Principle #6Universality (Multi-functionality)

3Measurement precision

If manual query construction is used, then translation accuracy is improved, but productivity deteriorates due to time-intensive process

Engineering Contradiction:

Improvequery syntactic correctnessVSAvoidquery construction speed

Core Design Contradiction:

Measurement precisionVSProductivity

Solution Approach 1:

The patent replaces the mechanical process of manual query construction with an automated AI-based system. The system uses natural language understanding, vector embeddings, and similarity search to automatically generate syntactically correct KQL queries, eliminating the need for manual syntax knowledge while maintaining accuracy and significantly improving speed

Inventive Principle:

Principle #28Mechanics substitution (Replace mechanical system)

4Adaptability or versatility

If unstructured log data is processed, then adaptability is improved, but device complexity increases due to lack of predefined schema

Engineering Contradiction:

Improvehandling of unstructured dataVSAvoidquery generation complexity

Core Design Contradiction:

Adaptability or versatilityVSDevice complexity

Solution Approach 1:

The patent transforms unstructured log data into structured vector representations by changing the data parameter from raw text to embedded vectors. This transformation simplifies the processing complexity by converting the unstructured data into a format that can be efficiently searched and queried through similarity metrics, making the system more adaptable while managing complexity

Inventive Principle:

Principle #35Parameter changes

Data Source

PatentUS12505095B1Method and system for constructing vector databases used for converting free text queries to cyber language queries

Publication Date: 2025.12.23 VEGA CYBER SOLUTIONS LTD

AI summary

A system and method for querying data sources for cybersecurity analysis is presented. The system and method include: receiving security logs from at least one data source, wherein the security logs lack pre-defined schema; generating a schema of the security logs based on at least a type of data of the security logs, wherein the generated schema includes fields of the security logs and values of the fields; embedding field vectors, wherein each field vector is a vector representation of a value of each respective field; embedding value vectors, wherein each value vector is a vector representation of a natural language description of each value in each respective field; and generating a query in a cyber language query, using an AI system, for execution on at least one target data source based, in part, on the generated schema, the embedded field vectors, and the embedded value vectors.