Security Language Query Generation with Probe-Guided Examples
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Security analysts face challenges in deciphering proprietary log structures from multiple security products, such as Microsoft® Sentinel®/Defender®, due to the unfamiliarity with Kusto Query Language (KQL), which hinders effective security investigations.
Innovation Solution
A computer-implemented method using a trained machine learning model to select examples and generate metadata for prompts, which are input to a large language model to produce accurate security language queries, such as KQL, by employing a probing procedure to assess the impact of included examples.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Extent of automation
If a large language model is used to generate security language queries directly from user input, then the system can provide automated query generation, but the opacity of the large language model makes it difficult to assess the impact of included examples and ensures reliability
Solution Approach 1:
A trained machine learning model is introduced as an intermediary between the user input and the large language model. This intermediary model selects relevant examples and generates metadata for prompts, making the otherwise opaque large language model more controllable and reliable. The intermediary model is trained using probe prompts that systematically assess the impact of different examples, creating a transparent layer that bridges automation and reliability.
Solution Approach 2:
The system implements a feedback mechanism through probe prompts that evaluate the impact of included examples on query generation. The trained machine learning model uses this feedback to optimize example selection and metadata generation. By systematically probing and learning from the results, the system continuously improves its ability to generate reliable security language queries while maintaining automation.
2Adaptability or versatility
If multiple security products with proprietary log structures are used to enhance security coverage, then security protection capability is improved, but the complexity of understanding and interrogating different log structures increases significantly
Solution Approach 1:
The system creates a universal interface that works across multiple security products with different proprietary log structures. The machine learning model is trained to handle various log formats and generate appropriate security language queries for different products. This universal approach allows the system to maintain compatibility with multiple security products while abstracting away the underlying complexity of each proprietary log structure.
3Productivity
If junior analysts with limited knowledge of table and schema definitions use KQL for security investigations, then productivity can be maintained, but the difficulty of becoming proficient in the query language and log structures increases the learning curve
Solution Approach 1:
The system enables junior analysts to perform security investigations without requiring deep expertise in KQL or log structures. The machine learning model automatically generates the complex query language code based on simpler user input, allowing analysts to serve themselves without needing to master the difficult query language. This self-service approach maintains productivity while eliminating the steep learning curve.
Data Source
AI summary
A computer-implemented method of generating a security language query from a user input query includes receiving, at a computer system, an input security hunting user query indicating a user intention; selecting, using a trained machine learning model and based on the input security hunting query, an example user security hunting query and corresponding example security language query; generating, using the trained machine learning model, query metadata from the input security hunting query; generating a prompt, the prompt comprising: the input security hunting user query; the selected example user security hunting query and the corresponding example security language query; and the generated query metadata; inputting the prompt to a large language model; receiving a security language query from the large language model corresponding to the input security hunting query reflective of the user intention.