Token-Level Source Code Vulnerability Localization With LLM Verification
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Software vulnerabilities are difficult to detect due to the absence of fail-stop symptoms and require high-precision accuracy, while existing methods often rely on low-precision models that can lead to false positives and negatives.
Innovation Solution
A two-stage vulnerability detection system using a classifier model to identify vulnerability types and token positions, followed by a large language model with few-shot examples to confirm the detection, and a repair engine to generate corrected code.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If a single-stage vulnerability detection model is used, then the detection speed is faster, but the detection precision is insufficient leading to false positives and negatives
Solution Approach 1:
The vulnerability detection system is segmented into three distinct stages: (1) a classifier model that identifies vulnerability types and locations, (2) a prompt generator that creates contextual prompts with few-shot examples, and (3) a large language model that makes final vulnerability determinations. This segmentation allows each component to specialize in specific tasks, improving overall precision while managing complexity through modular architecture.
Solution Approach 2:
A prompt generator acts as an intermediary between the classifier model and the large language model. It transforms the classifier's output into contextualized prompts that include few-shot examples, serving as a mediator that bridges the gap between initial detection and final verification, thereby improving detection accuracy without directly increasing model complexity.
2Measurement precision
If a large language model is used for vulnerability detection, then the detection precision is improved, but the computational resources and time required increase
Solution Approach 1:
The system segments the detection task so that the computationally intensive large language model only processes a small subset of code regions identified by the classifier model. The prompt generator creates compact prompts with few-shot examples that focus the LLM's computational resources only on relevant vulnerability patterns, reducing overall energy consumption while maintaining high precision.
Solution Approach 2:
The system applies partial action by using the large language model only for verification of suspected vulnerabilities rather than analyzing all code. The classifier model performs initial screening, and the LLM provides selective verification only where needed, avoiding excessive computational expenditure on code regions that are unlikely to contain vulnerabilities.
3Measurement precision
If few-shot examples are provided to the large language model, then the detection accuracy is improved, but the system complexity and data requirements increase
Solution Approach 1:
The system uses a small number of few-shot examples (partial action) rather than requiring extensive training data. The prompt generator creates prompts with only a few representative examples that capture the essential vulnerability patterns, allowing the large language model to achieve high accuracy without needing large quantities of training data for each vulnerability type.
Data Source
AI summary
A vulnerability detection and repair system utilize a classifier model to detect a software vulnerability in a source code snippet and the tokens in the source code snippet attributable to the vulnerability. A large language model is then given the vulnerable source code snippet, its vulnerability type, the vulnerability tokens, and a few-shot examples to determine whether or not the source code snippet includes the identified vulnerability. The few-shot examples include positive and negative samples of the type of vulnerability to guide the large language model towards the correct output.