Differential Fuzzing for LLM Code Verification
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing methods for checking the dynamic behavior of code generated by language models are inefficient, particularly for larger code bases, as they often rely on static methods that are not suitable for measuring runtime performance and can result in over-approximation errors.
Innovation Solution
The method involves providing two executable files, one generated by a language model and another as a reference, and using differential fuzzing to inject identical inputs into both files. This monitors their behavior and output, ensuring that the generated code meets the requirements by comparing it to the reference.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If static methods are used to check code correctness, then completeness of analysis is improved, but time consumption and computational resources increase significantly
Solution Approach 1:
The patent applies partial action by using fuzzing to test only critical paths and frequently executed code segments rather than performing exhaustive static analysis on the entire codebase. The differential approach focuses comparisons on specific behaviors and outputs, allowing thorough testing of important aspects while skipping less critical areas, thus reducing overall time consumption while maintaining reliability for key functionality.
2Reliability
If static methods are used for soundness guarantees, then correctness coverage is improved, but usability and ease of operation deteriorate due to multitude of warnings
Solution Approach 1:
The patent extracts the essential correctness verification from the comprehensive static analysis process. Instead of presenting all possible warnings and issues, the system extracts only the critical behavioral differences and correctness violations that matter. The differential fuzzing approach isolates and reports only the significant discrepancies between expected and actual behavior, filtering out noise and non-critical warnings to improve usability.
Solution Approach 2:
The patent implements feedback mechanisms where the fuzzing results are continuously monitored and used to guide further testing. The system provides targeted feedback about specific correctness violations rather than overwhelming users with all possible warnings. This feedback loop allows developers to address critical issues systematically while maintaining ease of operation through prioritized, actionable information.
3Reliability
If static methods are used for code generation checking, then theoretical completeness is improved, but applicability to performance measurement and runtime behavior deteriorates
Solution Approach 1:
The patent substitutes static mechanical analysis with dynamic runtime execution testing. Instead of relying on static method analysis that cannot capture runtime behavior, the system uses fuzzing to actually execute the generated code with various inputs and observe its behavior. This replacement enables performance measurement and runtime behavior verification while maintaining thoroughness through comprehensive input testing.
Solution Approach 2:
The patent transitions from static to dynamic verification methods. The fuzzing approach dynamically executes code with varying inputs to observe actual runtime behavior, making the verification process adaptable to performance measurement and behavioral analysis. This dynamic approach maintains completeness by testing diverse scenarios while gaining the versatility needed for performance evaluation and runtime behavior assessment.
4Measurement precision
If exhaustive testing is performed on LLM-generated code, then detection precision is improved, but productivity and development speed decrease
Solution Approach 1:
The patent applies partial action by focusing fuzzing efforts on critical code paths, frequently executed segments, and areas with higher risk of LLM generation errors. Rather than uniformly testing all code, the system prioritizes testing where detection precision matters most, achieving high detection precision for critical issues while maintaining productivity by reducing unnecessary testing of less important areas.
Data Source
AI summary
A method for checking the dynamic behavior of code generated using a language model. Th method includes: providing a first executable file from a program code generated using a language model; providing a second executable file, wherein the second executable file is a previous first executable file or is an original source code of the program code; executing differential fuzzing using a fuzzer, wherein the fuzzer injects identical inputs into the first executable file and into the second executable file; monitoring the behavior and the output of the first executable file and the second executable file; outputting the program code if the fuzzing found no inconsistencies, no errors and/or no worse runtime behavior of the first executable file compared to the second executable file.