Synthetic Labeled Data for Machine Learning Program Analysis
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Machine learning-based program analysis is imprecise due to the lack of labeled data, as source code from public repositories includes both correct and buggy code, but labels are unknown, leading to false positives and incomplete analysis, especially when code relies on external libraries or different languages.
Innovation Solution
Synthetically generated labeled data is created by using the results of non-machine learning-based program analysis to train machine learning models, where correct code is mutated to include errors, providing training data for error detection models to improve accuracy.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Extent of automation
If machine learning-based program analysis is used, then analysis coverage and automation are improved, but measurement precision and reliability deteriorate due to lack of labeled data and false positives
Solution Approach 1:
The system performs preliminary program analysis using traditional static analysis tools to identify potential bugs and generate candidate bug reports before training the machine learning model. These preliminary results serve as the foundation for creating synthetically labeled training data, allowing the ML model to learn from pre-processed, partially annotated information.
Solution Approach 2:
The system uses its own program analysis results to automatically generate training data labels without requiring external manual annotation. The static analysis tool's bug detections are automatically used as ground truth labels for training the machine learning model, enabling the system to self-generate training data and improve its own performance iteratively.
2Measurement precision
If traditional program analysis is used, then measurement precision is maintained through sound analysis, but adaptability deteriorates when analyzing code with external libraries or different languages
Solution Approach 1:
The machine learning model is trained to perform multiple bug detection tasks across different programming languages and code contexts. By using synthetically generated training data that encompasses various language constructs and library usage patterns, the model learns universal bug detection patterns that generalize across different languages and external libraries, making the analysis system adaptable to diverse codebases.
3Measurement precision
If synthetically generated labeled data is used for training, then measurement precision improves by reducing false positives, but device complexity increases due to the synthetic data generation pipeline
Solution Approach 1:
The system introduces an intermediary synthetic data generation layer between the traditional static analysis tool and the machine learning training process. This intermediary component automatically transforms static analysis results into structured training examples with synthetic labels, bridging the gap between traditional analysis outputs and ML training requirements without requiring complex manual data preparation pipelines.
Data Source
AI summary
Techniques for performing machine learning-based program analysis using synthetically generated labeled data are described. A method of performing machine learning-based program analysis using synthetically generated labeled data may include receiving a request to perform program analysis on code, determining a first portion of the code associated with a first error type, sending the first portion of the code to an endpoint of a machine learning service associated with an error detection model to detect the first error type, the error detection model trained using synthetically generated labeled data, and receiving inference results from the error detection model identifying one or more errors of the first error type in the first portion of the code.