Program Synthesis Language Models With Bootstrapped Code Repair

Resolve Bottlenecks,
Find Innovative SolutionsGenerate Solutions

Solution Overview

Problem

Conventional machine learning models for program synthesis rely on limited and low-quality training datasets, leading to inaccurate program generation and a large search space, especially in programming languages, which hinders their effectiveness.

Innovation Solution

A method involving bootstrapping and program repair, where the model generates, validates, and iteratively refines code using feedback from compilation errors and unit tests to improve the training process, utilizing the generated and refined code as fine-tuning data.

Engineering Contradictions & Design Principles

VSEngineering Contradiction Analysis

1Manufacturing precision

If conventional language models are trained on limited programming datasets, then training cost and time are reduced, but program synthesis accuracy deteriorates

Engineering Contradiction:

Improveprogram synthesis accuracyVSAvoidtraining data quantity

Core Design Contradiction:

Manufacturing precisionVSQuantity of substance

Solution Approach 1:

The system uses the language model itself to generate training data by synthesizing programs from programming problems. The model generates candidate programs, executes them, and uses execution results (success/failure cases) as training data, allowing the system to create its own training dataset without external human effort.

Inventive Principle:

Principle #25Self-service

Solution Approach 2:

The system transforms the training data generation process by changing from human-authored datasets to automatically generated datasets. It controls parameters such as the number of generation iterations, diversity of programming problems, and execution test conditions to produce high-quality training data with appropriate quantity and variety.

Inventive Principle:

Principle #35Parameter changes

2Manufacturing precision

If diverse and high-quality training data is generated through bootstrapping, then program synthesis accuracy is improved, but computational resources and time increase

Engineering Contradiction:

Improveprogram synthesis accuracyVSAvoidtraining data generation time

Core Design Contradiction:

Manufacturing precisionVSLoss of time

Solution Approach 1:

The system performs preliminary actions by pre-selecting a diverse set of programming problems from existing datasets before the bootstrapping process. It also pre-configures execution environments and test cases, so that during the actual training data generation, only the model inference and execution steps need to be performed, reducing overall time consumption.

Inventive Principle:

Principle #10Preliminary action

Solution Approach 2:

The bootstrapping process continuously generates training data by iteratively having the model generate programs, execute them, and collect results. This continuous generation process maximizes the utilization of computational resources by keeping the model and execution environment actively producing training examples without idle time between batches.

Inventive Principle:

Principle #20Continuity of useful action

3Reliability

If the model generates and refines programs through iterative feedback, then program quality is improved, but system complexity increases

Engineering Contradiction:

Improveprogram qualityVSAvoidsystem complexity

Core Design Contradiction:

ReliabilityVSDevice complexity

Solution Approach 1:

The system implements a feedback loop where the language model generates candidate programs, these programs are executed against programming problems, and the execution results (correct/incorrect outputs, runtime errors) are fed back to the model. The model uses this feedback to refine its predictions and generate improved programs in subsequent iterations, progressively enhancing program quality.

Inventive Principle:

Principle #23Feedback

4Manufacturing precision

If execution-based validation is used to verify generated programs, then training data quality is improved, but computational overhead increases

Engineering Contradiction:

Improvetraining data qualityVSAvoidcomputational overhead

Core Design Contradiction:

Manufacturing precisionVSUse of energy by moving object

Solution Approach 1:

The system uses disposable, lightweight execution environments that are quickly instantiated and terminated for each program validation. Rather than maintaining persistent complex test harnesses, it creates minimal execution contexts that run the generated program against the programming problem and immediately discard them, reducing the energy cost per validation operation.

Inventive Principle:

Principle #27Cheap short-living objects (Disposable)

Data Source

PatentUS12487912B2Bootstrapping program synthesis language models to perform repairing

Publication Date: 2025.12.02 QUALCOMM TECHNOLOGIES INC

AI summary

Certain aspects of the present disclosure provide techniques and apparatus for improved program synthesis using machine learning. An input indicating a programming task is accessed. A generated program is generated based on processing the input using a trained machine learning model. In response to determining that the generated program failed to satisfy the programming task, feedback is generated, and a revised program is generated based on processing the feedback using the trained machine learning model. In response to determining that the revised program satisfied the programming task, one or more parameters of the trained machine learning model are updated based on the revised program.