Code Generative Model Pretraining With Next AST Branch Prediction

Resolve Bottlenecks,
Find Innovative SolutionsGenerate Solutions

Solution Overview

Problem

Existing natural language processing (NLP) models for source code generation fail to accurately capture the strict grammatical structure of programming languages, leading to suboptimal code generation and increased finetuning duration due to the neglect of syntactic and semantic information, and alternative approaches struggle with handling syntactically invalid code.

Innovation Solution

A pretraining framework called next branch prediction (NBP) is introduced, which injects syntactic and semantic information of programming languages into a generative model by aligning vector representations of tokens with their corresponding graph traversal paths, using a multitask learning paradigm that includes next token prediction (NTP) and next branch prediction tasks.

Engineering Contradictions & Design Principles

VSEngineering Contradiction Analysis

1Adaptability or versatility

If bilingual biphasic training is used to pretrain with natural language and then finetune with programming language, then the model can learn general language patterns, but the model fails to capture strict grammatical structure of programming languages leading to suboptimal code generation

Engineering Contradiction:

Improvelanguage pattern learningVSAvoidcode generation accuracy

Core Design Contradiction:

Adaptability or versatilityVSManufacturing precision

Solution Approach 1:

The patent segments the training process into distinct phases: initial natural language pretraining followed by programming language-specific finetuning with structure-aware objectives. This segmentation allows the model to first learn general language patterns and then specialize in programming language grammar and structure, resolving the contradiction between versatility and precision.

Inventive Principle:

Principle #1Segmentation

Solution Approach 2:

The patent changes training parameters during the finetuning phase by introducing structure-aware objectives that explicitly model programming language grammar and syntax. This parameter change enables the model to transition from general language processing to precise code generation, addressing the accuracy deficit of standard bilingual training.

Inventive Principle:

Principle #35Parameter changes

2Quantity of substance

If bilingual biphasic training is used, then the model can be pretrained on large natural language corpora, but the finetuning duration increases to achieve desired final accuracy

Engineering Contradiction:

Improvetraining data volumeVSAvoidfinetuning duration

Core Design Contradiction:

Quantity of substanceVSLoss of time

Solution Approach 1:

The patent performs preliminary structure-aware pretraining on programming language data before final task-specific finetuning. This preliminary action with structure-aware objectives accelerates convergence during subsequent finetuning, reducing the time required to achieve desired accuracy while maintaining the benefit of large-scale natural language pretraining.

Inventive Principle:

Principle #10Preliminary action

3Manufacturing precision

If synthetic tokens representing syntactic structure are inserted into token sequence, then the model can capture program structure, but the approach cannot handle syntactically invalid source code

Engineering Contradiction:

Improvestructural accuracyVSAvoidhandling of invalid code

Core Design Contradiction:

Manufacturing precisionVSAdaptability or versatility

Solution Approach 1:

The patent introduces an intermediary AST (Abstract Syntax Tree) representation that mediates between raw code tokens and model processing. The AST provides structural information without requiring valid syntax, allowing the model to learn from both valid and invalid code. This intermediary structure enables structural awareness while maintaining adaptability to syntactically diverse inputs.

Inventive Principle:

Principle #24Intermediary (Mediator)

Data Source

PatentUS12547832B2Next AST branch prediction and next token prediction joint pre-training task for code generative models

Publication Date: 2026.02.10 ORACLE INT CORP

AI summary

During pretraining, a computer generates three trainable and untrained machine learning models that are a token sequence encoder, a token predictor, and a path predictor. A sequence of lexical tokens is generated that represents a lexical text in a training corpus. A graph is generated that represents the lexical text. In the graph, a next traversal path is selected that corresponds to a next lexical token that is adjacent to a sliding subsequence of the sequence of lexical tokens. From the subsequence, the token sequence encoder infers an encoded sequence that represents the subsequence. The path predictor and token predictor accept the encoded sequence as input for respective inferencing for which respective training losses are measured. Both training losses are combined into a combined loss that is used to increase the accuracy of the three machine learning models by, for example, backpropagation of the combined loss.