Graph Path Prediction Pretraining for Accurate Code Structure Modeling

Resolve Bottlenecks,
Find Innovative SolutionsGenerate Solutions

Solution Overview

Problem

Natural language processing models like BERT struggle with accurately representing the formal structure of programming languages due to the lack of unambiguous syntax and grammar, leading to suboptimal code representation and limited reusability, and existing specialized approaches are prone to design errors and require impractical pretraining without finetuning.

Innovation Solution

A pretraining framework called Graph Path Prediction (GPP) injects code structure knowledge into language models using multitask learning with Masked Language Modelling, allowing token-level knowledge input and implicit alignment of vector representations with graph topology, reducing computational cost and improving accuracy.

Engineering Contradictions & Design Principles

VSEngineering Contradiction Analysis

1Adaptability or versatility

If a natural language model like BERT is applied to programming language tasks, then the model can be reused across different applications, but the accuracy of code representation deteriorates due to the lack of formal structural patterns

Engineering Contradiction:

Improvemodel reusabilityVSAvoidcode representation accuracy

Core Design Contradiction:

Adaptability or versatilityVSMeasurement precision

Solution Approach 1:

The patent combines a language model (BERT) with graph analytics components to create a hybrid architecture. The language model processes lexical tokens while the graph analytics component processes parse trees and control flow graphs, merging both representations to achieve accurate code understanding while maintaining model reusability through the language model component

Inventive Principle:

Principle #5Merging (Combining)

Solution Approach 2:

The patent creates a composite representation by integrating multiple data structures (lexical tokens, parse trees, control flow graphs) into a unified code representation. This composite approach combines the strengths of different representation methods to achieve both accuracy in code representation and versatility in application

Inventive Principle:

Principle #40Composite materials

2Measurement precision

If a specialized machine learning approach is designed for a specific graph representation, then the accuracy for that specific application is improved, but the device complexity increases and reusability deteriorates

Engineering Contradiction:

Improvegraph representation accuracyVSAvoidmodel design complexity

Core Design Contradiction:

Measurement precisionVSDevice complexity

Solution Approach 1:

The patent designs a graph analytics component that can process multiple types of graphs (parse trees, control flow graphs) and multiple types of nodes, making it a universal component that handles various code representation tasks. This universality reduces the need for multiple specialized models while maintaining high accuracy for different graph types

Inventive Principle:

Principle #6Universality (Multi-functionality)

3Manufacturing precision

If a specialized machine learning approach is designed with handcrafted structural limitations, then the manufacturing precision for that specific domain is improved, but the ease of manufacture deteriorates due to extended development phase

Engineering Contradiction:

Improvedomain-specific accuracyVSAvoiddevelopment time

Core Design Contradiction:

Manufacturing precisionVSEase of manufacture

Solution Approach 1:

The patent employs dynamic graph construction where the graph structure is automatically generated from the input code rather than being statically defined. The graph analytics component dynamically adapts to different code structures and domains, eliminating the need for extensive handcrafting and domain-specific customization while maintaining high precision

Inventive Principle:

Principle #15Dynamics

4Productivity

If pretraining is performed without finetuning, then the productivity is improved by saving electricity and processor time, but the reliability deteriorates due to impractical pretraining requirements

Engineering Contradiction:

Improvecomputational efficiencyVSAvoidmodel performance reliability

Core Design Contradiction:

ProductivityVSReliability

Solution Approach 1:

The patent performs preliminary pretraining of the language model component on general language data, which establishes a strong baseline representation. This preliminary action allows the model to be reused across applications without full retraining, improving productivity while maintaining reliability through the prelearned language patterns

Inventive Principle:

Principle #10Preliminary action

Data Source

PatentUS12566596B2Graph path prediction and masked language modelling joint training algorithm for language models

Publication Date: 2026.03.03 ORACLE INT CORP

AI summary

In an embodiment providing natural language processing (NLP), a computer generates a histogram that correctly represents a graph that represents a lexical text, and generates a token sequence encoder that is trainable and untrained. During training such as pretraining, the token sequence encoder infers an encoded sequence that incorrectly represents the lexical text, and the encoded sequence is dense and saves space. To increase the accuracy of the sequence encoder by learning, the token sequence encoder is adjusted based on, as discussed herein, an indirectly measured numeric difference between the encoded sequence that incorrectly represents the lexical text and the histogram that correctly represents the graph.