Partial Graph Path Prediction for Source Code Generation
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current bilingual biphasic training methods for source code generation are suboptimal due to the neglect of source code syntax, leading to inaccurate and slow training processes, and are unable to handle syntactically invalid source code.
Innovation Solution
The proposed solution is a pretraining framework called Partial Graph Path Prediction (PGPP) that injects syntactic and semantic information of a programming language into a machine learning model by predicting partial graph paths associated with each token in a source code sequence, allowing the model to learn structural information during training.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If bilingual biphasic training is used to pretrain language models with natural language corpus and then finetune with programming language corpus, then the model can generate source code, but the training accuracy is suboptimal and training duration is increased
Solution Approach 1:
The patent applies preliminary action by injecting syntactic structure information into the training corpus before the actual training process. The method transforms natural language text into structured representations (such as parse trees or dependency graphs) that encode grammatical relationships, and incorporates these structures into the training data beforehand. This allows the model to learn syntax rules during pretraining rather than requiring extensive finetuning later, thereby improving accuracy while reducing finetuning duration.
2Measurement precision
If synthetic tokens representing syntactic structure are inserted into the original token sequence, then the model can learn program structure, but the approach cannot handle syntactically invalid source code
Solution Approach 1:
The patent applies partial action by selectively injecting syntactic structure information only for the portions of code that can be parsed, rather than requiring complete syntactic validity. The method processes available parse trees or dependency graphs for valid portions of the code and incorporates this partial structural information into the training tokens. This allows the model to learn from available structure while still being able to handle and generate code with syntax errors, improving both structure understanding and adaptability to invalid code.
Data Source
AI summary
During pretraining, a computer generates three untrained machine learning models that are a token sequence encoder, a token predictor, and a decoder that infers a frequency distribution of graph traversal paths. A sequence of lexical tokens is generated that represents a lexical text in a training corpus. A graph is generated that represents the lexical text. In the graph, multiple traversal paths are selected that collectively represent a sliding subsequence of the sequence of lexical tokens. From the subsequence, the token sequence encoder infers an encoded sequence that represents the subsequence of the sequence of lexical tokens. The decoder and token predictor accept the encoded sequence as input for respective inferencing for which respective training losses are measured. Both training losses are combined into a combined loss that is used to increase the accuracy of the three machine learning models by, for example, backpropagation of the combined loss.