AST Path Prediction for Syntax-Aware Code Language Models
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing natural language processing (NLP) models, such as BERT, struggle to accurately encode and analyze the structural and semantic information of programming languages due to the loss of important tree topology details during flattening, leading to suboptimal inference accuracy.
Innovation Solution
A novel encoded tree generator is used to finetune a pretrained logic encoder, incorporating structural and semantic information from abstract syntax trees (ASTs) into language models like BERT through a self-supervised training objective called AST path prediction, where AST information flows in a backward path using neural backpropagation.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Loss of information
If state of the art feature encoding is used to encode logical tree, then encoding can be performed, but important tree topology details are lost and inference accuracy is poor
Solution Approach 1:
The patent segments the logical tree into fixed-size encoded tree representations, dividing the complex tree structure into manageable fixed-size segments that preserve topological information while enabling efficient processing by machine learning models
Solution Approach 2:
The patent changes the encoding parameters by using fixed-size encoded tree representations with specific dimensional constraints, transforming the variable-size tree structure into a fixed-dimensional format that retains essential topological details while being suitable for model input
2Quantity of substance
If dense encoding is used to encode logical tree, then space efficiency is improved, but important information may be lost
Solution Approach 1:
The patent applies local quality by preserving critical topological information in specific regions of the encoded representation while using fixed-size constraints elsewhere, ensuring that important structural details are maintained without requiring excessive space
3Adaptability or versatility
If large language models are used for source codes, then model capacity is increased, but computational cost and data requirements increase significantly
Solution Approach 1:
The patent performs preliminary action by pre-processing source code into fixed-size encoded tree representations before feeding them to the language model, preparing the data in advance in a format that maximizes model effectiveness while minimizing computational requirements during training and inference
4Speed
If BERT forgoes parsing to accept lexical tokens, then processing speed is improved, but accuracy decreases
Solution Approach 1:
The patent performs parsing and tree encoding as a preliminary action before BERT processing, so that the structural information is prepared in advance in fixed-size encoded format, allowing BERT to process the encoded representation efficiently without performing parsing itself, thus maintaining both speed and accuracy
Data Source
AI summary
In an embodiment, a computer stores and operates a logic encoder that is an artificial neural network that infers a fixed-size encoded logic from textual or tokenized source logic. Without machine learning, a special parser generates a parse tree that represents the source logic and a fixed-size correctly encoded tree that represents the parse tree. For finetuning the logic encoder, an encoded tree generator is an artificial neural network that accepts the fixed-size encoded logic as input and responsively infers a fixed-size incorrectly encoded tree that represents the parse tree. The neural weights of the logic encoder (and optionally of the encoded tree generator) are adjusted based on backpropagation of error (i.e. loss) as a numerically measured difference between the fixed-size incorrectly encoded tree and the fixed-size correctly encoded tree.