Large Language Model Training with Abstract Syntax Trees

Solution Overview

Problem

Existing source code generation models struggle to accurately reflect the structural and grammatical differences between natural language and source code, leading to inefficiencies in generating high-quality source code.

Innovation Solution

The proposed method involves training a large language model using training data consisting of pairs of natural language specifications and abstract syntax trees (ASTs), allowing the model to better understand and replicate the structure of source code.

Engineering Contradictions & Design Principles

VSEngineering Contradiction Analysis

1Measurement precision

If existing source code generation models are used, then natural language processing capability is achieved, but structural and grammatical accuracy of source code is insufficient

Engineering Contradiction:

Improvestructural accuracyVSAvoidcode generation quality

Core Design Contradiction:

Measurement precisionVSReliability

Solution Approach 1:

The patent introduces an Abstract Syntax Tree (AST) as an intermediary representation between natural language specifications and source code. The model first generates an AST that captures the structural and grammatical properties of the target code, then converts this AST into actual source code. This intermediary approach allows the model to learn code structure more effectively while maintaining the ability to generate syntactically correct and semantically accurate source code.

Inventive Principle:

Principle #24Intermediary (Mediator)

2Loss of information

If traditional training data formats are used, then training efficiency is maintained, but learning of source code structure is insufficient

Engineering Contradiction:

Improvestructural informationVSAvoidtraining efficiency

Core Design Contradiction:

Loss of informationVSProductivity

Solution Approach 1:

The patent segments the source code into its structural components by representing it as an Abstract Syntax Tree (AST). This segmentation allows the model to learn hierarchical code structures more effectively, breaking down complex code into manageable structural elements that can be systematically processed and generated, thereby reducing information loss while maintaining training efficiency.

Inventive Principle:

Principle #1Segmentation

3Manufacturing precision

If direct natural language to source code generation is performed, then generation speed is maintained, but structural accuracy deteriorates

Engineering Contradiction:

Improvecode structure precisionVSAvoidgeneration process complexity

Core Design Contradiction:

Manufacturing precisionVSDevice complexity

Solution Approach 1:

The patent introduces an Abstract Syntax Tree (AST) as an intermediary representation between natural language specifications and source code. The model first generates an AST that captures the structural and grammatical properties of the target code, then converts this AST into actual source code. This intermediary approach allows the model to learn code structure more effectively while maintaining the ability to generate syntactically correct and semantically accurate source code.

Inventive Principle:

Principle #24Intermediary (Mediator)

Data Source

PatentUS20250190180A1Device and method for automatically generating source code

Publication Date: 2025.06.12 ELECTRONICS & TELECOMM RES INST

AI summary

Provided are a device and method for automatically generating source code. The device includes a training data generator that receives first training data consisting of a pair of natural language specification and source code and converts the first training data into second training data consisting of a pair of natural language specification and abstract syntax tree and a large language model trainer that trains a large language model based on the second training data, in which the abstract syntax tree includes structural information and semantic information of the source code.