Multi-lingual Code Completion via Subtokenization
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing software development environments face challenges with code completion systems that fail to recognize out-of-vocabulary code elements, require excessive memory, take too long to generate candidate lists, or provide irrelevant suggestions.
Innovation Solution
A multi-lingual line-of-code completion system using a neural transformer model trained on a large unsupervised dataset of source code programs from various programming languages, which predicts the most likely candidates to complete a line of source code by encoding frequently-used elements as tokens and less frequent elements as subtokens.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Adaptability or versatility
If a traditional code completion system uses a large vocabulary to cover all possible code elements, then it can recognize more code elements, but it requires excessive memory and takes too long to generate candidate lists
Solution Approach 1:
The patent segments the code completion task into two parts: a small vocabulary of common code elements (tokens) and a mechanism to handle rare elements (out-of-vocabulary tokens) through subtokenization. This segmentation allows the system to maintain a manageable vocabulary size while still being able to represent and complete rare code elements by breaking them down into smaller subtokens, thus resolving the contradiction between comprehensive recognition capability and vocabulary size.
2Adaptability or versatility
If a code completion system uses a large vocabulary to cover all possible code elements, then it can recognize more code elements, but it increases memory requirements
Solution Approach 1:
The patent applies segmentation by dividing the code vocabulary into frequent tokens and rare subtokens. This allows the system to store only the essential frequent tokens in the main vocabulary (reducing memory) while using subtokenization to handle rare elements on-demand, thus maintaining comprehensive recognition capability with reduced memory requirements.
3Reliability
If a code completion system generates a comprehensive list of candidates to ensure relevance, then it provides better completion suggestions, but it takes too long to generate the list
Solution Approach 1:
The patent applies partial action by generating only the top-k most likely candidates rather than exhaustively listing all possible completions. The system uses the segmented vocabulary (tokens and subtokens) to efficiently compute and rank the most relevant candidates, providing sufficient relevance without the time cost of generating comprehensive lists of all possible code elements.
Data Source
AI summary
A code completion tool uses a neural transformer model to generate candidate sequences to complete a line of source code. The neural transformer model is trained using a conditional language modeling objective on a large unsupervised dataset that includes source code programs written in several different programming languages. The neural transformer model is used within a beam search that predicts the most likely candidate sequences for a code snippet under development.