AST-Based Cooccurrence Matrices for Non-Local Code Embeddings
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Traditional natural language processing techniques fail to accurately analyze computer code due to its non-local structure, where tokens may be related across large distances, unlike natural language, leading to inaccurate code analyses.
Innovation Solution
The use of abstract syntax trees (ASTs) to generate cooccurrence matrices of token pairs, enabling the application of unsupervised learning techniques like word2vec and GloVe to identify vector-space embeddings in source code, capturing non-local relationships between tokens.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If traditional natural language processing techniques are used to analyze code, then the analysis process is simple and familiar, but the analysis accuracy is poor due to non-local token relationships
Solution Approach 1:
The patent segments the code into an abstract syntax tree (AST) structure, dividing the code into hierarchical components (nodes and edges) that represent syntactic relationships. This segmentation allows the system to capture non-local token relationships by traversing the tree structure, thereby improving analysis accuracy while managing complexity through structured decomposition
Solution Approach 2:
The patent introduces an intermediary representation (AST with node embeddings) between the raw code and the final analysis results. This intermediary structure captures non-local relationships by representing tokens as nodes in a tree and their relationships as edges, enabling accurate analysis of token pairs that are far apart in the original code while maintaining manageable processing complexity
2Measurement precision
If unsupervised learning techniques are applied to code tokens, then non-local relationships can be captured, but additional labels would be required for traditional supervised approaches
Solution Approach 1:
The patent implements self-service by using unsupervised learning techniques that automatically learn token relationships from the code structure itself without requiring external labels. The AST-based embeddings are generated solely from the syntactic structure of the code, allowing the system to capture non-local token relationships while eliminating the need for additional labeled data
3Measurement precision
If local context windows are used for token embeddings, then processing is efficient and simple, but non-local token relationships are missed
Solution Approach 1:
The patent transitions from a one-dimensional local context window approach to a multi-dimensional AST structure where tokens are positioned in a hierarchical tree space. This dimensional change allows the system to capture non-local relationships by traversing tree edges rather than relying on linear proximity, improving context capture accuracy while managing processing time through efficient tree traversal algorithms
Data Source
AI summary
Techniques for unsupervised learning of embeddings on source code from non-local contexts are described. Code can be processed to generate an abstract syntax tree (AST) which represents syntactic paths between tokens in the code. Once the AST(s) have been generated, the paths in the AST(s) can be crawled to identify terminals (e.g., leaf nodes in the AST) and paths between terminals can be identified. The pairs of tokens identified at the ends of each path can then be used to generate a cooccurrence matrix. For example, if X number of unique terminals are identified, a matrix of size X by X can be generated to indicate a frequency at which pairs of terminals cooccur. This cooccurrence matrix can then be used as input to existing techniques for learning vector-space embeddings, such as word2vec, GloVe, Swivel, etc.