Pre-trained Contextual Embeddings for Programming Languages

Resolve Bottlenecks,
Find Innovative SolutionsGenerate Solutions

Solution Overview

Problem

Natural language processing (NLP) and machine learning (ML) face challenges in handling programming code due to its repetitive nature and unique concept keywords, leading to the out-of-vocabulary (OOV) problem, where created identifiers may lose information during processing, and existing solutions like sub-word tokenization are non-deterministic and require large vocabularies.

Innovation Solution

A method for building pre-trained contextual embeddings using specialized programming language keywords, which involves collecting and preparing programming code, creating contextual embeddings, and storing them to improve machine learning model training and address the OOV problem, incorporating techniques like tokenization, word embeddings, and transformer architectures.

Engineering Contradictions & Design Principles

VSEngineering Contradiction Analysis

1Adaptability or versatility

If sub-word tokenization is used to handle programming code, then vocabulary coverage is improved, but the solution becomes non-deterministic and requires large vocabularies

Engineering Contradiction:

Improvevocabulary coverageVSAvoidvocabulary size

Core Design Contradiction:

Adaptability or versatilityVSDevice complexity

Solution Approach 1:

The patent segments programming code into tokens using a tokenizer that handles identifiers, keywords, and operators. This segmentation approach allows the system to process code systematically without requiring large vocabularies, as each token type is handled according to its category rather than requiring individual vocabulary entries for every possible identifier.

Inventive Principle:

Principle #1Segmentation

Solution Approach 2:

The patent changes the parameter of vocabulary representation by using contextual embeddings that capture the meaning and usage of tokens in context, rather than relying on fixed vocabulary sizes. This allows the system to handle diverse programming constructs with a manageable vocabulary by representing tokens in continuous vector spaces where semantic relationships are preserved.

Inventive Principle:

Principle #35Parameter changes

2Adaptability or versatility

If created identifiers are processed during language processing, then vocabulary expansion occurs, but information is lost during processing

Engineering Contradiction:

Improveidentifier handling capabilityVSAvoididentifier information

Core Design Contradiction:

Adaptability or versatilityVSLoss of information

Solution Approach 1:

The patent performs preliminary action by creating contextual embeddings for identifiers before they are processed by the language model. This pre-processing step captures the semantic meaning and contextual information of identifiers, preserving this information throughout subsequent processing steps rather than losing it during standard language processing operations.

Inventive Principle:

Principle #10Preliminary action

Solution Approach 2:

The patent introduces contextual embeddings as an intermediary representation between raw identifiers and the language model processing. This intermediary layer preserves identifier information by representing identifiers in a continuous vector space that captures their semantic meaning, allowing the model to process them without information loss while maintaining adaptability to different identifier types.

Inventive Principle:

Principle #24Intermediary (Mediator)

3Productivity

If natural language processing techniques are applied to programming code, then processing speed is improved, but accuracy decreases due to the repetitive nature and unique keywords of code

Engineering Contradiction:

Improveprocessing speedVSAvoidcode processing accuracy

Core Design Contradiction:

ProductivityVSMeasurement precision

Solution Approach 1:

The patent applies local quality by using a specialized tokenizer that treats different token types (identifiers, keywords, operators, literals) differently according to their specific characteristics. This allows the system to process code efficiently while maintaining accuracy by applying appropriate processing rules to each token type based on its local properties rather than using a uniform approach.

Inventive Principle:

Principle #3Local quality

Solution Approach 2:

The patent performs preliminary action by pre-processing programming code through specialized tokenization and creating contextual embeddings before feeding it to the language model. This pre-processing step adapts the code to the model's expectations while preserving its structural and semantic properties, enabling both fast processing and high accuracy by resolving the tension between speed and precision upfront.

Inventive Principle:

Principle #10Preliminary action

Data Source

PatentUS11429352B2Building pre-trained contextual embeddings for programming languages using specialized vocabulary

Publication Date: 2022.08.30 INTERNATIONAL BUSINESS MACHINE CORPORATION

AI summary

A method, a computer system, and a computer program product for building pre-trained contextual embeddings is provided. Embodiments of the present invention may include collecting programming code. Embodiments of the present invention may include loading and preparing the programming code using a specialized programming language keywords-based vocabulary. Embodiments of the present invention may include creating contextual embeddings for the programming code. Embodiments of the present invention may include storing the contextual embeddings.