Code Tokenization for Functional Similarity in Large Code Models

Resolve Bottlenecks,
Find Innovative SolutionsGenerate Solutions

Solution Overview

Problem

Existing natural language processing techniques fail to accurately understand and analyze code language due to textually different but functionally similar or identical code portions, lacking contextual understanding and failing to recognize functional effects, leading to inefficiencies in software analysis and resource usage prediction.

Innovation Solution

The development of a code language processing model that utilizes tokenization and training based on functional similarity, allowing for sophisticated analysis of code segments across multiple languages and environments, including emulation and resource usage prediction without execution, tailored to specific hardware and software attributes.

Engineering Contradictions & Design Principles

VSEngineering Contradiction Analysis

1Speed

If natural language processing techniques are used for code analysis, then processing speed is improved, but accuracy deteriorates due to inability to understand code-specific semantics and contextual attributes

Engineering Contradiction:

Improveprocessing speedVSAvoidcode analysis accuracy

Core Design Contradiction:

SpeedVSMeasurement precision

Solution Approach 1:

The patent segments code into meaningful tokens (identifiers, operators, literals, punctuation) rather than using word-level tokenization. This segmentation allows the model to preserve code-specific semantics and structural relationships, resolving the contradiction by enabling accurate code understanding while maintaining processing efficiency through optimized token-level processing

Inventive Principle:

Principle #1Segmentation

Solution Approach 2:

The patent changes the fundamental parameter of tokenization from natural language word-based to code-syntax-based tokenization. By adjusting the tokenization parameters to match code's unique grammatical structure and semantic categories, the model achieves both speed and accuracy in code analysis

Inventive Principle:

Principle #35Parameter changes

2Ease of operation

If code is tokenized using natural language methods, then ease of operation is improved, but reliability deteriorates due to failure to recognize functionally equivalent but textually different code portions

Engineering Contradiction:

Improveease of operationVSAvoidfunctional equivalence recognition

Core Design Contradiction:

Ease of operationVSReliability

Solution Approach 1:

The patent applies local quality by assigning different token types and representations to code elements based on their specific semantic categories (identifiers, operators, literals). This localized differentiation enables the model to recognize functional equivalence while preserving operational simplicity through standardized token processing protocols

Inventive Principle:

Principle #3Local quality

3Device complexity

If existing tokenization methods are used, then device complexity is reduced, but loss of information increases due to inability to capture contextual attributes and functional effects

Engineering Contradiction:

Improvedevice complexityVSAvoidcontextual information loss

Core Design Contradiction:

Device complexityVSLoss of information

Solution Approach 1:

The patent creates a universal tokenization framework that handles multiple code languages and contexts through a single unified approach. The token types and contextual representation mechanisms work across different programming paradigms, maintaining device simplicity while preventing information loss through comprehensive contextual capture

Inventive Principle:

Principle #6Universality (Multi-functionality)

4Loss of time

If code is analyzed without execution, then loss of time is reduced, but measurement precision deteriorates due to inability to determine actual resource usage

Engineering Contradiction:

Improveanalysis timeVSAvoidresource usage prediction accuracy

Core Design Contradiction:

Loss of timeVSMeasurement precision

Solution Approach 1:

The patent replaces actual code execution (mechanical system) with a neural network-based prediction system that processes tokenized code representations. This substitution eliminates the time-consuming execution step while achieving comparable or superior resource usage prediction accuracy through learned patterns from training data

Inventive Principle:

Principle #28Mechanics substitution (Replace mechanical system)

Data Source

PatentUS20260050533A1Tokenizers for Large Code Language Models

Publication Date: 2026.02.19 AURORA LABS LTD

AI summary

Disclosed herein are techniques for creating and using tokens representing portions of programming code. Techniques include identifying a body of programming code; associating a plurality of tokens with respective portions of the body of programming code; configuring model input data for a code language processing model, the model input data comprising the plurality of tokens; and analyzing at least a part of the body of programming code using the code language processing model influenced by the model input data.