Universal Lexical Analyzer Using Probability Parameters

Resolve Bottlenecks,
Find Innovative SolutionsGenerate Solutions

Solution Overview

Problem

Existing lexical analyzers are often specific to particular programming languages and require manual configuration with regular expressions, which is time-consuming, error-prone, and inefficient, especially when the language evolves or when analyzing unfamiliar code.

Innovation Solution

A universal lexical analyzer is developed that can be configured for any programming language using a set of probability parameters determined from training programs, eliminating the need for manual specification of regular expressions and allowing for easy updates as the language changes.

Engineering Contradictions & Design Principles

VSEngineering Contradiction Analysis

1Reliability

If manual configuration with regular expressions is used, then lexical analysis can be performed for specific programming languages, but the process becomes time-consuming and error-prone

Engineering Contradiction:

Improveaccuracy of lexical analysisVSAvoidtime for configuring lexical analyzer

Core Design Contradiction:

ReliabilityVSLoss of time

Solution Approach 1:

The system enables self-service by automatically generating lexical analyzers through machine learning. The trainer component learns from training programs and automatically creates probability parameters and regular expressions, eliminating the need for manual configuration and reducing both time consumption and human error in the lexical analysis setup process

Inventive Principle:

Principle #25Self-service

Solution Approach 2:

The system performs preliminary action by pre-training the lexical analyzer on multiple programming languages before actual use. The probability parameters are determined in advance from training programs, so when the analyzer needs to process code, it is already configured and ready, eliminating the time-consuming manual setup phase

Inventive Principle:

Principle #10Preliminary action

2Adaptability or versatility

If manual configuration with regular expressions is used, then lexical analysis can be performed, but the process becomes inefficient especially when the language evolves

Engineering Contradiction:

Improveadaptability to language changesVSAvoidefficiency of lexical analysis setup

Core Design Contradiction:

Adaptability or versatilityVSProductivity

Solution Approach 1:

The system implements dynamics by making the lexical analyzer adaptable and reconfigurable. When programming languages evolve or new languages need to be supported, the trainer can learn from new training programs and update the probability parameters automatically. This dynamic adaptation eliminates the inefficiency of manual reconfiguration for language changes

Inventive Principle:

Principle #15Dynamics

Solution Approach 2:

The system achieves universality by creating a multi-functional trainer that can handle multiple programming languages simultaneously. The same lexical analyzer framework, guided by probability parameters learned from diverse training programs, can adapt to different language syntaxes and evolve with language changes, improving productivity across multiple languages without requiring separate manual configurations

Inventive Principle:

Principle #6Universality (Multi-functionality)

3Adaptability or versatility

If a universal lexical analyzer is developed, then adaptability to various programming languages is improved, but the complexity of determining probability parameters increases

Engineering Contradiction:

Improvesupport for multiple programming languagesVSAvoidcomplexity of parameter determination process

Core Design Contradiction:

Adaptability or versatilityVSDevice complexity

Solution Approach 1:

The system uses an intermediary approach by introducing a trainer component that mediates between training programs and the lexical analyzer. The trainer automatically extracts and determines probability parameters from training programs, serving as an intermediary that simplifies the complexity of parameter determination while enabling universal support for multiple programming languages

Inventive Principle:

Principle #24Intermediary (Mediator)

Data Source

PatentUS10169324B2Universal lexical analyzers

Publication Date: 2019.01.01 MICRO FOCUS LLC

AI summary

An example lexical analysis system may include lexical analyzer instructions and training instructions. The lexical analyzer instructions, when executed, may form a universal lexical analyzer that is to lexically analyze an input program by evaluating a function that maps an input tuple of lexemes to an output tuple of tokens based on input probability parameters. The universal lexical analyzer may be configurable for any programming language by inputting a set of probability parameters for the respective programming language as the input probability parameters of the function. The training instructions, when executed, may generate a set of probability parameters for any programming language by: receiving training programs that are in the respective programming language, each comprising a tuple of lexemes with associated tokens, and determining the set of probability parameters for the respective programming language based on lexemes and tokens in the training programs that are in the respective programming language.