272 results about "Lexical analysis" patented technology

In an aspect, a computer-implemented method may include: performing word tokenization on identifiers in a first identifier set to obtain a first set of tokens; performing word tokenization on identifiers in a second identifier set to obtain a second set of tokens; obtaining a weighting value for each of a plurality of tokens in the first set of tokens and the second set of tokens based on a frequency of occurrence; for each identifier of a plurality of identifiers in the first identifier set and of a plurality of identifiers in the second identifier set, obtaining embeddings for the identifier based on the embeddings of tokens obtained by performing word tokenization on that identifier and based on the weighting value for such tokens; obtaining similarity metrics. The similarity metrics may be used for rapid query handling to generate user interface screens in real time.

Methods and systems for prompting a large language model (LLM) to generate token-efficient output are described. A task prompt is provided to the LLM for performing a natural language task and generating token-efficient output based on the task prompt. Output is received from the LLM based on the task prompt, where the output includes text related to the task and at least a portion of the text is in a token-efficient language. The text in the token-efficient language is tokenized by a tokenizer of the LLM into fewer tokens than would correspond to a tokenization of corresponding text in a different constructed language, such as a programming language.

The invention relates to the technical field of data processing, and discloses an intelligent code review method and system based on a large language model, and the method comprises the steps: obtaining a to-be-reviewed code, carrying out the lexical analysis, grammatical analysis and semantic analysis of the to-be-reviewed code, extracting the key information of the code, and obtaining a preprocessed code; multiple features of the code are extracted from the preprocessed code, and target code features are obtained; inputting the target code features into a large language model, and generating a code review result through the large language model; according to the method, complex code semantics and logic problems can be processed, the code review accuracy is improved, and omission and errors of manual review are reduced.

Disclosed in the present invention are a method and system for building a compilable customized module. The method comprises: customizing a module on the basis of a process, defining a modeling language syntax, and generating a modeling language file corresponding to the customized module; establishing a text matching rule and a grammar matching rule, using flex to generate a source program file of a lexical analyzer from the text matching rule, and using bison to generate a source program file of a syntax analyzer from the grammar matching rule; performing lexical analysis and generating an abstract syntax tree; traversing the syntax tree, generating a translation C++ file for a modeling language on the basis of a syntax-based translation configuration file, and compiling the translation C++ file; and importing the built customized module and an executable binary file into a simulation platform, and performing flow simulations for different scenarios including process simulation, optimization and parameter tuning. The present invention can realize the rapid commissioning of a process and can seamlessly switch a customized model to various application scenarios.

A computer-implemented method for improving language models includes tokenization processing to concatenate an obtained image and texts with designated tags. An image encoder is trained to encode image information of the image. A two token representation is generated by a multi-modal generative agent using the encoded image information and the obtained texts. A text encoder and the image encoder are trained to encode the obtained image and texts. A text decoder and an image decoder are combined for the multi-modal reasoning segmentation by training the text decoder and the image decoder to generate a segmented image and segmented textual information using the two token representation in combination with an output of the text encoder and an output of the image encoder. The method has applications including, but not limited to, use cases in medical Al / healthcare, for example, for optimizing medical diagnosis or treatment or supporting decision making.

The invention relates to the technical field of large language models, and discloses a zero-illusion large language model output method and system, and the method comprises the steps: sequentially carrying out the deterministic lexical analysis and standardized mapping of a received original query, and generating a standard term set; locking to-be-reasoned items through hierarchical matching to form a deterministic retrieval result set; executing five rounds of progressive verification processing on the result set to obtain an integrated result set with an LLM verification mark; and outputting a layered zero illusion result through multiple verification and auditing of structure compliance, semantic consistency and knowledge base double recheck. According to the method, probabilistic reasoning is replaced by full-link deterministic operation, so that each code can be traced back to a credible knowledge source, illusion is thoroughly eliminated, absolute reliability and complete traceability of an output result in a professional field are realized by a verifiable deterministic path, and a foundation is laid for zero-illusion application.

System (100) for detecting anomalous commands in network traffic using a Large Language Model (LLM), comprising: a. a traffic capture module configured to monitor and extract command-level data from incoming network traffic over one or more communication protocols; b. a preprocessing and tokenization module configured to normalize, filter and transform the extracted commands into a structured format suitable for analysis; c. a contextual embedding module comprising a pre-trained or fine-tuned LLM configured to generate semantic embeddings of the structured commands; d. an anomaly detection module configured to compare the generated embeddings with a baseline profile of legitimate command behavior and identify deviations using machine learning algorithms; e. a threat classification module configured to categorize the identified anomalies based on the type and severity of the threat; and f. a response and logging module configured to initiate automatic remedial actions and record the events for further analysis, g. the system operates in real time to detect, classify, and respond to anomalous commands in a networked environment.

The invention discloses an ANTLR4-based multi-manufacturer network equipment configuration analysis method and system, and the method comprises the steps: carrying out the preprocessing of a configuration file; based on a machine learning algorithm of multi-dimensional feature analysis, automatic identification of configuration file manufacturer types is realized; based on the identification result of the manufacturer type, loading an ANTLR4 grammar parser corresponding to the manufacturer type; a lexical analyzer in ANTLR4 realizes lexical analysis on the configuration file through lexical rule definition, a Token classification system and an optimized analysis algorithm; a syntactic analyzer in ANTLR4 constructs an abstract syntax tree according to a lexical analysis result and a syntactic rule defined based on a context-independent grammar; traversing the abstract syntax tree, and extracting key configuration information; and converting the specific configuration format of each manufacturer into a unified standard model. According to the method and the system, network equipment configuration files of different manufacturers can be automatically identified and analyzed, and key configuration information is extracted and converted into a unified model.

A drug design method based on an autoregressive model is provided, which relates to the field of drug design technologies. The method includes: applying a sub-word tokenization algorithm to biological text processing, training protein and ligand information in data sets to obtain a protein tokenizer and a ligand tokenizer, and constructing a tokenizer of the autoregressive model; processing and transforming original data in the data sets into a text form, and encoding by the tokenizer to construct a training data set for the autoregressive model; training the autoregressive model by the training data set, so that the autoregressive model can understand SMILES representations of ligands and learn an interaction mode between proteins and ligands; generating predicted ligands by using the trained autoregressive model, and post-processing through a chemical information tool to acquire candidate ligands with specific chemical structures; and evaluating and optimizing the candidate ligands to determine target candidate molecules.

The embodiment of the invention provides a code error repairing method based on a compiler, which comprises the following steps: receiving a target code written by using a target language, and executing a first repairing operation on the target code in the process of compiling the target code by the compiler to obtain a first repairing sequence, the first repairing operation comprises lexical error repairing corresponding to the lexical analysis process and grammatical error repairing corresponding to the grammatical analysis process. Performing a plurality of rounds of semantic repair operation based on an abstract syntax tree, wherein the abstract syntax tree used for the first round of semantic repair operation is generated after the first repair sequence is subjected to syntactic analysis by the compiler; any round of semantic repair operation comprises the steps of cloning a current abstract syntax tree to obtain a cloned syntax tree, and performing semantic attribute labeling based on the cloned syntax tree; and for any error node with the error attribute label in the clone syntax tree, performing semantic repair on a node corresponding to the abstract syntax tree.

The invention relates to the technical field of data analysis, in particular to a multi-mode heterogeneous data query method which comprises the following steps: receiving a query statement sent by a user side, performing lexical analysis on the query statement through an analyzer, and performing semantic check on the analyzed query statement to obtain an initial query statement; sending the initial query statement to a query optimizer to generate a corresponding query syntax tree; after the query syntax tree is sent to a Data-speed engine for prompt word engineering processing, indexing is carried out in combination with an index vector library to obtain an index result, and an execution plan is intelligently predicted based on the index result; inputting the intelligent prediction execution plan into a large language model system, outputting an execution code corresponding to a current execution task and operators of an available instruction set corresponding to different execution tasks in real time, and introducing an entity identifier into an attention mechanism of the large language model system for supervision and fine adjustment; and calling an operator corresponding to the execution code through a Data-speed JIT generator, and generating an execution result based on the operator.

A model-as-a-service (MaaS) platform performs cross-model resources allocation from a shared pool of GPU resources based on model-agnostic metrics generated by a metric standardizer. The metric standardizer receives, from model providers, model-specific benchmark metrics that define relationships between resource utilization and token processing according to the different model-specific tokenization schemes; receives, from one or more MaaS components, token-based job metrics pertaining to LLM processing tasks; and determines, based on the model-specific benchmark metrics and token-based job metrics, the model-agnostic metrics for multiple model pools executing instances of different large language models (LLMs) that generate and process text according to different model-specific tokenization schemes. The MaaS platform further includes one or more resource allocation components that dynamically reallocates resources of the shared pool based on the model-agnostic metric.

The invention provides a model training method for a transverse mixed attention mechanism, a medium, equipment and a program product, and the method comprises the steps: obtaining a data set containing a plurality of sample sequences, each sample sequence in the data set being formed by arranging a plurality of Token sequences obtained through word segmentation; constructing a to-be-trained model based on the pre-trained full attention model, and adding newly added parameters for linear attention calculation; in the same transverse mixed attention layer, executing total attention calculation on a Token set in a preset total attention calculation range, executing linear attention calculation on all Tokens, and fusing results of the total attention calculation and the linear attention calculation to obtain transverse mixed attention output used for forward reasoning and loss calculation; and based on the output and prediction result, only updating the newly added parameters to optimize the to-be-trained model until the to-be-trained model converges. According to the method, the calculation complexity and video memory occupation of long text sequence processing are reduced, and the reasoning speed and the resource utilization rate are improved.

A hybrid recommendation system for generating personalized film recommendations, the system comprising the following: a processing unit for receiving metadata, configured to receive descriptive data related to films, the descriptive data including plot summaries, genre identifiers, cast lists and director names; a text preprocessing module that is communicatively coupled with the metadata ingestion processing unit and is configured to perform tokenization, stopword removal, stemming, and lemmatization on the descriptive data to obtain a processed corpus; a TF-IDF vectorization unit configured to encode the processed corpus into high-dimensional semantic feature vectors by calculating TF-IDF inverse document frequency values ​​over the entire film dataset, with the semantic feature vectors representing the contextual meaning of terms for individual films; a collaborative filter engine comprising a user-element interaction matrix, wherein the engine is configured to compute latent preference signals using one or more techniques selected from the group consisting of cosine similarity, k-nearest neighbor similarity, and matrix factorization; a score fusion controller coupled to both the TF-IDF vectorization unit and the collaborative filter engine, wherein the module is configured to normalize the semantic feature vectors and the collaborative preference scores and dynamically combine them according to an adaptive weighting coefficient, the coefficient being determined as a function of data density, interaction sparsity, and user history length; and a recommendation output module configured to evaluate candidate films for each user based on the combined score and generate a top-N recommendation list.

The invention aims to provide an abnormal access detection method and system based on data analysis, and relates to the technical field of network data communication, user behavior classification is carried out based on a support vector machine, only normal behavior data is needed to establish a detection model, abnormal data is not needed, the model training complexity is highly simplified, and the user experience is improved. Compared with a traditional method, the OCSVM has distinct advantages in the aspects of single-class data processing and anomaly detection; according to the method, log data are analyzed by using an LEX lexical analyzer and a YACC syntactic analyzer, high-precision description of user behaviors is realized, feature selection is performed on original logs, sensitive information such as user names, operation time and client IPs is extracted, data redundancy is reduced, and high efficiency of model training and recognition is ensured; according to the anomaly detection method provided by the invention, the user behavior anomaly information in the database can be efficiently analyzed, and an alarm can be given in time and necessary countermeasures can be taken, so that potential safety risks or data leakage can be prevented.

The invention discloses a real-time analysis and view editing method and system based on streaming data, and the method comprises the steps: receiving streaming data which comes from a large language model and is organized in an editing protocol format, carrying out the real-time lexical analysis of the streaming data, and decomposing the streaming data into a semantic unit sequence through a semantic pattern recognition technology; performing grammar analysis on the semantic unit sequence, and converting the semantic unit into a structured editing instruction for modifying an existing rendering tree node according to an instruction pattern recognition mechanism and a target object analysis mechanism; converting the structured editing instruction into an atomic instruction sequence executable by a rendering tree editing interface; and modifying the rendering tree in real time according to the atomic instruction sequence and triggering update rendering of the view. According to the method, the content generated by the AI is analyzed in real time by adopting a streaming output mechanism, so that a user can observe the generation process of the view in real time, local redrawing of the view is realized by using an editing protocol, and the view editing efficiency is greatly improved.

The invention relates to the technical field of nuclear power plants, in particular to a nuclear power equipment operation defect grading method and device, electronic equipment and a storage medium. The invention provides a nuclear power equipment operation defect grading method, and aims to realize grading of nuclear power equipment operation defects through a systematic and multi-level fusion decision-making mechanism. The problems that rule grading lacks flexibility, graph grading is influenced by keyword sensitivity, large model grading field knowledge is insufficient, and precision and adaptability are limited due to the fact that a traditional fusion strategy is single in the prior art are effectively solved. The core of the method is that lexical meta processing and a value matrix mechanism are introduced, dynamic and fine-grained fusion of multi-base classifier output is achieved, and therefore the overall performance of a grading system is improved. Through lexical element processing, value matrix construction and dynamic updating and multi-classifier fusion decision making, the nuclear power equipment defect grading method with high precision, high adaptability and high reliability is realized, and many limitations in related technologies are effectively solved.

The invention discloses a code similarity detection method and system based on a large language model, and the method comprises the steps: obtaining a to-be-detected source code pair, and marking the to-be-detected source code pair as a source code A and a source code B; performing code cleaning and format standardization on the source code A and the source code B, and mapping a variable name and a function name which are customized by a user into a uniform placeholder; analyzing the source codes A and B based on the abstract syntax tree, respectively replacing variable names and function names in the source codes A and B with unified serialized placeholders, and maintaining a mapping table; meanwhile, expanding a lexical dictionary of the pre-training large language model, and inserting a special identifier; splicing the replaced code snippets with special identifiers, and constructing a structure sensing input sequence; inputting the structure perception input sequence into a pre-trained large language model backbone network for feature coding to obtain a high-dimensional semantic feature vector containing global context information; connecting a multi-task prediction head behind the large language model backbone network, inputting the feature vector into the multi-task prediction head, outputting probability distribution of code clone types through a classification task head, and respectively outputting a row level similarity score and a lexical element level similarity score through a regression task head; and according to the classification probability and the regression score, performing comprehensive judgment by combining a preset threshold, and generating a detection report. According to the method, similar codes after variable renaming, statement rearrangement or control flow transformation can be accurately recognized, and the accuracy and robustness of code similarity detection are improved.

The invention discloses a macro-based multi-mode circuit design method and device and a medium, and belongs to the technical field of electronic design automation (EDA) design, and the method comprises the steps: setting a working mode, and configuring a relation between the mode and a macro; performing grammar and lexical analysis on the RTL code, abstracting and converting modules, signals, parameters and macro in the code into class and data structures, constructing a relationship between the modules, forming a new data relationship model, and marking macro on each object; copying, dispersing and merging the modes; a new data relation model is generated after operation, and new RTL codes are constructed according to the sequence; checking circuit connectivity; and outputting the RTL code containing multiple modes according to the latest data model. According to the design method, the macro-based multi-mode circuit design is supported, the mode comprises the combination of a plurality of macros, copying, dispersing and merging operation can be carried out among the modes, connectivity can be checked based on circuits among the modes, and the efficiency and quality of chip design are improved.

Devices, systems, and methods for tokenizing search attributes and terms of a search query for an index-based search. A method may include receiving, by a search service of a provider network, a first search query to search a first searchable document set, the first search query including a first search term in a first language; applying a first tokenization rule to identify the first search term in the first search query; determining that the first search term is in the first language; applying a second tokenization rule to tokenize the first search term based on the first search term being in the first language; causing a launch of a search instance by a managed compute service of the provider network, the search instance to execute a search function for a keyword-based text search using the tokenized first search term.

The invention relates to a query statement display method and device, computer equipment, a readable storage medium and a program product. The method comprises the steps that in the process that a user inputs an initial query statement, a current character input by the user is scanned in real time based on a lexical analyzer in a regular matching mode; switching a dynamic lexical analysis mode of a lexical analyzer based on the character type of the current character, and analyzing the current character based on the lexical analyzer after mode switching; based on the analysis result of the current character, generating a structured lexical unit Token stream of the initial query statement, and based on a current Token corresponding to the current character in the structured Token stream, obtaining at least one recommendation value of the current character; and generating a target query statement based on a selection result of the recommended value by the user, and displaying the target query statement in a display interface of the user. According to the method provided by the invention, the actual input requirement of the user can be met.

The invention discloses a general data exchange system based on a configurable label structure, which relates to the technical field of data processing and information integration, and comprises a parameter module used for establishing an identity authentication mechanism of two exchange parties; the business document module is used for establishing a data structure definition based on a preset exchange demand; the receipt logic module is used for carrying out logic analysis and code decoupling on the data structure definition, generating a marked language script code and an execution instruction, constructing an execution matrix and realizing mapping between a database physical field and a memory logic variable; and the standard adaptation module is used for comparing and checking the content of the generated data and feeding back a checking result to the parameter module to correct the exchange rule and parameter configuration. A more intelligent logic separation strategy is implemented through structural insight, data reading, writing and transmission are cooperatively controlled, complete decoupling of service logic and program codes is achieved through a nested tag structure, and flexibility, universality and maintenance efficiency of data exchange are optimized.

The disclosure presents systems and methods for hierarchical classification of input data across a plurality of categories. A machine learning model processes various data formats, starting with dimensional reduction using tokenization techniques, such as Bert-tiny tokenization, to create model-readable representations. The system predicts super-categories, sub-categories, and granular categories through selective activation of sub-layers tied to identified super-categories, optimizing computational efficiency. Label smoothing during training mitigates overconfidence in predictions, while softmax normalization refines inference outputs. Synthetic data generation using Large Language Models (LLMs) supplements training datasets, and an automated data labeling pipeline efficiently generates hierarchical labels. Modifications to the model, such as stop word removal and file size limitations, further reduce latency. Inference analyzes logits to predict hierarchical paths, providing detailed classifications with clear outputs. The method is adaptable for multimodal formats, ensuring scalable and accurate predictions across diverse data types while minimizing computational costs and improving reliability.

The invention discloses a webshell security detection analysis method and system based on a semantic analysis engine, and relates to the technical field of security detection.The webshell security detection analysis method comprises the steps that lexical analysis and grammatical analysis are conducted on a to-be-detected file, the file is converted into a language fragment, and an abstract syntax tree is constructed; extracting the stain subtree, identifying and marking the part containing the potential danger node, and evaluating the risk level of the potential danger node; malicious code behavior characteristics are extracted, and a standard definition BNF is used to construct a characteristic graph; and matching the potential danger nodes with the constructed feature graph, calculating a matching value to evaluate the risk of each potential danger node, and outputting the specific risk value of the node. According to the webshell security detection analysis method based on the semantic analysis engine provided by the invention, the limitation of a traditional detection method is effectively solved. According to the method, potential malicious code behaviors can be accurately identified, and the risk level of each node can be dynamically evaluated, so that more accurate security detection is realized.

The invention provides a multi-database dialect fused language specification, a multi-database dialect fused language conversion method and a multi-database dialect fused language conversion system, and relates to the technical field of databases. A user inputs a unified SQL statement and selects the type of a target database, lexical analysis and grammatical analysis are performed on the unified SQL statement, an abstract syntax tree is constructed, semantic analysis is performed, dialect characteristics in the unified SQL statement are identified, the support condition of the dialect characteristics in the target database is verified, and a dialect conversion strategy is provided. And selecting a corresponding conversion rule in the unified SQL specification library, converting the grammar of the unified SQL statement into the dialect grammar of the target database, and generating the SQL statement. According to the method, one set of SQL and multi-database support is realized through unified SQL specification formulation, intelligent dialect adaptation and configuration driving, a complete technical solution is provided for cross-database application development, and technical challenges caused by database dialect differences are solved.

A model-as-a-service (MaaS) platform performs cross-model resources allocation from a shared pool of GPU resources based on model-agnostic metrics generated by a metric standardizer. The metric standardizer receives, from model providers, model-specific benchmark metrics that define relationships between resource utilization and token processing according to the different model-specific tokenization schemes; receives, from one or more MaaS components, token-based job metrics pertaining to LLM processing tasks; and determines, based on the model-specific benchmark metrics and token-based job metrics, the model-agnostic metrics for multiple model pools executing instances of different large language models (LLMs) that generate and process text according to different model-specific tokenization schemes. The MaaS platform further includes one or more resource allocation components that dynamically reallocates resources of the shared pool based on the model-agnostic metric.

The invention discloses a source code obfuscation method and system, and belongs to the technical field of software development. According to the source code obfuscation method, in the preprocessing obfuscation stage, all first compiling units in a source code to be obfuscated are subjected to lexical analysis one by one to obtain corresponding lexical units, and a preprocessing instruction tree is generated according to all the lexical units; confusion is carried out in a corresponding mode based on the category of each preprocessing node in the preprocessing instruction tree, so that the purposes of deleting the to-be-confused source code annotation and expanding an internal header file of the to-be-confused source code annotation are achieved; in the symbol name obfuscation stage, all second compiling units in a first source code are analyzed one by one to obtain an abstract syntax tree, a symbol table is constructed according to the abstract syntax tree, the first source code is compiled based on the symbol table to obtain an obfuscated code, and the purpose of renaming information such as variables, functions and types in the source code is achieved; according to the method, the difficulty of quickly analyzing the source code is improved, and the protection strength of the source code is improved.

The invention relates to the technical field of SQL (Structured Query Language) grammar conversion, and provides a general SQL grammar conversion method, device and equipment and a medium, the method comprises the following steps: performing lexical analysis on a source end SQL statement to obtain a lexical unit set; based on the type features and semantic attributes of the lexical unit set, determining parser combiners, and constructing a specific parser by infinitely combining the parser combiners; the specific resolver is used for conducting grammar analysis on the lexical unit set, a general abstract syntax tree is constructed through a mapping assembly, and nodes of the general abstract syntax tree correspond to the specific resolver one to one; and traversing the general abstract syntax tree, and generating a target end SQL statement according to an SQL syntax rule of a target end database. Through the technical scheme, any complex source end SQL can be accurately converted into the target end SQL.

The invention discloses a compiling method for compiling a C language source code into an RISC-V assembly code. The method comprises the following steps: acquiring a C language source code; performing lexical analysis on the C language source code to generate a mark flow; performing syntactic analysis on the mark flow, and constructing an abstract syntax tree; performing semantic analysis on the abstract syntax tree to generate a target abstract syntax tree; constructing a runtime environment of the RISC-V assembly code; generating an intermediate code and a control flow diagram corresponding to the intermediate code according to a rule in the runtime environment and the target abstract syntax tree; generating a target code by using the intermediate code and the control flow diagram corresponding to the intermediate code; according to the technical scheme, the intermediate representation more adaptive to RISC-V custom instruction mapping can be generated, so that the execution efficiency of assembly codes is improved; and meanwhile, by designing a lightweight runtime environment, the performance overhead is further reduced.

A system for autonomous compliance with legal regulations in smart contracts; the system includes: a regulatory data collection unit comprising a network interface controller physically connected to an external communications port, a hardware-based public key infrastructure circuit configured to validate digital certificates of regulatory servers, and a direct memory access controller configured to transfer authenticated regulatory update packets from the network interface controller to a volatile buffer memory without processor intervention; a Compliance Code Conversion Unit with a hardware lexicon scanner implemented as a finite state machine and embedded in reconfigurable FPGA logic blocks, a microcontroller executing a hardware-based lexical analysis pipeline stored in firmware registers, and a translation cache memory configured to temporarily store tokenized compliance rules before writing them to a non-volatile instruction memory; a policy evaluation and enforcement unit comprising a transaction verification processor connected to a secure enclave memory, a hardware comparator circuit configured to compare transaction parameters with compliance thresholds stored in the secure enclave memory, and a logic gate circuit configured to generate execution gate signals that selectively allow, block, or modify smart contract execution signals transmitted to a blockchain execution processor; and an audit logging unit with a cryptographic hashing circuit configured to generate block hashes of enforcement records, a timestamp oscillator configured to generate temporal signatures of compliance enforcement actions, and a Merkle tree generation circuit configured to create tamper-proof hierarchical hash structures, with the enforcement records being passed to a decentralized storage interface for anchoring in the chain or distributed ledger.