341 results about "Lexical analysis" patented technology

In an aspect, a computer-implemented method may include: performing word tokenization on identifiers in a first identifier set to obtain a first set of tokens; performing word tokenization on identifiers in a second identifier set to obtain a second set of tokens; obtaining a weighting value for each of a plurality of tokens in the first set of tokens and the second set of tokens based on a frequency of occurrence; for each identifier of a plurality of identifiers in the first identifier set and of a plurality of identifiers in the second identifier set, obtaining embeddings for the identifier based on the embeddings of tokens obtained by performing word tokenization on that identifier and based on the weighting value for such tokens; obtaining similarity metrics. The similarity metrics may be used for rapid query handling to generate user interface screens in real time.

There are disclosed herein systems and methods for human agency support and facilitation through an integrated use of context information, historical work product, biosensors, explicit user input and a generative AI or generalist agent. The system comprises input means, tokenization, a generative AI or generalist agent and an output stage capable of enacting agency on a user's behalf using output tokens from the language model.

In an aspect, a computer-implemented method may include: performing word tokenization on identifiers in a first identifier set to obtain a first set of tokens; performing word tokenization on identifiers in a second identifier set to obtain a second set of tokens; obtaining a weighting value for each of a plurality of tokens in the first set of tokens and the second set of tokens based on a frequency of occurrence; for each identifier of a plurality of identifiers in the first identifier set and of a plurality of identifiers in the second identifier set, obtaining embeddings for the identifier based on the embeddings of tokens obtained by performing word tokenization on that identifier and based on the weighting value for such tokens; obtaining similarity metrics. The similarity metrics may be used for rapid query handling to generate user interface screens in real time.

The invention discloses an internet-based intelligent bid invitation management platform, which comprises a data integration and analysis module configured to establish an intelligent comprehensive economic analysis model by integrating internal data and / or external data of the intelligent bid invitation platform; the standardized bid invitation file module is configured to construct an industry lexical analysis and language model by utilizing natural language processing and a knowledge graph; the intelligent question-and-answer type bidding file filling module is configured to complete compiling of a bidding file by guiding a bidder to adopt an intelligent question-and-answer mode; the intelligent bid evaluation tool set module is configured to assist the bid evaluation process by establishing an intelligent bid evaluation tool; the risk early warning module is configured to construct a knowledge graph of a supplier, a subject matter and a project; and the large model application module is configured to construct an AI bid evaluation model by using a large model technology. Through intelligent bid invitation management, bid evaluation work efficiency is improved, and fairness and accuracy of bid evaluation results are ensured.

A system for sentiment analysis in language texts based on deep learning, consisting of: a preprocessing unit configured to receive and normalize raw text input from various sources, with the unit performing tokenization, punctuation removal, text stemming, and standardization to prepare text data for analysis; an embedding unit that is operationally connected to the preprocessing unit and is configured to convert normalized text data into numerical representations, wherein the embedding unit also contains pre-trained language embeddings to capture semantic relationships in the text and dynamically selects embeddings based on the source and linguistic context of the text data; a sentiment analysis neural network (SANNet) operationally connected to the embedding unit, comprising layers of a convolutional neural network (CNN) for local pattern recognition and layers of a recurrent neural network (RNN) configured to capture sequential dependencies within text data, with the SANNet also including an attention mechanism to dynamically weight word relevance based on contextual meaning, and the SANNet being trained to interpret and classify complex sentiment patterns in the text; A context module integrated into the neural network for mood analysis. The module includes a bidirectional transformer layer for the simultaneous processing of forward and backward word dependencies, enabling the system to capture complex language structures, including sarcasm, irony, and mixed emotions, and thus perform an accurate mood classification; an output interpretation unit that is operationally connected to the neural network for sentiment analysis and is configured to convert probability-based classification results into human-readable sentiment categories with associated confidence levels, wherein the output interpretation unit also provides interface compatibility for external applications, thus enabling direct integration of sentiment classifications; A continuous training module connected to the neural network for sentiment analysis. The module is configured to use real-time user feedback and sentiment accuracy metrics to retrain the SANNet, thus ensuring adaptive learning for evolving language patterns and domain-specific contexts; a GPU / TPU-optimized processing card that is operationally connected to the neural network for sentiment analysis and is configured to accelerate deep learning calculations within the system, thereby enabling high-throughput, real-time sentiment analysis.

The invention relates to the technical field of computer code upgrading, and discloses an automatic code upgrading method and device based on a compiler and a storage medium. The method comprises the steps that firstly, lexical analysis, grammatical analysis and semantic analysis are conducted on a source code, an abstract syntax tree is generated, then a new standard grammatical rule mapping table is constructed, and a new standard grammatical rule mapping table is generated; and traversing the abstract syntax tree, querying whether the nodes of the abstract syntax tree can be queried in the mapping table, outputting the character strings mapped to the queried nodes, and normally outputting codes conforming to node semantics for the nodes which are not queried. The abstract syntax tree of the original code is automatically traversed, and node matching and conversion are performed according to the syntax rule mapping table of the new standard, so that the need of manual intervention is greatly reduced, the automation degree of code migration from an old standard to the new standard is improved, a large amount of manpower and time cost is saved, and the migration problem caused by human errors is reduced.

Methods and systems for prompting a large language model (LLM) to generate token-efficient output are described. A task prompt is provided to the LLM for performing a natural language task and generating token-efficient output based on the task prompt. Output is received from the LLM based on the task prompt, where the output includes text related to the task and at least a portion of the text is in a token-efficient language. The text in the token-efficient language is tokenized by a tokenizer of the LLM into fewer tokens than would correspond to a tokenization of corresponding text in a different constructed language, such as a programming language.

The invention relates to the technical field of data processing, and discloses an intelligent code review method and system based on a large language model, and the method comprises the steps: obtaining a to-be-reviewed code, carrying out the lexical analysis, grammatical analysis and semantic analysis of the to-be-reviewed code, extracting the key information of the code, and obtaining a preprocessed code; multiple features of the code are extracted from the preprocessed code, and target code features are obtained; inputting the target code features into a large language model, and generating a code review result through the large language model; according to the method, complex code semantics and logic problems can be processed, the code review accuracy is improved, and omission and errors of manual review are reduced.

The invention relates to the technical field of computers, and discloses a method for converting a C + + code into a CUDA-C code, and the method comprises the steps: receiving a C + + source code, carrying out lexical analysis and grammatical analysis, generating an abstract syntax tree, and carrying out semantic analysis to ensure that the source code is correct semantically; analyzing the abstract syntax tree, identifying a data dependency relationship among the calculation tasks, and determining loop statements and conditional statements in the source code; and converting a loop statement and a conditional statement in the source code into a kernel function, a thread block and a thread management code corresponding to the CUDA, and further generating a corresponding compilable source file. According to the method, the abstract syntax tree of the C + + code is analyzed, the parallelism in the abstract syntax tree is analyzed, the code is optimized, the CUDA-C efficient code is generated, then efficient automatic conversion of the code is achieved, and the method has great significance in accelerating program performance and shortening the development period.

Disclosed in the present invention are a method and system for building a compilable customized module. The method comprises: customizing a module on the basis of a process, defining a modeling language syntax, and generating a modeling language file corresponding to the customized module; establishing a text matching rule and a grammar matching rule, using flex to generate a source program file of a lexical analyzer from the text matching rule, and using bison to generate a source program file of a syntax analyzer from the grammar matching rule; performing lexical analysis and generating an abstract syntax tree; traversing the syntax tree, generating a translation C++ file for a modeling language on the basis of a syntax-based translation configuration file, and compiling the translation C++ file; and importing the built customized module and an executable binary file into a simulation platform, and performing flow simulations for different scenarios including process simulation, optimization and parameter tuning. The present invention can realize the rapid commissioning of a process and can seamlessly switch a customized model to various application scenarios.

There are disclosed herein systems and methods for human agency support and facilitation through an integrated use of context information, historical work product, biosensors, explicit user input and a generative Al or generalist agent. The system comprises input means, tokenization, a generative Al or generalist agent and an output stage capable of enacting agency on a user's behalf using output tokens from the language model.

A computer-implemented method for improving language models includes tokenization processing to concatenate an obtained image and texts with designated tags. An image encoder is trained to encode image information of the image. A two token representation is generated by a multi-modal generative agent using the encoded image information and the obtained texts. A text encoder and the image encoder are trained to encode the obtained image and texts. A text decoder and an image decoder are combined for the multi-modal reasoning segmentation by training the text decoder and the image decoder to generate a segmented image and segmented textual information using the two token representation in combination with an output of the text encoder and an output of the image encoder. The method has applications including, but not limited to, use cases in medical Al / healthcare, for example, for optimizing medical diagnosis or treatment or supporting decision making.

The invention discloses a PE compiling method of an AI chip, which comprises the following steps: acquiring a source code, preprocessing the acquired source code to obtain a preprocessed source code, sequentially performing lexical analysis and grammatical analysis on the preprocessed source code to obtain a syntax tree corresponding to the source code, and processing the syntax tree by utilizing a semantic analyzer to obtain a syntax tree corresponding to the source code; according to the syntax tree and the symbol table, using a code conversion module in a low-level virtual machine LLVM compiler to generate an intermediate representation corresponding to the source code after preprocessing, and according to a PE instruction set specification of an AI chip, generating an intermediate representation corresponding to the source code after preprocessing; and converting all operations in the intermediate representation corresponding to the preprocessed source code and each of all data structures into a corresponding PE instruction, and performing parallel optimization on each PE instruction through a plurality of PEs to obtain an execution result of the optimized PE instruction. According to the invention, the technical problem that the traditional compiler cannot give full play to the performance due to the fact that PE optimization is not carried out on the AI chip can be solved.

According to the abstract syntax tree optimization method based on the ST language, the ST programming language is optimized through the compiler front-end technology, the code structure is optimized from the source, the compiling efficiency is improved, and the execution efficiency of the finally generated instruction is improved. An abstract syntax tree optimization method based on an ST language comprises the steps that 1, an antlr tool is used for conducting lexical analysis and grammatical analysis of the ST language, and a complete abstract syntax tree is created; (2) traversing the abstract syntax tree to carry out constant merging, calculating constant tree nodes in advance, and merging the constant tree nodes into a tree node; (3) traversing the abstract syntax tree to eliminate dead codes, and deleting useless tree nodes; and (4) traversing the abstract syntax tree to calculate characteristic values of the tree nodes, and eliminating sub-expressions according to the similarity of the tree nodes, thereby reducing redundant calculation and improving execution efficiency.

The invention relates to the technical field of large language models, and discloses a zero-illusion large language model output method and system, and the method comprises the steps: sequentially carrying out the deterministic lexical analysis and standardized mapping of a received original query, and generating a standard term set; locking to-be-reasoned items through hierarchical matching to form a deterministic retrieval result set; executing five rounds of progressive verification processing on the result set to obtain an integrated result set with an LLM verification mark; and outputting a layered zero illusion result through multiple verification and auditing of structure compliance, semantic consistency and knowledge base double recheck. According to the method, probabilistic reasoning is replaced by full-link deterministic operation, so that each code can be traced back to a credible knowledge source, illusion is thoroughly eliminated, absolute reliability and complete traceability of an output result in a professional field are realized by a verifiable deterministic path, and a foundation is laid for zero-illusion application.

The invention discloses a script conversion method and a related device, which can be applied to the financial field or other fields, a script to be converted is subjected to lexical analysis and decomposition operation to obtain basic grammar units, the basic grammar units are subjected to combination operation to obtain an intermediate expression, and an abstract syntax tree corresponding to the intermediate expression is constructed subsequently. Script conversion is carried out based on the abstract syntax tree, and the purpose of SQL script conversion of a cross-database management system is achieved. Besides, script conversion is carried out based on the intermediate expression, and the intermediate expression only needs to be generated once under the scene that one script needs to be converted into other various scripts, so that the analysis operation of the source script can be reduced, and the script conversion efficiency is improved. Besides, during script conversion, a proper target mapping mode is selected, corresponding grammar conversion is performed in the conversion process, and a layer-by-layer recursive conversion mode is adopted, so that the script conversion accuracy can be improved in the script conversion process.

System (100) for detecting anomalous commands in network traffic using a Large Language Model (LLM), comprising: a. a traffic capture module configured to monitor and extract command-level data from incoming network traffic over one or more communication protocols; b. a preprocessing and tokenization module configured to normalize, filter and transform the extracted commands into a structured format suitable for analysis; c. a contextual embedding module comprising a pre-trained or fine-tuned LLM configured to generate semantic embeddings of the structured commands; d. an anomaly detection module configured to compare the generated embeddings with a baseline profile of legitimate command behavior and identify deviations using machine learning algorithms; e. a threat classification module configured to categorize the identified anomalies based on the type and severity of the threat; and f. a response and logging module configured to initiate automatic remedial actions and record the events for further analysis, g. the system operates in real time to detect, classify, and respond to anomalous commands in a networked environment.

The invention discloses an ANTLR4-based multi-manufacturer network equipment configuration analysis method and system, and the method comprises the steps: carrying out the preprocessing of a configuration file; based on a machine learning algorithm of multi-dimensional feature analysis, automatic identification of configuration file manufacturer types is realized; based on the identification result of the manufacturer type, loading an ANTLR4 grammar parser corresponding to the manufacturer type; a lexical analyzer in ANTLR4 realizes lexical analysis on the configuration file through lexical rule definition, a Token classification system and an optimized analysis algorithm; a syntactic analyzer in ANTLR4 constructs an abstract syntax tree according to a lexical analysis result and a syntactic rule defined based on a context-independent grammar; traversing the abstract syntax tree, and extracting key configuration information; and converting the specific configuration format of each manufacturer into a unified standard model. According to the method and the system, network equipment configuration files of different manufacturers can be automatically identified and analyzed, and key configuration information is extracted and converted into a unified model.

A drug design method based on an autoregressive model is provided, which relates to the field of drug design technologies. The method includes: applying a sub-word tokenization algorithm to biological text processing, training protein and ligand information in data sets to obtain a protein tokenizer and a ligand tokenizer, and constructing a tokenizer of the autoregressive model; processing and transforming original data in the data sets into a text form, and encoding by the tokenizer to construct a training data set for the autoregressive model; training the autoregressive model by the training data set, so that the autoregressive model can understand SMILES representations of ligands and learn an interaction mode between proteins and ligands; generating predicted ligands by using the trained autoregressive model, and post-processing through a chemical information tool to acquire candidate ligands with specific chemical structures; and evaluating and optimizing the candidate ligands to determine target candidate molecules.

The embodiment of the invention provides a code error repairing method based on a compiler, which comprises the following steps: receiving a target code written by using a target language, and executing a first repairing operation on the target code in the process of compiling the target code by the compiler to obtain a first repairing sequence, the first repairing operation comprises lexical error repairing corresponding to the lexical analysis process and grammatical error repairing corresponding to the grammatical analysis process. Performing a plurality of rounds of semantic repair operation based on an abstract syntax tree, wherein the abstract syntax tree used for the first round of semantic repair operation is generated after the first repair sequence is subjected to syntactic analysis by the compiler; any round of semantic repair operation comprises the steps of cloning a current abstract syntax tree to obtain a cloned syntax tree, and performing semantic attribute labeling based on the cloned syntax tree; and for any error node with the error attribute label in the clone syntax tree, performing semantic repair on a node corresponding to the abstract syntax tree.

The invention relates to the technical field of data analysis, in particular to a multi-mode heterogeneous data query method which comprises the following steps: receiving a query statement sent by a user side, performing lexical analysis on the query statement through an analyzer, and performing semantic check on the analyzed query statement to obtain an initial query statement; sending the initial query statement to a query optimizer to generate a corresponding query syntax tree; after the query syntax tree is sent to a Data-speed engine for prompt word engineering processing, indexing is carried out in combination with an index vector library to obtain an index result, and an execution plan is intelligently predicted based on the index result; inputting the intelligent prediction execution plan into a large language model system, outputting an execution code corresponding to a current execution task and operators of an available instruction set corresponding to different execution tasks in real time, and introducing an entity identifier into an attention mechanism of the large language model system for supervision and fine adjustment; and calling an operator corresponding to the execution code through a Data-speed JIT generator, and generating an execution result based on the operator.

A model-as-a-service (MaaS) platform performs cross-model resources allocation from a shared pool of GPU resources based on model-agnostic metrics generated by a metric standardizer. The metric standardizer receives, from model providers, model-specific benchmark metrics that define relationships between resource utilization and token processing according to the different model-specific tokenization schemes; receives, from one or more MaaS components, token-based job metrics pertaining to LLM processing tasks; and determines, based on the model-specific benchmark metrics and token-based job metrics, the model-agnostic metrics for multiple model pools executing instances of different large language models (LLMs) that generate and process text according to different model-specific tokenization schemes. The MaaS platform further includes one or more resource allocation components that dynamically reallocates resources of the shared pool based on the model-agnostic metric.

The invention provides a DSL (Digital Subscriber Language)-based public dimension data conversion method and system, and the method comprises the following steps: reading an input script as a file stream by using a business language script reading module; segmenting characters in the file stream into tokens by utilizing a lexical analysis module; using a grammar parser module to construct the token into a grammar tree; traversing all nodes of the syntax tree by using a syntax traversal module, and triggering a conversion rule monitor module at each node; and extracting information by using the conversion rule monitor module, and assembling the information into a computer language program. According to the technical scheme, modeling is carried out on the service function through the configuration data conversion assembly of the DSL, the model is formed, an existing model or a newly-built model is introduced during modeling, and model verification is carried out on a system, so that the condition of repeated modeling is avoided, and a bloated model can be simplified.

The invention provides a model training method for a transverse mixed attention mechanism, a medium, equipment and a program product, and the method comprises the steps: obtaining a data set containing a plurality of sample sequences, each sample sequence in the data set being formed by arranging a plurality of Token sequences obtained through word segmentation; constructing a to-be-trained model based on the pre-trained full attention model, and adding newly added parameters for linear attention calculation; in the same transverse mixed attention layer, executing total attention calculation on a Token set in a preset total attention calculation range, executing linear attention calculation on all Tokens, and fusing results of the total attention calculation and the linear attention calculation to obtain transverse mixed attention output used for forward reasoning and loss calculation; and based on the output and prediction result, only updating the newly added parameters to optimize the to-be-trained model until the to-be-trained model converges. According to the method, the calculation complexity and video memory occupation of long text sequence processing are reduced, and the reasoning speed and the resource utilization rate are improved.

Systems and methods are disclosed for generating a semi-structured term sheet. According to some embodiments, the systems and methods include receiving a brief description of a term sheet in a natural language format from a user interface connected to the electronic device; tokenizing the brief description to create a plurality of numerical values for a content of the brief description; using a large language model to retrieve information relevant to the brief description based on the tokenization; selecting a dynamic template database to the information retrieved by the large language model based on the retrieved information; querying the dynamic template database using the retrieved information to extract a term; and outputting the terms in a determined format based on the dynamic template database.

A hybrid recommendation system for generating personalized film recommendations, the system comprising the following: a processing unit for receiving metadata, configured to receive descriptive data related to films, the descriptive data including plot summaries, genre identifiers, cast lists and director names; a text preprocessing module that is communicatively coupled with the metadata ingestion processing unit and is configured to perform tokenization, stopword removal, stemming, and lemmatization on the descriptive data to obtain a processed corpus; a TF-IDF vectorization unit configured to encode the processed corpus into high-dimensional semantic feature vectors by calculating TF-IDF inverse document frequency values ​​over the entire film dataset, with the semantic feature vectors representing the contextual meaning of terms for individual films; a collaborative filter engine comprising a user-element interaction matrix, wherein the engine is configured to compute latent preference signals using one or more techniques selected from the group consisting of cosine similarity, k-nearest neighbor similarity, and matrix factorization; a score fusion controller coupled to both the TF-IDF vectorization unit and the collaborative filter engine, wherein the module is configured to normalize the semantic feature vectors and the collaborative preference scores and dynamically combine them according to an adaptive weighting coefficient, the coefficient being determined as a function of data density, interaction sparsity, and user history length; and a recommendation output module configured to evaluate candidate films for each user based on the combined score and generate a top-N recommendation list.

The invention aims to provide an abnormal access detection method and system based on data analysis, and relates to the technical field of network data communication, user behavior classification is carried out based on a support vector machine, only normal behavior data is needed to establish a detection model, abnormal data is not needed, the model training complexity is highly simplified, and the user experience is improved. Compared with a traditional method, the OCSVM has distinct advantages in the aspects of single-class data processing and anomaly detection; according to the method, log data are analyzed by using an LEX lexical analyzer and a YACC syntactic analyzer, high-precision description of user behaviors is realized, feature selection is performed on original logs, sensitive information such as user names, operation time and client IPs is extracted, data redundancy is reduced, and high efficiency of model training and recognition is ensured; according to the anomaly detection method provided by the invention, the user behavior anomaly information in the database can be efficiently analyzed, and an alarm can be given in time and necessary countermeasures can be taken, so that potential safety risks or data leakage can be prevented.

The invention discloses a real-time analysis and view editing method and system based on streaming data, and the method comprises the steps: receiving streaming data which comes from a large language model and is organized in an editing protocol format, carrying out the real-time lexical analysis of the streaming data, and decomposing the streaming data into a semantic unit sequence through a semantic pattern recognition technology; performing grammar analysis on the semantic unit sequence, and converting the semantic unit into a structured editing instruction for modifying an existing rendering tree node according to an instruction pattern recognition mechanism and a target object analysis mechanism; converting the structured editing instruction into an atomic instruction sequence executable by a rendering tree editing interface; and modifying the rendering tree in real time according to the atomic instruction sequence and triggering update rendering of the view. According to the method, the content generated by the AI is analyzed in real time by adopting a streaming output mechanism, so that a user can observe the generation process of the view in real time, local redrawing of the view is realized by using an editing protocol, and the view editing efficiency is greatly improved.

The invention relates to the technical field of nuclear power plants, in particular to a nuclear power equipment operation defect grading method and device, electronic equipment and a storage medium. The invention provides a nuclear power equipment operation defect grading method, and aims to realize grading of nuclear power equipment operation defects through a systematic and multi-level fusion decision-making mechanism. The problems that rule grading lacks flexibility, graph grading is influenced by keyword sensitivity, large model grading field knowledge is insufficient, and precision and adaptability are limited due to the fact that a traditional fusion strategy is single in the prior art are effectively solved. The core of the method is that lexical meta processing and a value matrix mechanism are introduced, dynamic and fine-grained fusion of multi-base classifier output is achieved, and therefore the overall performance of a grading system is improved. Through lexical element processing, value matrix construction and dynamic updating and multi-classifier fusion decision making, the nuclear power equipment defect grading method with high precision, high adaptability and high reliability is realized, and many limitations in related technologies are effectively solved.

The invention discloses a code similarity detection method and system based on a large language model, and the method comprises the steps: obtaining a to-be-detected source code pair, and marking the to-be-detected source code pair as a source code A and a source code B; performing code cleaning and format standardization on the source code A and the source code B, and mapping a variable name and a function name which are customized by a user into a uniform placeholder; analyzing the source codes A and B based on the abstract syntax tree, respectively replacing variable names and function names in the source codes A and B with unified serialized placeholders, and maintaining a mapping table; meanwhile, expanding a lexical dictionary of the pre-training large language model, and inserting a special identifier; splicing the replaced code snippets with special identifiers, and constructing a structure sensing input sequence; inputting the structure perception input sequence into a pre-trained large language model backbone network for feature coding to obtain a high-dimensional semantic feature vector containing global context information; connecting a multi-task prediction head behind the large language model backbone network, inputting the feature vector into the multi-task prediction head, outputting probability distribution of code clone types through a classification task head, and respectively outputting a row level similarity score and a lexical element level similarity score through a regression task head; and according to the classification probability and the regression score, performing comprehensive judgment by combining a preset threshold, and generating a detection report. According to the method, similar codes after variable renaming, statement rearrangement or control flow transformation can be accurately recognized, and the accuracy and robustness of code similarity detection are improved.