31 results about "Lexical analysis" patented technology

A Cybersecurity Event Handling Processor (CEHP) and method for processing security alerts includes: a File System containing a Universal Target Schema (UTS) of target language representations (UTS JSONs); a Normalizer running Feature Extraction and Word Embeddings algorithms; a Tree Converter; and a Transformer running linguistic and structural matching algorithms. The CEHP: (a) captures threat events in one or more native formats generated by cybersecurity tools; (b) runs Feature Extraction and Word Embeddings algorithms for tokenization and categorization of the captured events to create normalized events; (c) converts the normalized events into trees and then translates the trees into event representations in JSON (or XML) format (Event JSONs); and (d) runs nearest neighbor and / or linguistic and structural matching algorithms to compare the Event JSONs to the UTS JSONs to generate output JSONs (Translation JSONs) from the UTS corresponding to the captured events.

The application discloses a COBOL-to-Java program conversion method and system based on a syntax tree, belongs to the technical field of computer program language conversion, and comprises the following steps: obtaining a COBOL source program file to be converted and configuration rules; performing lexical analysis and syntax analysis to generate an abstract syntax tree; converting COBOL syntax elements into corresponding Java semantic representations according to pre-defined semantic mapping rules; establishing an accurate access chain of a main program and COPYBOOK internal variables through variable path tracking; detecting and solving naming conflicts of COBOL different-level same-name variables based on a scope tree, and renaming variables by adding a parent class name prefix; and generating the converted Java semantic representations into a Java source code file. The application realizes high-fidelity and high-automation COBOL-to-Java conversion, significantly reduces the migration cost of a legacy system, and generates code with strong readability and excellent performance.

This invention discloses a method for porting a digital signal processor (DSP) assembler based on a processor architecture description, comprising the following steps: S1. Based on the processor architecture description, a lexical analysis and syntax analysis part of the assembler is constructed to realize the parsing of special assembly instructions; S2. Taking the parsed instruction data or instruction binary code as input, dynamic encoding and decoding of instructions are implemented according to the instruction definition, thereby realizing the conversion between the object file and the assembly file. This DSP assembler porting method further improves the performance of the assembler and automates the porting process, adapting it to the digital signal processor architecture. It also provides support for single instruction multiple data, very long instruction words, and instruction validity checks. While ensuring that the program execution result meets expectations, it greatly reduces the number of clock cycles required for execution, possesses universality, and meets the needs of rapid processor architecture updates.

The application discloses an AI driving-based structured data migration method and system, belongs to the technical field of big data migration, and aims to solve the technical problem of how to improve the efficiency of large-scale data migration and reduce resource waste. The method comprises the following steps: forming a private large language model with field adaptation capability; performing lexical analysis and syntax analysis on non-standard structured data in a source database based on an AST analysis method, generating an abstract syntax tree AST, traversing and optimizing AST nodes, and converting the AST nodes into standard structured data supported by a target database; calling the large language model, and converting unstructured data into structured data supported by the target database through instruction fine-tuning, thought chain reasoning and step-by-step analysis; generating a migration script by the large language model; monitoring the migration process in real time, recording abnormal events, automatically retrying or repairing the abnormal events by the large model, and optimizing the large language model configuration strategy and data conversion rules in combination with feedback data.

A system and method guide an Artificial Intelligence engine to determine the semantic and grammatical correctness of a user-expanded sentence in real-time. The sentence validation process involves receiving input from the user, the input includes sentence fragment that the user wishes to expand and user-expanded sentence that the user constructs on the fragment provided. The inputs are broken down into tokens. The word-level tokenization algorithm is used, which identifies tokens by splitting the text into spaces, punctuation marks, and other delimiters. Further, a token comparison algorithm is used to assess the relationship between the sentence fragment and the user-expanded sentence to analyze order and placement. Once the token comparison is complete, a prompt is generated using prompt generator to evaluate grammatical and semantic evaluation of the user-expanded sentence. Real-time feedback is provided to the user based on grammatical and semantic evaluation.

This invention discloses a method and apparatus for isolating agent ontology pollution based on memory layering and tokenized access control, belonging to the field of large language model agent technology. This invention establishes a three-layer logically isolated architecture within the MCP service cluster: a dialogue text layer, an ontology structure layer, and an execution protocol layer. It employs a differentiated update strategy of cumulative appending, complete replacement in each round, and constant updates. Each round generates a unique token strongly bound to the current round's ontology subgraph's TTL, atomically invalidating historical tokens and completing atomic switching of the ontology version. The business service entry point performs triple mandatory checks on existence, state validity, and version currentity in a fixed order through a token verification interceptor. The agent is configured in a zero-memory, stateless mode, with context and ontology data obtained through external services. Simultaneously, dynamic memory compression and summary condensation are implemented in the dialogue text layer.

This invention provides a code sequence compression method, apparatus, and device that combines semantic analysis and attention, relating to the field of large language model processing technology. The method includes: performing lexical and syntactic analysis on a source code sequence to generate a syntax tree corresponding to the source code sequence; generating an original syntax topology distribution network based on the dependencies between nodes in the syntax tree; determining the comprehensive maintenance value of each node in the original syntax topology distribution network; and removing redundant nodes from the original syntax topology distribution network based on a comprehensive maintenance value threshold and the comprehensive maintenance value of each node in the original syntax topology distribution network, resulting in an updated syntax topology distribution network and a compressed code sequence. The method of this application avoids the problems of traditional compression methods such as accidentally deleting key nodes, disrupting program connectivity, and causing semantic understanding bias in the model, providing high-quality input with standardized structure, semantic coherence, and reliable reasoning for large language models.

The application discloses a knowledge base and multi-agent collaborative academic review automatic generation method, system, device and storage medium, relates to the field of artificial intelligence and natural language processing technology, and comprises the following steps: adopting a mixed extraction mechanism to extract an agent to original unstructured literature, and constructing a bottom-layer JSON knowledge base containing technical indexes and quantitative data; mapping the knowledge base information to a preset target hierarchical structure through a planning agent, and generating a global review outline; adopting a block retrieval enhancement strategy, a writing agent carries out chapter text synthesis and graph generation under the constraint of forced binding quantitative evidence; comparing the source text and the generated text through entity lexical analysis, and performing quality evaluation.The application adopts the above-mentioned knowledge base and multi-agent collaborative academic review automatic generation method, system, device and storage medium, effectively improves the local fact accuracy and global logical coherence of the generated literature, and realizes automatic and high-fidelity compilation of the academic review.

Disclosed in the present application are a robot control language generation method, system and apparatus, and a storage medium. The method comprises: on the basis of a user task description and system information, constructing target prompt information, and inputting the target prompt information into a large language model, so as to obtain a robot skill language; performing lexical analysis, syntax analysis and a semantic check on the robot skill language by means of a robot skill language compiler, so as to determine whether a robot skill language rule is met; when the robot skill language rule is not met, generating error feedback information by means of a robot skill language debugger, updating the target prompt information on the basis of the error feedback information, and returning to the step of inputting the target prompt information into a large language model; and when the robot skill language rule is met, compiling the robot skill language by means of the robot skill language compiler, so as to generate a robot control language. The present application improves the accuracy and executability of a robot control language, and can be applied to the technical field of robot control.

The application provides a system and method for parsing PLC intermediate code based on compilation technology and automatically generating data bridge description. The system comprises: a preprocessing module for normalizing the C intermediate code generated by matIEC; a lexical analysis module for converting the code into a Token stream; a syntax analysis module for constructing an abstract syntax tree; an abstract syntax tree processing module for traversing the AST to extract type definitions, fields and memory layouts, and converting them into an internal data model; a code generation module for serializing the data model into a standardized JSON file, including data item name, type, size, offset, memory type and nested sub-items; a version management module for automatically updating the compiler version number and embedding the JSON based on CMake during building. The application realizes full-automatic extraction and standardized output of PLC data structure, eliminates manual parsing, ensures accurate memory layout, supports cross-platform integration and version consistency verification.

The application provides a three-mode redundant circuit netlist compiling method and device based on state machine conversion, belongs to the technical field of anti-radiation integrated circuit netlist compiling processing, and comprises the following steps: generating a netlist ASCII text stream and a keyword-regular set by inputting core source data suitable for an anti-radiation scene, triggering a lexical analyzer, forming an ordered lexical unit sequence through lexical analysis, constructing a state transition graph and checking the state transition graph, integrating the graph and a fragment, extracting a syntax tree, converting the syntax tree into a flow graph data structure, finally generating an IR intermediate representation mapped with SoC hardware, and outputting the IR intermediate representation. The three-mode redundant circuit netlist compiling method and device based on state machine conversion are used, so that the problems of low recognition efficiency, large cost and easy deletion of redundant logic in the process of processing a three-mode redundant circuit netlist by a traditional hardware compiler are solved.

Subword tokenization is provided. The method comprises receiving a text document comprising n bytes and specifying a maximum token width of L bytes. An initial vocabulary of tokens is defined, wherein the tokens comprise a number of different n-grams of L or less bytes. The document is tokenized with the fewest number of tokens from the vocabulary according to a minimum total weight through a directed acyclic graph comprising nodes that represent intervals between the bytes in the document and edges that represent potential tokens from the vocabulary appearing in text of the document. Natural language processing is then performed on the text document according to the tokenization.

The application provides a model training method, medium, equipment and program product of a transverse mixed attention mechanism. The method comprises: acquiring a data set containing multiple sample sequences, each sample sequence in the data set being composed of multiple Tokens arranged in sequence after tokenization processing; constructing a to-be-trained model based on a pre-trained full attention model and adding new parameters for linear attention calculation; in the same transverse mixed attention layer, performing full attention calculation on a Token set located within a preset full attention calculation range, performing linear attention calculation on all Tokens, and fusing the results of the two to obtain transverse mixed attention output for forward reasoning and loss calculation; based on the output and a prediction result, only updating the new parameters to optimize the to-be-trained model until the to-be-trained model converges. The application reduces the computational complexity and memory occupation of long text sequence processing, and improves the reasoning speed and resource utilization rate.

The proposed utility model belongs to the field of computer technology and is a standalone hardware device for automatically detecting referential relationships in natural language text data. The utility model is a hardware device for determining referential relationships in text data, implemented in the form of a hardware hub and located in a housing, including a power control module, a data input / output interface module, a processor, RAM, non-volatile memory, as well as functional processing units. The feature of the utility model is that text data processing operations are carried out not as a software method, but in a separate hardware hub, and that the device provides for the complex automatic detection of referential relationships in text data through the presence of a tokenization and morphological analysis block, a reference detection and conflict resolution block, a result generation and user interface block. The device receives text data from external systems, processes it in the tokenization and morphological analysis block, and then forms referent-antecedent relationships through the reference detection and conflict resolution block. The processing results are provided to the user or external systems through the output generation and user interface block. The technical result of the utility model is to increase the productivity of the device by reducing the time for processing text data and reducing the latency when processing data streams. The proposed device is intended for use in electronic document management systems, information and analytical platforms, educational, legal and corporate information systems.

This invention discloses an online grammar synthesis-assisted symbolic execution method and apparatus. The method includes the following steps: Step S1. Performing character-level symbolic execution on the lexical analysis code of the test program to extract a token digest; Step S2. The symbolic execution engine receives the initial input or candidate input generated from the grammar, performs token-level symbolization, collects token-level path constraints, and continues to explore uncovered path branches using a grammar-oriented search strategy after the initial input or candidate input generated from the grammar has been explored; Step S3. When a valid input that cannot be received by the original grammar is found, grammar synthesis is performed and the input grammar of the test program is updated; Step S4. Samples and generates multiple new candidate inputs from the currently updated input grammar and feeds them back to the symbolic execution engine. This invention can improve the effectiveness and efficiency of symbolic execution of programs with complex input formats.

The application discloses a source code processing method, computer equipment and program product, and relates to the technical field of computer program code analysis, and the method is characterized in that: an abstract syntax tree is converted into a hybrid code tree that integrates data semantics and control logic, unified standard node basic attributes and semantic information of data nodes and control nodes that inherit a unified base class are relied on, multiple independent intermediate representations are avoided to be built, and operation overhead in a code analysis process is reduced; the original parent-child hierarchical relationship of the abstract syntax tree is completely retained in a conversion process by using the hybrid code tree, sibling nodes in the same layer are arranged according to a source code lexical analysis sequence, and syntax, data and control semantic information are integrated into a hierarchical clear tree structure by combining the design of the data nodes and the control nodes in the hybrid code tree, without additionally adding jump edges, control flow edges and data flow edges, so that the query overhead of the analysis process and the system maintenance complexity are reduced, and the method is suitable for various source code analysis application scenarios such as vulnerability detection and compliance audit.

Methods and systems for improving retrieval accuracy in a Retrieval Augmented Generation (RAG) framework are disclosed. Input documents (102) are analyzed using a Small Language Model (SLM) to identify document types and extract content using type-specific methods. An adaptive chunking process generates chunked content based on variables including use case type, speed, cost, latency requirements, and Large Language Model (LLM) context window size. The chunked content undergoes tokenization using strategies based on accuracy and vocabulary requirements. An embedding model is selected based on user requirements, and quantization policies are applied to generate optimized vector representations. Document portions are retrieved using a multi-step strategy comprising hybrid search, meta filters, multi-query retrieval, and hypothetical document embeddings. Performance is evaluated through context relevance, groundedness, and answer relevance scores, implementing parameter adjustments and re-evaluation processes when scores fall below thresholds.

Introduced here are computer programs and associated computer-implemented techniques for discovering the presence of filler words through tokenization of a transcript derived from audio content. When audio content is obtained by a media production platform, the audio content can be converted into text content as part of a speech-to-text operation. The text content can then be tokenized and labeled using a Natural Language Processing (NLP) library. Tokenizing / labeling may be performed in accordance with a series of rules associated with filler words. At a high level, these rules may examine the text content (and associated tokens / labels) to determine whether patterns, relationships, verbatim, and context indicate that a term is a filler word. Any filler words that are discovered in the text content can be identified as such so that appropriate action(s) can be taken.

The application provides a vulnerability type identification method based on information retrieval and CWE knowledge graph, and belongs to the technical field of software engineering. The technical problems of low accuracy of traditional vulnerability type identification methods are solved. The technical scheme is as follows: the method comprises the following steps: (1) collecting the codes, descriptions and corresponding CWE types of authoritative vulnerability sources, and processing them as local instance libraries; (2) calculating the similarity of the codes and descriptions of the target vulnerability respectively, adopting a hybrid reordering strategy based on AST syntax and Token lexical analysis to optimize the code retrieval result, and obtaining the top-k most similar instance samples according to a weighted fusion strategy; (3) designing a chain thinking prompt template. The application can accurately predict the CWE type to which the software vulnerability belongs.

The application discloses a kind of general data exchange systems based on configurable label structure, it is related to the technical field of data processing and information integration, including parameter module for establishing the identity authentication mechanism of exchange party;Business document module is used to establish data structure definition based on preset exchange demand;Document logic module is used for logical analysis and code decoupling to data structure definition, generates tokenization language script code and execution instruction, constructs execution matrix, realizes the mapping of database physical field and memory logic variable;Standard adaptation module is used to compare and check the generated data content, and the verification result is fed back to the parameter module to correct the exchange rule and parameter configuration.The application implements more intelligent logic separation strategy through structured insight, cooperatively controls data reading, writing and transmission, and realizes complete decoupling of business logic and program code using nested label structure, optimizes the flexibility, universality and maintenance efficiency of data exchange.

This invention provides an automatic UML sequence diagram generation method compatible with multilingual code analysis. By capturing the characteristics of C++ and Java code, it identifies the code types of each part of the mixed source code and analyzes different code types. For code marked as C++, regular expressions are used to match keywords, class information, and inheritance information of the C++ source code to be analyzed, and these are converted into parameter names. For code marked as Java, syntax and lexical analysis are used to extract class information and method call information of the Java source code to be analyzed. Corresponding Plant UML code is generated and drawn as a comprehensive UML sequence diagram. This invention's automatic UML sequence diagram generation method compatible with multilingual code analysis can intuitively reflect the functional and logical composition of mixed C++ and Java code, and intuitively reflect the interaction relationship between them. It avoids the need to view tedious source code to obtain program information and functional relationships, improving code readability and maintainability.

A textbook text question and answer method and system based on multi-level attention, the method comprising: inputting the question text and the corresponding chapter context paragraph into the first attention model after tokenization and coding, performing self-attention calculation and pooling within the sentence to obtain the sentence representation vector; calculating the cosine similarity of the question sentence representation vector and the representation vector of all context paragraph sentences, retaining only the representation vector corresponding to the sentence with the maximum similarity of each paragraph as the context representation vector of each paragraph corresponding to the question; inputting the question and the context representation vector of each paragraph corresponding to the question into the second attention model, performing self-attention calculation and pooling between the paragraphs and the question to obtain the answer representation vector corresponding to the question as the input of the classifier; outputting the answer options by the classifier, and obtaining the context paragraph of the answer text corresponding to the chapter. The application can select more accurate answer options for textbook text questions.

The present disclosure relates to a commodity search method and device, electronic equipment and storage medium, and relates to the technical field of electronic commerce. The method comprises the following steps: obtaining a commodity search request of a user; performing lexical analysis on the commodity search request to obtain a search word; searching in a first search system according to the search word to obtain a first search result; returning the first search result to the user to search in a second search system according to the selection of the user on the first search result. The present disclosure can accurately search for commodities.

A system and a method for unstructured synthetic data generation is disclosed. The system a processor, and memory with instructions to: The system includes a processor. A memory coupled to the processor comprises instructions causing reception of real-time input data from sources, the input comprising textual / non- textual data; analyse structure to select encoding per modality; encode textual data via tokenization, embeddings or character encoding and non-textual data via normalization, one-hot or scaling to produce encoded data; segregate encoded textual, numerical and categorical portions into groups; train and fine-tune transformer on the encoded data using metrics; generate conditional textual data from contextual cues; combine modalities by aligning formats to produce unified dataset; process the unified dataset to preserve textual semantics and statistical compatibility; and output the unified dataset as a fusion retaining semantic integrity and statistical fidelity.

This invention discloses a method for dynamic language syntax extension and type description based on Prefer blocks. The method includes: performing lexical and syntactic analysis on the dynamic language source code containing Prefer blocks to construct an abstract syntax tree (AST) containing Prefer nodes; parsing the Prefer nodes in the AST to extract type constraint information declared in the Prefer blocks; associating the type constraint information with the corresponding function and variable nodes in the AST to obtain an associated syntax tree; and performing type checking and code generation on the dynamic language source code based on the associated syntax tree to obtain a bytecode file. By setting non-intrusive Prefer blocks and describing type constraints using syntax classes, type information is introduced into the dynamic language without modifying the core syntax or intruding on business logic code, thus solving the problems of tight coupling, poor extensibility, and strong intrusion between the type description mechanism and the core syntax.

A system for magnitude-invariant image-text agentic interface automation is disclosed. A bit vectorization logic is configured to convert image patches in a plurality of image patches into magnitude-invariant bit vectors, and generate a plurality of lines of magnitude-invariant bit vectors. A tokenization logic is configured to translate the input text sequence into a sequence of input text tokens, and to translate the successive lines of magnitude-invariant bit vectors interleaved with a newline character into a sequence of input magnitude-invariant bit vector tokens. A linear projection logic is configured to linearly project a single token stream of the sequence of input text tokens and the sequence of input magnitude-invariant bit vector tokens into a decoder-only Transformer logic, wherein the linear projection of the single token stream bypasses any embedding lookup.

This invention discloses a method for extending the syntax of a programming language. The method includes: receiving source code containing lexical annotations; the lexical annotations are defined by annotation definition statements; performing lexical and syntactic analysis on the source code using a preset programming language to generate an abstract syntax tree (AST); identifying the type of annotation nodes during the compilation preparatory stage; processing the AST based on the type of annotation nodes; and compiling the processed AST into a bytecode file using a bytecode compiler, so that the bytecode file can be run through a virtual machine. Through the technical solution of this invention, annotation definition statements are introduced to define the names and scope of lexical annotations in a standardized manner, integrating lexical annotations into the entire compilation process, achieving effective syntax extension; annotation processing is completed during the compilation stage, and the generated bytecode has no additional overhead at runtime, balancing development efficiency and execution performance.

The application discloses an underwater sound source positioning method and system based on underwater acoustic-positioning text generation alignment, which acquires multi-channel underwater acoustic signals and corresponding structured positioning text labels, and performs Tokenization processing on the positioning text labels; an underwater acoustic multi-modal converter model with an encoder-decoder structure is constructed, a generative alignment of underwater positioning features and decoding end text features is realized through a cross attention mechanism, and a positioning text sequence containing depth and distance information is generated in a self-recurrence manner. During training, the model is used as a student model, an exponential moving average copy thereof is constructed as a teacher model, a loss is calculated based on clean input and noise disturbance input, and distance numerical Token and depth numerical Token are subjected to a weight higher than that of ordinary structure Token. During reasoning, the generated structured positioning text sequence is parsed to obtain underwater sound source depth values and distance values, so that the positioning result is determined.

The tokenization process of input text and visuals (e.g., images, videos, or frames of videos) can be separated into two stages. In a first stage, a large batch size can be used for text encoding and visual encoding while focusing on the first objective of an alignment loss and mean square loss objectives. In a second stage, the text encoder can be stopped, and the visual encoder can be prevented from making additional changes. The second stage focuses on a second loss objective of a weighted sum of the mean square loss, the perceptual loss, and the generative adversarial network loss objectives. In the second stage, a discrete set of tokens can be generated from the inputs, and the set of tokens can be further fine-tuned. A transformer, with an autoregressive model, can be applied to the set of discrete tokens.

The application discloses a code snippet summary generation method fusing data dependency and control structure information, and belongs to the technical field of software engineering and natural language processing. A corresponding token sequence is obtained through lexical analysis; a data dependency structure is constructed in combination with the definition and use relationship of variables in a program, and control structure semantic labels are assigned to each token according to the code syntax structure. The code semantic representation and the control structure embedded vector are fused and processed, and are input into a Transformer-based encoder for structure-enhanced coding. In the coding process, a data dependency adjacency matrix is introduced, so that the model can pay more attention to code elements with a dependency relationship when calculating attention weights; then, a natural language summary corresponding to the code snippet semantics is generated through a decoder. The application can improve the understanding ability of the model for code semantics and execution logic, and further generate a code summary with higher semantic consistency and a more complete structure.