62 results about "Language modelling" patented technology

A novel method provides an AI language virtual agent having self-improvement features and which uses language modeling and tree search techniques. The AI language virtual agent exchanges textual discussion with users and other simulated agents. The method includes receiving a current situational description depicting natural language user input, temperament qualities and textual tendencies of the virtual agent, and indicia regarding subject matter context of a present conversation. The indicia regarding subject matter context include textual logs from recent conversational exchanges. The current situational description includes audio, visual, and tactile inputs collected proximate to the virtual agent. The method preferably utilizes an MCTS tree search in combination with self-moving modules, one or more language models, tree search techniques outputting textual responses to the current situation description, and the virtual agent responding with textual expression to verbal input in combination with the audio, visual, tactile, and other sensory inputs.

Systems and methods for attribute recognition can include obtaining an image and a text string. The text string can be processed with a language model to generate a set of candidate attributes based on sequence based prediction. The image and the candidate attributes can be processed with an image-text model to determine a likelihood that the respective candidate attribute is depicted in the image. The likelihood determination can then be utilized to determine a predicted attribute for the object of interest.

Embodiments described herein provide a training framework for generative NLP models that operate on previously learnt knowledge from pretrained large language models. Specifically, to train an NLP model to generate a response to a user utterance (e.g., “resolve login issue”), document embeddings of support IT documents encoded by a pretrained LLM are fed to an NLP decoder together with a training dialogue (e.g., a dialogue between the chat agent on how to “resolve login issue”). The NLP decoder can thus be trained by a causal language modeling loss computed based on the predicted next token and the ground-truth token from the training dialogue.

In an embodiment providing natural language processing (NLP), a computer generates a histogram that correctly represents a graph that represents a lexical text, and generates a token sequence encoder that is trainable and untrained. During training such as pretraining, the token sequence encoder infers an encoded sequence that incorrectly represents the lexical text, and the encoded sequence is dense and saves space. To increase the accuracy of the sequence encoder by learning, the token sequence encoder is adjusted based on, as discussed herein, an indirectly measured numeric difference between the encoded sequence that incorrectly represents the lexical text and the histogram that correctly represents the graph.

The invention discloses a professional noun synonym replacement assisting method and system based on deep learning. The method comprises the steps of knowledge fusion pre-training, dynamic context sensing modeling, domain adaptation optimization, multi-dimensional output optimization and professional noun synonym replacement assisting. The invention relates to the technical field of natural language text generation, and adopts multi-modal learning and knowledge fusion technologies in a deep learning technology to construct a dual-channel model, cross-modal alignment operation and domain knowledge extraction in a knowledge fusion model pre-training process. The language modeling accuracy and the context information processing capability of the basic model are optimized through the combination of the three, and the targeted optimization of professional noun synonym replacement assistance and the availability of the whole scheme are realized; performing dynamic context sensing modeling by adopting a generative adversarial verification network combining improved attention and dynamic context coding; a lightweight model adaptation improvement method combined with domain classification is adopted, and a domain adaptation effect is optimized through adaptive adjustment and an elastic weight mechanism.

Methods, systems, and computer programs are presented for abstractive summarization of text by viewing sequence transduction as a language modeling problem. One method comprises an operation for training a machine-learning program to create a machine-learning model that estimates a word to be added to a running summary for the text being summarized. The method further comprises operations for detecting the text to be summarized, initializing the running summary, and performing a plurality of iterations. Each iteration comprises providing, to the machine-learning model, the source text and the running summary, and adding, using the machine-learning model, a new word to the running summary. Further, the method comprises an operation for storing, on a memory, the running summary as the summary of the text.

The model generation method comprises the following steps: acquiring a natural language description of an optimization problem; according to the natural language description, a mathematical model of the optimization problem is obtained, and the mathematical model is expressed through DSL. When modeling of the optimization problem is carried out, the model of the optimization problem is constructed through natural language description of the optimization problem, the model is expressed through the DSL, and the DSL has the advantages that DSL description information is similar to a high-level language, the modeling threshold is low, the marking cost of a training sample is low, and the modeling efficiency is improved. Therefore, a high-precision conversion model for constructing the DSL model based on the natural language description of the optimization problem can be obtained through training at a low cost, and the DSL model is well understood by a user, so that the user can modify the DSL model in the reasoning process to obtain the more accurate DSL model which better meets the own requirements, and the user experience is improved. The reliability and the accuracy of a model generation result can be ensured under the condition that the labor cost and the user requirements are greatly reduced.

The invention provides a method and system for pre-training a sign language understanding framework based on semantic enhancement of skeletons, and relates to the technical field of sign language recognizing.The method comprises the steps that sign language video data and skeleton sequences and text data matched with the sign language video data are obtained; extracting a skeleton key point sequence from the sign language video data, modeling to form skeleton features, and performing word segmentation processing on a text; in a pre-training stage, skeleton features and word segmentation texts are input into a fusion network to generate bidirectional enhanced features, and global and local similarities are obtained through double-layer semantic alignment; calculating the comparison loss based on the similarity, and coordinating the weight through the balance parameter to obtain the hierarchical loss; executing the matching task and the language modeling task to obtain corresponding loss, weighting and combining the three types of loss into pre-training total loss, and adjusting parameters to complete training; and finally, part of parameters are optimized in combination with specific task types in a fine tuning stage, and enhanced understanding of sign language semantics is realized. The sign language understanding accuracy is improved.

The invention relates to the technical field of model-driven system engineering modeling, in particular to an MBSE-oriented multi-agent collaborative automatic modeling method and system. The method comprises the following steps: acquiring a natural language modeling demand of a user, retrieving in a shared knowledge environment to obtain a knowledge sub-graph related to the demand, constructing a modeling context, and calling a generation agent to generate a current SysML v2 model; calling a verification agent to perform grammar verification and semantic verification based on the domain knowledge graph on the model, and outputting a verification result; when a verification result represents that grammar does not pass or semantic inconsistency exists or an engineering constraint rule is violated, calling a repair agent to execute deterministic repair and iterative verification according to the domain knowledge graph until a preset condition is met, and outputting a final SysML v2 model; notch marking or pocket bottom repairing can be carried out under the condition that the notch is covered. According to the method, the correctness, interpretability and convergence efficiency of complex system modeling are improved.

The present application provides a data processing method and apparatus. The method comprises: acquiring a natural language processing (NLP) training data set; inputting the NLP training data set into a first large language model to train the first large language model, the trained first large language model comprising a first language modeling parameter; acquiring the first language modeling parameter; obtaining first text data on the basis of the NLP training data set; and obtaining a first prompt word set on the basis of the first text data, the first language modeling parameter, and a second large language model, wherein the first prompt word set and the first text data are used for training a third large language model, and the first large language model and the second large language model are based on the same model architecture. The method provided by the present application can obtain the first prompt word set of which the data volume is much smaller than that of the NLP training data set. In this way, using the first prompt word set to train the third large language model can improve the speed and accuracy of large language model training.

The invention relates to the technical field of natural language processing, and discloses a Yi language speech recognition method based on self-supervision and attention feature fusion. The method comprises a feature encoder module, a comparative learning module, a mask language modeling module, a joint optimization and feature fusion module and a decoder module, the feature encoder module adopts a convolutional neural network structure and converts continuous waveform signals into feature representation suitable for subsequent modeling, and the comparative learning module performs feature fusion on the continuous waveform signals through a Gumbel-Softmax technology. The method comprises the following steps that: a mask language modeling module and a feature fusion module are integrated, deviation caused by manual definition or clustering is avoided, the mask language modeling module obviously enhances semantic understanding and tone modeling capabilities of a model in a low-resource scene, a self-attention feature fusion mechanism is introduced into the feature fusion module, continuous features, discrete unit representation and semantic context representation from an acoustic level are spliced, and a self-attention feature fusion mechanism is introduced into the self-attention feature fusion mechanism. The decoder module adopts a decoder structure based on connection time sequence classification, and the corresponding relation between the voice and the text can be achieved without strict alignment labeling.

Techniques for dynamically developing a contextual set of prompts based on relevant aspects extracted from s set of training data. One technique includes obtaining training data comprising text examples and associated labels, extracting aspects from the training data, generating prompting templates based on the training data and the extracted aspects, concatenating each of the text examples with the respective generated prompting template to create prompting functions, training a machine learning language model on the prompting functions to predict a solution for a task, where the training is formulated as a masked language modeling problem with blanks of the prompting templates being set as text labels and expected output for the task being set as specified solution labels, and the training learns or updates model parameters of the machine learning language model for performing the task. The machine learning language model is provided with the learned or updated model parameters.

The invention provides a hierarchical auxiliary sparse attention method for a long text large language model. The method comprises the following steps: S1, adding a parameter sharing offset branch in each large language model layer to obtain a new large language model; s2, segmenting the context into a plurality of segments, inputting the segments into a large language model layer to obtain local feature output, pooling each segment, and inputting the segments into offset branches of the same layer to obtain low-resolution representation; s3, splicing the low-resolution representation output by the offset branch of the previous layer to the local feature of the large language model layer of the current layer, and outputting the low-resolution representation to the large language model layer of the next layer; and S4, performing fine adjustment on the new large language model, and accessing a language modeling header behind the last large language model layer to output a processing result of a downstream task. By adopting the method disclosed by the invention, the reasoning efficiency can be effectively improved, the model performance is not reduced, and the possible obvious delay when the first lexical element is generated is reduced.

A machine learning legal natural language generation computing system for developing a pre-trained and instruction fine-tuned MLLNLG generative language model that aligns with human instructions and preferences using causal language modeling for next-token prediction. The training method employs a scaled version of RoPE to compress token position indices, to manage longer sequences beyond the computing hardware devices (GPU's) DRAM memory capacity. This system is optimized for various legal NLP and NLG tasks by receiving input tasks from users through an I / O component. The system enables efficient pre-training of a generative legal language model from scratch on a single computing hardware device (GPU), achieving an efficient MFU of 41.35 and through weight tying, the system reduces model parameters compared to LLMs, eliminating the need for multiple GPUs. Tailored for the legal domain, the model excels in processing legal jargon and demonstrates advanced reasoning capabilities, outperforming existing models in summarization tasks.

Techniques are disclosed herein for focused training of language models and end-to-end hypertuning of the framework. In one aspect, a method is provided that includes obtaining a machine learning model pre-trained for language modeling, and post-training the machine learning model for various tasks to generate a focused machine learning model. The post-training includes: (i) training the machine learning model on an unlabeled set of training data pertaining to a task that the machine learning model was pre-trained for as part of the language modeling, and the unlabeled set of training data is obtained with respect to a target domain, a target task, or a target language, and (ii) training the machine learning model on a labeled set of training data that pertains to another task that is an auxiliary task related to a downstream task to be performed using the machine learning model or output from the machine learning model.

Embodiments described herein provide a training framework for generative NLP models that operate on previously learnt knowledge from pretrained large language models. Specifically, to train an NLP model to generate a response to a user utterance (e.g., “resolve login issue”), document embeddings of support IT documents encoded by a pretrained LLM are fed to an NLP decoder together with a training dialogue (e.g., a dialogue between the chat agent on how to “resolve login issue”). The NLP decoder can thus be trained by a causal language modeling loss computed based on the predicted next token and the ground-truth token from the training dialogue.

A computer-implemented method may use a neural network-based language model for generating a 3D structure of a molecule conditioned by atomic connectivity or a binding of the molecule with a target protein pocket. The language model may be trained with the large-scale training data set to obtain a pre-trained language model, wherein the pre-trained language model may be trained to generate the 3D structures of molecules and / or protein pockets. The pre-trained language model may be trained with the finetuning training data to obtain a finetuned language model capable of at least one of: generating 3D structures of molecules capable of binding to the target protein pockets or generating 3D structures of molecules bound in the target protein pockets or generating 3D structures of the target protein pockets. The trained language model capable of generating the 3D structure of a generated molecule that binds with the protein pocket.

Example solutions for zero-shot domain transfer with a text-to-text model train a text-to-text model for a target domain using unlabeled in-domain text training data, and concurrently train the model using labeled general-domain task training data. The in-domain training comprises masked language modeling (MLM) training, and the task training comprises both natural language generation (NLG) training and natural language understanding (NLU) training. The NLG training comprises natural language inference (NLI) training and the NLU training comprises summarization training. The trained model acquires domain-specific task competency, sufficient to perform a language task within the target domain. Suitable target domains include radiology, biomedical, and other medical, legal, and scientific domains. This approach leverages large volumes of general-domain task training data and plentiful unlabeled in-domain text, even as labeled in-domain training data may be unavailable or prohibitively expensive for certain specialized domains.

Methods, systems, and computer programs are presented for abstractive summarization of text by viewing sequence transduction as a language modeling problem. One method comprises an operation for training a machine-learning program to create a machine-learning model that estimates a word to be added to a running summary for the text being summarized. The method further comprises operations for detecting the text to be summarized, initializing the running summary, and performing a plurality of iterations. Each iteration comprises providing, to the machine-learning model, the source text and the running summary, and adding, using the machine-learning model, a new word to the running summary. Further, the method comprises an operation for storing, on a memory, the running summary as the summary of the text.

The invention discloses a multi-modal large language model training method and device, equipment and a medium, which are applied to financial / medical scenes, and are characterized in that a target multi-modal data training set is obtained by preprocessing an obtained multi-modal data training set containing sample images and corresponding sample texts; according to the target multi-modal data training set, a visual encoder and a large language model are connected in series to obtain a first training structure, loss calculation is carried out on the first training structure, and language modeling loss is obtained; and performing loss calculation on a second training structure obtained by connecting the visual representation alignment middle layer, the visual basic model, the visual-language projection module and the alignment loss module in series according to the target multi-modal data training set to obtain alignment loss, and training a multi-modal large language model based on language modeling loss and the alignment loss to obtain the multi-modal large language model. And the target multi-modal large language model is obtained until the model training ending condition is reached, so that the alignment effect of the multi-modal information is improved, and the understanding ability of the model for the image visual information is improved.

Systems and methods are disclosed for predicting a next text. A method may include receiving one or more documents, such as a document associated with a healthcare provider. The document is then processed to generate one or more tokens which are representative of the document. The document is then processed with a machine-learning model, such as a topic model, and a topic vector is output for the document. Based at least in a part on this topic vector, the document is then processed by one or more expert machine-learning models, which each output a probability vector. The various probability vectors are then further processed to calculate a total probability vector for the document. Based at least in part on the total probability vector for the document, a text output is selected.

The present disclosure provides a training method for a pre-trained language model, a training method for a language model and a device, comprising: obtaining a sample text, performing at least two of the pre-training tasks according to the sample text, and obtaining a pre-trained language model, wherein the pre-training tasks include: a forward causal language modeling task, a reverse causal language modeling task and a masked language modeling task, the forward causal language modeling task is a modeling task of predicting a subsequent word in the sample text by a preceding word in the sample text, the reverse causal language modeling task is a modeling task of predicting a preceding word in the sample text by a subsequent word in the sample text, and the masked language modeling task is a modeling task of predicting a word in a masked position in the sample text by a word in a non-masked position in the sample text, and pre-training can be achieved from multiple dimensions, thereby achieving diversity and flexibility in training, and the pre-trained language model obtained by training from multiple dimensions can have higher accuracy and reliability.

A computer-implemented method may use a neural network-based language model for generating a 3D structure of a molecule conditioned by atomic connectivity or a binding of the molecule with a target protein pocket. The language model may be trained with the large-scale training data set to obtain a pre-trained language model, wherein the pre-trained language model may be trained to generate the 3D structures of molecules and / or protein pockets. The pre-trained language model may be trained with the finetuning training data to obtain a finetuned language model capable of at least one of: generating 3D structures of molecules capable of binding to the target protein pockets or generating 3D structures of molecules bound in the target protein pockets or generating 3D structures of the target protein pockets. The trained language model capable of generating the 3D structure of a generated molecule that binds with the protein pocket.

A code completion tool uses a neural transformer model to generate candidate sequences to complete a line of source code. The neural transformer model is trained using a conditional language modeling objective on a large unsupervised dataset that includes source code programs written in several different programming languages. The neural transformer model is used within a beam search that predicts the most likely candidate sequences for a code snippet under development.

Methods, apparatuses, or computer program products that process service message data objects via large language modeling and classification machine learning to provide service message classifications. In some examples, a large language model is applied to a plurality of service message data objects associated with an application framework to generate a feature set for the plurality of service message data objects, a classification machine learning model is applied to the feature set to generate a plurality of classification data objects associated with the plurality of service message data objects that classify a respective service message data object as belonging to a predefined class of a plurality of predefined classes, and a rendering of a dashboard visualization is initiated via an electronic interface based at least in part on the plurality of classification data objects.

The invention relates to an mRNA (messenger Ribonucleic Acid) codon sequence language modeling method based on a bidirectional Mama framework, which comprises the following steps of: acquiring an mRNA coding sequence, performing sequence tokenization on the mRNA coding sequence, and querying by adopting a word list to obtain an embedding characteristic; the embedded features are sequentially input into a convolution feature extraction module and a state space module of a bidirectional Mamba model to generate collaborative representation, and meanwhile, each time step serves as an affine pair and is used for carrying out inter-block synthesis on block-level pairs to obtain a synthesis result; and inputting the collaborative representation and synthesis result into a bidirectional branch and linear fusion module to obtain a linear transformation fusion result, projecting the linear transformation fusion result to a word list dimension, obtaining a cross entropy and a model parameter used for training a bidirectional Mama model, and generating a target dimension vector of each sequence by using the trained bidirectional Mama model. According to the invention, the accuracy and expandability of mRNA sequence analysis and design can be improved.

Introduced here is an interactive digital health system (or simply “interactive system”) that is able to update a knowledge base used to facilitate semi-automated conversations to account for feedback. The interactive system can include a collection of modules that, in operation, can provide or support coaching services in the digital health domain, Upon receiving input indicative of a question from a patient, these modules allow the interactive system to perform language modeling to identify content objects in a knowledge base that are relevant to the question. Moreover, the interactive system may update the knowledge base to account for feedback, thereby making the knowledge base more robust to a broad range of questions.

An online system trains a machine-learning model to generate an embedding in real time for a current session of a user with the online system. The machine-learning model is trained by applying a masked language modeling algorithm to training data including a training sequence of actions and a masked action to predict a user’s action that follows the training sequence of actions. The online system captures current session data describing a sequence of actions of the user performed during the current session. The online system applies the trained machine-learning model to predict a next user’s action and generate a session embedding that encodes information about the sequence of actions and the next action. Using the session embedding, the online system ranks a list of objects. The online system generates a user interface signal causing a user’s device to display a user interface with the ranked list of objects.

The present invention provides a method for Latin Zhuang language vector representation and calculation, including a representation and calculation model composed of a Latin Zhuang language preprocessing module, a Latin Zhuang language language modeling module, a Latin Zhuang language vector representation normalization module, and a Latin Zhuang language vector representation and calculation module connected in sequence. After a text written in Latin Zhuang language is processed by this representation and calculation model, the vector representation of the Latin Zhuang language text can finally be obtained. The method of the present invention solves the problems of vector representation and calculation of Latin Zhuang language and can be used for the automated processing and application of Latin Zhuang language.

The invention discloses an electric power operation personal risk static assessment method, system, equipment and medium based on comparative learning and meta-learning prompts.The electric power operation personal risk static assessment method and system based on comparative learning and meta-learning prompts.The electric power operation personal risk static assessment method and system based on comparative learning and meta-learning prompts.The electric power operation personal risk static assessment method and system based on comparative learning and meta-learning prompts.The electric power operation personal risk static assessment method comprises the steps that deep semantic alignment of text and numerical bimodal features is achieved through comparative Introducing a meta-learning mechanism to enable the deep learning model to obtain cross-task generalization ability in a small sample scene; the classification task is reconstructed into a language modeling task through prompt learning, and end-to-end joint optimization is carried out by means of learnable prompt vectors and features. According to the method, an improved comparative learning framework is adopted, and a loss function of cross-modal attention and innovation is utilized, so that deep semantic alignment of text and numerical features in a unified semantic space is realized, and the problem of a shallow layer of traditional multi-modal fusion is solved.