Dynamic token loading strategy for large scale embedding learning

Abstract

During training of a language model including samples of outcomes of requests determined by previous input data, a computer system receives a training set of training samples. Each training sample represented by a set of tokens and including an outcome of a request. The computer system determines a batch of training samples from the training set of training samples and determines a set of distinct tokens from the batch of training samples. The computer system determines state-related tokens that correspond to possible outcomes of requests. The computer system generates a token embedding batch from token embeddings of the set of distinct tokens and the state-related tokens. The computer system loads the batch of training samples and the token embedding batch into a memory of a processor of the computer system. The processor trains a machine learning model using the batch of training samples and the token embedding batch.

Description

PATENT Attorney Docket No.: 079900-1472383 Client Reference No.: 9459WO01DYNAMIC TOKEN LOADING STRATEGY FOR LARGE SCALE EMBEDDING LEARNINGBACKGROUND

[0001] Large language models can have a vocabulary size ranging from 30k to 50k. For example GPT-3 has a vocabulary size of 50k tokens, BERT has a vocabulary size of 30k tokens, and RoBERTa has a vocabulary size of 50k tokens. The vocabulary size can significantly affect a model's performance and is usually chosen to balance between the model's complexity7and its ability7to handle a wide range of inputs.

[0002] When training the large language model, all tokens in the vocabulary7are loaded into memory7. This becomes problematic as the vocabulary size grows larger and all of the tokens cannot be stored in memory at once.

[0003] Natural language processing methods using large language models can utilize subword tokenization (e.g., byte pair encoding, WordPiece, etc.) to reduce total size of the vocabulary. For example, the word “working” can be split into the tokens of “work” and “ing.” The token “ing” can be reused in conjunction with other tokens.

[0004] However, subword tokenization can only be used when tokens have common patterns that can allow for the reuse of subword portions, such as prefixes and sufixes. When the vocabulary of tokens includes tokens that do not have subwords, then such methods cannot be used to solve the problem of vocabulary7size.

[0005] Embodiments of the disclosure address this problem and other problems individually and collectively.SUMMARY

[0006] Embodiments provide for systems and methods of generating batches of training samples and token embedding batches to iteratively train machine learning models. The token embedding batches can allow the machine learning model to access a portion of token embeddings (e.g., in a training batch) from a large number of token embeddings during a given training iteration. As such, the machine learning model can have access to a large178958504V 1PATENT Attorney Docket No.: 079900-1472383 Client Reference No.: 9459WO01 number of tokens in the overall vocabulary' (e.g., one billion tokens). A distinct portion of the token embeddings in a library to be used for training can be augmented with additional token embeddings (e.g., state-related tokens), thereby generating a training batch that is more accurate while also reducing the amount of memory required, since the entire library of token embeddings is not needed during a given training iteration.

[0007] One embodiment is related to a method of training a machine learning model including samples of outcomes of requests determined by previous input data. The method can be performed by a computer system. The computer system can receive a training set of training samples. Each training sample can be represented by a set of tokens and including an outcome of a request. The computer system can determine a batch of training samples from the training set of training samples. The computer system can determine a set of distinct tokens from the batch of training samples. The computer system can then determine state-related tokens. The state-related tokens can correspond to possible outcomes of requests. The computer system can generate a token embedding batch from token embeddings of the set of distinct tokens and the state-related tokens. The computer system can load the batch of training samples and the token embedding batch into a memory of a processor of the computer system. The computer system can tram, by the processor, the machine learning model using the batch of training samples and the token embedding batch.

[0008] Another embodiment is related to a computer product comprising a computer readable medium storing a plurality' of instructions for controlling a computer system to perform operations of any of the methods above.

[0009] Another embodiment is related to a system comprising the aforementioned computer product and one or more processors for executing instructions stored on the computer readable medium.

[0010] Further details regarding embodiments of the disclosure can be found in the Detailed Description and the Figures.BRIEF DESCRIPTION OF THE DRAWINGS

[0011] FIG. 1 shows a diagram illustrating an exemplary' architecture of a transformer model according to embodiments.278958504V 1PATENT Attorney Docket No.: 079900-1472383 Client Reference No.: 9459WO01

[0012] FIG. 2 shows a diagram illustrating a self-attention mechanism according to embodiments.

[0013] FIG. 3 shows a diagram illustrating converting an input sequence to an output sequence according to embodiments.

[0014] FIG. 4 shows a block diagram of an interaction processing system according to embodiments.

[0015] FIG. 5 shows a block diagram illustrating an overview of a dynamic token loading method according to embodiments.

[0016] FIG. 6 shows a block diagram of components of a computer system according to embodiments.

[0017] FIG. 7 shows a diagram illustrating details of token batching and model training according to embodiments.

[0018] FIG. 8 shows a flow diagram illustrating a token batching and model training method according to embodiments.

[0019] FIG. 9 shows a diagram illustrating a token batch according to embodiments.

[0020] FIG. 10 shows a diagram illustrating sampling tokens according to similarity according to embodiments.

[0021] FIG. 11 shows a flow diagram illustrating a token sampling method according to embodiments.

[0022] FIG. 12 shows a block diagram illustrating a computer according to embodiments.TERMS

[0023] Prior to discussing embodiments of the disclosure, some terms can be described in further detail.

[0024] A '‘machine learning model” (ML model) can refer to a software module configured to be run on one or more processors to provide a classification or numerical value of a property7of one or more samples. An ML model can include various parameters (e.g., for 378958504V 1PATENT Attorney Docket No.: 079900-1472383 Client Reference No.: 9459WO01 coefficients, weights, thresholds, functional properties of function, such as activation functions). As examples, an ML model can include at least 10, 100, 1,000, 5,000. 10,000, 50,000. 100,000, or one million parameters. An ML model can be generated using sample data (e.g., training samples) to make predictions on test data. Various number of training samples can be used, e.g., at least 10, 100, 1,000, 5,000, 10,000, 50,000, 100,000, or at least 200,000 training samples. One example is an unsupervised learning model such as hidden Markov model (HMM), clustering (e.g., hierarchical clustering, k-means, mixture models, model-based clustering, density-based spatial clustering of applications with noise (DBSCAN), and OPTICS algorithm), approaches for learning latent variable models such as Expectation-maximization algorithm (EM), method of moments, and blind signal separation techniques (e.g., principal component analysis, independent component analysis, nonnegative matrix factorization, singular value decomposition), and anomaly detection (e.g., local outlier factor and isolation forest). Another example ty pe of model is supervised learning that can be used with embodiments of the present disclosure. Example supervised learning models may include different approaches and algorithms including analytical learning, statistical models, artificial neural network (e.g. including convolutional and / or transformer layers) that may have 1-10 layers as examples, recurrent neural network (e.g., long short term memory, LSTM), boosting (meta-algorithm), bootstrap aggregating (bagging) such as random forests, support vector machine (SVM), support vector (SVR), Bayesian statistics, case-based reasoning, decision tree learning, inductive logic programming, linear regression, logistic regression, Gaussian process regression, genetic programming, group method of data handling, kernel estimators, learning automata, learning classifier systems, minimum message length (decision trees, decision graphs, etc.), multilinear subspace learning, naive Bayes classifier, maximum entropy classifier, conditional random field, nearest neighbor algorithm, probably approximately correct learning (PAC) learning, ripple down rules, a knowledge acquisition methodology7, symbolic machine learning algorithms, subsymbolic machine learning algorithms, minimum complexity7machines (MCM), ordinal classification, data pre-processing, handling imbalanced datasets, statistical relational learning, or Proaftn (a multicriteria classification algorithm), or an ensemble of any of these types. Supervised learning models can be trained in various ways using various cost / loss functions that define the error from the known label (e.g., least squares and absolute difference from known classification) and various optimization techniques, e.g., using478958504V 1PATENT Attorney Docket No.: 079900-1472383 Client Reference No.: 9459WO01 backpropagation, steepest descent, conjugate gradient, and Newton and quasi-Newton techniques.

[0025] A “deep neural network (DNN)” may be a neural network in which there are multiple layers between an input and an output. Each layer of the deep neural network may represent a mathematical manipulation used to turn the input into the output. In particular, a “recurrent neural network (RNN)” may be a deep neural network in which data can move forward and backward between layers of the neural network.

[0026] A “model database’7may include a database that can store machine learning models. Machine learning models can be stored in a model database in a variety' of forms, such as collections of parameters or other values defining the machine learning model.Models in a model database may be stored in association with keywords that communicate some aspect of the model. For example, a model used to evaluate news articles may be stored in a model database in association with the keywords “news,” “propaganda,” and “information.” A computer can access a model database and retrieve models from the model database, modify models in the model database, delete models from the model database, or add new7models to the model database.

[0027] A “feature vector” may include a set of measurable properties (or “features”) that represent some object or entity-. A feature vector can include collections of data represented digitally in an array or vector structure. A feature vector can also include collections of data that can be represented as a mathematical vector, on which vector operations such as the scalar product can be performed. A feature vector can be determined or generated from input data. A feature vector can be used as the input to a machine learning model, such that the machine learning model produces some output or classification. The construction of a feature vector can be accomplished in a variety of w ays, based on the nature of the input data. For example, for a machine learning classifier that classifies w ords as correctly spelled or incorrectly spelled, a feature vector corresponding to a word such as “LOVE” could be represented as the vector (12, 15, 22, 5), corresponding to the alphabetical index of each letter in the input data w'ord. For a more complex “input,” such as a human entity-, an exemplary- feature vector could include features such as the human's age, height, weight, a quantitative representation of relative happiness, etc. Feature vectors can be represented and stored electronically in a feature store. Further, a feature vector can be578958504V 1PATENT Attorney Docket No.: 079900-1472383 Client Reference No.: 9459WO01 normalized (i.e., be made to have unit magnitude). As an example, the feature vector (12, 15, 22, 5) corresponding to LOVE " could be normalized to approximately (0.40, 0. 1, 0.74. 0.17).

[0028] A '‘language model” can include a probabilistic model relating to evaluating natural language. A language model can include a large language model (LLM). A large language model can include a transformer and can be utilized to evaluate data other than natural language.

[0029] A “training set of training samples” can include a set of data used for training. A training set of training samples can include a plurality of training samples.

[0030] A '‘training sample” can include data used to train a model. A training sample can include a vector, or other data structure. A training sample can include access request data.

[0031] A “token” can include a word, character set, or combination of words and punctuation that. For example, a token can represent a character set that indicates a particular user account (e g., 1234567890). As another example, a token can represent a word of “authorized.” A token can be embedded to create a token embedding such that the token can be utilized in a machine learning model.

[0032] A “distinct token"’ can include a token that occurs in one or more training samples in a training set. A distinct token can indicate a specific token that is present in a training set.

[0033] A '‘state-related token” can include a token that corresponds to possible outcomes of requests as indicated in each training sample. A state-related token can be, for example, “authenticated” or “not authenticated."’

[0034] “Outcomes of requests” can include an indication of an outcome of a particular request. An outcome of a request can indicate how the request was finalized. For example, an outcome of a request can be “authorized” or “not authorized.” An outcome of a request can be represented by a state-related token that is included in a training sample that represents the request. A request can be an interaction.678958504V 1PATENT Attorney Docket No.: 079900-1472383 Client Reference No.: 9459WO01

[0035] An “appearance threshold token” can include a token that can be selected based on the number of times that the token appears in a particular dataset. For example, a computer can determine ten appearance threshold tokens for the ten most present tokens in a training set.

[0036] “Tokens sampled by similarity” can include tokens that can be selected based on their similarity' to other tokens. A token can be determined to be similar to another token using similarity determination methods, such as a clustering process.

[0037] An “embedding” can include a value that represents an object. An embedding can represent an object such as quantitative data, categorical data, text, images, audio, etc. An embedding can be in a particular dimensional space. An embedding can represent semantically meaningful information of the originating object. An embedding can be numerical value such as a vector.

[0038] A “batch” can include a group of records processed as a single unit. A batch can include a subset of data examples from a training set that a machine learning model uses during an iteration of the training process. A number of data examples in a batch can be specified by a batch size, which can be a machine learning model hyperparameter.

[0039] A “topological graph” or “graph” can include a representation of a graph in a plane of distinct vertices connected by edges. The distinct vertices in a topological graph may be referred to as “nodes.” Each node may represent specific information for an event or may represent specific information for a profile of an entity or object. The nodes may be related to one another by a set of edges, E. An “edge” may be described as an unordered pair composed of two nodes as a subset of the graph G=(V, E), where is G is a graph comprising a set V of vertices (nodes) connected by a set of edges E. For example, a topological graph may represent a transaction network in which a node representing a transaction may be connected by edges to one or more nodes that are related to the transaction, such as nodes representing information of a device, a user, a transaction type, etc. An edge may be associated with a numerical value, referred to as a “weight,” that may be assigned to the pairwise connection between the two nodes. The edge weight may be identified as a strength of connectivity between two nodes and / or may be related to a cost or distance, as it often represents a quantity that is required to move from one node to the next.778958504V 1PATENT Attorney Docket No.: 079900-1472383 Client Reference No.: 9459WO01

[0040] A “subgraph” or “sub-graph” can include a graph formed from a subset of elements of a larger graph. The elements may include vertices and connecting edges, and the subset may be a set of nodes and edges selected amongst the entire set of nodes and edges for the larger graph. For example, a plurality of subgraph can be formed by randomly sampling graph data, wherein each of the random samples can be a subgraph. Each subgraph can overlap another subgraph formed from the same larger graph.

[0041] A ““cluster” can include a group of nodes in a graph that are densely connected within the group. A cluster may be a subgraph or a portion / derivative thereof and a subgraph may or may not be a cluster and / or comprise one or more clusters. A cluster may be identified from a graph using a graph learning algorithm, such as a graph learning algorithm for mapping protein complexes. Clusters identified using historical data can be used to classify new data for making predictions. For example, identifying clusters can be used as part of a machine learning process, in which predictions about information elements can be made based on their relation to one another.

[0042] The term “node” can include a discrete data point representing specified information. Nodes may be connected to one another in a topological graph by edges, which may be assigned a value known as an edge weight in order to describe the connection strength between the two nodes. For example, a first node may be a data point representing a first device in a network, and the first node may be connected in a graph to a second node representing a second device in the network. The connection strength may be defined by an edge weight corresponding to how quickly and easily information may be transmitted between the two nodes. An edge weight may also be used to express a cost or a distance required to move from one state or node to the next. For example, a first node may be a data point representing a first position of a machine, and the first node may be connected in a graph to a second node for a second position of the machine. The edge weight may be the energy required to move from the first position to the second position.

[0043] A “user device” may be a device that is operated by a user. Examples of user devices may include a mobile phone, a smart phone, a card, a personal digital assistant (PDA), a laptop computer, a desktop computer, a server computer, a vehicle such as an automobile, a thin-client device, a tablet PC, etc. Additionally, user devices may be any type of wearable technology device, such as a watch, earpiece, glasses, etc. The user device may878958504V 1PATENT Attorney Docket No.: 079900-1472383 Client Reference No.: 9459WO01 include one or more processors capable of processing user input. The user device may also include one or more input sensors for receiving user input. As is known in the art, there are a variety of input sensors capable of detecting user input, such as accelerometers, cameras, microphones, etc. The user input obtained by the input sensors may be from a variety of data input types, including, but not limited to, audio data, visual data, or biometric data. The user device may comprise any electronic device that may be operated by a user, which may also provide remote communication capabilities to a network. Examples of remote communication capabilities include using a mobile phone (wireless) network, wireless data network (e.g., 3G, 4G or similar networks), Wi-Fi, Wi-Max, or any other communication medium that may provide access to a network such as the Internet or a private network.

[0044] "Access request” can include a request to access a resource. An access request can include a request from a first device or a first entity to access a resource that is provided by a second device or a second entity. For example, an access request can be a request generated by a user device that is requesting access to a resource provided by a resource provider computer. An access request may include authorization information, such as a user name, account number, password, credential, etc. The access request may also include an access request identifier, a resource identifier, a timestamp, a date, a device or computer identifier, a geo-location, or any other suitable information. Example access requests include a transaction between tw o parties and a data exchange betw een two devices. In some embodiments, an access request can include a user requesting access to secure data, a secure webpage, a secure location, a resource, a service, etc. In other embodiments, an access request can include a payment transaction in which two devices can interact to facilitate a payment. Access request data can include data related to and / or recorded during an access request. In some embodiments, access request data can be transaction data that includes a primary account number, a user device identifier, an amount (e.g.. a transaction amount), a resource provider computer identifier, a transport computer identifier, a network processing computer identifier, an authorizing entity computer identifier, a resource identifier, an IP address, a mailing address, a billing address, and / or other information related to the devices involved, the steps performed, or the interaction itself.978958504V 1PATENT Attorney Docket No.: 079900-1472383 Client Reference No.: 9459WO01

[0045] A “user"’ may include an individual. In some embodiments, a user may be associated with one or more personal accounts and / or mobile devices. The user may also be referred to as a cardholder, account holder, or consumer in some embodiments.

[0046] A '‘user device” may be a device that is operated by a user. Examples of user devices may include a mobile phone, a smart phone, a card, a personal digital assistant (PDA), a laptop computer, a desktop computer, a server computer, a vehicle such as an automobile, a thin-client device, a tablet PC. etc. Additionally, user devices may be any type of wearable technology device, such as a watch, earpiece, glasses, etc. The user device may include one or more processors capable of processing user input. The user device may also include one or more input sensors for receiving user input. As is known in the art, there are a variety of input sensors capable of detecting user input, such as accelerometers, cameras, microphones, etc. The user input obtained by the input sensors may be from a variety of data input types, including, but not limited to, audio data, visual data, or biometric data. The user device may comprise any electronic device that may be operated by a user, which may also provide remote communication capabilities to a network. Examples of remote communication capabilities include using a mobile phone (wireless) network, wireless data network (e.g., 3G, 4G, 5G, or similar networks), Wi-Fi, Wi-Max, or any other communication medium that may provide access to a network such as the Internet or a private network.

[0047] A “user identifier” can include any piece of data that can identify a user. A user identifier can comprise any suitable alphanumeric string of characters. In some embodiments, the user identifier may be derived from user identifying information. In some embodiments, a user identifier can include an account identifier associated with the user.

[0048] An “access device” may be any suitable device that provides access to a remote system. An access device may also be used for communicating with a coordination computer, a communication network, or any other suitable system. An access device may generally be located in any suitable location, such as at the location of a merchant. An access device may be in any suitable form. Some examples of access devices include POS or point of sale devices (e.g., POS terminals), cellular phones, personal digital assistants (PDAs), personal computers (PCs), tablet PCs, hand-held specialized readers, set-top boxes, electronic1078958504V 1PATENT Attorney Docket No.: 079900-1472383 Client Reference No.: 9459WO01 cash registers (ECRs), vending machines, automated teller machines (ATMs), virtual cash registers (VCRs), kiosks, security’ systems, access systems, and the like.

[0049] An access device may use any suitable contact or contactless mode of operation to send or receive data from, or associated with, a mobile communication or payment device. For example, access devices can have card readers that can include electrical contacts, radio frequency (RF) antennas, optical scanners, bar code readers, or magnetic stripe readers to interact with portable devices such as payment cards.

[0050] The term "‘resource’' can include any asset that may be used or consumed. For example, the resource may be an electronic resource (e.g., stored data, received data, a computer account, a network-based account, an email inbox), a physical resource (e.g., a tangible object, a building, a safe, or a physical location), or other electronic communications between computers (e.g., a communication signal corresponding to an account for performing a transaction).

[0051] A ‘‘resource provider” may be an entity that can provide a resource such as goods, services, information, and / or access. Examples of resource providers includes merchants, data providers, transit agencies, governmental entities, venue and dwelling operators, etc.

[0052] An “authorization request message” may be an electronic message that requests authorization for an interaction. In some embodiments, it is sent to a transaction processing computer and / or an issuer of a payment card to request authorization for a transaction. An authorization request message according to some embodiments may comply with International Organization for Standardization (ISO) 8583. which is a standard for systems that exchange electronic transaction information associated with a payment made by a user using a payment device or payment account. The authorization request message may include an issuer account identifier that may be associated with a payment device or payment account. An authorization request message may also comprise additional data elements corresponding to “identification information” including, by way of example only: a service code, a CVV (card verification value), a dCVV (dynamic card verification value), a PAN (primary account number or “account number”), a payment token, a user name, an expiration date, etc. An authorization request message may also comprise “transaction information,” such as any information associated with a current transaction, such as the transaction value,1178958504V 1PATENT Attorney Docket No.: 079900-1472383 Client Reference No.: 9459WO01 merchant identifier, merchant location, acquirer bank identification number (BIN), card acceptor ID, information identifying items being purchased, etc., as well as any other information that may be utilized in determining whether to identify and / or authorize a transaction.

[0053] An ‘'authorization response message” may be a message that responds to an authorization request. In some cases, it may be an electronic message reply to an authorization request message generated by an issuing financial institution or a transaction processing computer. The authorization response message may include, by way of example only, one or more of the following status indicators: Approval — transaction was approved; Decline - transaction was not approved; or Call Center — response pending more information, resource provider is to call the toll-free authorization phone number. The authorization response message may also include an authorization code, which may be a code that a credit card issuing bank returns in response to an authorization request message in an electronic message (either directly or through the transaction processing computer) to the merchant's access device (e.g., POS equipment) that indicates approval of the transaction. The code may serve as proof of authorization.

[0054] An “authorizing entity” may be an entity that authorizes a request. Examples of an authorizing entity may be an issuer, a governmental agency, a document repository, an access administrator, etc. An authorizing entity may operate an authorizing entity computer. An “issuer” may refer to a business entity7(e.g., a bank) that issues and optionally maintains an account for a user. An issuer may also issue payment credentials stored on a user device, such as a cellular telephone, smart card, tablet, or laptop to the consumer, or in some embodiments, a portable device.

[0055] A “processor” may include a device that processes something. In some embodiments, a processor can include any suitable data computation device or devices. A processor may comprise one or more microprocessors working together to accomplish a desired function. The processor may include a CPU comprising at least one high-speed data processor adequate to execute program components for executing user and / or systemgenerated requests. The CPU may be a microprocessor such as AMD's Athlon, Duron and / or Opteron; IBM and / or Motorola's PowerPC; IBM's and Sony's Cell processor; Intel's Celeron. Itanium, Pentium, Xeon, and / or XScale; and / or the like processor(s).1278958504V 1PATENT Attorney Docket No.: 079900-1472383 Client Reference No.: 9459WO01

[0056] A “memory” may be any suitable device or devices that can store electronic data. A suitable memory may comprise a non-transitory computer readable medium that stores instructions that can be executed by a processor to implement a desired method.Examples of memories may comprise one or more memory chips, disk drives, etc. Such memories may operate using any suitable electrical, optical, and / or magnetic mode of operation.

[0057] A “server computer” may include a powerful computer or cluster of computers. For example, the server computer can be a large mainframe, a minicomputer cluster, or a group of servers functioning as a unit. In one example, the server computer may be a database server coupled to a Web server. The server computer may comprise one or more computational apparatuses and may use any of a variety of computing structures, arrangements, and compilations for servicing the requests from one or more client computers.DETAILED DESCRIPTION

[0058] Embodiments provide for systems and methods of efficiently training a machine learning model that uses a large number of tokens by generating training batches of token embeddings that can be loaded into a processor as the tokens are needed. For example, a computer can load token embedding batches and batches of training samples into a secondary’ processor, such as a graphics processing unit, from a primary processor, such as a central processing unit, to train the machine learning model. Embodiments solve the technical problem of having a large number tokens available in the machine learning model (e.g., one billion) for determining a next token in a sequence such that they cannot all be loaded in a processor (e.g., a graphics processing unit (GPU)) managed memory at the same time. Most language-based machine learning models, such as large language models (LLMs), have around 30-50 thousand tokens in a vocabulary library. It is impossible to load one billion token embeddings into a secondary processor managed memory' at a time to train the machine learning model. The secondary processor can be a training processor.

[0059] Embodiments provide a technical solution to this technical problem with an efficient data and token batching process to train batches of data at a time using the graphics processing unit. However, typical batching techniques that are utilized for LLMs, which have smaller token counts, such as subword tokenization cannot be utilized with data such as1378958504V 1PATENT Attorney Docket No.: 079900-1472383 Client Reference No.: 9459WO01 access request data that does not have prefixes or suffixes to convert into subwords. A computer, according to embodiments, can generate batches of training sample as well as batches of token embeddings that relate to the training samples. By creating token embedding batches, the computer can segment the total number of token embeddings into smaller groups of token embeddings that are relevant to the current batch of training samples. The computer can include token embeddings corresponding to tokens that occur in the batch of training samples (e.g.. distinct tokens), but can advantageously introduce additional tokens to ensure good accuracy within the batch based on additional criteria. The additional tokens can include state-related tokens, appearance threshold tokens, and tokens sampled by similarity7.

[0060] According to embodiments, a computer system can obtain token embeddings and a training set of training samples from a database. The computer system can determine a batch of training samples batch from the training set of training samples. The batch of training samples can be a subset of the training set of training samples.

[0061] After determining the batch of training samples, the computer system can determine a token batch based on the batch of training samples . The token batch can include tokens that relate to the tokens in the input data batch. The token batch can include tokens that are in the input data batch (e.g., distinct tokens). For example, the computer system can determine distinct tokens used in the batch of training samples to include into the token batch.

[0062] The token batch can also include state-related tokens, appearance threshold tokens, and tokens sampled by similarity7. The state-related tokens are tokens that are related to task questions and task classifications (e.g., approved, declined, fraud, chargeback, etc.). The appearance threshold tokens can include tokens that appear in the training set or a plurality of training sets more than or equal to a threshold number of times. The tokens sampled by similarity can include tokens that are similar to the distinct tokens used in the training set of training samples, as determined by the computer using a sampling and clustering process.

[0063] The computer system can determine a token embedding batch from token embeddings of the set of distinct tokens, the state-related tokens, the appearance threshold tokens, and a token sampled by similarity. The computer system can use the batch of training samples and the token embedding batch to train a machine learning model, such as a large1478958504V 1PATENT Attorney Docket No.: 079900-1472383 Client Reference No.: 9459WO01 language model (LLM). The machine learning model can be trained to predict a next token from a vocabulary (library) of tokens.

[0064] The vocabulary of tokens for a machine learning model can include all potential tokens. The token batch (corresponding to the token embedding batch) can include a portion of the vocabulary7of tokens. For a training batch, the machine learning model can be provided token embeddings for a particular portion of the vocabulary of tokens since the overall number of tokens cannot be loaded at the same time. Embodiments provide for systems and methods of constructing token batches that provide for accurate model performance even though the model does not have access to all tokens at once.

[0065] During training, the machine learning model can learn new token embeddings for tokens in the token batch. The token embeddings can be updated based on the new token embeddings learned by the machine learning model.I. EXAMPLE LANGUAGE MODEL USING TOKEN LIBRARY

[0066] Embodiments can utilize a model that predicts a next token in a sequence. Such a model can include transformer models to evaluate sequences of tokens, e.g.. as part of a large language model or other type of foundation model. A transformer model can be a type of neural network architecture designed to process sequential data by analyzing relationships between elements within a sequence.A. Encoder and decoder

[0067] A transformer model can include an encoder and a decoder. An encoder can map an input sequence to a sequence of continuous representations, which is then fed into the decoder. The decoder can receive the output of the encoder together with the decoder output at the previous time step to generate an output sequence.

[0068] FIG. 1 shows an exemplary architecture of a transformer model 100, according to some embodiments. The transformer model 100 is a pretrained language model that uses a unifiedc'text-to-text” format for all text based NLP problems. The architecture of the transformer model 100 is designed to support any natural language processing task (e.g., classification, named entity recognition (NER), question answering, etc.). The example of the tasks performed by the transformer model 100 are generative tasks (such as machine1578958504V 1PATENT Attorney Docket No.: 079900-1472383 Client Reference No.: 9459WO01 translation, summarization, text generation, etc.) where the task format may require the model to generate text conditioned on some input, and classification tasks where the transformer model 100 is trained to output the literal text of the label (e.g.. “positive” or “negative” for sentiment analysis).

[0069] The transformer model 100 uses an encoder-decoder transformer architecture. The transformer model 100 includes an encoder part 101 and a decoder part 102. First, an input sentence is tokenized into distinct elements (e.g., tokens). These tokens are typically integer indices in a vocabulary dataset. To feed those tokens into the neural network, each token is converted into an embedding vector by an input embedding layer 103. Further, a positional encoding layer 104 (e.g., a linear encoding layer) is provided and injects positional encoding into each embedding so that the model can know word positions without recurrence. The outputs of the input embedding layer 103 and the positional encoding layer 104 are combined and passed on to a multi -head attention layer 106 of the encoder part 101. Thus, an input to the transformer is not the characters of the input text but a sequence of embedding vectors. Each vector represents the semantics and position of a token.

[0070] The output of the encoder part 101 is provided as an input to the decoder part 102 that generates an output sequence (e.g., an output vector). Then, the output vector from the decoder part 102 goes through a linear transformation layer 110 that changes the dimension of the vector from the embedding vector size into the size of vocabulary. The softmax layer 120 further converts the vector into probabilities that are then provided as an output of the transformer model 100. For example, an output of the transformer model 100 are probabilities associated with the words distributed within the English language.

[0071] During training of the transformer model 100, the decoder 102 can receive both the context vector and a desired target output sequence (e.g., ground truth that is a known next token in the training data set). The transformer model 100 can generate a predicted output sequence. The transformer model 100 can compare the predicted output sequence to the desired target output sequence. The transformer model 100 can update the w eights based on the comparison within the transformer model 100 to adjust the model to more accurately predict output sequences in subsequent training rounds.1678958504V 1PATENT Attorney Docket No.: 079900-1472383 Client Reference No.: 9459WO01

[0072] During inference using the transformer model 100, the decoder part 102 can receive the context vector from the encoder part 101. The decoder part 102 can determine the output sequence based on the context vector.

[0073] The decoder part 102 uses the context vector to comprehend the input sequence and create the corresponding output sequence. The decoder part 102 can engage in autoregressive generation, producing individual elements sequentially. At each time step, during a training or inference process, the decoder part 102 can use the context vector, and a previous output token to generate a probability distribution of a possible next tokens. The token with the highest probability is then chosen as the output, and the process continues until the end of the output sequence is reached.

[0074] The output sequence can include a sequence of tokens. The output sequence can include any number of tokens. The output sequence can be the same length as the input sequence or a different length as the input sequence. The output sequence can include a sequence of access request data, where each element of the access request data (e.g., an identifier, an amount, a time, etc.) is a token.B. Self-attention

[0075] Self-attention mechanisms can capture interactions between tokens within an input sequence and within an output sequence. The self-attention mechanism can be included in a self-attention layer. In a self-attention layer, a computer can first compute a key vector, a query vector, and a value vector. The computer can then compute a matrix multiplication between the key vector and query vector. After a softmax transformation, this matrix of interactions can be referred to as an attention matrix. The resulting output hidden states of the attention layer are the matrix multiplication of the attention matrix and the value vector. Each of the resulting hidden states coming from the attention layer can be understood as a weighted average of the values, with the attentions being the weights.

[0076] FIG. 2 shows a diagram illustrating a self-attention mechanism according to embodiments. FIG. 2 includes a self-attention layer 200. A transformer can include the selfattention layer 200.

[0077] The self-attention layer 200 can allow the transformer model to focus on different parts of the sequence data dynamically. FIG. 2 includes an input sequence 202 that 1778958504V 1PATENT Attorney Docket No.: 079900-1472383 Client Reference No.: 9459WO01 includes a sequence of tokens. The input sequence 202 includes tokens from a vocabulary' that can be embedded into token embeddings. The input sequence 202 includes an example that includes the five tokens of “how,” “are,” “you,” “doing,” and “?.”

[0078] For the input sequence 202, the transformer model can determine token embeddings 204 for each token in the input sequence 202. Each token embedding of the token embeddings 204 can be a vector representation of the token. For example, the token of “how” can be converted into a vector with 128 elements.

[0079] The transformer model can determine three sets of vectors from the token embeddings, key weights 206, query weights 208, and value weights 210. The transformer model can determine the three sets of vectors of: key vectors 212, query' vectors 214, and value vectors 216. The transformer model can generate the key vectors 212 by multiplying the token embeddings by the key weights 206. The transformer model can generate the query vectors 214 by multiplying the token embeddings by the query weights 208. The transformer model can generate the value vectors 216 by multiplying the token embeddings by the value weights 210. The key weights 206, the query weights 208, and the value weights 210 are model parameters that the transformer model can adjust during training of the transformer model. The key vectors 212, the query vectors 214, and the value vectors 216 can represent different aspects of the input sequence 202.

[0080] The transformer model can then compute attention scores 218 by combining the key vectors 212 and the query vectors 214. The transformer model can combine the key vectors 212 with the query vectors 214 using matrix multiplication, element-wise dot product, or other suitable combination process. The transformer model can then user a softmax function 220 to generate an attention matrix 222 from the attention scores 218.

[0081] After generating the attention matrix 222, the transformer model can combine the attention matrix 222 with the value vectors 216 to generate hidden states 224. The hidden states 224 can be representative of weighted averages of the value vectors 216 of the input sequence 202, with the attention matrix 222 being the weights applied to the value vectors 216.

[0082] Hidden states 224 can be used to determine which token (e g., a word) has the highest probability of occurring next in the sequence. To determine which token is next, a set1878958504V 1PATENT Attorney Docket No.: 079900-1472383 Client Reference No.: 9459WO01 of tokens can each be tried and a probability determined for each. The set of tokens can correspond to a vocabulary of tokens from which an optimal sequence can be determined.C. Use of Vocabulary to determine next token in sequence

[0083] A machine learning model (such as a transformer model) can have access to a vocabulary library' of tokens. When the machine learning model determines what token appears next in a sequence, the machine learning model can select a token from the vocabulary library of tokens. The vocabulary library of tokens can include tokens as indicated by a current token batch.

[0084] FIG. 3 shows a diagram illustrating converting an input sequence to an output sequence according to embodiments. A computer can perform a process using a machine learning model to determine the output sequence from the input sequence.

[0085] The computer can receive an input sequence 302. For example, the machine learning model can receive the input sequence 302 that includes four tokens: “Paris,” “is,” “the,” and “city.”

[0086] The computer can input the input sequence 302 into a tokenizer 304 to determine token embeddings for each token in the input sequence 302. The tokenizer 304 can determine token embeddings for tokens that are included in the vocabulary library of tokens as indicated by the token batch. The token batch can indicate, for example, 1,000 tokens that are available in the vocabulary library of tokens, 4 of which are used to tokenize the input sequence 302.

[0087] The computer can obtain the token embeddings 306 for each token in the input sequence 302. For example, for the input sequence 302 of “Paris is the city.” the computer can obtain the token embedding sequence 306 of “16525 42 171.”

[0088] The computer can provide the sequence of token embeddings 306 that represent the input sequence 302 into a machine learning model 308 such as a large language model (LLM). The machine learning model 308 can output logits 310 based on the sequence of token embeddings 306 using a self-attention mechanism. The logits 310 can be unnormalized final scores of the machine learning model 308.1978958504V 1PATENT Attorney Docket No.: 079900-1472383 Client Reference No.: 9459WO01

[0089] After obtaining the logits, the computer can apply a softmax function 312 to the logits 310 to obtain a probability distribution 314 over the potential next tokens for the input sequence 302. The probability distribution 314 includes a normalized score for each potential next token for the input sequence 302.

[0090] For example, the probability distribution 314 can include five potential next tokens for the input sequence 302. The five potential next tokens include “of,” “that,” “where.” “with,” and “in.” The score for each potential next token indicates a probability that the potential next token is the correct next token. For example, the token “of’ has a score of 0.337, the token “that” has a score of 0.153, the token “where” has a score of 0.094, the token “with” has a score of 0.083, and the token “in” has a score of 0.036.

[0091] The computer can select the potential next token with the highest score to be the next token in the input sequence 302. For example, the token “of can be added to the end of the input sequence 302 to form the sequence of “Paris is the city of’. The computer can repeat the aforementioned process to determine additional next tokens in the sequence. For example, the computer can determine a next token of “light,” thus making the overall sequence “Paris is the city of light”. The computer can repeat the process again to determine a final token in the sequence. For example, the computer can determine a next token of (e.g., a period), thus making the overall sequence “Paris is the city' of light.”.II. EXAMPLE USE FOR ACCESS REQUESTS

[0092] Embodiments can utilize the systems described herein to, at least create, train, and utilize machine learning models that can utilize large numbers of tokens. For example, systems can train a machine learning model, such as a transformer, with one billion tokens. An example use case for such a model is determining whether to grant access to a request from another device, e.g., access to an electronic resource or a physical resource.

[0093] Embodiments provide for systems that can efficiently determine training sample batches and token embedding batches that can be used to train the machine learning model. The training sample batches can be created from a training set of training samples. The training sets can include any suitable data, such as access request data captured by an access request system.2078958504V 1PATENT Attorney Docket No.: 079900-1472383 Client Reference No.: 9459WO01

[0094] FIG. 4 shows a system 400 according to embodiments of the disclosure. The system 400 comprises a user device 402. an access device 404, a resource provider computer 406, a transport computer 408. a network processing computer 410. an authorizing entity computer 412, a data database 414, a computer system 416, a model database 418, and a resource 420. The system 400 can be an access request network.

[0095] The user device 402 can be in operative communication with the access device 404 and / or the resource provider computer 406. The resource provider computer 406 can be in operative communication with the access device 404. the user device 402, and the transport computer 408. The network processing computer 410 can be in operative communication with the transport computer 408, the authorizing entity computer 412, and the data database 414. The computer system 416 can be in operative communication with the data database 414 and the model database 418. Access to the resource 420 (e.g., an electronic resource such as a computer or account or a physical location such as a building) can be controlled by the resource provider computer 412 or the access device 410.

[0096] For simplicity of illustration, a certain number of components are shown in FIG. 4. It is understood, however, that embodiments of the invention may include more than one of each component. In addition, some embodiments of the invention may include fewer than or greater than all of the components shown in FIG. 4.

[0097] Messages between the devices of the system 400 illustrated in FIG. 4 can be transmitted using a secure communications protocols such as, but not limited to, File Transfer Protocol (FTP); HyperText Transfer Protocol (HTTP); Secure Hypertext Transfer Protocol (HTTPS), SSL. ISO (e.g., ISO 8583) and / or the like. The communications network may include any one and / or the combination of the following: a direct interconnection; the Internet; a Local Area Network (LAN); a Metropolitan Area Network (MAN); an Operating Missions as Nodes on the Internet (OMNI); a secured custom connection; a Wide Area Network (WAN); a wireless network (e.g.. employing protocols such as, but not limited to a Wireless Application Protocol (WAP), I-mode, and / or the like); and / or the like. The communications network can use any suitable communications protocol to generate one or more secure communication channels. A communications channel may, in some instances, comprise a secure communication channel, which may be established in any known manner,2178958504V 1PATENT Attorney Docket No.: 079900-1472383 Client Reference No.: 9459WO01 such as through the use of mutual authentication and a session key, and establishment of a Secure Socket Layer (SSL) session.

[0098] The user device 402 can include one or more computers, portable computers, laptop computers, tablet computers, mobile devices, cellular phones, wearable devices (e.g., watches, glasses, lenses, clothing, etc.), personal digital assistants (PDAs), Internet of Things (loT) devices, and / or the like. The user device 402 can initiate interactions (e.g., transactions, secure location access requests, cybersecurity requests, etc.) with resource provider computers and / or access devices to request access to resources such as the resource 420. For example, the user device 402 can select one or more items for the interaction at a resource provider location (e.g., a grocery' store). During checkout, the user can be instructed to tap (e.g., bring into near-field communication range) the user device 402 against the access device 404. As another example, the user device 402 can select one or more items for the interaction on a website hosted by the resource provider computer 406.

[0099] The resource 420 can include an asset that may be used or consumed. The user device 402 can request access to the resource 420 from the resource provider computer 406. The resource 420 may be an electronic resource (e.g., stored data, received data, a computer account, a network-based account, an email inbox, etc.), a physical resource (e.g., a tangible object, a building, a safe, a physical location, etc.), or other electronic communications between computers (e.g., a communication signal corresponding to an account for performing a transaction, etc.).

[0100] The access device 404 can include a device operated by a resource provider. The access device 404. for example, can include a mobile device, a POS terminal, a laptop, etc. The access device 404 can communicate with another device (e.g., a user device 402) to perform an interaction. During the interaction, the access device 404 can receive credentials from the user device and can provide interaction data to the resource provider computer 406 for authorization of the interaction. In some embodiments, the access device 404 can generate an authorization request message comprising at least the interaction data. The access device 404 can provide the authorization request message to the resource provider computer 406.

[0101] The resource provider computer 406 can include any suitable computational apparatus operated by a resource provider (e.g., a merchant). In some embodiments, the resource provider computer 406 may include one or more server computers that may host one 2278958504V 1PATENT Attorney Docket No.: 079900-1472383 Client Reference No.: 9459WO01 or more websites associated with the resource provider (e.g., a merchant). In some embodiments, the resource provider computer 406 may be configured to send data to the network processing computer 410 via the transport computer 408 as part of a payment verification and / or authentication process for a transaction between the user (e.g., consumer) and the resource provider. The resource provider computer 406 may also be configured to generate authorization request messages for transactions between a resource provider and a user, and route the authorization request messages to the authorizing entity computer 412 for transaction processing.

[0102] The transport computer 408 can include a server computer. The transport computer 408 may be associated with an acquirer, which may be an entity (e.g., a commercial bank) that has a business relationship with a particular merchant or other entity. Some entities can perform both issuer and acquirer functions. Some embodiments may encompass such single entity issuer-acquirers.

[0103] The network processing computer 410 can include a server computer. The network processing computer 410 may be disposed between the transport computer 408 and the authorizing entity computer 412. The network processing computer 410 may include data processing subsystems, networks, and operations used to support and deliver authorization services, exception file services, and clearing and settlement services. For example, the network processing computer 410 may comprise a server coupled to a network interface (e.g., by an external communication interface), and databases of information. The network processing computer 410 may be representative of a transaction processing network. An exemplary transaction processing network may include VisaNet™. Transaction processing networks such as VisaNet™ are able to process credit card transactions, debit card transactions, and other types of commercial transactions. VisaNet™, in particular, includes a VIP system (Visa Integrated Payments system) which processes authorization requests and a Base II system which performs clearing and settlement services. The network processing computer 410 may use any suitable wired or wireless network, including the Internet.

[0104] The network processing computer 410 can store access request data(e.g., transaction data) into the data database 414 during the interaction between the user device 402 and the resource provider computer 406. The network processing computer 410 can store training samples that can include the access request data into the data database 414.2378958504V 1PATENT Attorney Docket No.: 079900-1472383 Client Reference No.: 9459WO01

[0105] The authorizing entity computer 412 can include a server computer operated by an authorizing entity. The authorizing entity computer 412 may be associated with an authorizing entity, which may be an entity that authorizes a request. An example of an authorizing entity' may be an issuer, which may typically refer to a business entity (e.g., a bank) that maintains an account for a user. An issuer may also issue and manage an account associated with the user device 402.

[0106] As an illustrative example, the user may conduct an access request (e.g.. a transaction) at the resource provider (e.g., a merchant) using a user device 402. The transaction may be a payment transaction (e.g., for the purchase of a good or service), an access transaction (e.g., for access to a transit system), or any other suitable transaction. The user device 402 can interact with an access device 404 at a resource provider associated with resource provider computer 406. For example, the user may tap the user device 402 against an NFC reader in the access device 404. Alternately, the user may indicate payment account information to the resource provider electronically, such as in an online transaction. In some cases, the user device 402 may transmit to the access device 404 an account identifier, such as a payment token.

[0107] In order to authorize a transaction, an authorization request message may be generated by access device 404 or resource provider computer 406 and then forwarded to the transport (e.g., acquirer) computer 408. After receiving the authorization request message, the authorization request message is then sent to the network processing computer 410. The network processing computer 410 then forwards the authorization request message to the corresponding authorizing entity computer 412 associated with an authorizing entity associated with the user's payment account.

[0108] After the authorizing entity computer 412 receives the authorization request message, the authorizing entity computer 412 sends an authorization response message back to the network processing computer 410 to indicate whether the current transaction is authorized (or not authorized). The network processing computer 410 then forwards the authorization response message back to the transport computer 408. In some embodiments, network processing computer 410 may decline the transaction even if the authorizing entity' computer 412 has authorized the transaction, for example depending on a value of the fraud2478958504V 1PATENT Attorney Docket No.: 079900-1472383 Client Reference No.: 9459WO01 risk score. The transport computer 408 then sends the response message back to the resource provider computer 406.

[0109] After the resource provider computer 406 receives the authorization response message, the resource provider computer 406 may then provide the authorization response message for the user. The response message may be displayed by the access device 404, or may be printed out on a physical receipt. Alternately, if the transaction is an online transaction, the resource provider may provide a webpage or other indication of the authonzation response message as a virtual receipt. The receipts may include transaction data for the transaction.

[0110] If the authorization response message indicates that the access request is authorized, then the user device 402 and / or the user operating the user device 402 can access the resource 420. In some embodiments, the resource provider computer 406 can provide the resource 420 to the user device 402.

[0111] Over time, more access request data is stored into the data database 414. The computer system 416 can obtain the access request data from the data database 414 to train the machine learning model. The computer system 416 can develop the machine learning model over time with additional access request data.

[0112] The data database 414 can store data that can be utilized by the computer system 416 to train, maintain, and / or utilize a machine learning model, such as a language model. The data database 414 can store interaction data (e.g., transaction data, access request data, etc.).

[0113] The computer system 4416 can include a computer that can train, maintain, and utilize machine learning models. The computer system 4416 can obtain data from the data database 414, train a machine learning model using the data, and store the machine learning model in the model database 418.

[0114] The model database 418 can store machine learning models. The machine learning models can include language models that are trained by the computer system 416.

[0115] The data database 414 and the model database 418 can include can include any suitable databases. The databases may be a conventional, fault tolerant, relational, scalable, secure databases such as those commercially available from Oracle™ or Sybase™.2578958504V 1PATENT Attorney Docket No.: 079900-1472383 Client Reference No.: 9459WO01 III. DYNAMIC TOKEN LOADING

[0116] Embodiments provide for systems and methods of dynamically loading tokens to train a machine learning model. Dynamically loading tokens and associated token embeddings for training samples in a current batch of training samples can allow the computer system 416 to train a machine learning model that can handle billions of different tokens in a vocabulary library of tokens. The computer system 416 can tram the machine learning model in batches and can update subsets of token embeddings over time as the machine learning model learns new updated token embeddings.

[0117] The vocabulary library7of tokens can include all tokens know n to the machine learning model. The vocabulary library7of tokens can include a large number of tokens such that it is impossible to actively load all token embeddings corresponding to the tokens at once in memory accessible by the machine learning model. The token batches can be constructed from the vocabulary library of tokens based on what tokens are in a batch of training samples. By batching tokens, the machine learning model can have access to tokens that are currently needed, but can also maintain a larger vocabulary library of tokens.

[0118] The machine learning model, which can be a large language model, can utilize tokens from the token batch to predict a next token in an input sequence. For example, an input sequence can include a training sample. The machine learning model can process the training sample and can predict a next token for the training sample from the token batch.A. Batching process

[0119] A computer system (e.g., the computer system 416) can dynamically determine and load batches of training samples and token embedding batches into a GPU to train a machine learning model. The sy stem computer 416 can generate a batch of training samples from a training set. The system computer 416 can then generate a token embedding batch. The token embedding batch can include distinct token embeddings corresponding to distinct tokens and state-related token embeddings corresponding to state-related tokens. In some embodiments, the token embedding batch can also include appearance threshold token embeddings corresponding to appearance threshold tokens and set of token embeddings sampled by similarity corresponding to tokens sampled by similarity.2678958504V 1PATENT Attorney Docket No.: 079900-1472383 Client Reference No.: 9459WO01

[0120] The distinct tokens can include tokens that occur in the training samples. The state-related tokens can correspond to task questions and task classifications (e.g., approved, declined, fraud, chargeback, etc.). Including the state-related tokens into the token batch along with the distinct tokens can improve machine learning prediction accuracy since the machine learning model can have access to tokens relating to the task questions and task classifications.

[0121] FIG. 5 shows a block diagram illustrating an overview of a dynamic token loading method according to embodiments. As an example, the method illustrated in FIG. 5 will be described in the context of the computer system 416 dynamically loading tokens during training of a model (e.g., a language model) when a vocabulary7library of tokens is too large to load into a memory7accessible by the machine learning model. Reference to computer system 416 is simply explanatory and other computer systems can be used.

[0122] In the example shoyvn, the computer system 416 obtains a training set of training samples 502. The computer system 416 can obtain the training set of training samples 502 from the data database 414. Each training sample in the training set of training samples 502 can be represented by a set of tokens and can include an outcome of a request. The training sample can include interaction data obtained from an interaction processing system. In some embodiments, the outcomes of the requests included in each training sample can indicate yvhether or not a user involved in the associated interaction is authenticated or not authenticated. In other embodiments, the outcomes of the requests included in each training sample can indicate whether or not the associated interaction is authorized or not authorized.

[0123] Each training sample can include one or more tokens. A training sample can include data related to an interaction. A training sample can include data related a transaction between a user of a user device and a resource provider of a resource provider computer. For example, a training sample can include an account number (e.g., a primary account number (PAN)), a date, a time, a resource provider identifier, an authorizing entity identifier, a transport computer identifier, an amount identifier, or an interaction type. Each training sample can be of a same format, such that the account number occurs in the first element of each training sample and the outcomes of requests occurs in the last element of each training sample. For example, the training sample can be in the following format:2778958504V 1PATENT Attorney Docket No.: 079900-1472383 Client Reference No.: 9459WO01 [account number, date, time, resource_provider_identifier, authorizing entity identifier.transport computer identifier, interaction type, amount, outcome of request]

[0124] The computer system 416 can also obtain token embeddings 504. The token embeddings 504 can include token embeddings corresponding to tokens in the vocabulary library of tokens. The token embeddings 504 can include a large number of token embeddings (e.g., 500,000,000 token embeddings, 1,000,000,000 token embeddings, 3,000,000,000 token embeddings, etc.).

[0125] The tokens in the training samples can correspond to different items. For example, tokens can be words, numbers, or combinations thereof. Example tokens can include “1234567890123456,” “2023-09-01,” “10:00 am,” “John_Doe,” “XYZCorporation,” “card not present.” “1.99 USD,” “approved,” “declined,” etc. As additional examples, a token can relate to a bank (e.g., “XYZ Bank”), a business (e g., “XYZ Business”), a country (e.g., “USA”), a card type (e.g., “credit card”), a transaction type (e.g., “domestic”), transaction events (e.g., “clearing,” “fraud,” “not fraud,” etc.) or other information relating to an access request, entities involved in the access request, computers involved in the access request, and / or resources involved in the access request.

[0126] The computer system 416 can load the training set of training samples 502 and the token embeddings 504 into a memory coupled to the CPU 506. The memory can be random access memory (RAM), a CPU memory cache, or other suitable data storage means.

[0127] The computer system 416 can perform a batching process 508 to batch training samples and to batch tokens such that the batches can be utilized to train a language model.

[0128] The computer system 416 can determine a batch of training samples from the training set of training samples. For a first batch 510, the computer system 416 can determine a first data batch 512 that includes a subset of training samples from the training set of training samples 502. The first data batch 510 can include, for example, 5,000 training samples, 10,000 training samples, 100.000 training samples, or other suitable number of training samples. The batch size can be selected based on model complexity and available computing resources. A batch size can range from small batch sizes (e g., 4, 16, etc.) to large batch sizes (e.g., 4096, 8192, etc.). In some embodiments, the batch size can be selected as2878958504V 1PATENT Attorney Docket No.: 079900-1472383 Client Reference No.: 9459WO01 512 or 1024. A batch size of a power of 2 can be advantageous for computing efficiency. A batch can be at least any of these numbers.

[0129] The computer system 416 can determine a first token embedding batch 514 in the first batch 510. The first token embedding batch 514 can include token embeddings corresponding to tokens that relate to the first data batch 512. To determine the token embeddings in the first token embedding batch 514, the computer system 416 can determine a set of distinct tokens from the batch of training samples. The distinct tokens can include tokens that are included in one or more training samples included in the first data batch 512.

[0130] The computer system 416 can also determine state-related tokens that can be included in the first token batch 514 along with the distinct tokens. The state-related tokens correspond to possible outcomes of requests as indicated in each training sample. A state-related token can be, for example, "‘authenticated” or “not authenticated.”

[0131] The computer system 416 can determine token embeddings for the distinct tokens and the state-related tokens from the token embeddings 504. For example, the computer system 416 can determine distinct tokens and state-related tokens from the vocabulary library7of tokens. The computer system 416 can generate a token embedding batch from the token embeddings of the distinct tokens and the state-related tokens.

[0132] The computer system 416 can similarly generate a second batch 516 that includes a second data batch 518 and a second token embedding batch 520. The computer system 416 can generate an Nth batch 522 that includes an Nth data batch 524 and an Nth token embedding batch 526.

[0133] After performing the batching process 508, the computer system 416 can perform model training 528. The computer system 416 can load a batch, which includes a data batch and a token embedding batch, into a memory of a processor of the computer system. For example, the computer system 416 can load the first batch 510 into the memory. The computer system 416 can train the machine learning language model using the batch of training samples and the token embedding batch. The computer system 416 can train the machine learning language model using one or more GPUs, for example, including the first GPU 530 and the Nth GPU 532.2978958504V 1PATENT Attorney Docket No.: 079900-1472383 Client Reference No.: 9459WO01

[0134] The computer system 416 can iteratively train the machine learning language model using the created batches. For example, the computer system 416 can train the machine learning language model using the first batch 510, the second batch 516. and then the Nth batch 522. Between each batch, the computer system 416 can unload the previous batch of training samples and the previous token embedding batch from the memory7.

[0135] During training, the computer system 416 can update weights included in the machine learning language model based on the performance of the machine learning language model with each batch.B. Computer system

[0136] The computer system 416 can determine training sample batches and token embedding batches in order to iteratively train a machine learning model that is capable of processing billions of different input tokens.

[0137] FIG. 6 shows a block diagram of the computer system 416 according to embodiments. The exemplary computer system 416 may comprise a processor 604. The processor 604 may be coupled to a memory7602. a network interface 606, a computer readable medium 608, and a graphics processing unit 610. The computer readable medium 608 can comprise a batching module 608A and a training module 608B.

[0138] The memory 602 can be used to store data and code. For example, the memory 602 can store interaction data, tokens, training samples, machine learning models, etc. The memory7602 may be coupled to the processor 604 internally or externally (e.g., cloud based data storage), and may comprise any combination of volatile and / or non-volatile memory, such as RAM, DRAM, ROM, flash, or any other suitable memory device.

[0139] The computer readable medium 608 may comprise code, executable by the processor 604, for performing a method of training a machine learning model including samples of outcomes of requests determined by previous input data. The method includes the computer system 416 receiving a training set of training samples. Each training sample can be represented by a set of tokens and including an outcome of a request. The computer system 416 can determine a batch of training samples from the training set of training samples. The computer system 416 can determine a set of distinct tokens from the batch of training samples. The computer system 416 can determine state-related tokens. The state- 3078958504V 1PATENT Attorney Docket No.: 079900-1472383 Client Reference No.: 9459WO01 related tokens correspond to possible outcomes of requests. The computer system 416 can generate a token embedding batch from token embeddings of the distinct tokens and the state-related tokens. The computer system 416 can load the batch of training samples and the token embedding batch into a secondary processor managed memory of the computer system. The computer system 416 can train the machine learning large language model using the batch of training samples and the token embedding batch.

[0140] The batching module 608A may comprise code or software, executable by the processor 604, for batching data. The batching module 608A, in conjunction with the processor 604, can determine a batch of training samples from the training set of training samples and can determine a token embedding batch.

[0141] The batching module 608A, in conjunction with the processor 604, can create any suitable number of batches for a training set containing any suitable number of elements based on the number of training samples in the training set and a hyperparameter that can indicate a batch size (e.g., the number of elements to include in the batch).

[0142] The batching module 608A, in conjunction with the processor 604, can segment the training set of training samples into a plurality' of batches. The batching module 608A, in conjunction with the processor 604, can then determine token embedding batches to pair with each batch of training samples.

[0143] The batching module 608A, in conjunction with the processor 604, can determine a set of distinct tokens from the batch of training samples. The distinct tokens can include tokens that appear in the training samples in the batch. The batching module 608A, in conjunction with the processor 604, can also determine state-related tokens that correspond to possible outcomes of requests related to the training samples in the batch. The batching module 608A, in conjunction with the processor 604, can generate the token embedding batch from token embeddings of the set of distinct tokens and the state-related tokens.

[0144] The training module 608B can include may comprise code or software, executable by the processor 604, for training machine learning models. The training module 608B. in conjunction with the processor 604, can train a machine learning language model (e.g., a large language model (LLM)) using the batches created by the batching module 608A.3178958504V 1PATENT Attorney Docket No.: 079900-1472383 Client Reference No.: 9459WO01

[0145] The network interface 606 may include an interface that can allow the computer system 416 to communicate with external computers. The network interface 606 may enable the computer system 416 to communicate data to and from another device (e.g., the data database 414, the model database 418, etc.). Some examples of the network interface 606 may include a modem, a physical network interface (such as an Ethernet card or other Network Interface Card (NIC)), a virtual network interface, a communications port, a Personal Computer Memory Card International Association (PCMCIA) slot and card, or the like. The wireless protocols enabled by the network interface 606 may include Wi-Fi™. Data transferred via the network interface 606 may be in the form of signals which may be electrical, electromagnetic, optical, or any other signal capable of being received by the external communications interface (collectively referred to as “electronic signals"’ or “electronic messages”). These electronic messages that may comprise data or instructions may be provided between the network interface 606 and other devices via a communications path or channel. As noted above, any suitable communication path or channel may be used such as. for instance, a wire or cable, fiber optics, a telephone line, a cellular link, a radio frequency (RF) link, a WAN or LAN network, the Internet, or any other suitable medium.

[0146] The graphics processing unit 610 can include a specialized electronic circuit designed for parallel processing. The graphics processing unit 610 can be an electronic circuit that can perform mathematical calculations at a high speed. Computing tasks such as graphics rendering and machine learning can require the application of similar mathematical operations on a large dataset. The design of the graphics processing unit 610 can allow for the graphics processing unit 610 to perform the same operation on multiple data values in parallel in order to increase processing efficiency for compute-intensive tasks.

[0147] The graphics processing unit 610 may contain a number of multiprocessors. Each multiprocessor can have a shared memory block, a number of processors, and corresponding registers. The graphics processing unit 610 itself can manage a memory and may have access to a memory device on a board (e.g., motherboard) that the graphics processing unit 610 is housed on. The graphics processing unit 610 can be a standalone chip, known as a discrete GPU, or can be integrated with other computing hardware, known as integrated GPUs (iGPUs).3278958504V 1PATENT Attorney Docket No.: 079900-1472383 Client Reference No.: 9459WO01 C. Embeddings and model training

[0148] A computer system (e.g., the computer system 416) can dynamically determine and load batches of training samples and token embedding batches that are used to predict a next token in a sequence of token in the training samples. The system computer 416 can sample token embeddings (e g., from a plurality of token embeddings stored in memory) based on the batch of training samples and can feed the batches into a secondary processor managed memory' (e.g., a GPU) for each training iteration, while keeping a copy of the token embeddings in a primary' processor managed memory (e.g., a CPU). The sampled token embeddings can be updated during a training iteration, and the unsampled token embeddings may not be updated in the training iteration. The primary processor can include a processor that is designed to handle a wide range of tasks. The primary processor can be optimized to perform sequential processing. The primary processor can be a CPU. The secondary processor can include a processor designed to handle multiple simple calculations simultaneously. The secondary processor can be optimized to perform parallel processing. The secondary processor can be a GPU.

[0149] At the end of each training iteration, the embeddings determined by the machine learning model are updated in the copy of token embeddings that are stored in the primary processor managed memory. This can allow the computer to fit trainable token embeddings in the limited secondary processor managed memory even though there can be billions of token embeddings overall.

[0150] For example, the machine learning model can be a transformer. During each training iteration, an embedding layer of the transformer (such as the input embedding layer 103) can update weights that indicate how the input tokens are converted into token embeddings. As the weights are adjusted a token embedding for a token can change. The computer system 416 can store any changed token embeddings at the end of each training iteration into the copy of token embeddings that are stored in the primary processor managed memory.

[0151] FIG. 7 shows a diagram illustrating a token batching and model training method according to embodiments. The method illustrated in FIG. 7 will be described in the context of the computer system 416 updating token embeddings in batches using a machine3378958504V 1PATENT Attorney Docket No.: 079900-1472383 Client Reference No.: 9459WO01 learning model. The method described in reference to FIG. 7 is separated into four phases, a determine batch phase 710. an embedding selection phase 720, a model training phase 730. and an update embeddings phase 740.

[0152] During the determine batch phase 710, the computer system 416 can determine a batch of training samples 712 and can determine a token batch 714. The computer system 416 can determine the batch of training samples 712 from a training set of training samples stored in primary processor managed memory. The batch of training samples 712 can include a subset of training samples that are in the training set. The computer system 416 can generate the batch of training samples as further described in reference to FIG. 5 and FIG. 8.

[0153] The computer system 416 can also determine a token batch 714 that includes a plurality’ of tokens. The token batch 714 can include distinct tokens, state-related tokens, appearance threshold tokens, and tokens sampled by similarity. The computer system 416 can generate the token batch 714 as described herein.

[0154] During the embedding selection phase 720, after generating the token batch 714, the computer system 416 can generate a token embedding batch 726 using the token batch 714 and a plurality of stored token embeddings 722. The plurality of stored token embeddings 722 can include token embeddings that are stored in primary processor managed memory. The plurality of stored token embeddings 722 can include a token embedding for each potential token.

[0155] The computer system 416 can identify token embeddings in the plurality' of stored token embeddings 722 that are associated with tokens in the token batch 714. For example, the computer sy stem 416 can identify that a first token in the token batch 714 corresponds to a first stored token embedding 724 in the plurality of stored token embeddings 722. The computer system 416 can obtain the first stored token embedding 724 and include the first stored token embedding 724 into the token embedding batch 726. The computer system 416 can obtain a stored token embedding for each token in the token batch 714 and include the obtained stored token embeddings in the token embedding batch 726. The token embedding batch 726 can include a same number of entries as the token batch 714.3478958504V 1PATENT Attorney Docket No.: 079900-1472383 Client Reference No.: 9459WO01

[0156] During the model training phase 730, after generating the token embedding batch 726, the computer system 416 can load the batch of training samples 712 and the token embedding batch 726 into a secondary processor managed memory.

[0157] The computer system 416 can train a machine learning model 732 using the batch of training samples 712 and the token embedding batch 726. The machine learning model 732 can be a machine learning language model (LLM). The machine learning model 732 can include a transformer.

[0158] A transformer can be a type of neural network architecture that can process an input data sequence by utilizing a self-attention mechanism to understand relationships and contexts between different elements within the input sequence, allowing the transformer to effectively learn long-range dependencies. Transformers can be trained using training samples and can later be utilized to generate predictions using access request data.

[0159] During training of the machine learning model 732, the machine learning model 732 can leam updated token embeddings 734 based on the batch of training samples 712 and the token embedding batch 726. The updated token embeddings 734 can correspond to the same tokens as the token embedding batch 726.

[0160] During the update embeddings phase 740, after training the machine learning model 732, the computer system 416 can update stored token embeddings in the plurality of stored token embeddings 722 using the updated token embeddings 734. The computer system 416 can use the updated token embeddings 734 learned by the machine learning model 732 to update the token embeddings that are stored in the primary processor managed memory'.

[0161] As an example, the computer system 416 can obtain a first updated token embedding 744 in the updated token embeddings 734. The first updated token embedding 744 can correspond to the first stored token and the corresponding first stored token embedding 724. The computer system 416 can update the first stored token embedding 724 to be the first updated token embedding 744. For example, the computer system 416 can write the first updated token embedding 744 into the memory location occupied by the first stored token embedding 724.3578958504V 1PATENT Attorney Docket No.: 079900-1472383 Client Reference No.: 9459WO01 D. Example training process

[0162] A computer system (e.g., computer system 416) can train the machine learning model using batches of training samples and token embedding batches. The computer system 416 can generate the batches of training samples from a training set. The computer system 416 can generate the token embedding batches based on the batches of training samples.

[0163] FIG. 8 shows a flow diagram illustrating a token batching and model training method according to embodiments. The method illustrated in FIG. 8 can be performed by the computer system 416.

[0164] At step 802, the computer system 416 can receive a training set of training samples. The computer system 416 can obtain the training set of training samples from the data database 414. The training set of training samples can include one or more training samples. Each training sample can be represented by a set of tokens and can include an outcome of a request. The output of the request can be, for example, an outcome of an access request.

[0165] As an illustrative example, the computer system 416 can obtain a training set of training samples that includes 1000 training samples. Each training sample can include data that relates to an access request. The access requests can be requests to access one or more resources. An exemplary training sample can include the tokens as follows:[account_number, date, time, resource_provider_identifier, amount, outcome_of_request]

[0166] A first training sample in the training set of training samples can include the following tokens:[1234567801234567, 2025-01-01, 10:00 AM, RP-001, 8.99 USD, approved]

[0167] At step 804. after obtaining the training set of training samples, the computer system 416 can determine a batch of training samples from the training set of training samples. The computer system 416 can determine one or more batches of training samples from the training set of training samples.

[0168] The computer system 416 can separate the training samples from the training set into the one or more batches of training samples. The computer system 416 can determine the number of batches of training samples and the number of training samples to include in a 3678958504V 1PATENT Attorney Docket No.: 079900-1472383 Client Reference No.: 9459WO01 batch of training samples based on a batch count hyperparameter and / or a batch size hyperparameter.

[0169] As an illustrative example, the computer system 416 can store a batch size hyperparameter with a value of 200. The computer system 416 can generate 5 different batches of training samples from the 1000 training samples in the training set based on the batch size hyperparameter. Each batch of training samples can include 200 training samples.

[0170] It is understood that the batch size can be a different value and can be used to generate smaller or larger training sample batches. For example, the computer system 416 can obtain 100,000 training samples in the training set. The computer system 416 can separate the 100,000 training samples into 10 training samples based on a batch size hyperparameter with a value of 10,000. The computer system 416 can utilize any suitable batch size hyperparameter (e g., at least 800. 1,000, 2,000. etc ).

[0171] The computer system 416 can generate the batches of training samples from the training samples in the training set in any suitable manner. For example, the computer system 416 can randomly sample the training samples in the training set to generate the batches of training samples.

[0172] After step 804, the computer system 416 can iteratively perform steps 806-814 for each batch training samples.

[0173] At step 806. the computer system 416 can determine a set of distinct tokens from the batch of training samples. The set of distinct tokens can include all tokens that are included in the training samples of the batch of training samples. The computer system 416 can evaluate each training sample in the batch of training samples and can include the tokens in each training sample into the set of distinct tokens. If a training sample includes a token that is already included in the set of distinct tokens, then the computer system 416 can skip the token an continue to the next token.

[0174] At step 808, the computer system 416 can determine state-related tokens. The state-related tokens can correspond to possible outcomes of requests that are included in the training samples in the batch of training samples. The computer system 416 can identify each possible outcome of a request included in the batch of training samples and include each possible outcome of a request in a set of possible outcomes of requests. For example, the3778958504V 1PATENT Attorney Docket No.: 079900-1472383 Client Reference No.: 9459WO01 outcomes of requests can be included in each training sample as the last token in the training sample. The computer system 416 can determine each unique outcome of requests token in the training samples in the batch of training samples.

[0175] As an illustrative example, the computer system 416 can have the batch of training samples that includes 200 training samples. The computer system 416 can iterate through each of the 200 training samples and add the last token of a current training sample to the set of state-related tokens if the last token does not match a token already included in the state-related tokens. For example, the computer system 416 can determine that the batch of training samples, which includes 200 training samples, includes 2 state-related tokens of “approved” and “not approved.”

[0176] In some embodiments, the computer system 416 can determine a set of appearance threshold tokens and a set of tokens sampled by similarity, respectively described in reference to step 810 and step 812.

[0177] At step 810, the computer system 416 can determine a set of appearance threshold tokens. An appearance threshold token can be a token that appears into the training set more than a threshold number of times or is in the top N number of appearing tokens.

[0178] In some embodiments, the computer system 416 can determine a number of times that each token appears in the training set. The computer system 416 can determine a predetermined number of most common tokens (e.g., 10, 50, 400, 1,000, etc.) in the training set based on the number of times that each token appears in the training set. The computer system 416 can include the predetermined number of most common tokens into the set of appearance threshold tokens.

[0179] In other embodiments, the computer system 416 can determine the tokens in the training set that occur more than a threshold number of times (e.g., more than 10 times, 100 times, 1,000 times, etc.).

[0180] At step 812, the computer system 416 can determine a set of tokens sampled by similarity. The computer system 416 can determine the set of tokens sampled by similarity' using a clustering process. The computer system 416 can determine the set of tokens sampled by similarity as described in further detail in FIG. 10 and FIG. 11.3878958504V 1PATENT Attorney Docket No.: 079900-1472383 Client Reference No.: 9459WO01

[0181] The computer system 416 can generate a token batch that includes a plurality7of tokens that can be utilized to train the machine learning model in conjunction with the batch of training samples. FIG. 9 shows a diagram illustrating an example token batch 902 according to embodiments. The token batch 902 includes a set of distinct tokens 904, state-related tokens 906, a set of appearance threshold tokens 908, and a set of tokens sampled by similarity 910. The token batch 902 can include any number of total tokens that include at least every token that is included in the batch of training samples.

[0182] Returning to FIG. 8, at step 814, after generating the set of distinct tokens and the state-related tokens and, in some embodiments, the set of appearance threshold tokens and the set of tokens sampled by similarity7, the computer system 416 can generate a token embedding batch. The computer sy stem 416 can generate the token embedding batch from token embeddings that correspond to the tokens included in the set of distinct tokens, the state-related tokens, the set of appearance threshold tokens, and the set of tokens sampled by similarity7.

[0183] The computer system 416 can store token embeddings that are associated with each token. The computer system 416 can identify the token embeddings that are associated with the tokens included in the set of distinct tokens and the state-related tokens and, in some embodiments, the set of appearance threshold tokens and the set of tokens sampled by similarity7. The computer system 416 can compile the token embeddings into a token embedding batch.

[0184] At step 816, after generating the token embedding batch, the computer system 416 can load the batch of training samples and the token embedding batch into a memory of a processor of the computer system 416. In some embodiments, the computer system 416 can load the batch of training samples and the token embedding batch into a secondary processor (e.g., a GPU) managed memory7.

[0185] At step 818, after loading the batch of training samples and the token embedding batch, the computer system 416, in conjunction with the processor, can train the machine learning language model using the batch of training samples and the token embedding batch. During training of the machine learning model, the machine learning model can leam updated token embeddings for the token embeddings, which can be used to update the stored token embeddings.3978958504V 1PATENT Attorney Docket No.: 079900-1472383 Client Reference No.: 9459WO01

[0186] After training the machine learning model using the batch of training samples and the token embedding batch, the computer system 416 can unload the batch of training samples and the token embedding batch from the secondary processor managed memory.

[0187] In some embodiments, the computer system 416 can determine a second batch of training samples from the training set of training samples. The computer system 416 can determine a second set of distinct tokens from the second batch of training samples and can determine second state-related tokens. The computer system 416 can generate a second token embedding batch from token embeddings of the second set of distinct tokens and the second state-related tokens. After obtaining the second token embedding batch and the second batch of training samples, the computer system 416 can load the second batch of training samples and the second token embedding batch into the secondary processor managed memory. The computer system 416 can train the machine learning language model using the second batch of training samples and the second token embedding batch.IV. TOKEN SAMPLING

[0188] After determining a set of distinct tokens, state-related tokens, and a set of appearance threshold tokens, the computer system 416 can generate a set of token embeddings sampled by similarity to tokens included in the training samples in the batch of training samples. The computer system 416 can determine tokens that are not currently included in a token batch that are similar to tokens that are included in the token batch. The system computer 416 can generate a set of token embeddings sampled by similarity from the tokens sampled by similarity. The set of token embeddings sample by similarity' can be included in the token batch.A. Token clustering and sampling process

[0189] A computer system (e.g. computer system 416) can determine a set of tokens sampled by similarity using other already selected tokens for the token batch, which include distinct tokens, state-related tokens, and appearance threshold tokens. The computer system 416 can determine the set of tokens sampled by similarity using a clustering process.

[0190] FIG. 10 shows a diagram illustrating sampling tokens according to similarity' according to embodiments. FIG. 10 includes a legend 1000 that indicates meaning of the different hatching types used in FIG. 10 to indicate different token type categories. The 4078958504V 1PATENT Attorney Docket No.: 079900-1472383 Client Reference No.: 9459WO01 legend 1000 indicates distinct token hatching 1002, state-related token hatching 1004, appearance threshold token hatching 1006, and token sampled by similarity hatching 1008.

[0191] An initial plurality of tokens 1010 can include a set of all tokens with indicated selected tokens. The initial plurality of tokens 1010 can include tokens selected by the system computer 416 to be included in the token batch. The initial plurality of tokens 1010 can include, as an example indicated in FIG. 10, 11 tokens that are selected to be included in the token batch. The selected tokens include a distinct token 1012, a first state-related token 1014. a second state-related token 1016. and an appearance threshold token 1018.

[0192] After obtaining the set of distinct tokens, the state-related tokens, and the set of appearance threshold tokens, the computer system 416 can determine the a of tokens sampled by similarity using a clustering process. The clustering process is indicated in steps 1020. 1030, 1040, and 1050.

[0193] At step 1020, the computer system 416 can sample set of candidate tokens in a token type category’. A token type category' can indicate a ty pe of token for the tokens in the token batch. Token ty pes include distinct tokens, state-related tokens, appearance threshold tokens, and tokens sampled by similarity.

[0194] The computer system 416 can sample the set of candidate tokens in the token type category of distinct tokens from the initial plurality of tokens 1010. The computer system 416 can include at least one token of the token type category that is in the token batch into the set of candidate tokens.

[0195] For example, the computer system 416 can randomly sample the initial plurality of tokens 1010 for the token type category of distinct tokens to obtain the set of candidate tokens. The computer system 416 can sample a predetermined number of candidate tokens. For example, the computer system 416 can randomly select 100 distinct tokens, 1,000 distinct tokens, 100,000 distinct tokens, etc. to included into the set of candidate tokens. The computer system 416 can also include the first distinct token 1012 into the set of candidate tokens.4178958504V 1PATENT Attorney Docket No.: 079900-1472383 Client Reference No.: 9459WO01

[0196] The set of candidate tokens can include, as depicted in step 1020 of FIG. 10, the first distinct token 1012, a first candidate token 1022, and a plurality of other candidate tokens.

[0197] In some embodiments, the computer system 416 can perform the clustering process for each different token type category to determine tokens sampled by similarity by token type categories.

[0198] At step 1030, after obtaining the set of candidate tokens, the computer system 416 can cluster the tokens of the set of candidate tokens into clusters. The computer system 416 can cluster the set of candidate tokens using any suitable clustering technique, where each candidate token is represented as a node in a graph. For example, the computer system 416 can cluster the nodes using a centroid based clustering process, a k-means clustering process, or other clustering process that can cluster nodes in a graph to form one or more communities.

[0199] During the clustering process, the first distinct token 1012 can be treated as an anchor token. The set of candidate tokens can be clustered into one or more clusters based on the properties of the tokens themselves. The computer system 416 can determine one or more centroids for the one or more clusters. The clusters can be clustered around the centroids. For example, the computer system 416 can determine a first centroid 1032 and a second centroid 1034. The computer system 416 can cluster the tokens around the first centroid 1032 and the second centroid 1034.

[0200] As illustrated in step 1030 in FIG. 10, the first distinct token 1012 and the first candidate token 1022 can be clustered around the second centroid 1034. The first distinct token 1012 and the first candidate token 1022 can be included in the same cluster.

[0201] At step 1040, after clustering the set of candidate tokens, the computer system 416 can sample one or more selected tokens based on the clustering process. The computer system 416 can sample (e.g., randomly select) a candidate token that is in the same cluster as the anchor token (e.g., the first distinct token 1012). For example, the computer system 416 can select the first candidate token 1022 to be a first selected token 1042.

[0202] In some embodiments, at step 1050, after sampling one or more selected tokens, the computer system 416 can repeat the clustering process, update the centroids, and4278958504V 1PATENT Attorney Docket No.: 079900-1472383 Client Reference No.: 9459WO01 select one or more additional selected tokens. The computer system 416 can repeat the clustering process any suitable number of times (e.g., 2 times, 10 times. 30 times, until N number of candidate tokens are selected, etc.).

[0203] For example, in step 1050, the computer system 416 can repeat the clustering process, update the centroids, and can select a second selected token 1052. The computer system 416 can maintain a list of all selected tokens.

[0204] After obtaining the selected tokens, the computer system 416 can add the selected tokens into the token batch. The selected tokens can be a set of tokens sampled by similarity. For example, the computer system can include the set of tokens sampled by similarity into an updated plurality of tokens 1060. The updated plurality of tokens 1060 indicates all tokens that are included in the token batch. For example, the updated plurality of tokens 1060 indicates the distinct tokens, the state-related tokens, the set of appearance threshold tokens, and the set of tokens sampled by similarity. These tokens can make up the token batch.

[0205] For example, the updated plurality of tokens 1060 can tokens included in the token batch including the distinct token 1012, the first state-related token 1014, the second state-related token 1016, the appearance threshold token 1018, the first selected token 1042 (e.g., a first token sampled by similarity), and the second selected token 1052 (e.g.. a second token sampled by similarity).B. Generating a set of token embeddings sampled by similarity

[0206] A computer system (e.g. computer system 416) can determine a set of tokens sampled by similarity based on other tokens in the token batch. The computer 416 can use the set of tokens sampled by similarity to generate a set of token embeddings sampled by similarity to included into the token embedding batch that is used to training the machine learning model.

[0207] FIG. 11 shows a flow diagram illustrating a token sampling method according to embodiments. The method illustrated in FIG. 11 can be performed by the computer system 416.4378958504V 1PATENT Attorney Docket No.: 079900-1472383 Client Reference No.: 9459WO01

[0208] At step 1102, the computer system 416 can sample a set of candidate tokens in a token type category. The computer system 416 can sample the set of candidate tokens from all available tokens. To sample the set of candidate tokens, the computer system 416 can randomly select a predetermined number of tokens of the token type category7(e.g., distinct tokens).

[0209] The computer system 416 can include one or more of the tokens corresponding of the token type category that are in the token batch into the set of candidate tokens as anchor tokens. For example, the computer system 416 can select a distinct token from the token batch to include in the set of candidate token such that the distinct token is clustered along with the other candidate tokens. The anchor token can aid the computer system 416 in selected tokens in step 1106.

[0210] At step 1104, after sampling the set of candidate tokens, the computer system 416 can cluster the tokens of the set of candidate tokens into clusters. The computer system 416 can cluster the candidate tokens, as nodes, around determined centroids to form clusters. The computer system 416 can perform any suitable clustering process. On such clustering process can include k-means clustering. K-means clustering can be a machine learning algorithm that groups data points into clusters based on their similarity. As additional examples, the clustering process can include a hierarchical clustering process or can utilize Gaussian mixture models.

[0211] At step 1106, after clustering the set of candidate tokens, the computer system 416 can sample selected tokens based on the anchor token in a cluster of the clusters. The anchor token that was clustered along with the set of candidate tokens can be included in a particular cluster of the clusters. The computer system 416 can randomly select one or more selected tokens in the same cluster as the anchor token. The one or more selected tokens can be tokens sampled by similarity.

[0212] At step 1108, after obtaining the selected tokens, the computer system 416 can determine token embeddings that are associated with the selected tokens. Each token can correspond to a token embedding.

[0213] At step 1110, the computer system 416 can include the token embeddings associated with the selected tokens into a set of token embeddings sampled by similarity. The4478958504V 1PATENT Attorney Docket No.: 079900-1472383 Client Reference No.: 9459WO01 set of token embeddings sampled by similarity can be included into the token embedding batch.

[0214] The computer system 416 can perform steps 1102-1110 for each different token type category’. For example, the computer system 416 can first obtain token embeddings sampled by similarity using the distinct tokens in the token batch. The computer system 416 can then obtain token embeddings sampled by similarity' using the appearance threshold tokens in the token batch.V. ADVANTAGES

[0215] Embodiments of the disclosure have a number of advantages. For example, embodiments provide for systems and methods of a dynamic token loading that allows the for a machine learning model to process data, such as access request data, that is represented by billions of tokens. The machine learning model can leam and update a subset of token embeddings during training.

[0216] Embodiments solve the technical problem of having a large number tokens available in the machine learning model (e.g., one billion) such that they cannot all be loaded in a secondary processor managed memory’ at the same time. Embodiments provide for a technical solution to this technical problem with an efficient data and token batching process to train batches of data at a time using the secondary’ processor by’ selectively selecting token embeddings to provide to the secondary’ processor during a training batch.

[0217] Further, systems according to embodiments cannot utilize typical batching techniques that are utilized for LLMs, such as subword tokenization, with data such as access request data that does not have prefixes or suffixes to convert into sub words. Embodiment provide for a batching technique that can be utilized with data that cannot be split into subwords to save on the number of loaded tokens.VI. COMPUTER SYSTEM

[0218] Any of the computer systems mentioned herein may utilize any suitable number of subsystems. Examples of such subsystems are shown in FIG. 12 in computer system 1200. In some embodiments, a computer system includes a single computer apparatus, where the subsystems can be the components of the computer apparatus. In other4578958504V 1PATENT Attorney Docket No.: 079900-1472383 Client Reference No.: 9459WO01 embodiments, a computer system can include multiple computer apparatuses, each being a subsystem, with internal components. A computer system can include desktop and laptop computers, tablets, mobile phones and other mobile devices.

[0219] The subsystems shown in FIG. 12 are interconnected via a system bus 1224. Additional subsystems such as a printer 1208, keyboard 1216, storage device(s) 1218, monitor 1222 (e.g., a display screen, such as an LED), which is coupled to display adapter 1212, and others are shown. Peripherals and input / output (I / O) devices, which couple to I / O controller 1202, can be connected to the computer system by any number of means known in the art such as input / output (I / O) port 1214 (e.g., USB, FireWire®). For example, I / O port 1214 or external interface 1220 (e.g., Ethernet, Wi-Fi, etc.) can be used to connect computer system 1200 to a wide area network such as the Internet, a mouse input device, or a scanner. The interconnection via system bus 1224 allows the central processor 1206 to communicate with each subsystem and to control the execution of a plurality of instructions from system memory 1204 or the storage device(s) 1218 (e.g., a fixed disk, such as a hard drive, or optical disk), as well as the exchange of information between subsystems. The system memory 1204 and / or the storage device(s) 1218 may embody a computer readable medium. Another subsystem is a data collection device 1210, such as a camera, microphone, accelerometer, and the like. Any of the data mentioned herein can be output from one component to another component and can be output to the user.

[0220] A computer system can include a plurality of the same components or subsystems, for example, connected together by external interface 1220, by an internal interface, or via removable storage devices that can be connected and removed from one component to another component. In some embodiments, computer systems, subsystem, or apparatuses can communicate over a network. In such instances, one computer can be considered a client and another computer a server, where each can be part of a same computer system. A client and a server can each include multiple systems, subsystems, or components. In various embodiments, methods may involve various numbers of clients and / or servers, including at least 10, 20, 50, 100, 200, 500, 1,000, or 10,000 devices. Methods can include various numbers of communication messages between devices, including at least 100, 200, 500, 1,000. 10.000, 50.000, 100.000, 500,00, or one million communication4678958504V 1PATENT Attorney Docket No.: 079900-1472383 Client Reference No.: 9459WO01 messages. Such communications can involve at least 1 MB, 10 MB, 100 MB, 1 GB, 10 GB, or 100 GB of data.

[0221] Aspects of embodiments can be implemented in the form of control logic using hardware circuitry (e.g., an application specific integrated circuit or field programmable gate array) and / or using computer software stored in a memory with a generally programmable processor in a modular or integrated manner, and thus a processor can include memory storing software instructions that configure hardware circuitry, as well as an FPGA with configuration instructions or an ASIC. As used herein, a processor can include a singlecore processor, multi-core processor on a same integrated chip, or multiple processing units on a single circuit board or networked, as well as dedicated hardware. Based on the disclosure and teachings provided herein, a person of ordinary skill in the art will know and appreciate other ways and / or methods to implement embodiments of the present disclosure using hardware and a combination of hardware and software.

[0222] Any of the software components or functions described in this application may be implemented as software code to be executed by a processor using any suitable computer language such as. for example, Java, C, C++, C#, Objective-C, Swift, or scripting language such as Perl or Python using, for example, conventional or object-oriented techniques. The software code may be stored as a series of instructions or commands on a computer readable medium for storage and / or transmission. A suitable non-transitory computer readable medium can include random access memory (RAM), a read only memory (ROM), a magnetic medium such as a hard-drive or a floppy disk, or an optical medium such as a compact disk (CD) or DVD (digital versatile disk) or Blu-ray disk, flash memory, and the like. The computer readable medium may be any combination of such devices. In addition, the order of operations may be re-arranged. A process can be terminated when its operations are completed but could have additional steps not included in a figure. A process may correspond to a method, a function, a procedure, a subroutine, a subprogram, etc. When a process corresponds to a function, its termination may correspond to a return of the function to the calling function or the main function.

[0223] Such programs may also be encoded and transmitted using carrier signals adapted for transmission via wired, optical, and / or wireless networks conforming to a variety of protocols, including the Internet. As such, a computer readable medium may be created4778958504V 1PATENT Attorney Docket No.: 079900-1472383 Client Reference No.: 9459WO01 using a data signal encoded with such programs. Computer readable media encoded with the program code may be packaged with a compatible device (e.g., as firmware) or provided separately from other devices (e.g., via Internet download). Any such computer readable medium may reside on or within a single computer product (e.g., a hard drive, a CD, or an entire computer system), and may be present on or within different computer products within a system or network. A computer system may include a monitor, printer, or other suitable display for providing any of the results mentioned herein to a user.

[0224] Any of the methods described herein may be totally or partially performed with a computer system including one or more processors, which can be configured to perform the steps. Any operations performed with a processor may be performed in real-time. The term “real-time’' may refer to computing operations or processes that are completed within a certain time constraint. As examples, a time constraint may be 30 seconds. 1 minute, 10 minutes, 30 minutes, 1 hour, 4 hours, 1 day, or 7 days. Thus, embodiments can be directed to computer systems configured to perform the steps of any of the methods described herein, potentially with different components performing a respective step or a respective group of steps. Although presented as numbered steps, steps of methods herein can be performed at a same time or at different times or in a different order. Additionally, portions of these steps may be used with portions of other steps from other methods. Also, all or portions of a step may be optional. Additionally, any of the steps of any of the methods can be performed with modules, units, circuits, or other means of a system for performing these steps.

[0225] Although the steps in the flowcharts and process flows described above are illustrated or described in a specific order, it is understood that embodiments of the invention may include methods that have the steps in different orders. In addition, steps may be omitted or added and may still be w ithin embodiments of the invention.

[0226] The specific details of particular embodiments may be combined in any suitable manner without departing from the spirit and scope of embodiments of the disclosure. However, other embodiments of the disclosure may be directed to specific embodiments relating to each individual aspect, or specific combinations of these individual aspects.

[0227] The above description of example embodiments of the present disclosure has been presented for the purposes of illustration and description. It is not intended to be 4878958504V 1PATENT Attorney Docket No.: 079900-1472383 Client Reference No.: 9459WO01 exhaustive or to limit the disclosure to the precise form described, and many modifications and variations are possible in light of the teaching above.

[0228] A recitation of "a", "an" or "the" is intended to mean "one or more" unless specifically indicated to the contrary. The use of “or” is intended to mean an “inclusive or,” and not an “exclusive or” unless specifically indicated to the contrary. Reference to a “first” component does not necessarily require that a second component be provided. Moreover, reference to a “first” or a “second” component does not limit the referenced component to a particular location unless expressly stated. The term “based on” is intended to mean “based at least in part on.”

[0229] The claims may be drafted to exclude any element which may be optional. As such, this statement is intended to serve as antecedent basis for use of such exclusive terminology as “solely”, “only”, and the like in connection with the recitation of claim elements, or the use of a “negative” limitation.

[0230] All patents, patent applications, publications, and descriptions mentioned herein are incorporated by reference in their entirety for all purposes. None is admitted as prior art. Where a conflict exists between the instant application and a reference provided herein, the instant application shall dominate.4978958504V 1

Claims

PATENT Attorney Docket No.: 079900-1472383 Client Reference No.: 9459WO01 WHAT IS CLAIMED IS:

1. A method of training a machine learning model including samples of outcomes of requests determined by previous input data, the method comprising performing by a computer sy stem:receiving a training set of training samples, each training sample represented by a set of tokens and including an outcome of a request;determining a batch of training samples from the training set of training samples;determining a set of distinct tokens from the batch of training samples; determining state-related tokens, wherein the state-related tokens correspond to possible outcomes of requests;generating a token embedding batch from token embeddings of the set of distinct tokens and the state-related tokens;loading the batch of training samples and the token embedding batch into a memory of a processor of the computer system; andtraining, by the processor, the machine learning model using the batch of training samples and the token embedding batch.

2. The method of claim 1, wherein the token embedding batch further comprises: a set of appearance threshold token embeddings and a set of token embeddings sampled by similarity.

3. The method of claim 2 further comprising:determining the set of token embeddings sampled by similarity from a set of tokens sampled by similarity7; anddetermining the set of appearance threshold token embeddings from a set of appearance threshold tokens.

4. The method of claim 3, wherein determining the set of appearance threshold token embeddings comprises:determining a number of times that each token appears in the training set; determining a number of most common tokens in the training set based on the number of times that each token appears in the training set;5078958504V 1PATENT Attorney Docket No.: 079900-1472383 Client Reference No.: 9459WO01 determining an appearance threshold token embedding for each of the most common tokens; andgenerating the set of appearance threshold token embeddings from the appearance threshold token embeddings for each of the most common tokens.

5. The method of claim 3, wherein determining the set of token embeddings sampled by similarity compnses:sampling a set of candidate tokens in a token ty pe category';clustering tokens of the set of candidate tokens into clusters;sampling selected tokens based on a first distinct token of the set of distinct tokens in a cluster of the clusters; andincluding token embeddings associated with the selected tokens in the set of token embeddings sampled by similarity'.

6. The method of claim 5, wherein token type category includes distinct tokens.

7. The method of claim 5, wherein token type category includes appearance threshold tokens.

8. The method of claim 1, wherein the outcomes of requests indicate authenticated or not authenticated for a request for authentication.

9. The method of claim 1, wherein the machine learning model is trained to determine token embeddings, wherein after training of the machine learning model with the batch of training samples and the token embedding batch, the method further comprises:updating stored token embeddings associated with tokens represented in the token embedding batch.

10. The method of claim 1 , wherein a total number of tokens included in all training sets is greater than one million tokens.

11. The method of claim 1, wherein a token in the set of tokens represents a date, a time, a resource provider identifier, an authorizing entity' identifier, or an interaction type.5178958504V 1PATENT Attorney Docket No.: 079900-1472383 Client Reference No.: 9459WO0112. The method of claim 1, wherein the outcome of the request is included in the set of tokens in each training sample as the last token in the set of tokens.

13. The method of claim 1, wherein the batch of training samples is a first batch of training samples, the set of distinct tokens is a first set of distinct tokens, the state-related tokens are first state-related tokens, the token embedding batch is a first token embedding batch, wherein the method further comprises:unloading the first batch of training samples and the first token embedding batch from the memory;determining a second batch of training samples from the training set of training samples;determining a second set of distinct tokens from the second batch of training samples;determining second state-related tokens;generating a second token embedding batch from token embeddings of the second set of distinct tokens and the second state-related tokens;loading the second batch of training samples and the second token embedding batch into the memory of the processor of the computer system; andtraining, by the processor, the machine learning model using the second batch of training samples and the second token embedding batch.

14. The method of claim 1, wherein each training sample of the training set includes access request data including a user identifier, a resource provider identifier, a time, and a date.

15. The method of claim 1, wherein the token embedding batch includes a subset of token embeddings that correspond to tokens in a vocabulary library of tokens.

16. The method of claim 1, wherein the set of distinct tokens includes all tokens that are included in the training samples of the batch of training samples.

17. The method of claim 1 , wherein generating the token embedding batch from token embeddings of the set of distinct tokens and the state-related tokens comprises:5278958504V 1PATENT Attorney Docket No.: 079900-1472383 Client Reference No.: 9459WO01 determining a distinct token embedding for each distinct token of the set of distinct tokens using a plurality of stored token embeddings; anddetermining a state-related token embedding for each state-related token of the state-related tokens using the plurality of stored token embeddings, wherein the plurality of stored token embeddings are stored in a CPU managed memory'.

18. The method of claim 1 , wherein the machine learning model includes a transformer.

19. A computer product comprising a computer readable medium storing a plurality' of instructions for controlling a computer system to perform operations of any of the methods above.

20. A system comprising:the computer product of claim 19; andone or more processors for executing instructions stored on the computer readable medium.5378958504V 1

Images