Transaction data processing using SEQ2SEQ machine learning model

Abstract

A method includes obtaining time-dependent transaction data. After obtaining time-dependent transaction data, the method includes preprocessing the time-dependent transaction data to form pre-processed time-dependent transaction data. A training dataset is created from the pre-processed time-dependent transaction data. A sequence to sequence machine learning model is trained using the training dataset. Test time-dependent transaction data is input into the trained sequence to sequence machine learning model to obtain predicted time-dependent transaction data.

Description

PATENT Attorney Docket No.: 079900-1461958 Client Reference No.: 8153WO01TRANSACTION DATA PROCESSING USING SEQ2SEQ MACHINE LEARNING MODEL CROSS REFERENCE TO RELATED APPLICATIONS

[0001] None.BACKGROUND

[0002] Traditional algorithmic models may not efficiently capture complex temporal dependencies in transaction time-series data such as time-dependent computer network data, stock market data, and user financial transaction data. Existing algorithmic models may also lack the ability to effectively handle the noisy and non-stationary nature of transaction systems. Lastly, existing algorithmic models have difficulty in capturing the relative importance of past events in predicting future transactions. Traditional algorithmic models can take a number of inputs and make predictions, such that those inputs may be treated equally. This may lead to inaccurate and imprecise predictions when transaction data has time-dependencies associated with them.

[0003] Embodiments of the disclosure address this problem and other problems individually and collectively.SUMMARY

[0004] One embodiment is related to a method comprising: obtaining timedependent transaction data; preprocessing the time-dependent transaction data to form pre-processed time-dependent transaction data; creating a training dataset from the pre-processed time-dependent transaction data; training a sequence to sequence machine learning model using the training dataset; and inputting, test time-dependent transaction data into the trained sequence to sequence machine learning model to obtain predicted time-dependent transaction data.

[0005] Another embodiment is related to a computer comprising: a processor; and a computer-readable medium coupled to the processor, the computer-readable medium comprising code executable by the processor for implementing a methodPATENT Attorney Docket No.: 079900-1461958 Client Reference No.: 8153WO01 comprising: obtaining time-dependent transaction data; preprocessing the timedependent transaction data to form pre-processed time-dependent transaction data; creating a training dataset from the pre-processed time-dependent transaction data; training a sequence to sequence machine learning model using the training dataset; and inputting, test time-dependent transaction data into the trained sequence to sequence machine learning model to obtain predicted time-dependent transaction data.

[0006] Another embodiment is related to a system comprising: a data database that stores time-dependent transaction data; a model database that stores one or more machine learning models; and a model computer in communication with the data database and the model database, the model computer comprising: a processor; and a computer-readable medium coupled to the processor, the computer-readable medium comprising code executable by the processor for implementing a method comprising: obtaining the time-dependent transaction data from the data database; preprocessing the time-dependent transaction data to form pre-processed timedependent transaction data; creating a training dataset from the pre-processed timedependent transaction data; training a sequence to sequence machine learning model using the training dataset; and inputting, test time-dependent transaction data into the trained sequence to sequence machine learning model to obtain predicted timedependent transaction data.

[0007] Further details regarding embodiments of the disclosure can be found in the Detailed Description and the Figures.BRIEF DESCRIPTION OF THE DRAWINGS

[0008] FIG. 1 shows a block diagram of a model system according to embodiments.

[0009] FIG. 2 shows a block diagram of components of a model computer according to embodiments.

[0010] FIG. 3 shows a block diagram of a transaction processing system according to embodiments.PATENT Attorney Docket No.: 079900-1461958 Client Reference No.: 8153WO01

[0011] FIG. 4 shows a diagram illustrating a sequence to sequence machine learning model according to embodiments.

[0012] FIG. 5 shows a flow diagram of illustrating a method of processing and utilizing a sequence to sequence machine learning model according to embodiments.

[0013] FIG. 6 shows a flow diagram of illustrating a method of processing timedependent transaction data according to embodiments.DETAILED DESCRIPTION

[0014] Prior to discussing embodiments of the disclosure, some terms can be described in further detail.

[0015] An “interaction” may include a reciprocal action or influence. An interaction can include a communication, contact, or exchange between parties, devices, and / or entities. Example interactions include a transaction between two parties and a data exchange between two devices. In some embodiments, an interaction can include a user requesting access to secure data, a secure webpage, a secure location, and the like. In other embodiments, an interaction can include a payment transaction in which two devices can interact to facilitate a payment.

[0016] “Interaction data” can include data related to and / or recorded during an interaction. In some embodiments, interaction data can be transaction data.

[0017] “Transaction data” can include interaction data related to a transaction. Transaction data can comprise a plurality of data elements with data values. Transaction data can include payment card transaction data, machine performance data, user financial transaction data (an example of user-specific data), securities market data, etc.

[0018] “Time-dependent transaction data” can include transaction data that is dependent on time. In some cases, time-dependent transaction data can be data that is of the same type (e.g., all stock data). In some cases, transactions can occur over time and may be related to one another. A transaction may explicitly or implicitly relate to another transaction. For example, in the case where a second transaction explicitly relates to a first transaction, a second transaction can include a reference to a prior first transaction. The reference can be an identifier that ties the first and secondPATENT Attorney Docket No.: 079900-1461958 Client Reference No.: 8153WO01 transactions together. The identifier can be in the data for the first and second transactions. In another example, in the case where a second transaction implicitly relates to the first transaction, a first transaction can be for a credit card purchase, while a second transaction can be for a credit card payment. The two transactions can implicitly relate to one another since there is no explicit reference identifier linking the two transactions.

[0019] The term "artificial intelligence model" or "Al model" can include a model that may be used to predict outcomes in order achieve a pre-defined goal. The Al model may be developed using a learning algorithm, in which training data is classified based on known or inferred patterns. An Al model may also be referred to as a "machine learning model" or "predictive model."

[0020] "Machine learning" can include an artificial intelligence process in which software applications may be trained to make accurate predictions through learning. The predictions can be generated by applying input data to a predictive model formed from performing statistical analyses on aggregated data. A model can be trained using training data, such that the model may be used to make accurate predictions. The prediction can be, for example, a classification of an image (e.g., identifying images of cats on the Internet) or as another example, a recommendation (e.g., a movie that a user may like or a restaurant that a consumer might enjoy).

[0021] In some embodiments, a model may be a statistical model, which can be used to predict unknown information from known information. For example, a learning module may be a set of instructions for generating a regression line from training data (supervised learning) or a set of instructions for grouping data into clusters of different classifications of data based on similarity, connectivity, and / or distance between data points (unsupervised learning). The regression line or data clusters can then be used as a model for predicting unknown information from known information. Once model has been built from learning module, the model may be used to generate a predicted output from a new request. A new request may be a request for a prediction associated with presented data. For example, a new request may be a request for classifying an image or for creating a recommendation for a user.

[0022] “Sequence to sequence” (seq2seq) can include a family of machine learning approaches typically used for natural language processing. Sequence toPATENT Attorney Docket No.: 079900-1461958 Client Reference No.: 8153WO01 sequence uses sequence transformation where sequence to sequence models can turn one sequence into another sequence. Sequence to sequence models can map an input sequence into a real-numerical vector by a neural network (e.g., an encoder), then map the vector back to an output sequence using another neural network (e.g., a decoder). A sequence to sequence machine learning model can include an attention mechanism. The encoder and the decoder in a sequence to sequence machine learning model can be transformers.

[0023] “Attention” can include a machine learning method that determines a relative importance of each component in a sequence relative to other components in the sequence. As an example, in natural language processing, importance is represented by soft weights assigned to each word in a sentence. More generally, attention encodes vectors called embeddings (or in some cases token embeddings) across a fixed-width sequence. Unlike hard weights, which are computed during the backwards training pass, soft weights exist only in the forward pass and therefore change with every step of the input. Transformers can implement a parallel attention scheme such as a self-attention mechanism.

[0024] “Preprocessing” can include processing a thing prior to performing a main process on the thing. Preprocessing can include a process of modifying raw data into data that is ready for analysis. Preprocessing can include a variety of methods to ensure that the data is consistent, comparable, and organized. Preprocessing can include data cleaning (e.g., removing incorrect data), data transformation (e.g., normalizing data), data reduction (e.g., performing principal component analysis), data integration (e.g., combining data from multiple sources), feature engineering (e.g., generating new variables), data encoding (e.g., generating embeddings for the data), sequence padding (e.g., normalizing a length of data), and / or other data processing techniques.

[0025] A “processor” may include a device that processes something. In some embodiments, a processor can include any suitable data computation device or devices. A processor may comprise one or more microprocessors working together to accomplish a desired function. The processor may include a CPU comprising at least one high-speed data processor adequate to execute program components for executing user and / or system-generated requests. The CPU may be a microprocessorPATENT Attorney Docket No.: 079900-1461958 Client Reference No.: 8153WO01 such as AMD's Athlon, Duron and / or Opteron; IBM and / or Motorola's PowerPC; IBM's and Sony's Cell processor; Intel's Celeron, Itanium, Pentium, Xeon, and / or XScale; and / or the like processor(s).

[0026] A “memory” may be any suitable device or devices that can store electronic data. A suitable memory may comprise a non-transitory computer readable medium that stores instructions that can be executed by a processor to implement a desired method. Examples of memories may comprise one or more memory chips, disk drives, etc. Such memories may operate using any suitable electrical, optical, and / or magnetic mode of operation.

[0027] A “server computer” may include a powerful computer or cluster of computers. For example, the server computer can be a large mainframe, a minicomputer cluster, or a group of servers functioning as a unit. In one example, the server computer may be a database server coupled to a Web server. The server computer may comprise one or more computational apparatuses and may use any of a variety of computing structures, arrangements, and compilations for servicing the requests from one or more client computers.

[0028] Embodiments provide for improved methods of transaction data analysis (e.g., credit risk assessment) using machine learning techniques. Embodiments utilize a sequence to sequence machine learning model, combined with self-attention mechanisms and relative positional encodings, to generate predictions related to timedependent transaction data. Unlike traditional models, systems and methods according to embodiments are capable of capturing long dependencies in timedependent transaction data such as credit history data, market data, etc. While sequence to sequence machine learning models are known for use in language processing, they are not known to be used to process time-dependent transaction data.

[0029] Traditional transaction data analysis models lack the ability to capture long-term dependencies in time-dependent transaction data such as historical credit data. For example, a keyword frequency analysis might count the occurrences of terms in a document (e.g., a report). For instance, the keyword frequency analysis could determine that the term "profit" appears frequently, but would not account for the positioning of the term in the document (e.g., at the beginning of the documents) or itsPATENT Attorney Docket No.: 079900-1461958 Client Reference No.: 8153WO01 contextual relevance over time. Traditional models struggle with understanding the significance of the varying positions of these terms. Embodiments solve such problems using a sequence to sequence machine learning model that includes the use of self-attention and relative positional encodings. The use thereof addresses this problem by not only processing the data sequentially, but also by understanding the relative significance of the terms in the data.

[0030] The use of self-attention and relative positional encodings allows the model to account for the relative positions of different elements within the timedependent transaction data (e.g., credit history data), rather than relying on their absolute positions. In the context of evaluating credit history data, the model's ability to capture long-term dependencies in credit history data can potentially lead to more accurate credit risk assessments, thereby increasing efficiency and processing speed. For example, systems and methods according to embodiments can result in a smaller total training time than conventional LSTM (long short-term memory) based systems.

[0031] The processed time-dependent transaction data can include payment card transaction data, machine performance data, user specific data such as user financial transaction data, securities market data, etc. In some embodiments, the timedependent transaction data can include credit data, which can include information related to credit lending to a particular entity. The credit data can include payment histones, credit scores, income levels, employment statuses, personal information, interest rates, credit limits, inflation rates, unemployment rates, etc.

[0032] FIG. 1 shows a system 100 according to embodiments of the disclosure. The system 100 comprises a data computer 102, a model computer 104, a model database 106, a downstream computer 108, a transaction data database 110.

[0033] The model computer 104 can be in operative communication with the model database 106, the downstream computer 108, and the transaction data database 110. The data computer 102 can be in operative communication with the data computer 102.

[0034] For simplicity of illustration, a certain number of components are shown in FIG. 1. It is understood, however, that embodiments of the invention may includePATENT Attorney Docket No.: 079900-1461958 Client Reference No.: 8153WO01 more than one of each component. In addition, some embodiments of the invention may include fewer than or greater than all of the components shown in FIG. 1.

[0035] Messages between the devices of the system 100 illustrated in FIG. 1 can be transmitted using a secure communications protocols such as, but not limited to, File Transfer Protocol (FTP); HyperText Transfer Protocol (HTTP); Secure Hypertext Transfer Protocol (HTTPS), SSL, ISO (e.g., ISO 8583) and / or the like. The communications network may include any one and / or the combination of the following: a direct interconnection; the Internet; a Local Area Network (LAN); a Metropolitan Area Network (MAN); an Operating Missions as Nodes on the Internet (OMNI); a secured custom connection; a Wide Area Network (WAN); a wireless network (e.g., employing protocols such as, but not limited to a Wireless Application Protocol (WAP), l-mode, and / or the like); and / or the like. The communications network can use any suitable communications protocol to generate one or more secure communication channels. A communications channel may, in some instances, comprise a secure communication channel, which may be established in any known manner, such as through the use of mutual authentication and a session key, and establishment of a Secure Socket Layer (SSL) session.

[0036] The data computer 102 can include a computer that can obtain and store data. The data computer 102 can obtain and store time-dependent transaction data. For example, the data computer 102 can obtain time-dependent transaction data from a transaction processing network. The data computer 102 can store the timedependent transaction data in the transaction data database 110.

[0037] The model computer 104 can include a computer configured to create, train, maintain, and utilize models. The model computer 104 can create, train, maintain, and utilize machine learning models. The machine learning models can include a sequence to sequence machine learning model. The model computer 104 can obtain time-dependent transaction data from the transaction data database 110. The model computer 104 can utilize the time-dependent transaction data to train the sequence to sequence machine learning model. The model computer 104 can also utilize the time-dependent transaction data to determine a predicted time-dependent transaction data. The model computer 104 can store created machine learning models into the model database 106.PATENT Attorney Docket No.: 079900-1461958 Client Reference No.: 8153WO01

[0038] The model database 106 can include any suitable database. The database may be a conventional, fault tolerant, relational, scalable, secure database such as those commercially available from Oracle™ or Sybase™. The model database 106 can store models such as machine learning models.

[0039] The downstream computer 108 can include a computer or device that is downstream of the model database 106. The downstream computer 108 can obtain predicted time-dependent transaction data from the model computer 104. In some embodiments, the downstream computer 108 can generate a prediction request message comprising test time-dependent transaction data or a test time-dependent transaction data identifier. The downstream computer 108 can provide the predicted request message to the model computer 104. The downstream computer 108 can, in response, receive a prediction response message comprising the predicted timedependent transaction data from the model computer 104.

[0040] The transaction data database 110 can include any suitable database. The database may be a conventional, fault tolerant, relational, scalable, secure database such as those commercially available from Oracle™ or Sybase™. The transaction data database 110 can store time-dependent transaction data. The transaction data database 110 can store hundreds, thousands, millions, etc. of data entries that are related to transactions.

[0041] FIG. 2 shows a block diagram of a model computer 104 according to embodiments. The exemplary model computer 104 may comprise a processor 204. The processor 204 may be coupled to a memory 202, a network interface 206, and a computer readable medium 208. The computer readable medium 208 can comprise a data preprocessing module 208A, a training module 208B, and a prediction module 208C.

[0042] The memory 202 can be used to store data and code. For example, the memory 202 can store time-dependent transaction data, pre-processed timedependent transaction data, test time-dependent transaction data, pre-processed test time-dependent transaction data, sequence to sequence machine learning models, etc. The memory 202 may be coupled to the processor 204, and may comprise any combination of volatile and / or non-volatile memory, such as RAM, DRAM, ROM, flash, or any other suitable memory device.PATENT Attorney Docket No.: 079900-1461958 Client Reference No.: 8153WO01

[0043] The computer readable medium 208 may comprise code, executable by the processor 204, for performing a method comprising: obtaining time-dependent transaction data; preprocessing the time-dependent transaction data to form pre-processed time-dependent transaction data; creating a training dataset from the pre-processed time-dependent transaction data; training a sequence to sequence machine learning model using the training dataset; and inputting, test time-dependent transaction data into the trained sequence to sequence machine learning model to obtain predicted time-dependent transaction data.

[0044] The data preprocessing module 208A may comprise code or software, executable by the processor 204, for preprocessing data. The data preprocessing module 208A, in conjunction with the processor 204, can process data such that the data is in a format for further use.

[0045] The data preprocessing module 208A, in conjunction with the processor 204, can preprocess data by performing data cleaning, (e.g., removing incorrect data), data transformation (e.g., normalizing data), data reduction (e.g., performing principal component analysis), data integration (e.g., combining data from multiple sources), feature engineering (e.g., generating new variables), data encoding (e.g., generating embeddings for the data), sequence padding (e.g., normalizing a length of data), and / or other data processing techniques.

[0046] The data preprocessing module 208A, in conjunction with the processor 204, can preprocess time-dependent transaction data. For example, the data preprocessing module 208A, in conjunction with the processor 204, can perform data encoding on any categorical variables in the time-dependent transaction data, such as a transaction type. The data preprocessing module 208A, in conjunction with the processor 204, can also, for example, perform sequence padding on a sequence of time-dependent transaction data such that the sequence of time-dependent transaction data can be input into a sequence to sequence machine learning model.

[0047] The training module 208B can include may comprise code or software, executable by the processor 204, for training machine learning models. The training module 208B, in conjunction with the processor 204, can train a sequence to sequence machine learning model using time-dependent transaction data. In some embodiments, the time-dependent transaction data used to train the sequence toPATENT Attorney Docket No.: 079900-1461958 Client Reference No.: 8153WO01 sequence machine learning model can be associated with a ground truth. The ground truth can be a known next transaction in a sequence of transactions. The known next transaction can be compared to the next predicted transaction to adjust the weights of the sequence to sequence machine learning model to improve performance.

[0048] The training module 208B, in conjunction with the processor 204, can train a sequence to sequence machine learning model that includes an encoder and a decoder. The encoder can read an input sequence and produce a context vector of the input sequence. The decoder can utilize the context vector to generate an output sequence. The training module 208B, in conjunction with the processor 204, can train the encoder and decoder to accurately generate an output sequence based on the input sequence. Specifically, the training module 208B, in conjunction with the processor 204, can train the sequence to sequence machine learning model to predict probability distributions of a next token in a sequence given previous context sequence (e.g., a source sequence and previously predicted target tokens). At each step, the training module 208B, in conjunction with the processor 204, can maximize the probability that the model assigns to the correct token. To maximize the probability of the correct token being assigned, the training module 208B, in conjunction with the processor 204, can minimize a loss function, such as a cross-entropy loss function, which indicates how well the machine learning model is performing. At each training step, the training module 208B, in conjunction with the processor 204, can maximize the probability that the model assign to the correct token as the next token in the output sequence.

[0049] The prediction module 208C can include may comprise code or software, executable by the processor 204, for generating predictions. The prediction module 208C, in conjunction with the processor 204, can input test data (e.g., test time-dependent transaction data) into the trained sequence to sequence machine learning model to generate an output sequence. The output sequence includes a sequence of predicted tokens.

[0050] The network interface 206 may include an interface that can allow the model computer 104 to communicate with external computers. The network interface 206 may enable the model computer 104 to communicate data to and from another device (e.g., the data computer 102, the model database 106, and the downstreamPATENT Attorney Docket No.: 079900-1461958 Client Reference No.: 8153WO01 computer 108, etc.). Some examples of the network interface 206 may include a modem, a physical network interface (such as an Ethernet card or other Network Interface Card (NIC)), a virtual network interface, a communications port, a Personal Computer Memory Card International Association (PCMCIA) slot and card, or the like. The wireless protocols enabled by the network interface 206 may include Wi-Fi™. Data transferred via the network interface 206 may be in the form of signals which may be electrical, electromagnetic, optical, or any other signal capable of being received by the external communications interface (collectively referred to as “electronic signals” or “electronic messages”). These electronic messages that may comprise data or instructions may be provided between the network interface 206 and other devices via a communications path or channel. As noted above, any suitable communication path or channel may be used such as, for instance, a wire or cable, fiber optics, a telephone line, a cellular link, a radio frequency (RF) link, a WAN or LAN network, the Internet, or any other suitable medium.

[0051] FIG. 3 shows a system 300 according to embodiments of the disclosure. The system illustrated in FIG. 3 can be used to generate and store time dependent transaction data.

[0052] The system 300 comprises the model computer 104, the model database 106, the downstream computer 108, and the transaction data database 110. The system 300 also comprises a user device 122, an access device 112, a resource provider computer 114, a transport computer 116, a network processing computer 118, and an authorizing entity computer 120.

[0053] The model computer 104 can be in operative communication with the model database 106, the downstream computer 108, and the network processing computer 118. The user device 122 can be in operative communication with the access device 112 and the resource provider computer 114. The resource provider computer 114 can be in operative communication with the access device 112 and the transport computer 116. The network processing computer 118 can be in operative communication with the transport computer 116 and the authorizing entity computer 120.

[0054] For simplicity of illustration, a certain number of components are shown in FIG. 3. It is understood, however, that embodiments of the invention may includePATENT Attorney Docket No.: 079900-1461958 Client Reference No.: 8153WO01 more than one of each component. In addition, some embodiments of the invention may include fewer than or greater than all of the components shown in FIG. 3.

[0055] Messages between the devices of the system 300 illustrated in FIG. 3 can be transmitted in similar manners to the devices of the system 100 and will not be repeated here.

[0056] The user device 122 can include one or more computers, portable computers, laptop computers, tablet computers, mobile devices, cellular phones, wearable devices (e.g., watches, glasses, lenses, clothing, etc.), personal digital assistants (PDAs), Internet of Things (loT) devices, and / or the like. The user device 122 can initiate interactions (e.g., transactions) with resource provider computers and / or access devices. For example, in some embodiments, the user device 122 can select one or more items for the interaction at a resource provider location (e.g., a grocery store). During checkout, the user can be instructed to tap (e.g., bring into nearfield communication range) the user device 122 against the access device 112. In other embodiments, the user device 122 can connect to a website hosted by the resource provider computer 114 to communicate with the resource provider computer 114. For example, the during checkout, the user can be instructed to select a checkout button for the items in a digital shopping cart.

[0057] The access device 112 can include a device operated by a resource provider. The access device 112, for example, can include a mobile device, a POS terminal, a laptop, etc. The access device 112 can communicate with another device (e.g., a user device 122) to perform an interaction. During the interaction, the access device 112 can receive credentials from the user device and can provide interaction data to the resource provider computer 114 for authorization of the interaction. In some embodiments, the access device 112 can generate an authorization request message comprising at least the interaction data. The access device 112 can provide the authorization request message to the resource provider computer 114.

[0058] The resource provider computer 114 can include any suitable computational apparatus operated by a resource provider (e.g., a merchant). In some embodiments, the resource provider computer 114 may include one or more server computers that may host one or more websites associated with the resource provider (e.g., a merchant). In some embodiments, the resource provider computer 114 mayPATENT Attorney Docket No.: 079900-1461958 Client Reference No.: 8153WO01 be configured to send data to the network processing computer 118 via the transport computer 116 as part of a payment verification and / or authentication process for a transaction between the user (e.g., consumer) and the resource provider. The resource provider computer 114 may also be configured to generate authorization request messages for transactions between a resource provider and a user, and route the authorization request messages to the authorizing entity computer 120 for transaction processing.

[0059] The transport computer 116 can include a server computer. The transport computer 116 may be associated with an acquirer, which may be an entity (e.g., a commercial bank) that has a business relationship with a particular merchant or other entity. Some entities can perform both issuer and acquirer functions. Some embodiments may encompass such single entity issuer-acquirers.

[0060] The network processing computer 118 can include a server computer. The network processing computer 118 may be disposed between the transport computer 116 and the authorizing entity computer 120. The network processing computer 118 may include data processing subsystems, networks, and operations used to support and deliver authorization services, exception file services, and clearing and settlement services. For example, the network processing computer 118 may comprise a server coupled to a network interface (e.g., by an external communication interface), and databases of information. The network processing computer 118 may be representative of a transaction processing network. An exemplary transaction processing network may include VisaNet™. Transaction processing networks such as VisaNet™ are able to process credit card transactions, debit card transactions, and other types of commercial transactions. VisaNet™, in particular, includes a VIP system (Visa Integrated Payments system) which processes authorization requests and a Base II system which performs clearing and settlement services. The network processing computer 118 may use any suitable wired or wireless network, including the Internet.

[0061] The network processing computer 118 can be an example of the data computer 102 illustrated in FIG. 1. The network processing computer 118 can provide transaction data to the transaction data database 110.PATENT Attorney Docket No.: 079900-1461958 Client Reference No.: 8153WO01

[0062] The authorizing entity computer 120 can include a server computer operated by an authorizing entity. The authorizing entity computer 120 may be associated with an authorizing entity, which may be an entity that authorizes a request. An example of an authorizing entity may be an issuer, which may typically refer to a business entity (e.g., a bank) that maintains an account for a user. An issuer may also issue and manage an account associated with the user device 122.

[0063] In FIG. 3, in a typical transaction, a user operating the user device 122 can interact with the access device 112 and / or the resource provider computer 114 by passing a credential such as a primary account number to the access device 112 and / or the resource provider computer 114. The access device 112 and / or the resource provider computer 114 can generate an authorization request message comprising information including at least the credential and a transaction amount, and then transmit it to the authorizing entity computer 120 via the transport computer 116 and the network processing computer 118 for authorization.

[0064] The authorizing entity computer 120 can receive the authorization request message. It can then determine if the transaction is authorized, and can generate an authorization response message and transmit it to the resource provider computer and / or the access device 112 via the network processing computer 118 and the transport computer 116.

[0065] At the end of the day or any other period of time, a clearing and settlement process can occur between the transport computer 116, the network processing computer 118, and the authorizing entity computer 120.

[0066] Transaction data regarding the authorization and settlement process described above can be captured by the network processing computer 118 and stored in the transaction data database 110. The model computer 104 can use the stored transaction data to create models, which can be stored in the model database 106 and used by the downstream computer 108.

[0067] FIG. 4 shows a diagram illustrating a sequence to sequence machine learning model according to embodiments. FIG. 4 includes an input sequence 402, a sequence to sequence machine learning model 404 that includes an encoder 406 and a decoder 408, and an output sequence 410.PATENT Attorney Docket No.: 079900-1461958 Client Reference No.: 8153WO01

[0068] The input sequence 402 can include a sequence of tokens. The input sequence 402 can include any number of tokens. The input sequence 402 can include a sequence of transaction data, where each instance of a transaction can be a token. For example, a token can include transaction data for a transaction such as a time of the transaction, an amount of the transaction, parties to the transaction, a description of assets involved in the transaction, etc.

[0069] The sequence to sequence machine learning model 404 includes the encoder 406 and the decoder 408. The sequence to sequence machine learning model 404 can accept the input sequence 402 as input and can output the output sequence 410.

[0070] The encoder 406 can process the input sequence 402. The encoder 406 can capture information from the input sequence 402 in a fixed-size context vector. The encoder 406 can process each element of the input sequence 402 using one or more of a neural network and / or a transformer architecture. Throughout this process, the encoder 406 can determine and maintain a context vector that encapsulates a compressed representation of the entire input sequence 402. The context vector can capture the semantic meaning and important information of the input sequence 402. The encoder 406 can output the context vector, which is provided to the decoder 408.

[0071] The decoder 408 can be similar to the encoder 406. The decoder 408 can process the context vector from the encoder 406 to generate the output sequence 410 incrementally.

[0072] During training of the sequence to sequence machine learning model 404, the decoder 408 can receive both the context vector and a desired target output sequence (e.g., ground truth that is a known next transaction in the training data set). The sequence to sequence machine learning model 404 can generate a predicted output sequence. The sequence to sequence machine learning model 404 can compare the predicted output sequence to the desired target output sequence. The sequence to sequence machine learning model 404 can update the weights based on the comparison within the sequence to sequence machine learning model 404 to adjust the model to predict output sequences more accurately in subsequent training rounds.PATENT Attorney Docket No.: 079900-1461958 Client Reference No.: 8153WO01

[0073] During inference using the sequence to sequence machine learning model 404, the decoder 408 can receive the context vector from the encoder 406. The decoder 408 can determine the output sequence 410 based on the context vector.

[0074] The decoder 408 uses the context vector to comprehend the input sequence 402 and create the corresponding output sequence 410. The decoder 408 can engage in autoregressive generation, producing individual elements sequentially. At each time step, during a training or inference process, the decoder 408 can use the context vector, and a previous output token to generate a probability distribution of a possible next tokens. The token with the highest probability is then chosen as the output, and the process continues until the end of the output sequence 410 is reached.

[0075] The output sequence 410 can include a sequence of tokens. The output sequence 410 can include any number of tokens. In some embodiments, the output sequence 410 can be the same length as the input sequence 402. The output sequence 410 can include a sequence of transaction data, where each transaction is a token. The sequence to sequence machine learning model 404 can generate the output sequence 410 by predicting a transaction that occurs next in the output sequence 410 of transactions.

[0076] The sequence to sequence machine learning model 404 can utilize a self-attention mechanism. The self-attention mechanism allows the sequence to sequence machine learning model 404 to focus on different parts of the sequence data dynamically. For each transaction, the sequence to sequence machine learning model 404 computes three vectors: a query vector, a key vector, and a value vector. The query associated with the query vector represents the information that one is searching for, the key associated with the key vector represents the potential information to compare against, and the value associated with the value vector is the actual data associated with each key. The model computer 104 can generate the query vector by multiplying the encoded input sequence by a query weight matrix. The model computer 104 can generate the key vector by multiplying the encoded input sequence by a key weight matrix. The model computer 104 can generate the value vector by multiplying the encoded input sequence by a value weight matrix. The query weight matrix, the key weight matrix, and the value weight matrix are model parameters that the model computer 104 can adjust during training of the sequence to sequencePATENT Attorney Docket No.: 079900-1461958 Client Reference No.: 8153WO01 machine learning model 404. The query vector, the key vector, and the value vector can represent different aspects of the transaction.

[0077] The sequence to sequence machine learning model 404 can then compute attention scores by taking the dot product of the query vector of a given transaction with the key vectors of all transactions. This gives a measure of how much attention to pay to each transaction when considering the current transaction. The attention scores are then used to compute a weighted sum of the value vectors using, for example, a softmax function. By doing so, the sequence to sequence machine learning model 404 creates a context vector for each transaction that is a weighted sum of all the transactions, with the weights determined by the attention scores.

[0078] As an illustrative example, when evaluating a time-dependent transaction data sequence, as the input sequence 402, that is representative of a credit card payment on May 15, the sequence to sequence machine learning model 404 might give higher attention scores to a salary deposit on April 5 and previous repayments, as these are likely to be more relevant for predicting future credit behavior.

[0079] Relative positional encoding can aid the sequence to sequence machine learning model 404 in understanding the order and distance between different transactions. Rather than using absolute positions (e.g., the 1st, 2nd, or 3rd transaction), the model computer 104 can utilize relative positional encoding, which captures the distance between transactions (e.g., how many days apart two transactions are). Using relative positional encoding can allow the sequence to sequence machine learning model 404 to recognize that a transaction that occurs one month after a salary deposit is different than a transaction that occurs six months later, even if both are credit card payments. As such, the sequence to sequence machine learning model 404 can understand that a salary deposit on April 5 is one month before a credit card payment on May 15. This timing might indicate that the credit card payment is funded by the salary deposit. The sequence to sequence machine learning model 404 understands that the salary deposit precedes the credit card payment, indicating that the user has funds to make payments.

[0080] By combining self-attention with relative positional encoding, the sequence to sequence machine learning model 404 can identify key transactions,PATENT Attorney Docket No.: 079900-1461958 Client Reference No.: 8153WO01 understand temporal patterns, and capture long-term dependencies. The sequence to sequence machine learning model 404 can focus on high-impact transactions, such as large salary deposits or significant loan repayments, to identify key transactions. The sequence to sequence machine learning model 404 can recognize recurring financial behaviors, such as monthly salary deposits followed by expenditures, to understand temporal patterns. The sequence to sequence machine learning model 404 can relate distant transactions to each other, such as understanding how a loan repayment might impact future borrowing capacity, to capture long-term dependencies.

[0081] As an illustrative example, the model computer 104 can train, maintain, and utilize the sequence to sequence machine learning model 404 that can perform credit risk assessment. For example, the sequence to sequence machine learning model 404 can be utilized to predict a likelihood of default for a new loan application.

[0082] The sequence to sequence machine learning model 404 can also evaluate long-term dependences. The sequence to sequence machine learning model 404 can relate the loan repayment on February 10 to the credit card purchase on March 20, recognizing a pattern of borrowing and repayment. This nuanced understanding of the credit history data enables the model to make more accurate predictions about credit risk, considering both the importance of individual transactions and their temporal relationships.

[0083] The sequence to sequence machine learning model 404 can be trained using a loss function. The sequence to sequence machine learning model 404 can be trained with any suitable loss function (e.g., mean squared error loss for regression tasks, cross-entropy loss for classification tasks, etc.). The gradients determined based on the loss function can be backpropagated through the sequence to sequence machine learning model 404 to update the weights within the sequence to sequence machine learning model 404. The process can be repeated for a number of epochs until the performance of the sequence to sequence machine learning model 404 reaches a threshold performance value, stops improving, and / or other condition for ending training.

[0084] The sequence to sequence machine learning model 404, after having been trained, can be tested using performance testing values to evaluatePATENT Attorney Docket No.: 079900-1461958 Client Reference No.: 8153WO01 performance. The performance of the sequence to sequence machine learning model 404 can be measured using metrics such as mean absolute error (MAE) for regression tasks, or accuracy for classification tasks.

[0085] Table 1 illustrates an example program that utilizes a self-attention mechanism and relative positional encoding in a sequence to sequence machine learning model for transaction data analysis.PATENT Attorney Docket No.: 079900-1461958 Client Reference No.: 8153WO01PATENT Attorney Docket No.: 079900-1461958 Client Reference No.: 8153WO01PATENT Attorney Docket No.: 079900-1461958 Client Reference No.: 8153WO01Table 1: example program that utilizes a self-attention mechanism and relative positional encoding in a Transformer model for credit risk assessment

[0086] The example program illustrated in Table 1, creates the sequence to sequence machine learning model 404, trains the sequence to sequence machine learning model 404 on a time-dependent transaction data (e.g., from a credit risk assessment dataset), and prints a training loss every 10 epochs. The sequence to sequence machine learning model 404 includes two transformers, the encoder 406 and the decoder 408, that use a self-attention mechanism and relative positional encoding. The self-attention mechanism is shown in Table 1 in the definition of the transformer encoder within the TransformerModel class. The relative positional encoding is show in Table 1 in the definition and application of the pos_encoder within the TransformerModel class.

[0087] The combination of the self-attention mechanism and relative positional encoding allows the model to handle input sequences of different lengths. This isPATENT Attorney Docket No.: 079900-1461958 Client Reference No.: 8153WO01 advantageous in credit risk assessment since different users may have credit histories of different lengths.

[0088] In language models, the input sequence is typically discrete and categorical, such as words or characters from a text. These are often transformed into embeddings, which are continuous vector representations that capture semantic meanings. The sequence of these embeddings forms the input to the sequence to sequence machine learning model. Sequence to sequence machine learning models are described in “Attention is All You Need" by Ashish Vaswani, et al., Advances in Neural Information Processing Systems. 30. Curren Associates, Inc. arXiv: 1706.03762 (2017), which is herein incorporated by reference in its entirety.

[0089] In contrast, for transaction processing (e.g., credit risk assessment, etc.), the input data sequences typically include a sequence of transactions or other financial activities associated with a user. Each transaction can be represented by numerous features, such as the amount, type of transaction, time of transaction, etc. For example, a credit card transaction can be represented as follows:[transaction_amount, transaction_type, merchant_category, time_of_transaction, location_of_transaction],

[0090] As another example, the time-dependent transaction data can include securities market data. The securities market data can include data related to a particular security (e.g., a stock, a bond, a mutual fund, a promissory note, etc.). The securities market data can be represented as follows: [name, price, category, securities_type, date, volume],

[0091] As another example, the time-dependent transaction data can include user financial transaction data. The user financial transaction data can include data related to a particular user. The user financial transaction can be represented as follows: [name, account_number, routing_number, account_amount, transaction_amount, date, transaction],

[0092] As another example, the time-dependent transaction data can include machine performance data. The machine performance data can include data related to the operation of a machine. The machine performance data can be represented asPATENT Attorney Docket No.: 079900-1461958 Client Reference No.: 8153WO01 follows: [machine dentifier, machine_state, response_time, bandwidth, processing speed, latency, temperature, power_consumption_amount].

[0093] As another example, the time-dependent transaction data can relate to credit history data. Credit history data can include a series of events and transactions related to a user's credit behavior, such as: 1) [loan_amount, loan_type, interest_rate, loan_term, date_of_loan, repayment_status], 2) [credit_card_limit, credit_utilization, payment_due_date, payment_amount, payment_status, date_of_payment], 3) [credit_score, date_of_score_update, score_change_reason], or 4) [account_open_date, account_type, current_balance, date_of_last_activity],

[0094] As another example, the time-dependent transaction data can relate to credit card data. Credit card data for transaction processing can include: 1) [transaction_amount, transaction_type, merchant_category, time_of_transaction, location_of_transaction, card_type, currency], 2) [transaction_amount, transaction_type, merchant_name, date_of_transaction, transaction d, card ssuer], or 3) [transaction_amount, transaction_type, merchant_category_code, transaction_description, transaction_status, cardholder_name],

[0095] As another example, the time-dependent transaction data can relate to stock data. Stock transactions and related financial activities can be represented as: 1) [stock_symbol, trade_date, trade_type (buy / sell), trade_price, trade_volume, exchange, order_type], 2) [stock_symbol, dividend_date, dividend_amount, stock_split_ratio, split_date, corporate_action_type], or 3) [stock_symbol, earnings_report_date, earnings_per_share, revenue, net ncome, forecasted_earnings_per_share],

[0096] As another example, the time-dependent transaction data can relate to bank transactions. Bank transactions can include a variety of financial activities and can be represented as: 1) [transaction_amount, transaction_type, account_balance, time_of_transaction, location_of_transaction, transaction_description], 2) [deposit_amount, deposit_type, date_of_deposit, account_number, branch ocation], or 3) [withdrawal_amount, withdrawal_method, date_of_withdrawal, account_type, transaction_status].PATENT Attorney Docket No.: 079900-1461958 Client Reference No.: 8153WO01

[0097] As another example, the time-dependent transaction data can relate to loan data. Loan-related transactions and activities can be represented as: 1) [loan_amount, loan_type, interest_rate, loan_term, date_of_loan, repayment_date, repayment_amount], 2) [loan_amount, loan_purpose, loan_status, date_of_disbursement, collateral_value, date_of_collateral_assessment], or 3) [loan_amount, loan_term, monthly_repayment_amount, due_date, remaining_balance, late_payment_fees],

[0098] As another example, the time-dependent transaction data can relate to investment data. Investment transactions and activities may be represented as: 1) [investment_amount, investment_type, date_of_investment, return_rate, maturity_date, investment_status], 2) [mutual_fund_name, purchase_date, purchase_amount, unit_price, number_of_units, NAV, redemption_date], or 3) [bond_name, purchase_date, face_value, coupon_rate, maturity_date, yield_to_maturity].

[0099] As another example, the time-dependent transaction data can relate to insurance data. Insurance transactions and activities can include: 1) [policy_number, policy_type, premium_amount, coverage_amount, start_date, end_date, claim_status], 2) [claim d, claim_amount, claim_date, claim_type, settlement_date, claim_status, policy_number], 3) [premium_payment_date, premium_amount, payment_method, policy_number, coverage_start_date, coverage_end_date],

[0100] The information included in the transaction is often numerical, but can also be categorical. The categorical features are encoded into numerical representations (e.g., one-hot vectors or embeddings) so that they can be processed by the model.

[0101] The sequence of these transactions forms the input to the sequence to sequence machine learning model 404. The model then learns to predict the risk associated with a loan based on these sequences either due to utilizing a target attribute in the time-dependent transaction data or a second machine learning model, as described in further detail below.

[0102] In the context of credit risk assessment, the input data sequence can include a sequences of financial or transactional data. Each data point in the sequencePATENT Attorney Docket No.: 079900-1461958 Client Reference No.: 8153WO01 could be a representation of a transaction or credit activity, with various features such as the amount, type of transaction, time of transaction, etc. The target or label could be a binary indicator of whether a loan is defaulted or not, or it could be a continuous score representing the risk level.

[0103] FIG. 5 shows a flow diagram of illustrating a method of processing and utilizing a sequence to sequence machine learning model according to embodiments. The method illustrated in FIG. 5 will be described in the context of the model computer 104 processing transaction data to train a sequence to sequence machine learning model that is utilized to determine a prediction.

[0104] At step 502, the model computer 104 can obtain time-dependent transaction data. The model computer 104 can obtain the time-dependent transaction data from the model database 106.

[0105] The time-dependent transaction data can include data that changes over time and where one transaction may relate to another transaction in some manner. The time-dependent transaction data can include data relating to a plurality of transactions. The time-dependent transaction data can include payment card transaction data, machine performance data, user financial transaction data, securities market data, or other transaction related data that is time-dependent.

[0106] Table 2 illustrates sample credit history data as an example of timedependent transaction data."" """" " """""PATENTAttorney Docket No.: 079900-1461958Client Reference No.: 8153WO01 "" """" " """"""" """" " """"""" """" " """"""" """" " """""PATENT Attorney Docket No.: 079900-1461958 Client Reference No.: 8153WO01Table 2: example time-dependent transaction data

[0107] The time-dependent transaction data illustrated in Table 2 includes five example transactions for purposes of illustration. Each transaction includes a date, a transaction type, an amount, and an account balance. However, it is understood that each transaction can include other types of data such as times, user identifiers, resource provider identifiers, ticker symbols, machine identifiers, data amounts, authorizing entity identifiers, credit scores, credit limits, and other data related to a transaction. Also, embodiments of the invention can include many more transactions.

[0108] The five example transactions can be transactions that relate to an account associated with a user. The transactions include transaction types of credit card payment, loan repayment, credit card purchase, and salary deposit.

[0109] The model computer 104 can obtain time-dependent transaction data that includes transactions for a plurality of users. The model computer 104 can obtain time-dependent transaction data that includes thousands of transactions.

[0110] In some embodiments, the transaction data can include a targeted attribute that is targeted for evaluation. For example, the transaction data can include a data field of credit score, risk score, or other value that is related to the user at the time of the transaction. The targeted attribute may not have been included in the transaction data at the time of the transaction, but can be later introduced into the transaction data by a computer (e.g., the data computer 102, the model computer 104, etc.). The targeted attribute can be included at the end of the transaction data. For example, the transaction data can be in the format of [date, transaction_type, amount, account_balance, risk_score]. The targeted attribute can be targeted by the model computer 104 for evaluation. The model computer 104 can utilize a sequence to sequence machine learning model to predict a subsequent transaction from a sequence of transactions. The predicted subsequent transaction can include a predicted targeted attribute.

[0111] At step 504, after obtaining the time-dependent transaction data, the model computer 104 can pre-process the time-dependent transaction data to form prePATENT Attorney Docket No.: 079900-1461958 Client Reference No.: 8153WO01 processed time-dependent transaction data. The model computer 104 can pre-process the time-dependent transaction data such that the time-dependent transaction data can be utilized to train a sequence to sequence machine learning model.

[0112] Preprocessing the time-dependent transaction data is described in further detail in reference to FIG. 6.

[0113] At step 506, after preprocessing the time-dependent transaction data, the model computer 104 can create a training dataset. The model computer 104 can create the training dataset from the pre-processed time-dependent transaction data. The training dataset can include pre-processed time-dependent transaction data for 10,000, 400,000, 1,000,000, 5,000,000 transactions. In some embodiments, the training dataset includes more than 10,000 transactions.

[0114] At step 508, after creating the training dataset, the model computer 104 can train a sequence to sequence machine learning model using the training dataset comprising the pre-processed time-dependent transaction data.

[0115] As an illustrative example, the model computer 104 can represent the pre-processed time-dependent transaction data as a sequence of vectorized transactions. Each transaction in the input sequence can represent a specific transaction such as a loan application, a payment, a default, etc. The vectorized transaction can be an embedding generated for the transaction.

[0116] The model computer 104 can apply a self-attention mechanism to these vectorized transactions. For each state in the input sequence, the model computer 104 can compute a query Q, a key K, and a value V. The model computer 104 can determine attention weights using the query Q, the key K, and the value V using the formula:QKTAttention Q, K, V) = softmax(— =)V

[0117] The model computer 104 can determine a relative positional encoding for each vectorized transaction in the input sequence. For example, the model computer 104 can determine the relative positional encoding by computing:PE(pos, 2i) = sin(pos / 10000A(2i / d nodel )PATENT Attorney Docket No.: 079900-1461958 Client Reference No.: 8153WO01 PE(pos, 2i + 1) = cos(pos 110000A(2i I djnodeiy)

[0118] In the above equations, pos is the current position of the evaluated vectorized transaction in the sequence, i is the dimension of the vectorized transaction, and d_model is the dimension of the sequence to sequence machine learning model. The model computer 104 can then combine the output of the self-attention mechanism and the relative positional encoding to produce a final representation for each vectorized transaction in the input sequence. The exact method of combination can vary depending on the specific application. For example, one method of combining the attention and the relative positional encoding values can be to add the values together to form a combined output value, which is a combined transaction representation.Combine dOutput = Attention Q, K, 7) + PE(pos, 2i) + PE(pos, 2i + 1)

[0119] Another example, of utilizing relative positional encodings can be by using rotary position embeddings (RoPE) to, rather than add a separate positional encoding vector, apply a rotation to the existing token embeddings.

[0120] As such, the model computer 104 can combine the attention values and the relative positional encodings to form a combined transaction representation that represents the input sequence.

[0121] The model computer 104 can iteratively input the pre-processed timedependent transaction data from the training dataset into the sequence to sequence model to determine training predictions. The training predictions can be predicted next transactions in the sequence. The model computer 104 can compare the training predictions to known next transactions included in the training dataset. The model computer 104 can determine a difference between a known next transaction and a training prediction. The model computer 104 can update weight values in the sequence to sequence machine learning model based on the comparison.

[0122] At step 510, after training the sequence to sequence machine learning model, the model computer 104 can obtain test time-dependent transaction data. The model computer 104 can obtain the test time-dependent transaction data from the transaction data database 110. The test time-dependent transaction data can be previously stored in the transaction data database 110 by the data computer 102 such as the network processing computer 118.PATENT Attorney Docket No.: 079900-1461958 Client Reference No.: 8153WO01

[0123] The test time-dependent transaction data can be time-dependent transaction data. The test time-dependent transaction data can be new timedependent transaction data or may have been used to train the sequence to sequence machine learning model. The test time-dependent transaction data can correspond to a transaction for which the model computer 104 determines a prediction.

[0124] In some embodiments, the model computer 104 can receive a prediction request message comprising the test time-dependent transaction data or a test timedependent transaction data identifier from the downstream computer 108. The test time-dependent transaction data identifier can be an alphanumeric identifier and can uniquely identify the test time-dependent transaction data. If the model computer 104 receives the test time-dependent transaction data, the model computer 104 can generate a test time-dependent transaction data query message comprising the test time-dependent transaction data identifier. The model computer 104 can provide the test time-dependent transaction data query message to the transaction data database 110. The transaction data database 110 can obtain the test time-dependent transaction data that is stored in association with the test time-dependent transaction data identifier. The transaction data database 110 can generate a test time-dependent transaction data query response message comprising the test time-dependent transaction data. The transaction data database 110 can provide the test timedependent transaction data query response message to the model computer 104.

[0125] At step 512, after obtaining test time-dependent transaction data, the model computer 104 can input test time-dependent transaction data into the trained sequence to sequence machine learning model to obtain predicted time-dependent transaction data.

[0126] In some embodiments, the model computer 104 can preprocess the test time-dependent transaction data to form pre-processed test time-dependent transaction data prior to using the sequence to sequence machine learning model to determine the prediction.

[0127] The final output of the sequence to sequence machine learning model can be a sequence of transactions with a next predicted transaction in the sequence, referred to as the predicted time-dependent transaction data. The model computer 104PATENT Attorney Docket No.: 079900-1461958 Client Reference No.: 8153WO01 can evaluate the predicted time-dependent transaction data to determine one or more attributes related to the predicted time-dependent transaction data.

[0128] In some embodiments, the transaction data can include a targeted attribute that is targeted for evaluation. For example, the transaction data can include a risk score or other data that is included at the end of each transaction. The model computer 104 can evaluate the predicted time-dependent transaction data to determine the predicted targeted attribute. For example, the predicted targeted attribute can be the predicted risk score.

[0129] The model computer 104 can evaluate how the predicted timedependent transaction data may affect the user that is associated with the transactions. For example, the model computer 104 can evaluate the impact of the addition of the predicted time-dependent transaction data on a credit score associated with the user.

[0130] In some embodiments, a machine learning model can be trained to predict values based on the combined transaction representation determined by the sequence to sequence machine learning model from the attention value and the positional encoding value. For example, the model computer 104 can train a machine learning model to predict a risk of default based on the combined transaction representation.

[0131] After generating the predicted time-dependent transaction data, the model computer 104 can provide the predicted time-dependent transaction data to the downstream computer 108.

[0132] In some embodiments, if the evaluation of the test time-dependent transaction data was initiated by the downstream computer 108 via the prediction request message, then the model computer 104 can generate a prediction response message comprising the predicted time-dependent transaction data. The model computer 104 can provide the prediction response message to the downstream computer 108.

[0133] After receiving the predicted time-dependent transaction data, the downstream computer 108 can perform further processing based on the predicted time-dependent transaction data. For example, the downstream computer 108 canPATENT Attorney Docket No.: 079900-1461958 Client Reference No.: 8153WO01 generate an alert or notification based on the predicted time-dependent transaction data and provide the alert or notification to another device related to the predicted time-dependent transaction data (e.g., a user device). As additional examples, the downstream computer 108 can evaluate a targeted attribute, evaluate how the predicted time-dependent transaction data may affect the user that is associated with the transactions, train a machine learning model to predict values based on the predicted time-dependent transaction data to derive further information related to the prediction of the subsequent transaction, and / or any other processing of the predicted time-dependent transaction data. In another example, if the transaction data relates to network traffic or the operation of a machine, then an automated signal may be sent to any affected network computers or machines based on predictions made by the machine learning model.

[0134] FIG. 6 shows a flow diagram of illustrating a method of processing timedependent transaction data according to embodiments. The model computer 104 can perform any combination of the steps illustrated in FIG. 6 to preprocess the timedependent transaction data.

[0135] At step 602, the model computer 104 can perform data cleaning on the time-dependent transaction data. The model computer 104 can remove data that includes missing data elements. For example, if a particular transaction includes a NaN (not a number) field for the amount due to an error, then the model computer 104 can remove the transaction from the time-dependent transaction data.

[0136] At step 604, the model computer 104 can perform feature engineering as a part of the preprocessing of the time-dependent transaction data. For example, the model computer 104 can create new variables based on existing variables to capture additional information. For example, the time-dependent transaction data can include a variable of amount. The model computer 104 can create a new variable of running average amount that is a running average of the amount of each transaction over time.

[0137] At step 606, the model computer 104 can encode the time-dependent transaction data. For example, the model computer 104 can convert categorical variables into numerical variables so that they categorical variables can be processed by the sequence to sequence machine learning model. For example, a transaction inPATENT Attorney Docket No.: 079900-1461958 Client Reference No.: 8153WO01 the time-dependent transaction data can include a variable of authorization status that can indicate if the transaction is authorized or is not authorized. The model computer 104 can convert the categorical variable of authorization status to a numerical variable that includes a value of 1 if the transaction is authorized or a value of 0 if the transaction is not authorized.

[0138] As another example, the model computer 104 can encode categorical variables such as item categories, resource provider names, transaction type (e.g., loan repayment, credit card payment, salary deposit, etc.), authorizing entity name, stock name, and or any other non-numerical variable.

[0139] In some embodiments, the model computer 104 can also perform normalization as a part of the preprocessing of the time-dependent transaction data. For example, the model computer 104 can scale numerical features so that the numerical features all have a similar range. A numerical variable can be normalized such that all instances of the numerical variable fall within a range of 0 to 1.

[0140] At step 608, the model computer 104 can perform sequence padding as a part of the preprocessing of the time-dependent transaction data. The model computer 104 can perform sequence padding in order to make all sequences a same length (e.g., a dimension of 512) such that the sequence can be utilized by the sequence to sequence machine learning model. For example, the model computer 104 can add zeros (or other computer identifiable symbol) to the end of the timedependent transaction data such that the resulting modified time-dependent transaction data has a length of 512, 1028, or other predetermined length.

[0141] Embodiments of the disclosure have a number of advantages. The machine learning model with relative positional encodings, according to embodiments, has several advantages in handling data. For example, the machine learning model can handle variable-length inputs. Since the machine learning model uses sinusoidal encodings for relative position encodings, the machine learning model can handle input sequences of varying lengths, which is common in financial data.

[0142] Embodiments further provide for the advantage of utilizing directional bias. The machine learning model can assign different biases to different direction within an input sequence by using positive distance when the input elements (e.g.,PATENT Attorney Docket No.: 079900-1461958 Client Reference No.: 8153WO01 transactions) are to the left and negative distance otherwise. This can help the machine learning model understand the temporal relationships between the transactions. The machine learning model can learn from the patterns in the relative distances between transactions. This can improve the machine learning model’s ability to predict information related to the transactions, such as credit risk. This is particularly advantageous, for example, in credit risk assessment, as the order of the transactions can carry important information (e.g., a late payment followed by several on-time payments is not the same as several on-time payments followed by a late payment).

[0143] Embodiments provide for the advantage of handling long term dependencies. In credit risk assessment, the self-attention mechanism can capture long-term dependencies in the user’s credit history, which is difficult to do in previous methods that use LSTM models due to the vanishing gradient problem.

[0144] Embodiments provide for the advantage of efficiently capturing complex temporal dependencies in financial time-series data. Embodiments also provide for the advantage of effectively handling the noisy and non-stationary nature of transaction based markets. Embodiments also provide for the advantage of capturing the relative importance of past events in predicting future transactions. Embodiments provide for such advantages due to, at least, the attention mechanisms used in the model. The attention mechanisms can allow for each input token (e.g., which represents a transaction) to be related to every other token at each time step. As such, a computer can determine a value that represents how much every transaction in the input sequence relates to every other transaction.

[0145] For example, when evaluating a time-dependent transaction data sequence that is representative of a credit card payment on May 15, a salary deposit on April 5, and various other transactions, the attention mechanism can determine that the credit card payment on May 15 has a high attention score with the salary deposit on April 5. The high attention score can indicate that the two transactions are highly related. For example, the model can learn, using the attention score, that the salary deposit precedes the credit card payment and indicates that the user has funds to make payments.

[0146] Embodiments of the invention also compare favorably to other types of analyses using other types of machine learning models. For example, during anPATENT Attorney Docket No.: 079900-1461958 Client Reference No.: 8153WO01 experiment comparing systems and methods according to embodiments to long shortterm memory (LSTM) based methods was conducted. For a training data set, the experiment resulted in the response times illustrated in Table 3. Systems and methods according to embodiments provide for a lower total training time compared to LSTM based methods. Systems and methods further provide for a greater efficiency gain than LSTM methods. During the experiment, systems and methods according to embodiments and the LSTM started with an initial loss value of about 0.80. After training, systems and methods according to embodiments ended with a final loss value of ~0.25, whereas the LSTM ended with a final loss value of ~0.30. Embodiments obtained a better final loss value resulting in a greater efficiency gain than the LSTM method.Table 3: experiment response times and efficiency gains

[0147] Embodiments and the LSTM were also evaluated on prediction accuracy. Embodiments obtained a better accuracy on short sequences (<10 time steps) and on long sequences (> 50 time steps). The accuracy of predictions are illustrated in Table 4.Table 3: experiment response times and efficiency gainsPATENT Attorney Docket No.: 079900-1461958 Client Reference No.: 8153WO01

[0148] Overall, embodiments improve training efficiency. Models according to embodiments, train faster (~28.6% reduction in training time compared to the LSTM) and converge more efficiently due to parallel processing. Embodiments also improve prediction quality. Embodiments achieve higher accuracy (~12% overall improvement compared to the LSTM) and perform significantly better on long sequences (~23% improvement compared to the LSTM).

[0149] Although the steps in the flowcharts and process flows described above are illustrated or described in a specific order, it is understood that embodiments of the invention may include methods that have the steps in different orders. In addition, steps may be omitted or added and may still be within embodiments of the invention.

[0150] Any of the software components or functions described in this application may be implemented as software code to be executed by a processor using any suitable computer language such as, for example, Java, C, C++, C#, Objective-C, Swift, or scripting language such as Perl or Python using, for example, conventional or object-oriented techniques. The software code may be stored as a series of instructions or commands on a computer readable medium for storage and / or transmission, suitable media include random access memory (RAM), a read only memory (ROM), a magnetic medium such as a hard-drive or a floppy disk, or an optical medium such as a compact disk (CD) or DVD (digital versatile disk), flash memory, and the like. The computer readable medium may be any combination of such storage or transmission devices.

[0151] Such programs may also be encoded and transmitted using carrier signals adapted for transmission via wired, optical, and / or wireless networks conforming to a variety of protocols, including the Internet. As such, a computer readable medium according to an embodiment of the present invention may be created using a data signal encoded with such programs. Computer readable media encoded with the program code may be packaged with a compatible device or provided separately from other devices (e.g., via Internet download). Any such computer readable medium may reside on or within a single computer product (e.g., a hard drive, a CD, or an entire computer system), and may be present on or within different computer products within a system or network. A computer system mayPATENT Attorney Docket No.: 079900-1461958 Client Reference No.: 8153WO01 include a monitor, printer, or other suitable display for providing any of the results mentioned herein to a user.

[0152] The above description is illustrative and is not restrictive. Many variations of the invention will become apparent to those skilled in the art upon review of the disclosure. The scope of the invention should, therefore, be determined not with reference to the above description, but instead should be determined with reference to the pending claims along with their full scope or equivalents.

[0153] One or more features from any embodiment may be combined with one or more features of any other embodiment without departing from the scope of the invention.

[0154] As used herein, the use of "a," "an," or "the" is intended to mean "at least one," unless specifically indicated to the contrary.

Claims

PATENT Attorney Docket No.: 079900-1461958 Client Reference No.: 8153WO01 WHAT IS CLAIMED IS:

1. A method comprising:obtaining time-dependent transaction data;preprocessing the time-dependent transaction data to form pre-processed time-dependent transaction data;creating a training dataset from the pre-processed time-dependent transaction data;training a sequence to sequence machine learning model using training dataset; andinputting test time-dependent transaction data into the trained sequence to sequence machine learning model to obtain predicted time-dependent transaction data.

2. The method of claim 1 further comprising:performing further processing on the predicted time-dependent transaction data.

3. The method of claim 2, wherein further processing comprises: generating a score using the predicted time-dependent transaction data, the score being a risk score.

4. The method of claim 1 , wherein the time-dependent transaction data comprises machine performance data or user-specific data.

5. The method of claim 1 , wherein obtaining the time-dependent transaction data comprises:obtaining time-dependent transaction data from a transaction data database, wherein a network processing computer stores transaction data into the transaction data database.

6. The method of claim 1 further comprising:receiving a prediction request message comprising the test timedependent transaction data or a test time-dependent transaction data identifier from a downstream computer; andPATENT Attorney Docket No.: 079900-1461958 Client Reference No.: 8153WO01 providing a prediction response message comprising the predicted time-dependent transaction data to the downstream computer.

7. The method of claim 1 further comprising:preprocessing the test time-dependent transaction data.

8. The method of claim 1 , wherein the test time-dependent transaction data is included in the time-dependent transaction data.

9. The method of claim 1 , wherein training the sequence to sequence machine learning model comprises:iteratively inputting pre-processed time-dependent transaction data from the training dataset into the sequence to sequence model to determine training predictions;comparing the training predictions to known next transactions included in the training dataset; andupdating weight values in the sequence to sequence machine learning model based on the comparison.

10. The method of claim 1 , wherein preprocessing the timedependent transaction data to form pre-processed time-dependent transaction data comprises:performing data cleaning on the time-dependent transaction data; performing feature engineering on the time-dependent transaction data;generating encodings for the time-dependent transaction data; and performing sequence padding on the time-dependent transaction data.

11. The method of claim 1 , wherein training the sequence to sequence machine learning model using the training dataset comprises:for an input sequence from the time-dependent transaction data in the training dataset, wherein the input sequence represents a sequence of transactions:generating a query vector based on the input sequence and a query weight;PATENT Attorney Docket No.: 079900-1461958 Client Reference No.: 8153WO01 generating a key vector based on the input sequence and a key weight;generating a value vector based on the input sequence and a value weight; anddetermining an attention value for each transaction represented in the input sequence using the query vector, key vector, and the value vector;determining a relative positional encoding for each transaction represented in the input sequence; andcombining the attention values and the relative positional encodings to form a combined transaction representation that represents the input sequence.

12. The method of claim 1 , wherein the training dataset includes more than 10,000 transactions.

13. A computer comprising:a processor; anda computer-readable medium coupled to the processor, the computer-readable medium comprising code executable by the processor for implementing a method comprising:obtaining time-dependent transaction data;preprocessing the time-dependent transaction data to form pre- processed time-dependent transaction data;creating a training dataset from the pre-processed timedependent transaction data;training a sequence to sequence machine learning model using the training dataset; andinputting, test time-dependent transaction data into the trained sequence to sequence machine learning model to obtain predicted timedependent transaction data.

14. The computer of claim 13, wherein the method furthercomprises:PATENT Attorney Docket No.: 079900-1461958 Client Reference No.: 8153WO01 generating a score using the predicted time-dependent transaction data, the score being a risk score, wherein the time-dependent transaction data comprises user specific data.

15. The computer of claim 13, wherein the time-dependent transaction data includes data for a plurality of transactions.

16. The computer of claim 13, wherein the method further comprises:after training the sequence to sequence machine learning model, storing the trained sequence to sequence machine learning model into a model database; andprior to inputting the test time-dependent transaction data into the trained sequence to sequence machine learning model, obtaining the trained sequence to sequence machine learning model from the model database.

17. The computer of claim 13, wherein the sequence to sequence machine learning model includes an encoder and a decoder.

18. The computer of claim 13 further comprising:a data preprocessing module;a training module; anda prediction module.

19. A system comprising:a data database that stores time-dependent transaction data;a model database that stores one or more machine learning models; anda model computer in communication with the data database and the model database, the model computer comprising:a processor; anda computer-readable medium coupled to the processor, the computer-readable medium comprising code executable by the processor for implementing a method comprising:PATENT Attorney Docket No.: 079900-1461958 Client Reference No.: 8153WO01 obtaining the time-dependent transaction data from the data database;preprocessing the time-dependent transaction data to form pre-processed time-dependent transaction data;creating a training dataset from the pre-processed timedependent transaction data;training a sequence to sequence machine learning model using the training dataset; andinputting, test time-dependent transaction data into the trained sequence to sequence machine learning model to obtain predicted time-dependent transaction data.

20. The system of claim 19, wherein the method further comprises: storing, by the model computer, the trained sequence to sequence machine learning model in the model database.

Images