Method and system for processing contribution values

Abstract

Methods and systems for explaining the output of machine learning models using contribution values are described. In many applications, a complex machine learning model may be used to produce an output (e.g., a classification) or otherwise perform some analysis. However, due to its complexity, it may be difficult to generate explanations (e.g., relating the outputs of that machine learning model to its inputs) in a timely manner. In embodiments of the present disclosure, a lightweight surrogate model can be trained to generally copy the performance of a primary machine learning model. Explanations can be generated based on the outputs of the surrogate model, which can be used as surrogate explanations for the primary machine learning model. Such surrogate explanations can be generated more quickly due to the lesser complexity of the surrogate model, enabling such explanations to be used in real-time production environments.

Description

PATENTAttorney Docket No.: 079900-1461959Client Reference No.: 9115WO01METHOD AND SYSTEM FOR PROCESSING CONTRIBUTION VALUESBACKGROUND

[0001] Machine learning models see increasingly widespread use in various sciences, the field of information technology, and in other industries. Such machine learning models can learn to produce outputs automatically based on inputs. For example, a machine learning model could be used to forecast the demand for housing in a city based on features such as the location of the city, prior demand for housing, the price of housing, etc. Such a machine learning model could be used, e.g., for the purpose of planning the development of additional housing.

[0002] However, many machine learning models function effectively as “black boxes”. While it is possible to use such a model to generate output (e.g., predicted housing demand) based on inputs, it is difficult to determine why a machine learning model produced particular outputs based on its inputs.

[0003] Explainer systems are one solution to this problem. Such explainer systems can attempt to explain the output of a machine learning model, in terms of its inputs and in a human-understandable manner. For example, an explainer system could indicate the relative contribution of individual features (e.g., the location of the city) to the output of the model (e.g., housing demand). Unfortunately, existing explainer systems are slow and often involve very time-consuming training processes. Hence, they cannot be used to perform “real-time inference” (e.g., produce explanations under continuous or short-term time constraints), and consequently cannot be used in many applications of machine learning.

[0004] Embodiments address these and other problems, individually and collectively.SUMMARY

[0005] This Summary is provided to introduce a selection of concepts in a simplified form that are further described herein in the Detailed Description. This Summary is not intended to identify key factors or essential features of the claimedsubject matter, nor is it intended to be used to limit the scope of the claimed subject matter.

[0006] Embodiments of the present disclosure are directed to explanation systems and methods that can be used to explain the output of machine learning models. The term “explanation system” can refer to both a physical system (e.g., a computer system or other devices) implementing methods according to embodiments, or to machine learning systems (i.e., connected configurations of machine learning models and / or sub-models) instantiated by such physical systems and used to implement methods according to embodiments. Explanation systems according to embodiments can produce contribution values, reason codes, and descriptions, which can be used to explain the output of a machine learning model.

[0007] In general terms, a contribution value can comprise a numerical estimate of the contribution of a given feature to the output of a machine learning model. For example, for a machine learning model used to predict the value of a house on the market, a contribution value could indicate the contribution to the house’s value resulting from the proximity of public transportation. In such a case, the distance to a subway station could comprise a feature.

[0008] In some embodiments, reason codes and descriptions can be derived from such contribution values. A reason code can generally correspond to a “human understandable” reason for a particular output of a machine learning model, which can be further articulated by a corresponding description. For example, if contribution values indicate that the proximity to public transit, schooling, shops, and restaurants are the primary factors contributing to the value of a house, then a reason code and description could indicate that the house is valuable due to its location and proximity to amenities.

[0009] In general, in methods according to embodiments, for a given primary machine learning model used to implement a particular inference task (e.g., classifying input data as “normal” or “anomalous”), a “lightweight” surrogate machine learning model can be trained to generally mimic the primary machine learning model. While this surrogate machine learning model may not be as accurate as the original machine learning model (due to e.g., a smaller parameter set and lesser2KILPATRICK TOWNSEND 78918186 1complexity), the output of the surrogate machine learning model may generally be similar to the output of the primary machine learning model.

[0010] Rather than explaining the output of the primary machine learning model (which may be prohibitively time expensive for complex primary models) a surrogate explainer module according to embodiments can be used to explain the output of the lightweight surrogate machine learning model. Because the surrogate machine learning model is “more lightweight” than the primary machine learning model, it may be quicker and more efficient to generate explanations based on the surrogate machine learning model rather than the primary machine learning model. However, because the surrogate machine learning model has been trained to copy the primary machine learning model, the outputs of the surrogate machine learning model may be generally similar to the primary machine learning model. Consequently, explanations corresponding to those outputs (e.g., contribution values, reason codes, descriptions, etc.) may be similarly applicable to the outputs of the primary machine learning model. Hence surrogate machine learning models according to embodiments can be used to quickly and accurately produce explanations for primary machine learning models, which would generally be computationally infeasible in real time applications due to the complexity of those primary machine learning models.

[0011] In more detail, one embodiment is directed to a method performed by a computer system. As part of the method, the computer system can perform an iterative training process until a terminating condition has been met. During this iterative training process, the computer system can sample a batch of training interaction data sets. The batch of training interaction data sets can comprise one or more training interaction data sets corresponding to one or more training interactions. For each training interaction data set in the batch of training interaction data sets, the computer system can generate a first prediction using a first machine learning model. In this way, the computer system can generate one or more first predictions corresponding to the one or more training interaction data sets. The computer system can generate one or more sets of first contribution values based on the one or more first predictions. For each training interaction data set in the batch of training interaction data sets, the computer system can generate a second prediction using a second machine learning model. In this way the computer system3KILPATRICK TOWNSEND 78918186 1can generate one or more second predictions corresponding to the one or more training interaction data sets. The computer system can generate one or more sets of second contribution values based on the one or more second predictions. The computer system can determine one or more loss values based on at least the one or more sets of first contribution values and the one or more sets of second contribution values. The computer system can update a parameter set of at least the second machine learning model based on the one or more loss values, thereby training the second machine learning model. If the terminating condition has not been met, the computer system can repeat the iterative training process until the terminating condition has been met. Otherwise, the computer system can complete the iterative training process.

[0012] Another embodiment is directed to a method performed by a computer system. The computer system can receive a set of test interaction data. The computer system can generate a first test prediction based on the set of test interaction data using a first machine learning model. The computer system can generate a second test prediction based on the set of test interaction data using a second machine learning model. The computer system can generate a set of contribution values based on the second test prediction. The computer system can output the first test prediction and the set of contribution values generated based on the second test prediction.

[0013] Another embodiment is directed to a computer system comprising one or more processors and a non-transitory computer readable medium coupled to the one or more processors. The non-transitory computer readable medium can comprise code executable by the processor to perform any of the methods described herein.

[0014] Embodiments of the present disclosure are described in more detail with reference to the Detailed Description below.TERMS

[0015] A “server computer” may refer to a computer or cluster of computers. A server computer may be a powerful computing system, such as a large mainframe. Server computers can also include minicomputer clusters or a group of servers4KILPATRICK TOWNSEND 78918186 1functioning as a unit. In one example, a server computer can include a database server coupled to a web server. A server computer may comprise one or more computational apparatuses and may use any of a variety of computing structures, arrangements, and compilations for servicing requests from one or more client computers.

[0016] A “client computer” may refer to a computer or cluster of computers that receives some service from a server computer (or another computing system). The client computer may access this service via a communication network such as the Internet or any other appropriate communication network. A client computer may make requests to server computers including requests for data. As an example, a client computer can request a video stream from a server computer associated with a movie streaming service. As another example, a client computer may request data from a database server. A client computer may comprise one or more computational apparatuses and may use a variety of computing structures, arrangements, and compilations for performing its functions, including requesting and receiving data or services from server computers.

[0017] A “memory” may refer to any suitable device or devices that may store electronic data. A suitable memory may comprise a non-transitory computer readable medium that stores instructions that can be executed by a processor to implement a desired method. Examples of memories including one or more memory chips, disk drives, etc. Such memories may operate using any suitable electrical, optical, and / or magnetic mode of operation.

[0018] A “processor” may refer to any suitable data computation device or devices. A processor may comprise one or more microprocessors working together to achieve a desired function. The processor may include a CPU that comprises at least one high-speed data processor adequate to execute program components for executing user and / or system generated requests. The CPU may be a microprocessor such as AMD’s Athlon, Duron and / or Opteron; IBM and / or Motorola’s PowerPC; IBM’s and Sony’s Cell processor; Intel’s Celeron, Itanium, Pentium, Xenon, and / or Xscale; and / or the like processor(s).

[0019] A “feature” can be an individual measurable property or characteristic of a phenomenon. One or more features can be described using a “feature vector,”5KILPATRICK TOWNSEND 78918186 1e.g., a structured list of data (such as numerical data) representing those features. A feature can be input into a model to determine an output. As an example, in pattern recognition and machine learning, a feature vector can comprise an n-dimensional vector of numerical features that represent some object. In some machine learning contexts, a numerical representation of objects facilitate processing and statistical analysis. For image processing, for example, feature values might correspond to the pixels of an image. As another example, when feature vectors represent text, the features may comprise occurrence frequency of textual terms. Feature vectors can be equivalent to the vectors of explanatory variables used in statistical procedures such as linear regression.

[0020] The term “artificial intelligence model” or “machine learning model” can include a model that may be used to predict outcomes to achieve a pre-defined goal. A machine learning model may be developed using a learning process, in which training data is classified based on known or inferred patterns.

[0021] "Machine learning" can include an artificial intelligence process in which software applications may be trained to make accurate predictions through learning. The predictions can be generated by applying input data to a predictive model formed from performing statistical analyses on aggregated data. A model can be trained using training data, such that the model may be used to make accurate predictions. The prediction can be, for example, a classification of an image (e.g., identifying images of cats on the Internet) or as another example, a recommendation (e.g., a movie that a user may like or a restaurant that a consumer might enjoy).

[0022] A “machine learning model” may include an application of artificial intelligence that provides systems with the ability to automatically learn and improve from experience without explicitly being programmed. A machine learning model may include a set of software routines and parameters that can predict an output of a process (e.g., identification of an attacker of a computer network, authentication of a computer, a suitable recommendation based on a user search query, etc.) based on feature vectors or other input data. A structure of the software routines (e.g., number of subroutines and the relation between them) and / or the values of the parameters can be determined in a training process, which can use actual results of the process that is being modeled, e.g., the identification of different classes of input data.6KILPATRICK TOWNSEND 78918186 1Examples of machine learning models include support vector machines (SVM), models that classify data by establishing a gap or boundary between inputs of different classifications, as well as neural networks, collections of artificial “neurons” that perform functions by activating in response to inputs. A machine learning model can be trained using “training data” (e.g., to identify patterns in the training data) and then apply this training when it is used for its intended purpose. A machine learning model may be defined by “model parameters,” which can comprise numerical values that define how the machine learning model performs its function. Training a machine learning model can comprise an iterative process used to determine a set of model parameters that achieve the best performance for the model. One example of a machine learning model is an unsupervised learning model. Another example type of model is supervised learning. Example supervised learning models may include different approaches and algorithms including analytical learning, statistical models, artificial neural network, backpropagation, boosting (meta-algorithm), Bayesian statistics, case-based reasoning, decision tree learning, inductive logic programming, Gaussian process regression, genetic programming, group method of data handling, kernel estimators, learning automata, learning classifier systems, minimum message length (decision trees, decision graphs, etc.), multilinear subspace learning, naive Bayes classifier, maximum entropy classifier, conditional random field, nearest neighbor algorithm, probably approximately correct learning (PAC) learning, ripple down rules, a knowledge acquisition methodology, symbolic machine learning algorithms, subsymbolic machine learning algorithms, minimum complexity machines (MCM), random forests, ensembles of classifiers, ordinal classification, data pre-processing, handling imbalanced datasets, statistical relational learning, or Proaftn, a multicriteria classification algorithm. The model may include linear regression, logistic regression, deep recurrent neural network (e.g., long short term memory, LSTM), hidden Markov model (HMM), linear discriminant analysis (LDA), k-means clustering, density-based spatial clustering of applications with noise (DBSCAN), random forest algorithm, support vector machine (SVM), or any model described herein. Supervised learning models can be trained in various ways using various cost / loss functions that define the error from the known label (e.g., least squares and absolute difference from known classification) and various optimization techniques, e.g., using backpropagation, steepest descent, conjugate gradient, and Newton and quasi-Newton techniques.7KILPATRICK TOWNSEND 78918186 1

[0023] An “interaction” can refer to a reciprocal action or influence or an event or phenomenon corresponding to such an action or influence. As an example, an individual posting a comment on a social media website could be considered an interaction between that individual (or e.g., a personal computer system associated with that individual) and the social media website (or e.g., a server computer hosting that social media website). As another example, the collision of a fast-moving cold air front and a slow-moving warm air front (e.g., an “occluded front”) could be considered an interaction between the two air fronts. As another example, a credit card transaction between a cardholder and a business could be considered an interaction. In some cases, measurable properties of interactions can be analyzed or evaluated (e.g., using machine learning) to quantify or qualify those interactions.BRIEF DESCRIPTION OF THE DRAWINGS

[0024] FIG. 1 shows an exemplary Shapley value graph.

[0025] FIG. 2 shows an exemplary client-server system according to some embodiments.

[0026] FIG. 3 shows a table of exemplary features, reason codes, and descriptions of those reason codes, which can be used by an exemplary explanation system according to some embodiments.

[0027] FIG. 4 shows a diagram of a training process for an explanation system according to some embodiments.

[0028] FIG. 5 shows a flowchart depicting a training process for an explanation system according to some embodiments.

[0029] FIG. 6 shows a table describing exemplary regularization terms used in some loss functions according to some embodiments.

[0030] FIG. 7 shows a flowchart depicting an inference process for an explanation system according to some embodiments.

[0031] FIG. 8 shows a diagram of a stateless surrogate explanation system according to some embodiments.8KILPATRICK TOWNSEND 78918186 1

[0032] FIG. 9 shows a table detailing latencies and match rates for an explanation system according to some embodiments.

[0033] FIG. 10 shows an exemplary computer system according to some embodiments.DETAILED DESCRIPTION

[0034] As summarized above, some embodiments of the present disclosure relate to explainer systems for machine learning models. While it is assumed that a potential practitioner of methods according to embodiments has some general knowledge of the field of machine learning, a brief summary of machine learning is provided herein in order to better orient the reader.

[0035] At a high level, a machine learning model generally produces output data responsive to received input data. Such input data may comprise “features”, e.g., of some object, event, or phenomena being analyzed by a machine learning model, which may be quantified or qualified by the output data. For example, for a machine learning model used to evaluate comments posted on a social media website, the length of a comment (measured, e.g., in the number of words or the number of characters) could comprise a feature. The output of such a machine learning model could comprise scores used to describe or define such comments, e.g., a score indicating how “malicious” or “benign” a particular comment is.

[0036] Machine learning models are often defined by sets of parameters, which generally control how the machine learning model produces output data responsive to received input data. As an example, a support vector machine (SVM) is a type of machine learning model that divides data points using a hyperplane. Input data positioned on one “side” of the hyperplane is classified as one class (e.g., normal) while input data on the other side of the hyperplane is classified as another class (e.g., anomalous). The parameters of the support vector machine can comprise the coefficients used to define the hyperplane. Changing these parameters changes the shape of the hyperplane, and thus changes which data points the SVM classifies as normal or anomalous.

[0037] In general terms, the process of training a machine learning model can involve determining the set of parameters that achieve the “best” performance,9KILPATRICK TOWNSEND 78918186 1usually based on a loss or error function. A loss function relates the expected or ideal performance of the machine learning model to its actual performance on a (typically labeled) training data set. The loss function typically decreases in value as the model’s performance improves. As such, training a machine learning model often involves determining the set of parameters that minimize a loss function corresponding to that model. Sometimes a random parameter estimate is generated as an initial parameter “guess,” and then a process such as gradient descent is used to iteratively refine the parameter estimate, eventually resulting in a final set of parameters associated with the machine learning model.

[0038] This iterative refinement process can be performed in a series of training “rounds”, “epochs”, or other appropriate divisions. In each round, a machine learning model’s performance can be evaluated using the loss function, and the parameters can be updated based on this evaluation, e.g., with the goal of reducing the result over time. As an example, the gradient of the loss function can be determined in parameter space and can be used to reduce the value of the loss function in successive training rounds. Such a gradient corresponds to a change in model parameters that achieves the greatest immediate reduction in the loss function. By changing the model parameters based on the gradient, the loss function can be reduced during each successive training round. This process can be repeated until a terminating condition has been met. In embodiments of the present disclosure, one type of terminating condition is a defined number of training rounds. This terminating condition can be met if the number of training rounds performed (e.g., by a computer system training the machine learning model) equals or exceeds the defined number of training rounds, at which point the iterative training process has been completed. Another type of terminating condition in embodiments is a convergence condition. This terminating condition can be met if the machine learning model parameters “converge.” In general terms, convergence is achieved when the value of the loss function, and / or the values of the model parameters change in increasingly small amounts with each successive training round. For example, a convergence condition can be achieved if the value of the loss function decreases by less than 0.1 % in two successive training rounds.

[0039] It is possible to train multiple machine learning models simultaneously, e.g., by updating parameter sets corresponding to each of those machine learning10KILPATRICK TOWNSEND 78918186 1models simultaneously, e.g., using a single dataset and / or a combined loss function. As described in more detail below, some methods according to embodiments involve a first machine learning model (sometimes referred to as a “primary scoring model”), a second machine learning model (sometimes referred to as a “surrogate scoring model”), and an encoder. Each of these models can have its own parameter set, and in some embodiments, multiple machine learning models can be trained simultaneously based on a combined loss function. For example, a parameter set corresponding to the second machine learning model and an encoder parameter set corresponding to the encoder can each be updated in each training round.

[0040] As summarized above, some embodiments of the present disclosure relate to explainer systems for machine learning models. Such explainer systems can generate contribution values that can quantify the contribution of a given input feature to the output of a machine learning model. Shapley values (or approximations of Shapley values) are a type of contribution value which can be used in some embodiments of the present disclosure. As such, Shapley values are summarized below in order to facilitate a better understanding of embodiments of the present disclosure.

[0041] Shapley values are a solution concept in cooperative game theory. In general terms, for a cooperative game in which a coalition of players collectively scores a payout, Shapley values can be used to determine each player’s contribution to the payout, e.g., for the purpose of fairly distributing the payout among the players. As an example, Shapley values can be used to determine each worker’s contribution to the profit of a workers cooperative, and thereby distribute the profit among the workers. In some circumstances, Shapley values may not be necessary. For example, if each worker contributes linearly to revenue (e.g., the profits of the co-op are the sum of the revenue of each worker minus overhead), each worker’s contribution can be calculated directly, e.g., by subtracting each worker’s share of the overhead from their respective sales. In more complex circumstances however, it may not be possible to directly compute contributions in this manner. In such cases, Shapley values may be useful.

[0042] In very general terms, Shapley values can be computed using the average payout of various permutations of players, with and without a particular11KILPATRICK TOWNSEND 78918186 1player, in order to compute the contribution of that particular player to the payout. For example, for a worker’s co-op comprising workers Alice, Bob, and Carol, Alice’s payout can be calculated based on an average of: (1) what Alice contributes alone, (2) what Alice adds to Bob’s contribution, (3) what Alice adds to Carol’s contribution, and (4) what Alice adds to Bob and Carol’s collective contribution.

[0043] The concept of Shapley values can be applied to machine learning by treating each input feature as a player and treating the output of a machine learning model (or e.g., the output relative to some average or reference output) as the payout. In this way, the contribution of each individual input feature to the output can be modeled. For machine learning models, which typically function as black boxes, it can be difficult for people to understand why a model delivers a particular output based on its input features. Shapley values are useful because they provide a relatively human-understandable explanation of the outputs of a machine learning model in terms of their inputs.

[0044] FIG. 1 shows an example of a Shapley value graph, which can be used to summarize the relative contribution of different input features to a resulting machine learning model output. FIG. 1 relates to a hypothetical machine learning model used to evaluate comments posted on a social media website, which could be used as part of an automatic content moderation program. Such a machine learning model could evaluate comments and provide predictive scores indicating whether that comment is likely benign or likely malicious. A score of 0.00 could indicate a completely benign comment, while a score of 1.00 could indicate a completely malicious comment. In FIG. 1 , a prediction axis 102 shows the relatively “maliciousness” of comments, including a reference comment 104 with a score of 0.20 (a generally benign comment), and a test comment 106 with a score of 0.95 (a malicious comment). FIG. 1 also identifies some features 108 used by the machine learning model to produce its predictive scores.

[0045] FIG. 1 further shows five Shapley values 110, represented both numerically and by arrows. These Shapley values span the distance (on prediction axis 102) between the reference comment 104 (which could be representative of the score associated with an “average” comment) and the test comment 106 (e.g., a comment recently evaluated by the machine learning model), and thereby explain12KILPATRICK TOWNSEND 78918186 1the difference in their scores visually and numerically. For example, a feature corresponding to the presence of hyperlinks (which could be indicative of a scam comment) had a large positive contribution to the score of the test comment 106, indicating that the test comment 106 was likely malicious due to these hyperlinks. However, a “coherence score feature”, indicating a very coherent test comment 106 had a large negative contribution to the score of the test comment 106, indicating that the test comment 106 may be benign. As indicated by FIG. 1 a feature corresponding to the presence of suspicious words and a feature corresponding to a short message length have also indicated that the test comment is likely malicious. Finally, 16 other features are grouped and represented by a single Shapley value and arrow.

[0046] By viewing a Shapley value graph such as the Shapley value graph depicted in FIG. 1 , it is relatively easy for a person (e.g., a member of a content moderation staff) to understand and explain why a machine learning model produced a particular output. For example, if a user asks why their comment (e.g., test comment 106) was deleted, a content moderator could view the Shapley graph of FIG. 1 and explain that the presence of hyperlinks and suspicious words in the comment caused the system to flag the comment as malicious.

[0047] In practice, there are various ways in which Shapley Values can be generated or approximated, including both model-agnostic and model-specific techniques. Model-agnostic methods include kernel methods (such as Local Interpretable Model-Agnostic Explanations (LIME)), partition methods (such as Owen values), exact methods (such as optimized enumeration, which supports small feature size), permutation (such as permuting forward and reverse features), and sampling (which may assume independent features). Model-specific methods include tree model explainers, such as TreeSHAP for XGBoost models, deep explainers for neural networks, such as DeepLIFT, gradient explainers for neural networks, and linear explainers for linear models.

[0048] However, most methods for generating or approximating Shapley values are slow and inefficient for large and complex models, which make them poorly suited for production environments. For example, in order to approximate Shapley values, the LIME method involves training a local explanation model, which13KILPATRICK TOWNSEND 78918186 1is a time-consuming process. As another example, while random forest models such as XGBoost often achieve high performance, explainer systems for such tree models are typically very slow. For example, XGBoost’s TreeSHAP explainer has time complexity O(TLD2), where T is the number of trees, L is the number of leaves in a tree, and D is the depth of a tree. As a result, TreeSHAP scales poorly with large numbers of trees or highly complex trees. As such, most explainer systems for generating or approximating Shapley values (or generating explanations based on those Shapley values) are too slow for many production environments, which may have strict time and latency requirements. Additionally, it may be difficult to deploy such explainer systems for segmentation models, as multiple segmentation models may need to be deployed, and therefore multiple model explainers may need to be deployed.

[0049] However, embodiments of the present disclosure address these problems using a fast surrogate explainer. In general terms, rather than generating contribution value-based explanations (in some embodiments, such contribution values may comprise Shapley values) based on predictions from a “heavyweight” primary prediction model, such explanations can be generated based on predictions from a “lightweight” surrogate model. Due to the reduced complexity of the surrogate model, such explanations can be generated more quickly and efficiently.

[0050] As described in more detail below, some embodiments can use novel regularization terms in order to impart knowledge transfer between the primary prediction model and the surrogate model. In this way, embodiments of the present disclosure improve the quality of predictions generated by the surrogate model (e.g., by making them more similar to the primary model) and thereby improve the quality of explanations generated based on such predictions.

[0051] Having described some useful concepts related to embodiments of the present disclosure above, it may now be helpful to describe some systems according to embodiments of the present disclosure, including computer systems that can implement methods according to embodiments (described further below). FIG. 2 shows a computer system 202 that can be used to perform methods according to embodiments. As described in more detail below with reference to FIG. 10, a computer system such as computer system 202 can comprise a processor and a14KILPATRICK TOWNSEND 78918186 1non-transitory computer readable medium (e.g., a hard drive) coupled to the processor. The non-transitory computer readable medium can comprise code or instructions, executable by the processor for performing methods according to embodiments described herein.

[0052] In more detail, computer system 202 can receive features or other data (often referred to herein as “interaction data” or “test interaction data”) and use a primary scoring model 204 (i.e., a machine learning model, sometimes referred to as a “first machine learning model”) to analyze these features or other data, e.g., by producing scores, classifications, etc. For example, in the context of credit card fraud detection, computer system 202 can receive features corresponding to a credit card transaction (which can comprise a type of interaction) and use primary scoring model 204 to produce a fraud score indicating the probability that the credit card transaction is a fraudulent credit card transaction. As another example, in the context of an automatic content moderation system for a social networking website, computer system 202 can receive features corresponding to a post on the social networking website and use primary scoring model 204 to produce a content score indicating whether the post is objectionable or not (e.g., whether the post violates community guidelines). As another example, in the context of a power plant demand forecasting system, computer system 202 can receive features corresponding to various current and historical conditions (e.g., outdoor temperature, previous demand for power) and use primary scoring model 204 to produce an estimated power demand for some period of time in the future.

[0053] While primary scoring model 204 may be used to produce accurate and useful scores or other outputs (particularly if primary scoring model 204 includes a large number of model parameters and was trained on a large and useful set of training interaction data 214), it may be difficult or impossible to generate useful explanations for the outputs of primary scoring model 204 in a timely manner. For example, while an XGBoost primary scoring model 204 may produce accurate scores, a TreeSHAP explainer for such a model may take a comparatively long time (e.g., one or more seconds, which may be infeasible for high-throughput applications with large numbers of processed samples) to produce a useful explanation (e.g., a set of Shapley values, other contribution values, or data derived from such contribution values) that explains the output of the primary scoring model 204.15KILPATRICK TOWNSEND 78918186 1

[0054] As such, a computer system 202 according to embodiments can additionally use a surrogate scoring model 206 (sometimes referred to as a “second machine learning model”) and a surrogate explainer model 216 in order to generate explanations corresponding to outputs (e.g., scores) produced by the primary scoring model 204. In general terms, the surrogate scoring model 206 can comprise a lightweight surrogate for the primary scoring model 204. Explanations produced by the surrogate explainer module 216 based on the outputs of the surrogate scoring model 206 are generally applicable to the outputs of the primary scoring model 204 via the surrogacy relationship between the primary scoring model 204 and the surrogate scoring model 206. The surrogate scoring model 206 can be “lightweight” compared to the primary scoring model 204, e.g., because it can be characterized by a smaller parameter set (i.e. , a second parameter set) that takes up less data than a parameter set associated with the primary scoring model 204. Consequently, any processes for explaining the output of the surrogate scoring model 206 (e.g., implemented by the surrogate explainer module 216) may be executed more quickly for the surrogate scoring model 206 than corresponding processes executed for the primary scoring model 204.

[0055] Various types of machine learning models can be used in embodiments of the present disclosure and non-limiting examples are provided herein. As an example, the primary scoring model 204 could comprise a gradient boosted decision tree model (e.g., an XGBoost model) and the surrogate scoring model 206 could comprise a linear model, tree model, or neural network (e.g., a multilayer perceptron (MLP)). In some embodiments, the surrogate explainer module 216 can comprise a linear explainer, a tree explainer, a deep explainer, or a gradient explainer, e.g., corresponding to the model type of surrogate scoring model 206.

[0056] In an exemplary use case, the computer system 202 can use the primary scoring model 204 to generate a primary fraud score indicating whether a credit card transaction is legitimate or fraudulent. The computer system 202 can additionally use the surrogate scoring model 206 to generate a surrogate fraud score. The surrogate explainer module 216 can be used to explain the surrogate fraud score, e.g., in terms of the features used to produce that surrogate fraud score, e.g., using Shapley values or other contribution values. Because the surrogate fraud16KILPATRICK TOWNSEND 78918186 1score generally corresponds to the primary fraud score, the surrogate explanation produced by the surrogate explainer module 216 generally also explains the primary fraud score.

[0057] As another example, the computer system 202 can use the primary scoring model 204 to predict power demand for an electric power utility over the next eight hours. The computer system 202 can additionally use the surrogate scoring model 206 to generate a surrogate power demand prediction. The surrogate explainer module 216 can be used to explain the surrogate power demand prediction, e.g., in terms of the features used to produce that surrogate power demand prediction, e.g., using Shapley values or other contribution values. Because the surrogate power demand prediction generally corresponds to the primary power demand prediction, the surrogate explanation produced by the surrogate explainer module 216 generally also explains the primary power demand prediction.

[0058] In some embodiments, the primary scoring model 204 may be part of a larger segmentation model comprising multiple primary scoring models. In general terms, for a segmentation model, relevant feature data from an input dataset (e.g., an interaction or test interaction dataset) may be segmented into data subsets, each of which may be processed by a different scoring model (e.g., a different primary scoring model). The segmentation model can combine the outputs of the different scoring models to produce a single output, e.g., a single credit card fraud score from multiple credit card fraud stores, a single power demand prediction from multiple power demand predictions, a single comment score from multiple comment scores, etc. Segmentation models may be useful in situations in which it would be computationally slow, difficult, or infeasible to process a large set of features with a single model, as many models scale quadratically (or worse than quadratically) with the number and size of input features.

[0059] Regardless, even though FIG. 2 depicts only a single primary scoring model 204 and a single surrogate scoring model 206, in some embodiments the computer system 202 may instantiate, train, and / or utilize multiple primary scoring models and surrogate scoring models. It should be appreciated more generally that the numbers of devices, entities, and components shown in FIG. 2 were selected for simplicity of illustration and exposition. It should be understood however that17KILPATRICK TOWNSEND 78918186 1systems according to embodiments of the present disclosure can include more than one of each device, entity, component, computer system, etc. In addition, some systems according to embodiments may include a lesser number of devices, entities, and / or components or a greater number of devices, entities, and / or components than those shown in FIG. 2. For example, computer system 202 may comprise a distributed computing system comprising several computers collectively performing methods according to embodiments. Likewise, there may be multiple training data sources 208 from which computer system 202 retrieves training interaction data 214 for the purpose of training machine learning models and multiple communication networks 212 over which computer system 202 communicates with client computers(s) 210.

[0060] Contribution values (e.g., Shapley values) generated using the surrogate explainer module 216 can be used for various useful purposes. For example, an analysis at a power plant could analyze such contribution values to determine whether a particular feature or event (e.g., a winter storm) leads to a significant increase to the demand for electrical power. However, the analysis of such contribution values may require special knowledge of machine learning, game theory, and other associated topics. As such, in some cases it may be helpful to further summarize these contribution values so that the explained output of a machine learning model can be better understood by human users of computer system 202. In some embodiments, this can be accomplished through the generation of reason codes and descriptions.

[0061] As described above with reference to FIG. 1 , a Shapley value (or other contribution value) can correspond to a specific feature of an input data set and can explain the numerical contribution of that feature to the numerical output of a machine learning model. By contrast, a “reason code” can correspond to a general explanation of the output of a machine learning model and can correspond to any number of features. FIG. 3 shows a table 302 identifying some features 304, reason codes 306, and descriptions 308 corresponding to a fraud score output of a primary scoring model used to evaluate credit card fraud.

[0062] In FIG. 3 features Foand F2correspond to a reason code “1A”, which is described by the description “suspicious originator entity”. In general terms, this18KILPATRICK TOWNSEND 78918186 1could mean that the values of features Foand F2support the conclusion that an originator entity (e.g., a cardholder) is behaving in a suspicious manner, supporting a conclusion by a primary scoring model that a particular credit card transaction is fraudulent, and summarized by the reason code “1A”. As an example, the cardholder (or, e.g., an individual who stole the credit card from the rightful cardholder) could be making a purchase from a strange location at a strange time of night.

[0063] Likewise, features FltF3, and F4correspond to a reason code “1 B”, which is described by the description “suspicious beneficiary entity”. In general terms, this could mean that the values of features FltF3, and F4support the conclusion that a beneficiary entity (e.g., a merchant transacting with a cardholder) is behaving in a suspicious manner, supporting a conclusion by a primary scoring model that a particular credit card transaction is fraudulent, and summarized by the reason code “1 B”. As an example, a merchant could be charging a cardholder an exorbitant amount of money for a good or service with a well-established price.

[0064] Referring back to FIG. 2, generally, such reason codes and descriptions may be helpful to human operators of computer system 202 (or other individuals), in addition to (or instead of) contribution values. For example, a customer service representative of a social networking service can quickly determine, based on reason codes alone (or with the help of a corresponding description if the reason code is unknown to the customer service representative) why a particular post was flagged or removed by a machine learning based automated content moderation system. This can enable the customer service representative to explain to a user why their post was removed, e.g., in terms of the community guidelines violated by the post. By contrast, it may be difficult to explain such information to the user based on the contribution values alone.

[0065] Scores, contribution values, and other explanatory data (e.g., reason codes and descriptions) can be used or analyzed by an operator of computer system 202 or an organization that owns and / or operators computer system 202. For example, an analyst at a power plant could use reason codes and descriptions to explain a sudden increase (or projected increase) in power demand.19KILPATRICK TOWNSEND 78918186 1

[0066] As an alternative however, computer system 202 could be used to implement machine learning based analysis and machine learning model output explanation as a service for others, e.g., on behalf of client computer(s) 210 or users of client computer(s) 210 (which may also be referred to as “requestors”). For example, a client computer 210 could comprise a computer associated with a social networking web company. The social networking web company could outsource their content moderation to an organization associated with computer system 202 (which could comprise a server computer). In such a case, client computer 210 could request scores (e.g., scores indicating whether a posted comment violates community guidelines or is otherwise malicious) and explanations (e.g., reason codes, descriptions, contribution values, etc.) corresponding to those scores. Computer system 202 could use primary scoring model 204 to generate such scores, along with surrogate scoring model 206 and surrogate explainer module 216 to generate contribution values, reason codes, descriptions etc., and transmit those scores and explanations back to the client computer 210, which could then use them for some purpose (e.g., removing comments that violate community guidelines).

[0067] In such scenarios, client computer(s) 210 and computer system 202 can communicate over a communication network 212. A communication network such as communication network 212 can take any suitable form, and may include any one and / or the combination of the following: a direct interconnection; the Internet; a Local Area Network (LAN); a Metropolitan Area Network (MAN); an Operating Missions as Nodes on the Internet (OMNI); a secured custom connection; a Wide Area Network (WAN); a wireless network (e.g., employing protocols such as, but not limited to a Wireless Application Protocol (WAP), l-mode, and / or the like); and / or the like. Messages between computers and devices in the system of FIG. 2 and / or over communication network 212 may be transmitted using a secure communication protocol, such as, but not limited to, File Transfer Protocol (FTP); HyperText Transfer Protocol (HTTP); Secure HyperText Transfer Protocol (HTTPS); Secure Socket Layer (SSL), ISO (e.g., ISO 8583) and / or the like. Any suitable communication protocol can be used to communicate over the communication network 212, e.g., for the purpose of creating one or more communication channels. A communication channel may, in some instances, comprise a secure communication channel, which may be established in any known manner, such as20KILPATRICK TOWNSEND 78918186 1through the use of mutual authentication, a session key, and establishment of a Secure Socket Layer (SSL) session.

[0068] In some embodiments, such requests from client computer(s) 210 may contain all the information (e.g., interaction datasets, features, etc.) necessary for computer system 202 to generate predictions, scores, or any other data using primary scoring model 204 and generate contribution values, reason codes, explanations, etc., using surrogate scoring model 206 and surrogate explainer module 216. However, in other embodiments, computer system 202 may retrieve relevant interaction datasets, features, etc., from another data source 208, e.g., a memory element (e.g., a hard drive), database, or any other data structure or storage element, in order to service requests from client computer(s) 210.

[0069] For example, an analyst at a national weather service may request a machine learning based weather forecast in a distance region of their country. The analyst may use a client computer 210 to connect to computer system 202, which is located at a remote weather monitoring station. Using data stored on a hard drive located in the remote weather monitoring station, computer system 202 may use primary scoring model 204 to generate a weather forecast and additionally use surrogate scoring model 206 and surrogate explainer module 216 to generate an explanation (comprising e.g., contribution values, reason codes, descriptions, etc.) corresponding to that weather forecast. For example, a forecast indicating a likely storm could be explained by a reason code “WARM-COLD”, corresponding to a description of “collision of a warm air front and a cold air front”, indicating that a predicted storm is likely a result of these two colliding air fronts. The forecast and explanations can be returned to the client computer 210 and can then be interpreted by the weather analyst.

[0070] In embodiments, various combinations of scores, predictions, contribution values, reason codes, and descriptions can be provided to client computer(s) 210 and / or operators of computer system 202. For example, computer system 202 could transmit a score, reason code, description, and a contribution value to a client computer 210. As an alternative, computer system 202 could transmit a score, reason code, and a description to a client computer 210, but not transmit any corresponding contribution values, as the client computer 210 (or its21KILPATRICK TOWNSEND 78918186 1operator) may be ill-equipped to interpret such contribution values in a meaningful way.

[0071] In addition to using primary scoring model 204, surrogate scoring model 206, and surrogate explainer module 216 to generate scores or predictions, contribution values, reason codes, descriptions, etc., computer system 202 can also train these models for this purpose. Such training methods are described in more detail further below with reference to FIGs. 4, 5, and 8. In general terms however, the computer system 202 can train the surrogate scoring model 206 to approximate the primary scoring model 204. In some embodiments, the primary scoring model 204 may have been “pre-trained”, e.g., by computer system 202 or by another computer system. Computer system 214 may retrieve any relevant “training interaction data” (e.g., from data source 208) to train the surrogate scoring model 206 (and the primary scoring model 204, if necessary). Such data could comprise, e.g., data corresponding to historical interactions or other events. For example, for a primary scoring model 204 used to classify credit card transactions as normal or fraudulent, the training interaction data could comprise labelled data corresponding to historical credit card transactions.

[0072] Having described some systems according to embodiments of the present disclosure, it may now be helpful to describe some methods. As described above, the output of a surrogate scoring model can be used to produce explanations (e.g., contribution values and reason codes), which can serve as surrogate explanations for a primary scoring model. This may be useful in real-time applications, as it may be difficult or impossible to generate explanations for a large and complex (e.g., comprising a large number of parameters) primary scoring model in a reasonable timeframe.

[0073] As such, some embodiments are directed to methods for training a surrogate scoring model according to embodiments. Other embodiments are directed to methods for using a surrogate scoring model to generate explanations (e.g., contribution values, reason codes, etc.) corresponding to the output of a primary scoring model, thereby explaining the output of the primary scoring model in a fast and efficient manner. FIG. 4 depicts a block diagram of a surrogate model training system 402, which is also described below with reference to the flowchart of22KILPATRICK TOWNSEND 78918186 1FIG. 5. FIG. 7 depicts a flowchart of an inference method, e.g., during which a computer system can generate a prediction (or other output) using a primary scoring model and generate an explanation (e.g., contribution values, reason codes, etc.) using a surrogate scoring model. FIG. 8 depicts a stateless surrogate explanation system (described in more detail further below), an alternate to the system depicted in FIG. 4. While the stateless surrogate explanation system may be less complex and more straightforward to implement than the training system of FIG. 4, it may generally impart less knowledge transfer from the primary scoring model to the surrogate scoring model.

[0074] FIG. 4 shows a block diagram of a surrogate model training system 402 according to some embodiments. In general, the surrogate model training system 402 can be used to train a second machine learning model 412 (e.g., a surrogate scoring model, as described above with reference to FIG. 2) to generally act as a surrogate for a first machine learning model 404 (e.g., a primary scoring model, as described above with reference to FIG. 2). As described above, while it is possible to generate explanations (comprising, e.g., contribution values (which may themselves comprise Shapley values) or data derived from such contribution values (such as reason codes or descriptions) corresponding to outputs of the first machine learning model 404, it may take too much time to produce such explanations (e.g., due to model size, complexity, or other factors). As such, it may not be possible to generate such explanations in a real-time production setting. However, explanations generated based on the comparatively lightweight second machine learning model 412 can be generated much more quickly (e.g., on the order of milliseconds). Thus, by training the second machine learning model 412 to act as a surrogate for the first machine learning model 404, a computer system according to embodiments can enable rapid generation of useful explanations corresponding to outputs of the first machine learning model 404, based on the outputs of the second machine learning model 412.

[0075] In general terms, a computer system can acquire training interaction data sets 406 (corresponding to e.g., credit card transactions, posts or other interactions on social media websites, etc.), which may be labeled, and which can be used as training data. The computer system can sample a batch of training interaction data sets and use the first machine learning model 404 (which may have23KILPATRICK TOWNSEND 78918186 1been previously trained) to generate first predictions 408 corresponding to the training interaction data sets 406. Such first predictions 408 can comprise various types of data. In the context of e.g., predicting power demand, such first predictions 408 could comprise predicted power demand (e.g., in megawatt hours) based on training interaction data sets 406 comprising features predictive of future power demand (e.g., outdoor air temperature). Such first predictions 408 can be treated as “ground truth” data used to train the second machine learning model 412 to act as a surrogate for the first machine learning model 404. The computer system can likewise use the (currently untrained) second machine learning model 412 to produce second predictions 414 corresponding to the batch of training interaction data sets 406.

[0076] From these first predictions 408 and second predictions 414, the computer system can generate sets of first contribution values 410 (corresponding to the first predictions 408) and sets of second contribution values 418 (corresponding to the second predictions 414). In some embodiments these contribution values can comprise Shapley values. Various methods can be used to generate the sets of first contribution values 410 and sets of second contribution values 418, which may depend on what types of machine learning models are used in the surrogate model training system 402. For example, if the first machine learning model 404 comprises an XGBoost model, the sets of first contribution values 410 could be generated using a built-in TreeSHAP explanation method. Likewise, if the second machine learning model 412 comprises a multilayer perceptron or other neural network, the computer system could use a deep explainer to generate the sets of second contribution values 418. The computer system can use a contribution value computation method 416 in order to generate the sets of second contribution values.

[0077] Both the sets of first contribution values 410 and the sets of second contribution values 418 can be encoded using encoder 420, thereby generating first contribution value latent vectors 422 and second contribution value latent vectors 424. In general terms, such contribution value latent vectors can encode the distributions of the sets of first contribution values 410 and the sets of second contribution values 418. As described in more detail below with reference to FIG. 5, a loss function can be computed based on the first predictions 408, second predictions 414, the sets of first contribution values 410, the sets of second24KILPATRICK TOWNSEND 78918186 1contribution values 418, the first contribution value latent vectors 422, and the second contribution value latent vectors 424. Such a loss function can be used to train the second machine learning model 412, e.g., via backpropagation. Afterwards, a subsequent batch of training interaction data sets can be sampled, and the training process can be repeated until some terminating condition has been met (e.g., a set number of training rounds, model parameter convergence, etc.). After such training, the first machine learning model 404 and second machine learning model 412 can be used to generate predictions and explanations, e.g., the first machine learning model 404 can be used to generate first predictions, and the second machine learning model 412 can be used to generate second predictions, from which explanations can be generated. In some embodiments, such first predictions and explanations can be transmitted to a client computer, as described above with reference to FIG. 2.

[0078] A training method according to embodiments is described in more detail below with reference to the flowchart of FIG. 5. The method can be performed by a computer system (e.g., a computer system instantiating and training machine learning models according to embodiments) and can involve the computer system performing an iterative training process.

[0079] At step 502, after retrieving training data comprising a plurality of training interaction data sets (e.g., from a database, a data stream, a local memory element such as a hard drive, cloud storage, an I / O interface, or any other appropriate source), the computer system can perform a round of an iterative training process. The round of the iterative training process can comprise steps 504- 520, described in more detail below. The computer system can perform this iterative training process until a terminating condition has been met, e.g., a set number of training rounds or epochs, a convergence condition, or any other appropriate terminating condition.

[0080] Such training interaction data sets can comprise, e.g., sets of features corresponding to various types of interactions. For example, if embodiments of the present disclosure are used to generate predictions related to electrical power demand, such training interaction data sets could comprise sets of features predictive of electrical power demand (e.g., the time of year, the time of day, outdoor25KILPATRICK TOWNSEND 78918186 1temperature, etc.) as well as power demand values (e.g., in megawatt hours), which may comprise the “labels” of the training interaction data sets. As another example, if embodiments of the present disclosure are used to generate predictions related to the legitimacy (or fraudulence) of credit card transactions, the training interaction data sets could comprise sets of features predictive of credit card fraud (e.g., a purchase amount, a purchase location, a purchase time of day, frequency of purchases, etc.) and labels indicating whether such credit card transactions were legitimate or fraudulent. In some embodiments, training interaction data may comprise message data (e.g., corresponding to a message-based interaction, such as a communication between two computer systems over the Internet), transaction data (e.g., corresponding to credit card transactions), or machine performance data (e.g., corresponding to the performance of a computer system such as a server computer, relative to some benchmark).

[0081] At step 504, the computer system can sample a batch of training interaction data sets (e.g., from any training interaction data sets retrieved from a database or other appropriate data source). Such training interaction data sets can correspond to one or more training interactions, e.g., one or more training credit card transactions for a credit card fraud detection system. Each training interaction data set can comprise a plurality of training interaction data set features. For example, as described above, interaction data set features such as a purchase amount, a purchase location, a purchase time of day, or the frequency of purchases could comprise features used to evaluate the legitimacy of card-based interactions (e.g., credit card transactions transactions).

[0082] At step 506, the computer system can use a first machine learning model to generate a first prediction for each training interaction data set in the batch of training interaction data sets. In this way, the computer system can generate one or more first predictions corresponding to the one or more training interaction data sets in the batch of training interaction data sets. As described above, the first machine learning model can comprise a pre-trained primary scoring model used to produce accurate and high-quality predictions or scores (e.g., credit card fraud predictions or credit card fraud scores). As such, in some embodiments the one or more first predictions can comprise one or more primary predictions. Further, in26KILPATRICK TOWNSEND 78918186 1some embodiments the first machine learning model can comprise an XGBoost model, a type of highly accurate gradient boosted decision tree model.

[0083] At step 508, the computer system can generate one or more sets of first contribution values based on the one or more first predictions. In some embodiments, the one or more sets of first contribution values can comprise one or more sets of first Shapley values or one or sets of first Shapley value approximations. In some embodiments, the one or more sets of first contribution values can describe one or more first relationships between the one or more first predictions and the plurality of training interaction data set features. Expressed in other words, a particular contribution value can indicate how a given training interaction data set feature contributes to a respective first prediction, e.g., numerically, as described above with reference to the Shapley value graph of FIG. 1 .

[0084] There are a variety of ways in which the one or more first sets of contribution values can be generated, ways which may depend on the nature of the first machine learning model. For example, for an XGBoost model, a built-in tree explainer such as TreeSHAP can be used to generate the one or more sets of first contribution values. As another example, for a first machine learning model comprising a multilayer perceptron, a deep explainer method (e.g., DeepLIFT or another similar method) could be used by the computer system to generate the one or more first sets of first contribution values.

[0085] In general terms, a deep explainer such as DeepLIFT can be used to mathematically define the difference in the output of a neural network (relative to a reference output), in terms of the difference in the input features (relative to reference input features):where At is a difference between an output and the reference output, Ax, is the difference between an input xtand an analogous reference input, and CAx.Atis the “contribution” of the input difference Ax, to the output difference At. The contributions of the differences of each input feature Ax, are effectively the Shapley value (or approximates of the Shapley values).27KILPATRICK TOWNSEND 78918186 1

[0086] Generally, DeepLIFT or other deep explainers can use backpropagation, e.g., by “working backwards” from the output neurons of a neural network and computing the contributions CAxfAtto the output difference At for neurons in a given layer (e.g., layer n) based on differences in inputs Ax, produced by neurons in a previous layer (e.g., layer n - 1). By working through the entire neural network in this manner, a deep explainer can calculate the contributions CAX[Atto the differences At in the model output based on differences Ax, in the input features. These contributions can effectively comprise approximations of Shapley values.

[0087] It should be understood that the examples provided above (e.g., TreeSHAP and DeepLIFT) are non-limiting examples and that various other methods can be used by the computer system to generate the one or more first sets of contribution values and one or more sets of second contribution values, as described below with reference to step 512.

[0088] At step 510, the computer system can use a second machine learning model to generate a second prediction for each training interaction data set in the batch of training interaction data sets. In this way, the computer system can generate one or more second predictions corresponding to the one or more training interaction data sets in the batch of training interaction data sets. As described above, the second machine learning model can comprise a surrogate scoring model that can be trained (e.g., using the method of FIG. 5) to approximate a primary machine learning model. As such, in some embodiments the one or more second predictions can comprise one or more surrogate predictions, which can comprise one or more approximates of the one or more first (primary) predictions. Various types of second machine learning models can be used in embodiments of the present disclosure. As non-limiting examples, in some embodiments, the second machine learning model can comprise a linear model, tree model, or neural network model (e.g., a multilayer perceptron).

[0089] The second machine learning model may be a lightweight machine learning model compared to the first machine learning model. For example, a parameter set characterizing the second machine learning model (i.e., a “second parameter set”) may have a smaller logical size than a first parameter set used to28KILPATRICK TOWNSEND 78918186 1characterize the first machine learning model. In other words, a second data size (e.g., in megabytes, gigabytes, etc.) of the second parameter set may be smaller than a first data size of the first parameter set. Due to the smaller parameter set size, the computational complexity for generating second contribution values (e.g., Shapley values) based on second predictions produced by the second machine learning model may be lower than the computational complexity for generating first contribution values based on first predictions produced by the first machine learning model, and consequently a computer system may be able to generate such second contribution values much more quickly (e.g., on the order of milliseconds, as shown in Table 902 of FIG. 9).

[0090] At step 512, the computer system can generate one or more sets of second contribution values based on the one or more second predictions. In some embodiments, the one or more sets of second contribution values can comprise one or more sets of second Shapley values or one or sets of second Shapley value approximations. In some embodiments, the one or more sets of second contribution values can describe one or more second relationships between the one or more second predictions and the plurality of training interaction data set features. Expressed in other words, a particular contribution value can indicate how a given training interaction data set feature contributes to a respective second prediction, e.g., numerically, as described above with reference to the Shapley value graph of FIG. 1. As the second machine learning model is being trained by the computer system to generally approximate the first machine learning model, the one or more sets of second contribution values can approximate one or more first relationships between one or more first predictions (e.g., generated by the first machine learning model at step 506) and the plurality of training interaction data set features.

[0091] There are a variety of ways in which the one or more second sets of contribution values can be generated. For example, in embodiments in which the second machine learning model comprises a linear model, a tree model, or a neural network (e.g., a multilayer perceptron), the one or more sets of second contribution values can be generated using a linear explainer, a tree explainer, a deep explainer, or a gradient explainer. More details on the generation of contribution values (e.g., with reference to a deep explainer such as DeepLIFT) are provided above in the description of step 508.29KILPATRICK TOWNSEND 78918186 1

[0092] As described in more detail below (e.g., with reference to step 516), the computer system can determine one or more loss values based at least on the one or more sets of first contribution values and the one or more sets of second contribution values. Such loss values can be used to train the second machine learning model (e.g., at step 518). As described in more detail below, the computer system can additionally use the one or more first predictions and the one or more second predictions to determine the one or more loss values. Further the computer system may use one or more first contribution value latent vectors and one or more second contribution value latent vectors to determine the one or more loss values. As such, in some embodiments, determining the one or more loss values based on at least the one or more sets of first contribution values and the one or more sets of second contribution values may comprise determining the one or more first contribution value latent vectors and the one or more second contribution value latent vectors.

[0093] As such, at step 514 the computer system can generate the one or more first sets of contribution value latent vectors based on the one or more sets of first contribution values using an encoder (e.g., encoder 420 from FIG. 4). Likewise, the computer system can generate the one or more second sets of contribution value latent vectors based on the one or more sets of second contribution values using the encoder. The one or more first contribution value latent vectors can encode the distribution of the one or more sets of first contribution values, and the one or more second contribution value latent vectors can encode the distribution of the one or more sets of second contribution values. The numbers of first contribution value latent vectors and second contribution value latent vectors can comprise hyperparameters that can be set by an individual implementing methods according to embodiments, as different numbers of contribution value latent vectors may result in better performance by the second machine learning model in different contexts, resulting on more accurate contribution values and data derived from such contribution values (e.g., reason codes and descriptions). Additionally, the particular type of encoder used to generate the sets of contribution value latent vectors can also comprise a hyperparameter established by an individual implementing methods according to embodiments.30KILPATRICK TOWNSEND 78918186 1

[0094] At step 516, the computer system can determine one or more loss values based on at least the one or more sets of first contribution values and the one or more sets of second contribution values. In general terms, the one or more loss values can be determined by comparing the one or more sets of first contribution values and the one or more sets of second contribution values, e.g., the one or more loss values can be based on differences between the one or more sets of first contribution values and the one or more sets of second contribution values. If the sets of second contribution values are similar to the sets of first contribution values, then the loss values may be low, while if the sets of second contribution values are dissimilar to the sets of first contributions values, then the loss values may be high. Various statistical metrics can be used as loss values or used to derive loss values, such as the mean-squared error.

[0095] In some embodiments, the one or more loss values can be determined based on one or more first contribution value latent vectors and one or more second contribution value latent vectors in addition to the one or more sets of first contribution values and the one or more sets of second contribution values. In some embodiments, the one or more loss values can be determined based on the one or more first predictions and the one or more second predictions in addition to the other data described above.

[0096] In some embodiments, the one or more loss values can comprise several regularizer terms. These regularizer terms can be summed together in a weighted combination to generate a combined loss value. This combined loss value can be used to train the second machine learning model, e.g., as described further below. In some embodiments, these regularizer terms can include a first loss value, a second loss value, a third loss value, a fourth loss value, and a fifth loss value (although it should be understood that embodiments of the present disclosure can be practiced with any number of loss values).

[0097] In some embodiments, the first loss value can comprise a binary entropy loss BCE(Ytrn,Ptrn) between the one or more second predictions (Ptrn, as depicted in FIG. 4) and one or more labels Ytrncorresponding to the one or more training interaction data sets. In some embodiments, the first loss value can comprise a risk score regression regularizer. As shown in Table 602 of FIG. 631KILPATRICK TOWNSEND 78918186 1(which summarizes some of the insights behind some regularizer terms according to embodiments), the first loss value can be used to train the second machine learning model to match the scoring distribution of the first machine learning model, e.g., in order to produce second predictions that generally match the first predictions.

[0098] In some embodiments, the second loss value can comprise a regression losso* Lregressor(Pt:rn, Ptrn) between the one or more first predictions (Ptrn) and the one or more second predictions (Ptrn). This second loss value can additionally comprise a risk score regression regularizer and can be used to train the second machine learning model to match the scoring distribution of the first machine learning model, e.g., in order to produce second predictions that generally match the first predictions. The termocan comprise a weighing hyperparameter.

[0099] In some embodiments, the third loss valuecan be based on a Kullback-Leibler divergence between the one or more first contribution value latent vectors Veelistand the one or more second contribution value latent vectors V^ist. The third loss value can be used to train the second machine learning model to generate predictions that result in a similar distribution of contribution values (such contribution value distributions may be encoded by the first contribution value latent vectors and the one or more second contribution value latent vectors). The term can comprise a weighing hyperparameter.

[0100] In some embodiments, the fourth loss value can comprise a feature ranking loss regularizer term A2In order to determine the fourth loss value, the computer system can determine one or more first rankings R(S7trn) corresponding to the one or more sets of first contribution values and one or more second rankings corresponding to the one or more sets of second contribution values R(sPtrn). The computer system can determine the fourth loss value based on the one or more first rankings and the one or more second rankings, e.g., according to the formula provided above. These first rankings and second rankings can comprise ordered lists of contribution values and reason codes based on their magnitude (or other characteristics). The fourth loss value can generally relate to the second machine learning model’s ability to generate second predictions resulting in second contribution values that generally occupy the same relative rank as first32KILPATRICK TOWNSEND 78918186 1contribution values generated for first predictions. The fourth loss value can thereby be used to train the system to rank more important features (corresponding to such contribution values) more highly than less important features. The term A2can comprise a weighing hyperparameter.

[0101] The computer system can use various methods to generate reason codes based on contribution values, thereby enabling the computer system to generate rankings based on such reason codes. For example, a mapping function or program can be used to map particular contribution values or combinations of contribution values to reason codes, enabling such reason codes to be ranked. As an example, in the context of predicting power demand, high contribution values corresponding to features such as “outdoor temperature” and “precipitation” could be mapped to a reason code such as “INCL_WEATHER” using an application or other software, indicating that a power demand prediction produced by a machine learning model (e.g., a primary scoring model) may be a result of inclement weather conditions. As another example, in the context of predicting credit card fraud, high contribution values corresponding to features such as “unusual purchase location” and “unusual purchase time” could be mapped to a reason code “1A” (e.g., as depicted in table 302 of FIG. 3), indicating a suspicious originator entity. In some embodiments, first ranking lists of first contribution values and second ranking lists of second contribution values can be used by the computer system to generate rankings and ranking lists of reason codes. For example, the computer system could map the top two contribution values in a ranking list to a reason code, then map the third and fourth contribution value in a ranking list to a different reason code, and so on until a ranking list of reason codes has been generated.

[0102] In some embodiments, the fifth loss value can comprise a reason code ranking loss regularizer term A3Similar to the fourth loss value, the computer system can determine the fifth loss value based on the one or more first rankings and the one or more second rankings, e.g., according to the formula provided above. These first rankings and second rankings can comprise ordered lists of contribution values and reason codes based on their magnitude (or other characteristics). The fifth loss value can generally relate to the second machine learning model’s ability to generate second predictions resulting in reason codes that generally occupy the same relative rank as reason codes33KILPATRICK TOWNSEND 78918186 1generated for first predictions. As such, the fifth loss value can be used to train the system to rank more important reason codes (corresponding to such contribution values) more highly than less important features. The term3can comprise a weighing hyperparameter.

[0103] In some embodiments, a combined loss value Loss can be determined based on the first, second, third, fourth, and fifth loss value, e.g., according to the formula below:As described further below, e.g., with reference to step 518, the one or more loss values (e.g., via the combined loss value) can be used to train the second machine learning model, e.g., via backpropagation or other appropriate techniques.

[0104] Referring back to FIG. 5, at step 518, the computer system can update a parameter set of at least the second machine learning model based on the one or more loss values. In this way, the computer system can train the second machine learning model. In some embodiments, the computer system can additionally update an encoder parameter set of the encoder (which may have been used to generate contribution value latent vectors, e.g., at step 514), thereby training the encoder. The computer system can use any appropriate technique for updating the parameter set (and the encoder parameter set), such as using stochastic gradient descent to determine differential changes in the model parameters that result in the greatest immediate reduction to the one or more loss values (produced, e.g., at step 516), using backpropagation, and / or using any other appropriate technique.

[0105] At step 520, the computer system can determine if a terminating condition has been met. As described above, in some embodiments the terminating condition can comprise a defined number of training rounds, and the terminating condition can be met if a total number of training rounds performed equals or exceeds the defined number of training rounds. In other embodiments, the34KILPATRICK TOWNSEND 78918186 1terminating condition can comprise a convergence condition. This terminating condition can be met if the set of model parameters converge, e.g., exhibit little to no change in consecutive training rounds. If the terminating condition has not been met, the computer system can return to step 502 and repeat the iterative training process until the terminating condition has been met, e.g., by sampling a new batch of training interaction data sets. Otherwise at step 522 the computer system can complete the iterative training process. At this point, the parameters of the second machine learning model (and e.g., the encoder) can be fixed, and the second machine learning model can be used to generate second predictions (which can then be used to generate contribution values, reason codes, descriptions, etc.) during production.

[0106] As described above with reference to FIG. 2, in some embodiments the first machine learning model can be part of a segmentation model. A segmentation model can comprise multiple machine learning models, and each model can process a different segment of the input data features. As such, in some embodiments the first machine learning model can comprise a first primary machine learning model. A segmentation model according to embodiments can additionally comprise a second primary machine learning model. Likewise, the second machine learning model can comprise a first surrogate machine learning model (corresponding to the first primary machine learning model), and a second surrogate machine learning model can correspond to the second primary machine learning model. Such a segmentation model can comprise any number of additional machine learning models (e.g., a third primary machine learning model, a fourth primary machine learning model, etc.), however for ease of exposition, the description below will focus on a first primary machine learning model and a second primary machine learning model.

[0107] In such embodiments, the computer system can additionally train the second surrogate machine learning model (and e.g., a third surrogate machine learning model, fourth surrogate machine learning model, etc.). As such, the iterative training process described above can comprise a first iterative training process (terminating when a first terminating condition has been met), and the computer system can additionally perform a second iterative training process until a second terminating condition has been met. The second iterative training process can comprise training the second surrogate machine learning model corresponding35KILPATRICK TOWNSEND 78918186 1to the second primary machine learning model. This second iterative training process can comprise substantially the same steps described above (or any other appropriate training steps). The computer system can perform any number of additional training processes for any number of additional machine learning models in the segmentation model (e.g., a third iterative training process, a fourth iterative training process, etc.). In some embodiments, such iterative training processes can be performed concurrently, e.g., via a parallel computing system. In some embodiments, such iterative training processes can be performed using shared loss functions.

[0108] Regardless, after training the second machine learning model (and e.g., any number of additional surrogate machine learning models), the computer system can use the second machine learning model to generate explanations during an inference phase, as described below with reference to the flowchart of FIG. 7.

[0109] As summarized above, some embodiments are directed to methods (which can be performed by a computer system) for generating predictions and explanations (comprising e.g., Shapley value, reason codes, descriptions, etc.). FIG. 7 depicts a flowchart of one such method. At step 702, a computer system can receive a set of test interaction data. In some embodiments, the computer system can receive the set of test interaction from a database, data stream, a local memory element such as a hard drive, cloud storage, an I / O interface, or any other appropriate source. In other embodiments, the computer system can comprise a server computer and the computer system can receive the set of test interaction from a client computer, e.g., as described above with reference to FIG. 2, e.g., over a communication network such as the Internet. In such embodiments, the computer system can comprise a server computer that performs machine learning based analysis as a service for the client computer. The set of test interaction data can comprise a plurality of test interaction data features. For example, for test interaction data corresponding to electrical power demand, such test interaction data features could comprise sets of features predictive of electrical power demand (e.g., the time of year, the time of day, outdoor temperature, etc.). As another example, if embodiments of the present disclosure are used to generate predictions related to the legitimacy (or fraudulence) of credit card transactions, the test interaction data36KILPATRICK TOWNSEND 78918186 1set could comprise sets of features predictive of credit card fraud (e.g., a purchase amount, a purchase location, a purchase time of day, frequency of purchases, etc.).

[0110] At step 704, the computer system can use a first machine learning model to generate a first test prediction based on the set of test interaction data. As described above, the first machine learning model can comprise a primary scoring model used to produce accurate and high-quality predictions or scores (e.g., credit card fraud predictions or credit card fraud scores), and as such, the first test prediction can comprise a primary test prediction. In some embodiments, the first machine learning model can comprise an XGBoost model.

[0111] At step 706, the computer system can use a second machine learning model (e.g., trained using the methods described above with reference to FIG. 5) to generate a second test prediction based on the set of test interaction data. As described above, in some embodiments the second machine learning model can comprise a surrogate scoring model trained to approximate the first machine learning model. In such embodiments, the second test prediction can comprise a surrogate test prediction, which can comprise an approximation of the first test prediction. Various types of second machine learning models can be used in embodiments of the present disclosure. As non-limiting examples, in some embodiments, the second machine learning model can comprise a linear model, tree model, or neural network model (e.g., a multilayer perceptron).

[0112] At step 708, the computer system can generate a set of contribution values based on the second prediction. In some embodiments, the set of contribution values can comprise a set of Shapley values. The computer system can use any appropriate means to generate the set of contribution values, which may depend on the nature of the second machine learning model. For example, in embodiments in which the second machine learning model comprises a linear model, a tree model, or a neural network (e.g., a multilayer perceptron), the set of contribution values can be generated using a linear explainer, a tree explainer, a deep explainer, or a gradient explainer. More details on the generation of contribution values (e.g., with reference to a deep explainer such as DeepLIFT) are provided above in the description of FIG. 5. The set of contribution values can describe a relationship between the second test prediction and the plurality of test37KILPATRICK TOWNSEND 78918186 1interaction data features. For example, as described above with reference to FIG. 1 , the set of contribution values can describe the contribution of individual test interaction data features to the second test predictions. As the second machine learning model approximates the first machine learning model, this relationship can thereby approximate a primary relationship between the first test prediction and the plurality of test interaction data features. In this way, the set of contribution values can be used to explain the first prediction even though they were not directly generated from the first prediction.

[0113] At step 710, the computer system can output the first test prediction and the set of contribution values generated based on the second test prediction. As described above with reference to FIG. 2, there are various ways in which the first test prediction and the set of contribution values can be output. For example, the first test prediction and the set of contribution values can be provided to an operator of the computer system, e.g., by displaying the first test prediction and the set of contribution values on a monitor or by saving the first test prediction and the set of contribution values in a text document (or other appropriate file type).

[0114] In some embodiments, outputting the first test prediction and the set of contribution values can comprise determining one or more reason codes and / or one or more descriptions associated with the set of contribution values. In some embodiments, outputting the first test prediction and the set of contribution values can comprise performing further processing based on at least the first test prediction and the one or more reason codes. Various forms of further processing can be performed, which may depend on the particular context or use case for machine learning and machine learning explanation systems according to embodiments. As general examples, a computer system could automatically generate and relay a message (e.g., to another computer or device or an operator of such a computer or device) based on the first test prediction and / or the one or more reason codes. As another example, the computer system could automatically shut down or start up another machine (e.g., another computer system) based on the first test prediction and the set of contribution values.

[0115] As a more specific example, a computer system at a hydroelectric dam could be used to predict power demand using machine learning. Such a computer38KILPATRICK TOWNSEND 78918186 1system could interface with a hydroelectric governor used to control the water flow through turbines in the hydroelectric dam, and thereby controlling the speed of the turbines and / or the power output. In such cases, “further processing” could comprise signaling a governor controller to adjust the flow of water through the turbine based on the first test prediction (e.g., a power demand prediction) and the one or more reason codes.

[0116] As another example, a computer system associated with a payment processing network could be used to detect instances of credit card fraud using machine learning. Such a computer could perform additional processing by “freezing” or otherwise marking an account associated with a particular credit card holder responsive to generating a first test prediction (e.g., indicating likely credit card fraud) and one or more reason codes.

[0117] Performing further processing based on the one or more reason codes, in addition to the first test prediction, may provide additional benefits over performing further processing based on the first test prediction alone. The one or more reason codes may provide contextual information that could be useful for further processing operations. For example, a hydroelectric power station could increase water flow responsive to a prediction of increased power demand, without considering any reason codes. However, a top reason code such as “INCLEMENT WEATHER: HEATWAVE” could indicate that increased power demand likely stems from a heatwave, which may affect the quantity of available water, which may be a mitigating factor in increasing the flow rate of water through the hydroelectric turbines. Similarly, a credit card fraud detection system may decline to freeze a particular credit card, even if a transaction appears fraudulent (based on the first test prediction), given mitigating factors presented by the reason codes. As such, performing further processing based on the one or more reason codes and the first test prediction may be advantageous over performing further processing based on either alone.

[0118] As another alternative, outputting the first test prediction and the set of contribution values can comprise transmitting the first test prediction and / or the set of first contribution values and / or any data associated with the first test prediction and the set of contribution values (e.g., one or more reason codes or descriptions) to a39KILPATRICK TOWNSEND 78918186 1client computer. As such, in some embodiments step 710 can further comprise steps 712 and 714, described below.

[0119] At step 712, the computer system can determine and / or generate one or more reason codes and one or more descriptions associated with the set of contribution values. As described above, the computer system can use various methods to generate reason codes based on contribution values. For example, a mapping function or program can be used to map particular contribution values or combinations of contribution values to reason codes. As an example, in the context of predicting power demand, high contribution values corresponding to features such as “outdoor temperature” and “precipitation” could be mapped to a reason code such as “INCL_WEATHER” using an application or other software, indicating that a power demand prediction produced by a machine learning model may be a result of inclement weather conditions. As another example, in the context of predicting credit card fraud, high contribution values corresponding to features such as “unusual purchase location” and “unusual purchase time” could be mapped to a reason code “1A” (e.g., as depicted in table 302 of FIG. 3), indicating a suspicious originator entity. Such a mapping function could further connect reason codes to their corresponding descriptions (e.g., via relational data tables, pointers, resource locators, etc.), and thereby be used by the computer system to determine the one or more descriptions in addition to the one or more reason codes.

[0120] At step 714, the computer system can transmit the set of contribution values and / or the one or more reason codes and / or one or more descriptions to a client computer, e.g., a client computer from which the computer system received the set of test interaction data. In some embodiments, the computer system can additionally transmit the first test prediction to the client computer. As described above, in embodiments, various combinations of scores, predictions, contribution values, reason codes, and descriptions can be provided to client computers. For example, the computer system could transmit a score, reason code, description, and a contribution value to a client computer. As an alternative, the computer system could transmit a score, reason code, and a description to a client computer, but not transmit any corresponding contribution values, as the client computer (or its operator) may be ill-equipped to interpret such contribution values in a meaningful way.40KILPATRICK TOWNSEND 78918186 1

[0121] Afterwards, the client computer or an operator of the client computer can use any received data for some purpose. For example, the computer system could be part of a credit card fraud detection system implemented on behalf of issuing bank client computers. For a given credit card transaction (corresponding to the test interaction data), the computer system could transmit a fraud prediction (first test prediction) to an issuing bank computer (client computer) along with reason codes and descriptions, enabling employees of the issuing bank to interpret the fraud prediction and determine whether to cancel or reverse a transaction, freeze a credit card, etc. As another example, the computer system could be part of an automatic content moderation system implemented on behalf of social network clients. For a given post on a social networking website (corresponding to the test interaction data), the computer system could transmit an objectionable content prediction (i.e., the first test prediction, e.g., indicating whether the post likely contains objectionable content that should be moderated) to a social network server computer (client computer) along with reason codes and descriptions, enabling employees of the social networking company to determine whether to delete the post, ban an associated user, etc.

[0122] Embodiments of the present disclosure can use surrogate model explanation systems and methods other than those described above with reference to FIGs. 4, 5, and 7. For example, stateless surrogate explanation system 802 of FIG. 8, can be used in some embodiments of the present disclosure. Such a stateless surrogate explanation system 802 can include a pre-trained scoring model 804 (e.g., an XGBoost scoring model), along with a surrogate explainer 806. Such a surrogate explainer 806 can comprise (1 ) a multilayer perceptron, (2) an explainer system used to explain the output of the multilayer perceptron (e.g., a deep explainer that produces Shapley values or other contribution values) and (3) any process or logic used to produce reason codes 810 based on the output of the explainer system.

[0123] For a given set of training data 812, the pre-trained scoring model 804 can generate scores 808, which can be used to derive ground truth contribution values and reason codes, e.g., based on a TreeSHAP explainer or other appropriate explainer system. Such data can include some number (e.g., 10) of “most important” features (e.g., based on the magnitude of their respective contribution values) and41KILPATRICK TOWNSEND 78918186 1“most important” reason codes (e.g., derived from those contribution values and some process or logic used to derive reason codes). From a practical standpoint, the explainer used to generate these ground truth contribution values and reason codes based on scores 808 may have unacceptable latency and may be too slow to use in a production environment, e.g., due to the complexity of the pre-trained scoring model 804.

[0124] Hence, similar to the systems and methods described above with reference to FIGs. 4, 5, and 7, the surrogate explainer 806 (which may include a comparatively lightweight neural network) can be trained using the training data 812 and the ground truth contribution values and reason codes (produced based on scores 808) to provide explanations and reason codes 810 based on the output of the machine learning model (e.g., a multilayer perceptron) used in the surrogate explainer 806. Because this machine learning model is trained using the same training data 812 used to train the pre-trained scoring model 804, explanations and reason codes 810 generated using the surrogate explainer 806 may be generated considerably faster (e.g., in milliseconds). Additionally, the match rate for explanations between the pre-trained scoring model 804 and surrogate explainer 806 can be calculated offline, e.g., not during real-time operations, enabling the accuracy and effectiveness of the surrogate explainer 806 to be evaluated.

[0125] FIG. 9 shows a table 902 summarizing some initial pilot results for latency and match rates for the stateless surrogate explanation system 802 of FIG. 8. Table 902 shows average latency (in milliseconds) and match rate for various numbers of samples. The average latency per sample was calculated by determining the total amount of time needed to produce contribution values and reason codes for a given set of test interaction data (e.g., features corresponding to a particular credit card transaction, post on a social network website, demand for electrical power, etc.), then dividing by the sum of the time span over all interactions by the number of interactions. A match rate of e.g., 0.8: 4.7% indicates that 4.7% of interactions have a > 80% match rate on the top five reason codes over those produced from a primary scoring model (e.g., a TreeSHAP explainer for an XGBoost primary scoring model). As indicated by table 902, the relatively low (i.e., millisecond-level) average latency per sample suggests that the stateless surrogate42KILPATRICK TOWNSEND 78918186 1explanation system 802 of FIG. 8 can be used in many real-time production environments.

[0126] While the results depicted in table 902 are promising, unlike the training system depicted in FIG. 4, the stateless surrogate explanation system 802 of FIG. 8 may not impart any knowledge transfer from the pre-trained scoring model 804 to the surrogate explainer model 806. Hence, although the stateless surrogate explanation system 802 may be easier to implement, it may achieve a lower match rate than the systems described above with reference to FIGs. 4, 5, and 7, and hence it may be preferable to instead implement such systems instead of the stateless surrogate explanation system 802.

[0127] Generally, methods according to embodiments improve the rate of contribution value (e.g., Shapley value) generation (and the generation of derived data such as reason codes and descriptions) by a factor of about 100. Experiments indicated that (within a controlled context) the per sample latency for generating Shapley values based on predictions from a “heavyweight” primary model (e.g., using a TreeSHAP explainer for an XGBoost model) is approximately 1000 milliseconds, while the per sample latency for generating Shapley values based on predictions from a surrogate model (e.g., as disclosed herein) generally falls between 2 and 14.4 milliseconds, with an approximate average of 10 milliseconds. This hundredfold improvement means that methods and systems according to embodiments can be used in high-throughput (or otherwise low latency applications) such as in payment processing networks that process thousands of transactions per minute. Previous methods were too slow for such applications due to the approximate second of latency per sample.

[0128] As described herein, embodiments of the present disclosure provide novel methods and systems for machine learning model explanation systems that are fast, efficient, and useful in real-time production settings. As described above with reference to FIG. 6, embodiments of the present disclosure make use of novel regularization terms used in training loss functions, which can effectively distill and transfer knowledge from slow explanation systems (e.g., TreeSHAP explainer systems used for XGBoost scoring models). Additionally, embodiments of the present disclosure can be used to produce useful hierarchical explanations, which43KILPATRICK TOWNSEND 78918186 1can explain the outputs of machine learning models in human-readable terms, both at a low level (e.g., via contribution values explaining the contributions of individual features) and at the high level (e.g., via reason codes that provide more general explanations for the output of machine learning models).

[0129] Any of the computer systems mentioned herein may utilize any suitable number of subsystems. Examples of such subsystems are shown in FIG. 10 in computer system 1000. In some embodiments, a computer system includes a single computer apparatus, where the subsystems can be the components of the computer apparatus. In other embodiments, a computer system can include multiple computer apparatuses, each being a subsystem, with internal components. A computer system can include desktop and laptop computers, tablets, mobile phones and other mobile devices.

[0130] The subsystems shown in FIG. 10 are interconnected via a system bus 1012. Additional subsystems such as a printer 1008, keyboard 1018, storage device(s) 1020, monitor 1024 (e.g., a display screen, such as an LED), which is coupled to display adapter 1014, and others are shown. Peripherals and input / output (I / O) devices, which couple to I / O controller 1002, can be connected to the computer system by any number of means known in the art such as input / output (I / O) port 1016 (e.g., USB, FireWire®). For example, I / O port 1016 or external interface 1022 (e.g., Ethernet, Wi-Fi, etc.) can be used to connect computer system 1000 to a wide area network such as the Internet, a mouse input device, or a scanner. The interconnection via system bus 1012 allows the central processor 1006 to communicate with each subsystem and to control the execution of a plurality of instructions from system memory 1004 or the storage device(s) 1020 (e.g., a fixed disk, such as a hard drive, or optical disk), as well as the exchange of information between subsystems. The system memory 1004 and / or the storage device(s) 1020 may embody a computer readable medium. Another subsystem is a data collection device 1010, such as a camera, microphone, accelerometer, and the like. Any of the data mentioned herein can be output from one component to another component and can be output to the user.

[0131] A computer system can include a plurality of the same components or subsystems, e.g., connected together by external interface 1022, by an internal44KILPATRICK TOWNSEND 78918186 1interface, or via removable storage devices that can be connected and removed from one component to another component. In some embodiments, computer systems, subsystem, or apparatuses can communicate over a network. In such instances, one computer can be considered a client and another computer a server, where each can be part of a same computer system. A client and a server can each include multiple systems, subsystems, or components. In various embodiments, methods may involve various numbers of clients and / or servers, including at least 10, 20, 50, 100, 200, 500, 1 ,000, or 10,000 devices. Methods can include various numbers of communication messages between devices, including at least 100, 200, 500, 1 ,000, 10,000, 50,000, 100,000, 500,00, or one million communication messages. Such communications can involve at least 1 MB, 10 MB, 100 MB, 1 GB, 10 GB, or 100 GB of data.

[0132] Aspects of embodiments can be implemented in the form of control logic using hardware circuitry (e.g., an application specific integrated circuit or field programmable gate array) and / or using computer software stored in a memory with a generally programmable processor in a modular or integrated manner, and thus a processor can include memory storing software instructions that configure hardware circuitry, as well as an FPGA with configuration instructions or an ASIC. As used herein, a processor can include a single-core processor, multi-core processor on a same integrated chip, or multiple processing units on a single circuit board or networked, as well as dedicated hardware. Based on the disclosure and teachings provided herein, a person of ordinary skill in the art will know and appreciate other ways and / or methods to implement embodiments of the present disclosure using hardware and a combination of hardware and software.

[0133] Any of the software components or functions described in this application may be implemented as software code to be executed by a processor using any suitable computer language such as, for example, Java, C, C++, C#, Objective-C, Swift, or scripting language such as Perl or Python using, for example, conventional or object-oriented techniques. The software code may be stored as a series of instructions or commands on a computer readable medium for storage and / or transmission. A suitable non-transitory computer readable medium can include random access memory (RAM), a read only memory (ROM), a magnetic medium such as a hard-drive or a floppy disk, or an optical medium such as a45KILPATRICK TOWNSEND 78918186 1compact disk (CD) or DVD (digital versatile disk) or Blu-ray disk, flash memory, and the like. The computer readable medium may be any combination of such devices. In addition, the order of operations may be re-arranged. A process can be terminated when its operations are completed but could have additional steps not included in a figure. A process may correspond to a method, a function, a procedure, a subroutine, a subprogram, etc. When a process corresponds to a function, its termination may correspond to a return of the function to the calling function or the main function.

[0134] Such programs may also be encoded and transmitted using carrier signals adapted for transmission via wired, optical, and / or wireless networks conforming to a variety of protocols, including the Internet. As such, a computer readable medium may be created using a data signal encoded with such programs. Computer readable media encoded with the program code may be packaged with a compatible device (e.g., as firmware) or provided separately from other devices (e.g., via Internet download). Any such computer readable medium may reside on or within a single computer product (e.g., a hard drive, a CD, or an entire computer system), and may be present on or within different computer products within a system or network. A computer system may include a monitor, printer, or other suitable display for providing any of the results mentioned herein to a user.

[0135] Any of the methods described herein may be totally or partially performed with a computer system including one or more processors, which can be configured to perform the steps. Any operations performed with a processor may be performed in real-time. The term “real-time" may refer to computing operations or processes that are completed within a certain time constraint. As examples, a time constraint may be 30 seconds, 1 minute, 10 minutes, 30 minutes, 1 hour, 4 hours, 1 day, or 7 days. Thus, embodiments can be directed to computer systems configured to perform the steps of any of the methods described herein, potentially with different components performing a respective step or a respective group of steps. Although presented as numbered steps, steps of methods herein can be performed at a same time or at different times or in a different order. Additionally, portions of these steps may be used with portions of other steps from other methods. Also, all or portions of a step may be optional. Additionally, any of the steps of any of the methods can be46KILPATRICK TOWNSEND 78918186 1performed with modules, units, circuits, or other means of a system for performing these steps.

[0136] The specific details of particular embodiments may be combined in any suitable manner without departing from the spirit and scope of embodiments of the disclosure. However, other embodiments of the disclosure may be directed to specific embodiments relating to each individual aspect, or specific combinations of these individual aspects.

[0137] The above description of example embodiments of the present disclosure has been presented for the purposes of illustration and description. It is not intended to be exhaustive or to limit the disclosure to the precise form described, and many modifications and variations are possible in light of the teaching above.

[0138] A recitation of "a", "an" or "the" is intended to mean "one or more" unless specifically indicated to the contrary. The use of “or” is intended to mean an “inclusive or,” and not an “exclusive or” unless specifically indicated to the contrary. Reference to a “first” component does not necessarily require that a second component be provided. Moreover, reference to a “first” or a “second” component does not limit the referenced component to a particular location unless expressly stated. The term “based on” is intended to mean “based at least in part on.”

[0139] The claims may be drafted to exclude any element which may be optional. As such, this statement is intended to serve as antecedent basis for use of such exclusive terminology as “solely”, “only”, and the like in connection with the recitation of claim elements, or the use of a “negative” limitation.

[0140] All patents, patent applications, publications, and descriptions mentioned herein are incorporated by reference in their entirety for all purposes. None is admitted as prior art. Where a conflict exists between the instant application and a reference provided herein, the instant application shall dominate.47KILPATRICK TOWNSEND 78918186 1

Claims

WHAT IS CLAIMED IS:1 . A method performed by a computer system comprising: performing an iterative training process until a terminating condition has been met, the iterative training process comprising: sampling a batch of training interaction data sets comprising one or more training interaction data sets corresponding to one or more training interactions; for each training interaction data set in the batch of training interaction data sets, generating a first prediction using a first machine learning model, thereby generating one or more first predictions corresponding to the one or more training interaction data sets; generating one or more sets of first contribution values based on the one or more first predictions; for each training interaction data set in the batch of training interaction data sets, generating a second prediction using a second machine learning model, thereby generating one or more second predictions corresponding to the one or more training interaction data sets; generating one or more sets of second contribution values based on the one or more second predictions; determining one or more loss values based at least on the one or more sets of first contribution values and the one or more sets of second contribution values; updating a parameter set of at least the second machine learning model based on the one or more loss values, thereby training the second machine learning model; and if the terminating condition has not been met, repeating the iterative training process until the terminating condition has been met, otherwise completing the iterative training process.

2. The method of claim 1 , wherein the one or more sets of first contribution values are one or more sets of first Shapley values or one or more sets of first Shapley value approximations, and wherein the one or more sets of second48KILPATRICK TOWNSEND 78918186 1contribution values are one or more sets of second Shapley values or one or more sets of second Shapley value approximations.

3. The method of claim 1 , wherein the second machine learning model is a linear model, tree model, or neural network and wherein the one or more sets of second contribution values are generated using a linear explainer, a tree explainer, a deep explainer, or a gradient explainer.

4. The method of claim 1 , wherein the first machine learning model is an XGBoost model and wherein the second machine learning model is a multilayer perceptron (MLP) model.

5. The method of claim 1 , wherein the parameter set of the second machine learning model is a second parameter set, and wherein a second data size of the second parameter set is smaller than a first data size of a first parameter set of the first machine learning model.

6. The method of claim 1 , wherein determining the one or more loss values based on at least the one or more sets of first contribution values and the one or more sets of second contribution values comprises: generating one or more first contribution value latent vectors based on the one or more sets of first contribution values using an encoder; generating one or more second contribution value latent vectors based on the one or more sets of second contribution values using the encoder; and determining the one or more loss values based on at least the one or more first contribution value latent vectors and the one or more second contribution value latent vectors.

7. The method of claim 6, wherein updating the parameter set of at least the second machine learning model based on the one or more loss values additionally comprises updating an encoder parameter set of the encoder, thereby training the encoder.

8. The method of claim 6, wherein determining one or more loss values comprises:49KILPATRICK TOWNSEND 78918186 1determining a first loss value comprising a binary entropy loss between the one or more second predictions and one or more labels corresponding to the one or more training interaction data sets; determining a second loss value comprising a regression loss between the one or more first predictions and the one or more second predictions; determining a third loss value based on a Kullback-Leibler divergence between the one or more first contribution value latent vectors and the one or more second contribution value latent vectors; determining one or more first rankings corresponding to the one or more sets of first contribution values; determining one or more second rankings corresponding to the one or more sets of second contribution values; and determining a fourth and fifth loss value based on the one or more first rankings and the one or more second rankings.

9. The method of claim 1 , wherein the training interaction data comprises message data, transaction data, or machine performance data.

10. The method of claim 1 , wherein: each training interaction data set comprises a plurality of training interaction data set features; the first machine learning model comprises a primary scoring model; the second machine learning model comprises a surrogate scoring model trained to approximate the first machine learning model; the one or more first predictions comprise one or more primary predictions; the one or more second predictions comprise one or more surrogate predictions; the one or more second predictions comprise approximates of the one or more first predictions; the one or more sets of first contribution values describe one or more first relationships between the one or more first predictions and the plurality of training interaction data set features; and50KILPATRICK TOWNSEND 78918186 1the one or more sets of second contribution values describe one or more second relationships between the one or more second predictions and the plurality of training interaction data set features, and thereby approximate the one or more first relationships.11 . The method of claim 1 , wherein: the first machine learning model comprises a first primary machine learning model; the second machine learning model comprises a first surrogate machine learning model; a segmentation model comprises the first primary machine learning model and a second primary machine learning model; the iterative training process comprises a first iterative training process; the terminating condition comprises a first terminating condition; and the method further comprises performing a second iterative training process until a second terminating condition has been met, the second iterative training process comprising training a second surrogate machine learning model corresponding to the second primary machine learning model.

12. The method of claim 1 , further comprising: receiving a set of test interaction data; generating, using the first machine learning model and based on the set of test interaction data, a first test prediction; generating, using the second machine learning model and based on the set of test interaction data, a second test prediction; generating, based on the second test prediction, a set of contribution values; and outputting the first test prediction and the set of contribution values generated based on the second test prediction.

13. The method of claim 12, wherein outputting the first test prediction and the set of contribution values further comprises: determining one or more reason codes associated with the set of contribution values; and51KILPATRICK TOWNSEND 78918186 1responsive to generating the first test prediction and determining the one or more reason codes, performing further processing based on at least the first test prediction and the one or more reason codes.

14. The method of claim 12, wherein: the computer system comprises a server computer; the computer system receives the set of test interaction data from a client computer; outputting the first test prediction and the set of contribution values further comprises determining one or more reason codes and one or more descriptions associated with the set of contribution values; and outputting the first test prediction and the set of contribution values further comprises transmitting the one or more reason codes and the one or more descriptions to the client computer.

15. The method of claim 14, wherein outputting the first test prediction and the set of contribution values further comprises transmitting the first test prediction and / or the set of contribution values to the client computer.

16. A computer system comprising: a processor; and a non-transitory computer readable medium coupled to the processor, the non-transitory computer readable medium comprising code executable by the processor for performing a method comprising: performing an iterative training process until a terminating condition has been met, the iterative training process comprising: sampling a batch of training interaction data sets comprising one or more training interaction data sets corresponding to one or more training interactions; for each training interaction data set in the batch of training interaction data sets, generating a first prediction using a first machine learning model, thereby generating one or more first predictions corresponding to the one or more training interaction data sets; generating one or more sets of first contribution values based on the one or more first predictions;52KILPATRICK TOWNSEND 78918186 1for each training interaction data set in the batch, generating a second prediction using a second machine learning model, thereby generating one or more second predictions corresponding to the one or more training interaction data sets; generating one or more sets of second contribution values based on the one or more second predictions; determining one or more loss values based on at least the one or more sets of first contribution values and the one or more sets of second contribution values; updating a parameter set of at least the second machine learning model, thereby training the second machine learning model; and if the terminating condition has not been met, repeating the iterative training process until the terminating condition has been met, otherwise completing the iterative training process.

17. A method comprising: receiving, by a computer system, a set of test interaction data; generating, by the computer system, using a first machine learning model and based on the set of test interaction data, a first test prediction; generating, by the computer system, using a second machine learning model and based on the set of test interaction data, a second test prediction; generating, by the computer system, based on the second test prediction, a set of contribution values; and outputting, by the computer system, the first test prediction and the set of contribution values generated based on the second test prediction.

18. The method of claim 17, wherein: the computer system comprises a server computer; the computer system receives the set of test interaction data from a client computer; and outputting the first test prediction and the set of contribution values comprises:53KILPATRICK TOWNSEND 78918186 1generating, by the computer system, one or more reason codes and / or one or more descriptions corresponding to the set of contribution values, transmitting, by the computer system, the first test prediction to the client computer, and transmitting, by the computer system, the set of contribution values and / or the one or more reason codes and / or the one or more descriptions to the client computer.

19. The method of claim 17, wherein: the set of test interaction data comprises a plurality of test interaction data features; the first machine learning model comprises a primary scoring model; the second machine learning model comprises a surrogate scoring model trained to approximate the first machine learning model; the first test prediction comprises a primary test prediction; the second test prediction comprises a surrogate test prediction; the second test prediction comprises an approximation of the first test prediction; and the set of contribution values describe a relationship between the second test prediction and the plurality of test interaction data features, and thereby approximates a primary relationship between the first test prediction and the plurality of test interaction data features.

20. The method of claim 17, further comprising, prior to generating the second test prediction, performing an iterative training process until a terminating condition has been met, the iterative training process comprising: sampling a batch of training interaction data sets comprising one or more training interaction data sets corresponding to one or more training interactions; for each training interaction data set in the batch of training interaction data sets, generating a first prediction using the first machine learning model, thereby generating one or more first predictions corresponding to the one or more training interaction data sets;54KILPATRICK TOWNSEND 78918186 1generating one or more sets of first contribution values based on the one or more first predictions; for each training interaction data set in the batch, generating a second prediction using the second machine learning model, thereby generating one or more second predictions corresponding to the one or more training interaction data sets; generating one or more sets of second contribution values based on the one or more second predictions; determining one or more loss values based on at least the one or more sets of first contribution values and the one or more sets of second contribution values; updating a parameter set of at least the second machine learning model based on the one or more loss values, thereby training the second machine learning model; and if the terminating condition has not been met, repeating the iterative training process until the terminating condition has been met, otherwise completing the iterative training process.55KILPATRICK TOWNSEND 78918186 1

Images