Universal model routing for efficient neural network inference

Abstract

Systems and methods for effectively routing inputs between multiple neural networks with different computational costs. Each neural network is represented as a respective feature vector.

Description

[0001] UNIVERSAL MODEL ROUTING FOR EFFICIENT NEURAL NETWORK INFERENCE CROSS-REFERENCE TO RELATED APPLICATIONS This application claims priority to India Application No. 202511009370 filed on February 5, 2025. The disclosure of the prior application is considered part of and is incorporated by reference in the disclosure of this application.

[0002] BACKGROUND

[0003] This specification relates to processing inputs using neural networks.

[0004] Neural networks are machine learning models that employ one or more layers of nonlinear units to predict an output for a received input. Some neural networks include one or more hidden layers in addition to an output layer. The output of each hidden layer is used as input to the next layer in the network, i.e.. the next hidden layer or the output layer. Each layer of the network generates an output from a received input in accordance with current value inputs of a respective set of parameters.

[0005] SUMMARY

[0006] This specification describes a system implemented as computer programs on one or more computers in one or more locations that efficiently routes network inputs among multiple neural networks at inference time.

[0007] The subject matter described in this specification can be implemented in particular embodiments so as to realize one or more of the following advantages.

[0008] Neural networks’, e.g., large language models’ (LLMs), significant advances in capabilities are accompanied by significantly increasing inference costs.

[0009] Some existing techniques use model routing in an attempt to address this issue. Here, a system maintains a pool of candidate neural networks of various sizes and capabilities. Given a prompt, the system leams to predict the lowest-cost neural networks which can reasonably address the prompt. In doing so, the system can leam to use high-cost neural networks sparingly, only on the (relatively) few “hard” inputs. This is a conceptually simple but effective technique.

[0010] However, existing model routing techniques focus on routing over a fixed pool of neural networks. In practice, however, the pool of candidate neural networks can constantly change; e.g., older LLMs may be deprecated in favor of new, performantLLMs or new sizes of LLMs can be added to a family of models available for performing inference. To leverage such new neural networks, perhaps the simplest approach is to retrain the router that determines which neural network to route a given input to. However, with frequent changes to the neural network pool, this may be impractical owing to the non-trivial overhead of both model re-training and the difficulties obtaining sufficient training labels for each new neural network that would allow the router to be effectively trained.

[0011] This specification describes a new approach to this problem based on representing each neural network as a feature vector, e.g., derived from their prediction errors on a set of representative inputs. By performing routing over these neural network features, the described techniques enable generalization to previously unseen neural networks without any re-training. In other words, these techniques enable effective routing with a dynamic neural network pool without requiring any-retraining when the neural network pool changes. As a result, the described techniques can continue to make effective routing decisions to optimize a target trade-off between model output quality and computational efficiency even as the pool of available neural networks changes. For example, the described techniques can be used to determine a clustering of network inputs and then continue to use the same clustering in conjunction with the neural network feature vectors to effectively route network inputs even if none of the neural networks in the current set of available neural networks were available when the system initially began routing network inputs. The pool of available neural networks can operate in parallel. The pool of available neural networks can form a distributed system which as noted, can be dynamic in nature. The described techniques can allocate new inputs to particular neural networks of the parallel / distributed system based on a trade-off between model output quality and computational efficiency. For example, the described techniques can effectively minimize the network bandwidth required, i.e., the amount of data that is transmitted over a set of one or more data communication networks, to route inputs across different models within the distributed system while maintaining output quality’. The described techniques therefore provide an improved method of workload allocation in a parallel / distributed system.

[0012] The details of one or more embodiments of the subject matter of this specification are set forth in the accompanying drawings and the description below. Other features, aspects, and advantages of the subject matter will become apparent from the description, the drawings, and the claims.BRIEF DESCRIPTION OF THE DRAWINGS

[0013] FIG. 1 is a block diagram of an example neural network system.

[0014] FIG. 2 is a flow diagram of an example process for processing a new input using a set of neural networks.

[0015] FIG. 3 is a flow diagram of an example process for determining a feature vector for a neural network.

[0016] FIG. 4 is a flow diagram of an example process for determining an error measure for a neural network using clustering assignments.

[0017] FIG. 5 is a flow diagram of an example process for determining an error measure for a neural network using learned cluster maps

[0018] FIG. 6 shows an example of the operation of the system.

[0019] Like reference numbers and designations in the various drawings indicate like elements.

[0020] DETAILED DESCRIPTION FIG. 1 shows an example neural network system 100. The neural network system 100 is an example of a system implemented as computer programs on one or more computers in one or more locations in which the systems, components, and techniques described below are implemented.

[0021] The system 100 efficiently routes network inputs 102 among multiple neural networks 1 10A-N at inference time. For example, the neural networks can be respective generative neural networks, e.g., large language models (LLMs) or other types of generative neural networks.

[0022] Examples of generative neural networks and of various tasks that the system can perform using the generative neural networks are provided below.

[0023] That is, when anew network input 102 is received, the system 100 determines which neural network from a set of multiple neural networks 110A-N to use to process the network input 102 and then provides the new network input 102 as input to the determined neural network.

[0024] Generally, the neural networks in the set 110A-N are associated with different compute measures. A compute measure can also be referred to as a “cost measure.” The compute measure for a given neural network characterizes the amount of computational resources required to process a set of one or more inputs using the neural network.For example, the compute measure can be based on one or more: the latency required to send inputs to and receive outputs from the neural network, the amount of memory required to process a set of one or more inputs using the neural network, or the amount of compute, e.g., as measured in terms of processor cycles, processing power, energy7, and so on, required to process a set of one or more inputs using the neural network.

[0025] As one example, the compute measure can be equal to the output of a function, e.g., an identity7function, a normalizing function, and so on, applied to a single one of the above factors. As another example, the compute measure can be equal to the output of a function applied to multiple ones of the above factors, e.g., a weighted sum of two or more of the above factors or a weighted sum of respective normalizing or scaling functions applied to two or more of the above factors.

[0026] Different neural networks can have different compute measures for a variety of reasons.

[0027] For example, one or more of the neural networks can be deployed on the same set of one or more computers as the system 100 while one or more of the neural networks are deployed remotely from the system 100, e.g., and therefore require data to be sent over a data communication network in order to make use of the neural network. The system 100 can be a distributed system, i.e., a system that is distributed across multiple different computers in different physical locations. As a particular example, the system 100 and one or more of the neural networks can be deployed on a user device while one or more other neural networks can be deployed on a server that is remote from the user device.

[0028] As another example, one of the multiple neural networks in the set 110A-N may be a larger neural network, e g., can have more parameters, that one or more of the other neural networks, thereby requiring more computational resources to process a given input.

[0029] By ‘‘routing” different inputs to different neural networks based on which neural network is best-suited to generate an accurate output for a given input and on the respective computational compute measures for the neural networks, the system 100 can reliably generate high-quality outputs while maintaining inference efficiency, i.e., generate high-quality7outputs while performing inference for new inputs in a computationally-efficient manner.

[0030] In some implementations, the system 100 receives the compute measures for the neural networks as input. In some other implementations, the system 100 determines thecompute measure for a given neural network by processing a set of inputs using the given neural network and measuring the quantities required to compute the compute measure.

[0031] More specifically, the system 100 maintains feature data 120 that, for each of a plurality of neural networks in the set 110A-N, includes a respective feature vector 122 characterizing the neural network and a respective compute measure 124 characterizing an amount of computational resources required to process a set of one or more inputs using the neural network.

[0032] By representing the neural networks using feature vectors, the system 100 can dynamically add or remove neural networks from the set of multiple neural networks 110A-N, e.g., as the available neural networks for performing a given task change, without needing to perform any additional re-training.

[0033] Generating a feature vector that represents a given neural network will be described in more detail below.

[0034] That is, when a new neural network becomes available to process inputs, the system 100 can add a feature vector 122 for the new neural network to the maintained feature data 120 and begin using the new neural network to improve the efficiency of the inference process without requiring any computationally expensive training steps or other computationally expensive re-configuration.

[0035] In more detail, during inference operation, the system 100 receives a new network input 102.

[0036] The system 100 selects a neural network from the plurality of neural networks 112A-N based on the new network input 102, the respective feature vectors 122 for the neural networks, and the respective compute measures 124 for the neural networks.

[0037] Generally, the system 100 selects a neural network that best satisfies a trade-off between the likelihood of generating a high quality output for the new network input 102 and reducing the computational cost, i.e., the compute measure, of generating the output for the new network input 102.

[0038] For example, the system 100 can select the neural network r for processing a given input x as follows:

[0039]

[0040] where Mis the set of available neural networks, y(m)(x) is an estimate of the expected error of neural network m on input x, c(m) is the cost for the m-th neural network, and k isa non-negative hyper-parameter that controls the trade-off between model quality and cost, with a higher prioritizing lower cost.

[0041] Unlike other approaches, the system 100 uses the feature vector for the neural network m to determine the estimate of the expected error of neural network m on input x.

[0042] Selecting a neural network using the feature vectors is described in more detail below.

[0043] The system 100 provides the new network input 102 as input to the selected neural network to obtain a network output 112.

[0044] The system 100 then provides the network output 112 in response to the new network input 102.

[0045] For example, the system 100 can provide the network output 112 for presentation on a user device or to an external system.

[0046] FIG. 2 is a flow diagram of an example process 200 for processing a network input. For convenience, the process 200 will be described as being performed by a system of one or more computers located in one or more locations. For example, a training system, e.g., the neural network system 100 of FIG. 1, appropriately programmed in accordance with this specification, can perform the process 200.

[0047] As described above, the system maintains, for each of a plurality7of neural networks, a respective feature vector characterizing the neural network and a respective compute measure characterizing an amount of computational resources required to process a set of one or more inputs using the neural network (step 202).

[0048] For example, the respective feature vector characterizing a given neural network can include a respective error measure for the neural network for each of a plurality of clusters of network inputs. That is, the feature vector includes, for each cluster of network inputs, a value that represents the performance, e.g., in terms of expected error, of the neural network on network inputs that belong to the cluster.

[0049] Determining the feature vectors for the neural networks will be described in more detail below with reference to FIG. 3.

[0050] The system receives a new network input (step 204).

[0051] The system selects a neural network from the plurality of neural networks based on the new network input, the respective feature vectors for the neural networks, and the respective compute measures for the neural networks (step 206).

[0052] In particular, the system can determine, for each of the neural networks in the set of neural networks and using the feature vector for the neural network, a respective errormeasure for the new network input that measures an expected error of the neural network on the new network input.

[0053] For example, the system can represent both the network input and the neural networks as respective feature vectors and then compute the expected error for a given neural network on the new network input as follows:

[0054]

[0055] where <D(x) is the feature vector of the network input x andvP(7i) is the feature vector for a given neural network h. This allows the system to handle newly-added, previously unseen models by computing their feature vector.

[0056] That is. by making use of the feature vectors, the system can determine, for each neural network and without needing to process the new input using the neural network, an expected error of a network output generated by the neural network by processing the new network input.

[0057] Example techniques for determining the respective error measure for a given neural network will be described in detail below with reference to FIGS. 4 and 5.

[0058] The system can then select, based on the respective error measures and the respective compute measures for the neural networks, a neural network from the plurality7of neural networks.

[0059] For example, the system can determine a respective overall score for each neural network as the output of a scoring function that measures a trade-off between the likelihood of generating a high-quality output for the new network input and reducing the computational cost of generating the output for the new network input. In other words, for each neural network, the system can apply a scoring function to the respective error measure and the respective compute measure for the neural network to generate a respective overall score for the neural network. That is, the scoring function includes one term that is based on the respective error measure and another term that is based on the respective compute measure.

[0060] For example, the function can be a weighted sum of the respective error measure and the respective compute measure, where the weights in the weighted sum specify the importance of generating a high-quality output relative to minimizing computational cost and are received as input by the system.

[0061] More specifically, the scoring function can include a first term that is an increasing function of the respective error measures and a second term that is anincreasing function of the respective overall scores. In this example, for each neural network, the overall score is equal to a sum of an output of the first term and a product between a weight value and an output of the second term. Here, the weight value specifies the relative of importance of minimizing the computational cost during the operation of the system.

[0062] An example of such a scoring function is the function r(x) described above with reference to FIG. 1. More generally, however, any appropriate scoring function can be used.

[0063] The system can then select the neural network that has an overall score that optimizes the function. For example, the system can select the neural network with the lowest overall score as described above with FIG. 1.

[0064] The system provides the new network input as input to the selected neural network to obtain a network output (step 208) and then provides the network output in response to the new network input (step 210).

[0065] As a result of performing this process, the system effectively balances the tradeoff being maximizing output accuracy while minimizing computational cost. Moreover, because the system uses the feature vectors for the neural networks to select which neural network to use to process the new network input, the system can effectively handle modifications to the pool of available of neural networks. That is, if a neural network is removed from the pool, the system can remove the feature vector for the neural network from consideration. Similarly, if a neural network is added to the pool, the system can generate a feature vector for the neural network and obtain a compute measure for the neural network and then proceed to use the neural network when performing subsequent iterations of the process 200 without needing to re-train any components of the system.

[0066] FIG. 3 is a flow diagram of an example process 300 for generating a feature vector for a neural network. For convenience, the process 300 will be described as being performed by a system of one or more computers located in one or more locations. For example, a training system, e.g., the neural network system 100 of FIG. 1, appropriately programmed in accordance with this specification, can perform the process 300.

[0067] For example, the system can perform the process 300 for each neural network in an initial set of neural networks prior to deploying the initial set of neural networks for processing network inputs.As another example, the system can perform the process 300 for a new neural network in response to determining that a new neural network has been added to the set of neural networks.

[0068] Prior to performing the initial iteration of the process 300, the system generates data defining the clustering of the network inputs.

[0069] For example, the system can obtain a training data set that includes a plurality of training network inputs. Advantageously, the system does not require that the training network inputs be labeled and can therefore make use of any of a variety of large data sets of unlabeled training network inputs.

[0070] The system can then cluster the plurality of training network inputs to generate the plurality of clusters. For example, the system can generate a respective embedding of each training network input and then cluster the embeddings to generate the clusters of the corresponding network inputs.

[0071] The system can generate the embedding of any given training network input using any appropriate pre-trained embedding neural network. For example, when the network input is a text input, the system can make use of a text embedding neural network, e.g., an encoder-only Transformer neural network, a decoder-only Transformer neural network, a recurrent neural network, and so on. As another example, when the network input is an image, the system can make use of any appropriate vision embedding neural network, e.g., a convolutional neural network, a vision Transformer neural network, and so on. As yet another example, when the network input is a multi-modal input, the system can make use of any appropriate multi-modal embedding neural network, e.g., a vision language model (VLM).

[0072] The system can perform the clustering using any appropriate clustering technique. Examples of such techniques include k-means clustering techniques, agglomerative clustering techniques, and so on.

[0073] The system then uses the clusters to perform the process 300, as will be described below.

[0074] The system obtains a plurality of validation network inputs and a respective target output for each validation network input (step 302). The target output for each validation network input represents the “correct” or ground truth output that should be generated by processing the validation network input. Advantageously, the system can effectively perform the process 300 even when the number of labeled validation network inputs thatare available is significantly smaller than the number of unlabeled training network inputs.

[0075] The system also assigns each validation network input to a respective one of the plurality of clusters (step 304). That is, although the clusters were generated using the training network inputs, the system can assign each validation network input to one of the clusters. For example, the system can do this by embedding each validation network input as described above and then identifying the cluster that has a centroid that is closest to the embedding to the validation network input. Generally, the system only needs to perform step 304 during or prior to performing the first iteration of the process 300. For later iterations, the system can re-use the clustering assignments for the validation inputs that have already been generated.

[0076] For each validation network input, the system processes the validation network input using the neural network to generate a respective error for the validation network input relative to the respective target output for the validation network input (step 306).

[0077] The error can be a binary error, e.g., an error that is equal to 1 if the output generated by the neural network does not match the target output and equal to 0 if there is a match. Alternatively, the error can be a continuous score, e g., a score between zero and one, that measures the qualify of the generated output relative to the target output.

[0078] The system then determines the respective error measure for the neural network for each of the plurality of clusters based on the respective errors for the validation network inputs that are assigned to the cluster (step 308). For example, for each cluster, the system can aggregate the respective errors for the validation network inputs that are assigned to the cluster, e g., by computing an average or other aggregation measure, to generate the respective error measure for the neural network for the cluster.

[0079] The system can then set the feature vector for the given neural network to include the respective error measures for the neural network for the clusters.

[0080] FIG. 4 is a flow diagram of an example process 400 for determining an error measure for a neural network using clustering assignments. For convenience, the process 400 will be described as being performed by a system of one or more computers located in one or more locations. For example, a training system, e.g., the neural network system 100 of FIG. 1, appropriately programmed in accordance with this specification, can perform the process 400.

[0081] The system assigns the new network input to a particular cluster of the plurality of clusters (step 402).For example, after generating the clusters as described above with reference to FIG. 3, the system can maintain a respective centroid embedding for each of the clusters.

[0082] In this example, the system can assign the new network input to a particular cluster by generating an embedding of the new network input in the same embedding space as the embeddings of the training network inputs (step 404) and assigning the new network input to the cluster of the plurality of clusters that has a centroid that is closest to the embedding of the new network input in the embedding space (step 406).

[0083] As described above, the system can generate the embedding of the new network input using any appropriate pre-trained embedding neural network. For example, when the new network input is a text input, the system can make use of a text embedding neural network, e.g., an encoder-only Transformer neural network, a decoder-only Transformer neural network, a recurrent neural network, and so on. As another example, when the new network input is an image, the system can make use of any appropriate vision embedding neural network, e.g., a convolutional neural network, a vision Transformer neural network, and so on. As yet another example, when the new network input is a multi-modal input, the system can make use of any appropriate multi-modal embedding neural network, e.g., a vision language model (VLM).

[0084] The system determines, for each of the plurality7of neural networks, the respective error measure for the new network input based on the respective error measure for the neural network for the particular cluster (step 408).

[0085] For example, for each of the plurality of neural networks, the system can set the respective error measure for the new network input to be equal to or directly proportional to the respective error measure for the neural network for the particular cluster contained in the feature vector for the neural network.

[0086] In some implementations, the system instead uses a learned cluster map to generate the error measure.

[0087] FIG. 5 is a flow diagram of an example process 500 for determining an error measure for a neural network using learned cluster maps. For convenience, the process 500 will be described as being performed by a system of one or more computers located in one or more locations. For example, a training system, e.g., the neural network system 100 of FIG. 1, appropriately programmed in accordance with this specification, can perform the process 500.

[0088] The system generates, from the new network input, a respective likelihood score for each cluster of the plurality7of clusters (step 502).Generally, the respective likelihood score for each cluster measures a likelihood that the new network input belongs to the cluster.

[0089] For example, the system can process the new network input using a learned clustering function to generate the likelihood scores. For example, the learned clustering function can first generate an embedding of the new network input, e.g., as described above with reference to FIG. 3, and then process the embedding using one or more neural network layers, e.g., a linear layer followed by an activation function, to generate the likelihood scores.

[0090] The embedding neural network can be pre-trained as described above, while the system can have trained the one or more neural network layers to optimize the effectiveness of the likelihood scores. For example, the system can have trained the learned clustering function while holding the embedding neural network fixed on an objective, e.g., a log loss or other appropriate loss, that measures the performance of the neural networks on a corresponding training data set. An example of such an objective is as follows:

[0091] &

[0092]

[0093] where Str is the training set, Htr is the set of neural networks used for learning the clustering function, y is a target output for an input x from the training set, h(x) is the output generated by the neural network h, and 0 are the parameters of the learned clustering function.

[0094] The system can then determine, for each of the plurality of neural networks, the respective error measure for the new network input based on the respective likelihood score for the clusters and the respective error measures for the neural network for the clusters (step 504). For example, the system can determine the respective error measure for the new network input for a given neural network by determining a weighted sum of the respective error measures for the neural network for the clusters, with the respective error measure for each cluster being weighted by the respective likelihood score for the cluster in the weighted sum.

[0095] FIG. 6 shows an example 600 of the operation of the system when the system uses clustering assignments to determine performance measures. In the example 600, the set of neural networks includes two neural networks, LLM-1 610 and LLM-2620.

[0096] As shown in the example 600, the system can effectively route an input, i.e., a prompt 630, to LLM-2, which is a neural network that was not seen by the system duringtraining, i.e., when determining the clusters, and only added to set of neural networks after inference had already begun being performed.

[0097] In particular, in the example 600, the system first performs, using an LLM embedder 650, '-means clustering on a training set to find K centroids, and then partitions the validation set into K representative clusters. Each test-time LLM can then be represented as a A'-dimensional feature vector 670 of per-cluster errors. In the example 600. K is equal to 3 and thus both LLM-1 and LLM-2 are associated with respective 3 dimensional vectors that include respective per-cluster errors for the corresponding LLM for each of the 3 clusters.

[0098] Thus, as shown in the example 600, the system receives the new prompt 630 and processes the prompt using a prompt embedder 640 that uses the 3 centroids to assign the prompt to cluster 1, yielding a feature vector 660 for the prompt of (1, 0, 0).

[0099] The system then determines, from the cluster assignment and the feature vectors, a respective error score 680 of .3 for LLM-1 and a respective error score 680 of .6 for LLM-2. The system then applies a cost adjustment 690 that is based on the compute measure of the models and on X, yielding a final score of .8 for LLM-1 and a final score of .7 for LLM-2. The system then computes an argmin 692 of the final scores, resulting in the system routing the prompt to LLM-2 for processing.

[0100] In other words, for each prompt, the system routes to the LLM with the smallest cost-adjusted average error on the cluster the prompt belongs to. While not shown above, the prompt embedder can instead make use of a scoring function fitted via supervised learning using labels from a set of training LLMs that can be different from those seen during test time.

[0101] In some implementations, as an alternative to using a cluster-based approach, a feature vector for a neural network can be generated based upon the individual errors computed for each respective example of a validation set. The individual errors can be collected into a vector to be used as the feature vector, or the individual errors can be aggregated according to any appropriate scheme. For example, a projection can be applied to the vector of errors to generate a feature vector having a particular dimensionality. The projection can be a learned projection. The projection can be linear or non-linear.

[0102] An example of architectures and uses for the multiple neural networks now follows. It will be understood that, while the description below is with reference to asingle neural network, different ones of the multiple neural networks can have architectures that are the same as or different from one another.

[0103] For example, the neural network can be a generative neural network. As a particular example, in some situations, the neural networks can be referred to as an autoregressive neural network, i.e., because the neural network auto-regressively generates an output sequence of tokens. More specifically, the auto-regressively generated output is created by generating each particular token in the output sequence conditioned on a current input sequence that includes any tokens that precede the particular token in the output sequence, i.e., the tokens that have already been generated for any previous positions in the output sequence that precede the particular position of the particular token.

[0104] In particular, to generate a particular token at a particular position within an output sequence, the neural network can process the current input sequence to generate a score distribution (e.g., a probability distribution) that assigns a respective score, e.g., a respective probability’, to each token in a vocabulary of tokens. The neural network can then select, as the particular token, a token from the vocabulary using the score distribution. For example, the neural network can greedily select the highest-scoring token or can sample, e.g., using nucleus sampling or another sampling technique, a token from the distribution.

[0105] For example, the neural network can be an auto-regressive attention neural network that includes (i) a plurality of attention blocks that each apply a self-attention operation and (ii) an output subnetwork that processes an output of the last attention block to generate the score distribution.

[0106] In this example, the neural network can have any of a variety of Transformerbased neural network architectures. Examples of such architectures include those described in J. Hoffmann, S. Borgeaud, A. Mensch, E. Buchatskaya, T. Cai, E.

[0107] Rutherford, D. d. L. Casas, L. A. Hendricks, J. Welbl, A. Clark, et al. Training compute-optimal large language models, arXiv preprint arXiv:2203.15556, 2022; J.W. Rae, S. Borgeaud, T. Cai, K. Millican, J. Hoffmann, H. F. Song, J. Aslanides, S. Henderson, R. Ring, S. Young, E. Rutherford, T. Hennigan, J. Menick, A. Cassirer, R. Powell, G. van den Driessche, L. A. Hendricks, M. Rauh, P. Huang, A. Glaese, J. Welbl, S. Dathathri, S. Huang, J. Uesato, J. Mellor, I. Higgins, A. Creswell, N. McAleese, A.Wu, E. Eisen, S. M. Jayakumar. E. Buchatskaya, D. Budden. E. Sutherland, K. Simonyan, M. Paganini. L. Sifre, L. Martens, X. L. Li, A. Kuncoro, A. Nematzadeh, E. Gribovskaya, D. Donato, A.Lazaridou, A. Mensch, J. Lespiau, M. Tsimpoukelli, N. Grigorev, D. Fritz, T. Sottiaux, M. Pajarskas, T. Pohlen, Z. Gong, D. Toyama, C. de Masson d’Autume, Y. Li, T. Terzi, V. Mikulik, I. Babuschkin, A. Clark, D. de Las Casas, A. Guy, C. Jones, J. Bradbury, M. Johnson, B. A. Hechtman, L. Weidinger, I. Gabriel, W. S. Isaac, E. Lockhart, S.

[0108] Osindero, L. Rimell, C. Dyer, O. Vinyals, K. Ayoub, J. Stanway, L. Bennett, D. Hassabis, K. Kavukcuoglu. and G. Irving. Scaling language models: Methods, analysis & insights from training gopher. CoRR. abs / 2112.11446, 2021; Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. arXiv preprint arXiv: 1910.10683, 2019; Daniel Adiwardana, Minh-Thang Luong, David R. So, Jamie Hall, Noah Fiedel, Romal Thoppilan, Zi Yang, Apoorv Kulshreshtha.

[0109] Gaurav Nemade, Yifeng Lu, and Quoc V. Le. Towards a human-like open-domain chatbot. CoRR, abs / 2001.09977, 2020; and Tom B Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell. et al. Language models are few-shot learners. arXiv preprint arXiv:2005.14165, 2020.

[0110] More specifically, the neural network includes plurality of layers that include a plurality of attention layers.

[0111] Each attention layer receives a respective hidden state for each of the input positions and updates the respective hidden states for each of the input positions by applying an attention mechanism to the respective hidden states.

[0112] Generally, the task can be any task that requires generating an output sequence that includes a respective output token at each of multiple output positions. Examples of such tasks include computer code generation or editing tasks, text generation or editing tasks, image, video, or audio understanding tasks, and so on.

[0113] Some examples of machine learning tasks that a neural network when implemented using one of the architectures described above or other known architectures can be configured to perform follow.

[0114] In any of the implementations below, the neural network may be deployed as part of a chat bot, dialogue agent, or other software tool that receives inputs from users and provides outputs in response to the received input, e.g., as part of a conversation or dialogue. In these implementations, the input sequences received by the neural network are (generated from) user inputs and the output sequences generated by the neural network can be used to generate responses to the user inputs.In implementations the neural network may be configured as, or include, a generative (large) language model or a multi-modal model, e.g., a visual and language model, to perform these example machine learning tasks.

[0115] In some cases, the neural network is a neural network that is configured to perform an image processing task, i.e., receive an input image and to process the input image to generate a network output for the input image. For example, the task may be image classification and the output generated by the neural network for a given image may be scores for each of a set of object categories, with each score representing an estimated likelihood that the image contains an image of an object belonging to the category. As another example, the task can be image embedding generation and the output generated by the neural network can be a numeric embedding of the input image. As yet another example, the task can be object detection and the output generated by the neural network can identify locations in the input image at which particular types of objects are depicted. As yet another example, the task can be image segmentation and the output generated by the neural network can assign each pixel of the input image to a category from a set of categories. In some other cases, the neural network is a neural network that is configured to perform an image generation task, where the input is a conditioning input and the output is a sequence of intensity value inputs for the pixels of an image. The conditioning input can comprise text in natural language or comprise features based upon text in natural language that describes the content of the image to be generated. For example, the conditioning input can indicate the objects that are to be depicted in the generated image.

[0116] As one example, the task may be a neural machine translation task. For example, if the input to the neural network is a sequence of text, e.g., a sequence of words, phrases, characters, or word pieces, in one language, the output generated by the neural network may be a translation of the sequence of text into another language, i.e., a sequence of text in the other language that is a translation of the input sequence of text. The vocabulary for the input tokens may be words, wordpieces or characters of the first language, and the vocabulary for the output tokens may be words, wordpieces or characters of the other language. As a particular example, the task may be a multi-lingual machine translation task, where a single neural network is configured to translate between multiple different source language - target language pairs. In this example, the source language text may be augmented with an identifier that indicates the target language into which the neural network should translate the source language text.Some implementations may be used for automatic code generation. For example, the input tokens may represent words, wordpieces or characters in a first natural language and the output tokens may represent instructions in a computer programming or markup language, or instructions for controlling an application program to perform a task e.g. build a data item such as an image or w eb page.

[0117] As another example, the task may be an audio processing task. For example, if the input to the neural network is a sequence representing a spoken utterance, the output generated by the neural netw ork may be a score for each of a set of pieces of text, each score representing an estimated likelihood that the piece of text is the correct transcript for the utterance. As another example, if the input to the neural network is a sequence representing a spoken utterance, the output generated by the neural network can indicate whether a particular word or phrase (“hotword”) was spoken in the utterance. As another example, if the input to the neural network is a sequence representing a spoken utterance, the output generated by the neural network can be a classification of the spoken utterance into one of a plurality’ of categories, for example an identity of the natural language in which the utterance was spoken.

[0118] As another example, the task can be a natural language processing or understanding task, e.g., an entailment task, a paraphrase task, a textual similarity task, a sentiment task, a sentence completion task, a grammaticality task, and so on, that operates on a sequence of text in some natural language.

[0119] As another example, the task can be a text to speech task, where the input is text in a natural language or features of text in a natural language and the network output is a spectrogram, a waveform, or other data defining audio of the text being spoken in the natural language.

[0120] As another example, the task can be a health prediction task, where the input is a sequence derived from electronic health record data for a patient and the output is a prediction that is relevant to the future health of the patient, e.g., a predicted treatment that should be prescribed to the patient, the likelihood that an adverse health event will occur to the patient, or a predicted diagnosis for the patient. Such electronic health data may, for example, comprise one or more sequences of physiological data taken from a patient, with the output being a corresponding prediction that relates to those sequences of data. Examples of physiological data and a corresponding prediction include: blood glucose measurements, with the prediction being a predicted future blood glucose measurement or the prediction of a hyper- or hypo-glycemic event; a heart rate, with theprediction being the presence or absence of a heart condition, or a future cardiac event; blood pressure measurements, with the prediction being the risk of a future heart condition; or the like.

[0121] As another example, the task can be a text generation task, where the input is a sequence of text, and the output is another sequence of text, e.g., a completion of the input sequence of text, a response to a question posed in the input sequence, or a sequence of text that is about a topic specified by the first sequence of text. As another example, the input to the text generation task can be an input other than text, e.g., an image, and the output sequence can be text that describes the input.

[0122] In some implementations the input sequence represents data to be compressed, e.g. image data, text data, audio data, or any other type of data; and the output sequence a compressed version of the data. The input and output tokens may each comprise any representation of the data to be compressed / compressed data e.g. symbols or embeddings generated / decoded by a respective neural network.

[0123] As another example, the task can be an agent control task, where the input is a sequence of observations or other data characterizing states of an environment and the output defines an action to be performed by the agent in response to the most recent data in the sequence. The agent can be, e.g., a real-world or simulated robot, a control system for an industrial facility, or a control system that controls a different kind of agent. The observations may comprise sensor data captured by sensors associated with (e.g. part of) the agent, for example visual data, LIDAR data, sonar data, agent configuration data (e.g. joint angles), agent orientation data, or the like.

[0124] In some implementations, the environment is a real-world environment, the agent is a mechanical (or electro-mechanical) agent interacting with the real-world environment, e.g., a robot or an autonomous or semi-autonomous land, air, or sea vehicle operating in or navigating through the environment, and the actions are actions taken by the mechanical agent in the real-world environment to perform the task. For example, the agent may be a robot interacting with the environment to accomplish a specific task, e.g., to locate or manipulate an object of interest in the environment or to move an object of interest to a specified location in the environment or to navigate to a specified destination in the environment.

[0125] In these implementations, the observations may include, e.g., one or more of: images, object position data, and sensor data to capture observations as the agent interacts with the environment, for example sensor data from an image, distance, or position sensoror from an actuator. For example, in the case of a robot, the observations may include data characterizing the current state of the robot, e.g., one or more of: joint position, joint velocity, joint force, torque or acceleration, e.g., gravity-compensated torque feedback, and global or relative pose of an item held by the robot. In the case of a robot or other mechanical agent or vehicle the observations may similarly include one or more of the position, linear or angular velocity, force, torque or acceleration, and global or relative pose of one or more parts of the agent. The observations may be defined in 1, 2 or 3 dimensions, and may be absolute and / or relative observations. The observations may also include, for example, sensed electronic signals such as motor current or a temperature signal; and / or image or video data for example captured by a camera or a LIDAR sensor, e.g., data from sensors of the agent or data from sensors that are located separately from the agent in the environment.

[0126] In these implementations, the actions may be control signals to control the robot or other mechanical agent, e.g., torques for the joints of the robot or higher-level control commands, or the autonomous or semi-autonomous land, air, sea vehicle, e.g., torques to the control surface or other control elements e.g. steering control elements of the vehicle, or higher-level control commands. The control signals can include for example, position, velocity, or force / torque / accel eration data for one or more joints of a robot or parts of another mechanical agent. The control signals may also or instead include electronic control data such as motor control data, or more generally data for controlling one or more electronic devices within the environment the control of which has an effect on the observed state of the environment. For example, in the case of an autonomous or semi-autonomous land or air or sea vehicle the control signals may define actions to control navigation e.g. steering, and movement e.g.. braking and / or acceleration of the vehicle.

[0127] In some implementations the environment is a simulation of the above-described real-world environment, and the agent is implemented as one or more computers interacting with the simulated environment. For example, a system implementing the neural network may be used to select actions in the simulated environment during training or evaluation of the system and, after training, or evaluation, or both, are complete, the action selection policy may be deployed for controlling a real-world agent in the particular real-world environment that was the subject of the simulation. This can avoid unnecessary' wear and tear on and damage to the real-world environment or real-world agent and can allow the control neural network to be trained and evaluated on situations that occur rarely or are difficult or unsafe to re-create in the real-world environment. Forexample the system may be partly trained using a simulation of a mechanical agent in a simulation of a particular real-world environment, and afterwards deployed to control the real mechanical agent in the particular real-world environment. Thus in such cases the observations of the simulated environment relate to the real-world environment, and the selected actions in the simulated environment relate to actions to be performed by the mechanical agent in the real-world environment.

[0128] In some implementations, as described above, the agent may not include a human being (e g. it is a robot). Conversely, in some implementations the agent comprises a human user of a digital assistant such as a smart speaker, smart display, or other device. Then the information defining the task can be obtained from the digital assistant, and the digital assistant can be used to instruct the user based on the task.

[0129] For example, a system implementing the neural network may output to the human user, via the digital assistant, instructions for actions for the user to perform at each of a plurality7of time steps. The instructions may for example be generated in the form of natural language (transmitted as sound and / or text on a screen) based on actions chosen by the system. The system chooses the actions such that they contribute to performing a task. A monitoring system (e.g. a video camera system) may be provided for monitoring the action (if any) which the user actually performs at each time step, in case (e.g. due to human error) it is different from the action which the system instructed the user to perform. Using the monitoring system the system can determine whether the task has been completed. The system may identify actions which the user performs incorrectly with more than a certain probability. If so, when the system instructs the user to perform such an identified action, the system may warn the user to be careful. Alternatively or additionally, the system may learn not to instruct the user to perform the identified actions, i.e. ones which the user is likely to perform incorrectly.

[0130] More generally, the digital assistant instructing the user may comprise receiving, at the digital assistant, a request from the user for assistance and determining, in response to the request, a series of tasks for the user to perform, e.g. steps or sub-tasks of an overall task. Then for one or more tasks of the series of tasks, e.g. for each task, e.g. until a final task of the series the digital assistant can be used to output to the user an indication of the task, e.g. step or sub-task, to be performed. This may be done using natural language, e.g. on a display and / or using a speech synthesis subsystem of the digital assistant. Visual, e.g. video, and / or audio observations of the user performing the task may be captured, e.g. using the digital assistant. A system as described above may then be used to determinewhether the user has successfully achieved the task e.g. step or sub-task, i.e. from the answer as previously described. If there are further tasks to be completed the digital assistant may then, in response, progress to the next task (if any) of the series of tasks, e.g. by outputting an indication of the next task to be performed. In this way the user may be led step-by-step through a series of tasks to perform an overall task. During the training of the neural network, training rewards may be generated e.g. from video data representing examples of the overall task (if corpuses of such data are available) or from a simulation of the overall task.

[0131] In a further aspect there is provided a digital assistant device including a system as described above. The digital assistant can also include a user interface to enable a user to request assistance and to output information. In implementations this is a natural language user interface and may comprise a keyboard, voice input-output subsystem, and / or a display. The digital assistant can further include an assistance subsystem configured to determine, in response to the request, a series of tasks for the user to perform. In implementations this may comprise a generative (large) language model, in particular for dialog, e.g. a conversation agent such as Spanow (Glaese et al. arXiv:2209.14375) or Chinchilla (Hoffmann et al. arXiv:2203.15556). The digital assistant can have an observation capture subsystem to capture visual and / or audio observations of the user performing a task; and an interface for the above-described neural network (which may be implemented locally or remotely). The digital assistant can also have an assistance control subsystem configured to assist the user. The assistance control subsystem can be configured to perform the steps described above, for one or more tasks e.g. of a series of tasks, e.g. until a final task of the series. More particularly the assistance control subsystem and output to the user an indication of the task to be performed, capture, using the observation capture subsystem, visual or audio observations of the user performing the task, determine from the above-described answer whether the user has successfully achieved the task. In response the digital assistant can progress to a next task of the series of tasks and / or control the digital assistant, e.g. to stop capturing observations.

[0132] As another example, the task can be a genomics task, where the input is a sequence representing a fragment of a DNA sequence or other molecule sequence and the output is either an embedding of the fragment for use in a downstream task, e.g., by making use of an unsupervised learning technique on a data set of DNA sequence fragments, or an output for the downstream task. Examples of downstream tasks includepromoter site prediction, methylation analysis, predicting functional effects of non-coding variants, and so on.

[0133] In some cases, the machine learning task is a combination of multiple individual machine learning tasks, i.e., the system is configured to perform multiple different individual machine learning tasks, e.g., two or more of the machine learning tasks mentioned above. For example, the system can be configured to perform multiple individual natural language understanding tasks, with the network input including an identifier for the individual natural language understanding task to be performed on the network input.

[0134] In some cases, the machine learning task is a multi-modal processing task that requires processing multi-modal data. In general, multi-modal data is a combination of two or more different types of data, e.g., two or more of audio data, image data, text data, or graph data. As one example the multi-modal data may comprise audio-visual data, comprising a combination of pixels of an image or of video and audio data representing values of a digitized audio waveform. As another example the multi-modal data may comprise a combination of i) text data representing text in a natural language and li) pixels of an image or of video or audio data representing values of an audio waveform. Optionally, but not necessarily, the different types of data may represent the same or overlapping objects using the different modalities (types), and when processing multimodal data the data may be mapped into a common embedding space.

[0135] As a particular example, the task is a multi-modal processing task that requires processing both text and image inputs, so that the neural network includes both a computer vision neural network and a text processing neural netw ork. That is, the target output to be generated by the computer vision neural network for a given image depends on one or more outputs generated by the text processing neural network for one or more corresponding text inputs (and vice versa). Examples of such tasks include openvocabulary image classification, open- vocabulary object detection, image captioning, text-based image search, image-based retrieval, and so on.

[0136] More generally, the multi-modal processing task may correspond to any of the tasks previously described for any of the types of data making up the multi-modal combination. For example, an accuracy of the previously described tasks may be increased when the task is applied to multi-modal data combining the data for which the task has been previously described and another type of data. For example detection orclassification of an object or event may be improved when data of multiple different types (modalities) is processed.

[0137] More generally, the task to be performed by the neural network can be specified by the input sequence. As a particular example, the input sequence can include a prompt or an instruction that specifies the task that is to be performed by the neural network. Optionally, in this example, the input sequence also includes context for performing the task.

[0138] In this specification, the term "configured" is used in relation to computing systems and environments, as well as computer program components. A computing system or environment is considered "configured" to perform specific operations or actions when it possesses the necessary software, firmware, hardware, or a combination thereof, enabling it to cany' out those operations or actions during operation. For instance, configuring a system might involve installing a software library' with specific algorithms, updating firmware with new instructions for handling data, or adding a hardware component for enhanced processing capabilities. Similarly, one or more computer programs are "configured" to perform particular operations or actions when they contain instructions that, upon execution by a computing device or hardware, cause the device to perform those intended operations or actions.

[0139] The embodiments and functional operations described in this specification can be implemented in various forms, including digital electronic circuitry, software, firmware, computer hardware (encompassing the disclosed structures and their structural equivalents), or any combination thereof. The subject matter can be realized as one or more computer programs, essentially modules of computer program instructions encoded on a tangible non-transitory storage medium for execution by or to control the operation of a computing device or hardware. The storage medium can be a storage device such as a hard drive or solid-state drive (SSD), a storage medium, a random or serial access memory' device, or a combination of these. Additionally or alternatively, the program instructions can be encoded on a transmitted signal, such as a machine-generated electrical, optical, or electromagnetic signal, designed to carry information for transmission to a receiving device or system for execution by a computing device or hardware. Furthermore, implementations may leverage emerging technologies like quantum computing or neuromorphic computing for specific applications, and may be deployed in distributed or cloud-based environments where components reside on different machines or within a cloud infrastructure.The term "computing device or hardware" refers to the physical components involved in data processing and encompasses all types of devices and machines used for this purpose. Examples include processors or processing units, computers, multiple processors or computers working together, graphics processing units (GPUs), tensor processing units (TPUs), and specialized processing hardware such as field-programmable gate arrays (FPGAs) or application-specific integrated circuits (ASICs). In addition to hardware, a computing device or hardware may also include code that creates an execution environment for computer programs. This code can take the form of processor firmware, a protocol stack, a database management system, an operating system, or a combination of these elements. Embodiments may particularly benefit from utilizing the parallel processing capabilities of GPUs, in a General-Purpose computing on Graphics Processing Units (GPGPU) context, where code specifically designed for GPU execution, often called kernels or shaders, is employed. Similarly, TPUs excel at running optimized tensor operations crucial for many machine learning algorithms. By leveraging these accelerators and their specialized programming models, the system can achieve significant speedups and efficiency gains for tasks involving artificial intelligence and machine learning, particularly in areas such as computer vision, natural language processing, and robotics.

[0140] A computer program, also referred to as software, an application, a module, a script, code, or simply a program, can be written in any programming language, including compiled or interpreted languages, and declarative or procedural languages. It can be deployed in various forms, such as a standalone program, a module, a component, a subroutine, or any other unit suitable for use within a computing environment. A program may or may not correspond to a single file in a file system and can be stored in various ways. This includes being embedded within a file containing other programs or data (e.g., scripts within a markup language document), residing in a dedicated file, or distributed across multiple coordinated files (e.g., files storing modules, subprograms, or code segments). A computer program can be executed on a single computer or across multiple computers, whether located at a single site or distributed across multiple sites and interconnected through a data communication network. The specific implementation of the computer programs may involve a combination of traditional programming languages and specialized languages or libraries designed for GPGPU programming or TPU utilization, depending on the chosen hardware platform and desired performance characteristics.In this specification, the term "engine" broadly refers to a software-based system, subsystem, or process designed to perform one or more specific functions. An engine is typically implemented as one or more software modules or components installed on one or more computers, which can be located at a single site or distributed across multiple locations. In some instances, one or more dedicated computers may be used for a particular engine, while in other cases, multiple engines may operate concurrently on the same one or more computers. Examples of engine functions within the context of Al and machine learning could include data pre-processing and cleaning, feature engineering and extraction, model training and optimization, inference and prediction generation, and post-processing of results. The specific design and implementation of engines will depend on the overall architecture and the distribution of computational tasks across various hardware components, including CPUs, GPUs, TPUs, and other specialized processors.

[0141] The processes and logic flows described in this specification can be executed by one or more programmable computers running one or more computer programs to perform functions by operating on input data and generating output. Additionally, graphics processing units (GPUs) and tensor processing units (TPUs) can be utilized to enable concurrent execution of aspects of these processes and logic flows, significantly accelerating performance. This approach offers significant advantages for computationally intensive tasks often found in Al and machine learning applications, such as matrix multiplications, convolutions, and other operations that exhibit a high degree of parallelism. By leveraging the parallel processing capabilities of GPUs and TPUs, significant speedups and efficiency gains compared to relying solely on CPUs can be achieved. Alternatively or in combination with programmable computers and specialized processors, these processes and logic flows can also be implemented using specialized processing hardware, such as field-programmable gate arrays (FPGAs) or applicationspecific integrated circuits (ASICs), for even greater performance or energy efficiency in specific use cases.

[0142] Computers capable of executing a computer program can be based on general-purpose microprocessors, special-purpose microprocessors, or a combination of both. They can also utilize any other type of central processing unit (CPU). Additionally, graphics processing units (GPUs), tensor processing units (TPUs), and other machine learning accelerators can be employed to enhance performance, particularly for tasks involving artificial intelligence and machine learning. These accelerators often work inconjunction with CPUs, handling specialized computations while the CPU manages overall system operations and other tasks. Typically, a CPU receives instructions and data from read-only memory (ROM), random access memory (RAM), or both. The elements of a computer include a CPU for executing instructions and one or more memory' devices for storing instructions and data. The specific configuration of processing units and memory will depend on factors like the complexity of the Al model, the volume of data being processed, and the desired performance and latency requirements. Embodiments can be implemented on a wide range of computing platforms, from small embedded devices with limited resources to large-scale data center systems with high-performance computing capabilities. The system may include storage devices like hard drives, SSDs, or flash memory for persistent data storage.

[0143] Computer-readable media suitable for storing computer program instructions and data encompass all forms of non-volatile memory, media, and memory devices.

[0144] Examples include semiconductor memory devices such as read-only memory (ROM), solid-state drives (SSDs), and flash memory devices; hard disk drives (HDDs); optical media; and optical discs such as CDs, DVDs, and Blu-ray discs. The specific t pe of computer-readable media used will depend on factors such as the size of the data, access speed requirements, cost considerations, and the desired level of portability' or permanence.

[0145] To facilitate user interaction, embodiments of the subject matter described in this specification can be implemented on a computing device equipped with a display device, such as a liquid cry stal display (LCD) or an organic light-emitting diode (OLED) display , for presenting information to the user. Input can be provided by the user through various means, including a keyboard), touchscreens, voice commands, gesture recognition, or other input modalities depending on the specific device and application. Additional input methods can include acoustic, speech, or tactile input, while feedback to the user can take the form of visual, auditory', or tactile feedback. Furthermore, computers can interact with users by exchanging documents with a user's device or application. This can involve sending web content or data in response to requests or sending and receiving text messages or other forms of messages through mobile devices or messaging platforms. The selection of input and output modalities will depend on the specific application and the desired form of user interaction.

[0146] Machine learning models can be implemented and deployed using machine learning frameworks, such as TensorFlow or JAX. These frameworks offercomprehensive tools and libraries that facilitate the development, training, and deployment of machine learning models.

[0147] Embodiments of the subject matter described in this specification can be implemented within a computing system comprising one or more components, depending on the specific application and requirements. These may include a back-end component, such as a back-end server or cloud-based infrastructure; an optional middleware component, such as a middleware server or application programming interface (API), to facilitate communication and data exchange; and a front-end component, such as a client device with a user interface, a web browser, or an app, through which a user can interact with the implemented subject matter. For instance, the described functionality' could be implemented solely on a client device (e.g., for on-device machine learning) or deployed as a combination of front-end and back-end components for more complex applications. These components, when present, can be interconnected using any form or medium of digital data communication, such as a communication network like a local area network (LAN) or a wide area network (WAN) including the Internet. The specific system architecture and choice of components will depend on factors such as the scale of the application, the need for real-time processing, data security requirements, and the desired user experience.

[0148] The computing system can include clients and servers that may be geographically separated and interact through a communication network. The specific type of network, such as a local area network (LAN), a wide area network (WAN), or the Internet, will depend on the reach and scale of the application. The client-server relationship is established through computer programs running on the respective computers and designed to communicate with each other using appropriate protocols. These protocols may include HTTP, TCP / IP, or other specialized protocols depending on the nature of the data being exchanged and the security requirements of the system. In certain embodiments, a server transmits data or instructions to a user's device, such as a computer, smartphone, or tablet, acting as a client. The client device can then process the received information, display results to the user, and potentially send data or feedback back to the server for further processing or storage. This allows for dynamic interactions betw een the user and the system, enabling a wide range of applications and functionalities.

[0149] While this specification contains many specific implementation details, these should not be construed as limitations on the scope of any invention or on the scope ofwhat may be claimed, but rather as descriptions of features that may be specific to particular embodiments of particular inventions. Certain features that are described in this specification in the context of separate embodiments can also be implemented in combination in a single embodiment. Conversely, various features that are described in the context of a single embodiment can also be implemented in multiple embodiments separately or in any suitable subcombination. Moreover, although features may be described above as acting in certain combinations and even initially be claimed as such, one or more features from a claimed combination can in some cases be excised from the combination, and the claimed combination may be directed to a subcombination or variation of a subcombination.

[0150] Similarly, while operations are depicted in the drawings and recited in the claims in a particular order, this should not be understood as requiring that such operations be performed in the particular order show n or in sequential order, or that all illustrated operations be performed, to achieve desirable results. In certain circumstances, multitasking and parallel processing may be advantageous. Moreover, the separation of various system modules and components in the embodiments described above should not be understood as requiring such separation in all embodiments, and it should be understood that the described program components and systems can generally be integrated together in a single software product or packaged into multiple software products.

[0151] Particular embodiments of the subject matter have been described. Other embodiments are within the scope of the following claims. For example, the actions recited in the claims can be performed in a different order and still achieve desirable results. As one example, the processes depicted in the accompanying figures do not necessarily require the particular order shown, or sequential order, to achieve desirable results. In some cases, multitasking and parallel processing may be advantageous.

[0152] Innovative aspects of the present disclosure are also set out in the following numbered clauses.

[0153] Clause 1. A method performed by one or more computers, the method comprising:

[0154] maintaining, for each of a plurality of neural networks, a respective feature vector characterizing the neural netw ork and a respective cost measure characterizing an amount of computational resources required to process a set of one or more inputs using the neural netw ork;

[0155] receiving a new netw ork input;selecting, based on the new network input, the respective feature vectors for the neural networks, and the respective cost measures for the neural networks, a neural network from the plurality of neural networks;

[0156] providing the new network input as input to the selected neural network to obtain a network output; and

[0157] providing the network output in response to the new network input.

[0158] Clause 2. The method of clause 1, wherein the plurality of neural networks are each a respective generative neural network.

[0159] Clause 3. The method of any preceding clause, wherein selecting, based on the new network input, the respective feature vectors for the neural networks, and the respective cost measures for the neural networks, a neural network from the plurality of neural networks comprises:

[0160] determining, for each of the plurality of neural networks, a respective error measure for the new network input that measures an expected error of the neural network on the new network input; and

[0161] selecting, based on the respective error measures and the respective cost measures for the neural networks, a neural network from the plurality of neural networks.

[0162] Clause 4. The method of clause 3, wherein selecting, based on the respective error measures and the respective cost measures for the neural networks, a neural network from the plurality of neural networks comprises:

[0163] for each neural network, applying a scoring function to the respective error measure and the respective cost measure for the neural network to generate a respective overall score for the neural network; and

[0164] selecting, based on the respective overall scores for the neural networks, a neural network from the plurality of neural networks.

[0165] Clause 5. The method of clause 4, wherein selecting, based on the respective overall scores for the neural networks, a neural network from the plurality7of neural networks comprises:

[0166] selecting a neural network having a lowest overall score.Clause 6. The method of clause 4 or clause 5, wherein the scoring function comprises a first term that is an increasing function of the respective error measures and a second term that is an increasing function of the respective overall scores.

[0167] Clause 7. The method of clause 6, wherein, for each neural network, the overall score is equal to a sum of an output of the first term and a product between a weight value and an output of the second term.

[0168] Clause 8. The method of any preceding clause, wherein the respective feature vector characterizing the neural network comprises a respective error measure for the neural network for each of a plurality of clusters of network inputs.

[0169] Clause 9. The method of clause 8, when dependent on clause 3, wherein determining, for each of the plurality of neural networks, a respective error measure for the new network input that measures an expected error of the neural network on the new network input comprises:

[0170] assigning the new network input to a particular cluster of the plurality’ of clusters; and

[0171] determining, for each of the plurality of neural networks, the respective error measure for the new network input based on the respective error measure for the neural network for the particular cluster.

[0172] Clause 10. The method of clause 9, wherein each cluster is represented by a respective centroid in an embedding space, and wherein assigning the new network input to a particular cluster of the plurality of clusters comprises:

[0173] generating an embedding of the new network input in the embedding space; and assigning the new network input to a cluster of the plurality of clusters that has a centroid that is closest to the embedding of the new network input in the embedding space.

[0174] Clause 11. The method of clause 8, when dependent on clause 3, wherein determining, for each of the plurality of neural networks, a respective error measure for the new network input that measures an expected error of the neural network on the new network input comprises:generating a respective likelihood score for each cluster of the plurality of clusters, wherein the respective likelihood score for each cluster measures a likelihood that the new network input belongs to the cluster; and

[0175] determining, for each of the plurality of neural networks, the respective error measure for the new network input based on the respective likelihood score for the clusters and the respective error measures for the neural network for the clusters.

[0176] Clause 12. The method of clause 11, wherein determining, for each of the plurality of neural networks, the respective error measure for the new network input based on the respective likelihood score for the clusters and the respective error measures for the neural network for the clusters comprises:

[0177] for each of the plurality of neural networks, determining a weighted sum of the respective error measures for the neural network for the clusters, wherein the respective error measure for each cluster is weighted by the respective likelihood score for the cluster in the weighted sum.

[0178] Clause 13. The method of clause 11 or clause 12, wherein generating a respective likelihood score for each cluster of the plurality of clusters comprises processing the new network input using a learned clustering function.

[0179] Clause 14. The method of any one of clauses 8-13, further comprising:

[0180] obtaining a plurality7of training network inputs; and

[0181] clustering the plurality of training network inputs to generate the plurality of clusters.

[0182] Clause 15. The method of any one of clauses 8-14, further comprising:

[0183] obtaining a plurality7of validation network inputs and a respective target output for each validation network input;

[0184] assigning each validation network input to a respective one of the plurality of clusters; and

[0185] for each neural network, generating the respective feature vector characterizing the neural network, comprising:

[0186] for each validation network input, processing the validation network input using the neural network to generate a respective error for the validation network inputrelative to the respective target output for the validation network input; and determining the respective error measure for the neural network for each of the plurality of clusters based on the respective errors for the validation network inputs that are assigned to the cluster.

[0187] Clause 16. The method of any preceding clause, further comprising:

[0188] obtaining data specifying a new neural network for use in processing network inputs; and

[0189] in response:

[0190] generating a respective feature vector characterizing the new neural network, comprising:

[0191] for each of a plurality of validation network inputs, processing the validation network input using the new neural network to generate a respective error for the validation network input relative to the respective target output for the validation network input; and

[0192] determining a respective error measure for the new neural network for each of the plurality of clusters based on the respective errors for the validation network inputs that are assigned to the cluster.

[0193] Clause 17. A system comprising one or more computers and one or more storage devices storing instructions that when executed by the one or more computers cause the one more computers to perform the operations of the respective method of any one of clauses 1-16.

[0194] Clause 18. One or more computer storage media storing instructions that when executed by one or more computers cause the one more computers to perform the operations of the respective method of any one of clauses 1-16.

[0195] What is claimed is:

Claims

CLAIMS1. A method performed by one or more computers, the method comprising:maintaining, for each of a plurality of neural networks, a respective feature vector characterizing the neural network and a respective compute measure characterizing an amount of computational resources required to process a set of one or more inputs using the neural network;receiving a new network input;selecting, based on the new network input, the respective feature vectors for the neural networks, and the respective compute measures for the neural networks, a neural network from the plurality of neural networks;providing the new network input as input to the selected neural network to obtain a network output; andproviding the network output in response to the new network input.

2. The method of claim 1, wherein the plurality of neural networks are each a respective generative neural network.

3. The method of any preceding claim, wherein selecting, based on the new network input, the respective feature vectors for the neural networks, and the respective compute measures for the neural networks, a neural network from the plurality of neural networks comprises:determining, for each of the plurality of neural networks, a respective error measure for the new network input that measures an expected error of the neural network on the new network input; andselecting, based on the respective error measures and the respective compute measures for the neural networks, a neural network from the plurality of neural networks.

4. The method of claim 3, wherein selecting, based on the respective error measures and the respective compute measures for the neural networks, a neural network from the plurality of neural networks comprises:for each neural network, applying a scoring function to the respective error measure and the respective compute measure for the neural network to generate a respective overall score for the neural network; andselecting, based on the respective overall scores for the neural networks, a neural network from the plurality of neural networks.

5. The method of claim 4, wherein selecting, based on the respective overall scores for the neural networks, a neural netw ork from the plurality of neural networks comprises:selecting a neural network having a lowest overall score.

6. The method of claim 4 or claim 5, wherein the scoring function comprises a first term that is an increasing function of the respective error measures and a second term that is an increasing function of the respective overall scores.

7. The method of claim 6, wherein, for each neural network, the overall score is equal to a sum of an output of the first term and a product betw een a w eight value and an output of the second term.

8. The method of any preceding claim, wherein the respective feature vector characterizing the neural network comprises a respective error measure for the neural network for each of a plurality of clusters of network inputs.

9. The method of claim 8, when dependent on claim 3, wherein determining, for each of the plurality7of neural networks, a respective error measure for the new network input that measures an expected error of the neural network on the new network input comprises:assigning the new network input to a particular cluster of the plurality of clusters: anddetermining, for each of the plurality of neural netw orks, the respective error measure for the new network input based on the respective error measure for the neural network for the particular cluster.

10. The method of claim 9, wherein each cluster is represented by a respective centroid in an embedding space, and wherein assigning the new network input to a particular cluster of the plurality of clusters comprises:generating an embedding of the new network input in the embedding space; andassigning the new network input to a cluster of the plurality of clusters that has a centroid that is closest to the embedding of the new network input in the embedding space.

11. The method of claim 8, when dependent on claim 3, wherein determining, for each of the plurality of neural networks, a respective error measure for the new network input that measures an expected error of the neural network on the new network input comprises:generating a respective likelihood score for each cluster of the plurality of clusters, wherein the respective likelihood score for each cluster measures a likelihood that the new network input belongs to the cluster; anddetermining, for each of the plurality of neural networks, the respective error measure for the new network input based on the respective likelihood score for the clusters and the respective error measures for the neural network for the clusters.

12. The method of claim 11, wherein determining, for each of the plurality of neural networks, the respective error measure for the new network input based on the respective likelihood score for the clusters and the respective error measures for the neural network for the clusters comprises:for each of the plurality’ of neural networks, determining a weighted sum of the respective error measures for the neural network for the clusters, wherein the respective error measure for each cluster is weighted by the respective likelihood score for the cluster in the weighted sum.

13. The method of claim 11 or claim 12, wherein generating a respective likelihood score for each cluster of the plurality’ of clusters comprises processing the new network input using a learned clustering function.

14. The method of any one of claims 8-13. further comprising:obtaining a plurality of training network inputs; andclustering the plurality of training network inputs to generate the plurality of clusters.

15. The method of any one of claims 8-14. further comprising:obtaining a plurality of validation network inputs and a respective target output for each validation network input;assigning each validation network input to a respective one of the plurality of clusters; andfor each neural network, generating the respective feature vector characterizing the neural network, comprising:for each validation network input, processing the validation network input using the neural network to generate a respective error for the validation network input relative to the respective target output for the validation network input; and determining the respective error measure for the neural network for each of the plurality of clusters based on the respective errors for the validation network inputs that are assigned to the cluster.

16. The method of any preceding claim, further comprising:obtaining data specifying a new neural network for use in processing network inputs; andin response:generating a respective feature vector characterizing the new neural network, comprising:for each of a plurality of validation network inputs, processing the validation network input using the new neural network to generate a respective error for the validation network input relative to the respective target output for the validation network input; anddetermining a respective error measure for the new neural network for each of the plurality of clusters based on the respective errors for the validation network inputs that are assigned to the cluster.

17. A system comprising one or more computers and one or more storage devices storing instructions that when executed by the one or more computers cause the one more computers to perform the operations of the respective method of any one of claims 1-16.

18. One or more computer storage media storing instructions that when executed by one or more computers cause the one more computers to perform the operations of the respective method of any one of claims 1-16.

Images