Machine learning models at different bit-widths with nested precision

Abstract

Methods and systems for quantizing model parameters of a machine learning model to convert floating-point (e.g. FP32) model parameters to parameter values (e.g. INT8) having a smaller bit-width than the floating-point parameter values. The quantization produces a plurality of nested models at which the model parameters of one model (i.e. larger bit-width model) are nested within another (i.e. smaller bit-width). As one example, a system can apply a quantization function to convert the floating-point model parameters to reduced precision parameter values representing the model parameters of the larger bit-width model. The reduced precision parameter values may be sliced to extract model parameters of a smaller bit-width model. In each training iteration, a training objective is determined using nested models of different bit-widths. The training objective is optimized by adjusting parameters that include (i) the model parameters, (ii) quantization parameters associated with the quantization function, or (iii) both.

Description

[0001] MACHINE LEARNING MODELS AT DIFFERENT BIT-WIDTHS WITH NESTED PRECISION BACKGROUND

[0002] [1] This specification relates to quantizing the model parameters of a machine learning model.

[0003] [2] Machine learning models include neural networks that employ one or more layers of nonlinear units to predict an output for a received input. Some neural networks include one or more hidden layers in addition to an output layer. The output of each hidden layer is used as input to the next layer in the network, i.e., the next hidden layer or the output layer. Each layer of the network generates an output from a received input in accordance with current value inputs of a respective set of model parameters.

[0004] SUMMARY

[0005] [3] This specification describes a system implemented as computer programs on one or more computers in one or more locations that quantizes the model parameters of a machine learning model.

[0006] [4] Generally, the system can include any appropriate devices that can carry out operations required to perform machine learning model training. Examples of such devices include central processing units (CPUs) and hardware devices that include one or more hardware accelerators that have circuitry for performing matrix-vector multiplication in hardware, e.g., graphics processing units (GPUs), tensor processing units (TPUs), and other application-specific integrated circuits (ASICs) that are optimized for performing machine learning computations.

[0007] [5] The machine learning model may be trained to perform any of a number of possible tasks. The machine learning model may be a neural network. The neural network may, for example, comprise one or more transformer models that form part of a large language model (LLM) or a vision language model (VLM).

[0008] [6] In particular, the system can obtain a machine learning model comprising model parameters represented by floating-point parameter values (numbers). When the machine learning model is a neural network, the model parameters can, for example, comprise weights and / or biases. The floating-point parameter values may be 32-bit (e.g. FP32 orBF32) floating-point parameter values or 16-bit (FP16 or BF16) floating-point parameter values. The machine learning model may be a pre-trained model that is then subject to a further training process in order to quantize the floating-point parameter values to reduced precision parameter values having a smaller bit-width, such as 8-bit integers (INT8).

[0009] [7] The system performs a training process comprising one or more iterations (i.e. training iterations) for quantizing the model parameters. For each iteration, the system obtains training data compnsing input data for the iteration. The system applies a quantization function to convert at least some of the floating-point parameter values to a set of reduced precision parameter values each having a first number of bits (e.g. 8-bits). The quantization function may initially be applied using an initial (i.e. default) set of quantization parameters (e.g. determining scaling of the model parameters). The input data for the iteration (e.g. the training samples for the iteration) can be supplied to the machine learning model to calculate first output values based on (i.e., in accordance with) the quantized (i.e. reduced precision) parameter values. The system uses the first output values during the training, e.g., the system can calculate an associated first training objective term based on the first output values. For each of the reduced precision parameter values, the system then produces a respective updated reduced precision parameter value by extracting a second number of most significant bits (MSBs) belonging to the respective reduced precision parameter value. The updated reduced precision parameter values, therefore, have a precision that is less than that of the reduced precision parameter values from which they are produced, but can comprise the same number of bits in some implementations. For example, the system may slice out the second number of MSBs by zeroing the remaining bits of an 8-bit integer. The second number of MSBs may, for example, comprise 4-bits, so that the updated integers simulate 4-bit integers (INT4). In this example, the reduced precision parameter values are integers, but in another example the reduced precision parameter values could be floating-point parameter values (e.g. FP16) from which MSBs are extracted from the exponent and mantissa to simulate further reduced precision floating-point parameter values (e.g. FP8).

[0010] [8] The input data for the iteration is supplied to the machine learning model to then calculate second output values based on the updated reduced precision parameter values. The second output values are used during the training, e.g., the system can use the second output values to calculate a second training objective term. The system optimizes a training objective determined based on the first output values for the iteration and the second output values for the iteration, e.g., a training objective comprising a sum oftraining objective terms including the first training objective term and the second training objective term. The optimizing comprises adjusting parameter values of parameters that include (i) the at least some of the model parameters, (ii) quantization parameters associated with the quantization function, or (iii) both. For example, the optimizing can comprise adjusting the floating-point parameter values for the at least some of the model parameters and / or the parameter values for the quantization parameters.

[0011] [9] In this way. training is performed so as to provide quantization that is accurate for different bit-width parameter values (e.g. INT8, INT4, INT2) that are nested in each parameter value. The training objective terms may be loss terms. The first and second training objective terms may be loss terms. The third training objective term may be a third loss term.

[0012]

[0010] Once the one or more training iterations are complete, an updated version of the model is provided comprising model parameters represented by (i.e., set to) final parameter values determined based on the adjusted parameter values following the one or more iterations, the final parameter values each having the first number of bits (e.g. 8-bits). The updated version of the model is a nested model in which model parameters of different bit-widths are nested. In this way, a reduced bit-width model (e.g. in which each model parameter is represented by 4-bits or by 2-bits) may be extracted from a larger bitwidth model (e.g. in which each model parameter is represented by 8-bits) for use during inference. An advantage is that multiple nested models associated with different bitwidths may be stored with reduced storage consumption as compared to the storage of independent models.

[0013]

[0011] The system may therefore provide a machine learning model comprising model parameters, each of which comprises a plurality of nested parameter values having different bit-widths and provided according to a training process that minimizes a training objective term given by a sum of individual training objective terms associated with each of the different bit- widths.

[0014]

[0012] A computing device may perform a method of using a machine learning model provided by the above methods. The method of using the model may comprise: selecting a bit-width for use for the model parameters; if the selected bit-width is equal to the first number of bits, using the machine learning model with each of the model parameters having the first number of bits; and if the selected bit- width is less than the first number of bits, truncating the final parameter values to the selected bit- width and using the machine learning model with each of the model parameters having the selected bit-width.wherein using the machine learning model comprises processing input values to provide a result.

[0015]

[0013] Particular embodiments of the subject matter described in this specification can be implemented so as to realize one or more of the following advantages.

[0016]

[0014] By providing a nested representation of the model parameters, the system can store a single set of final parameter values having a first number of bits (e.g., 8-bits) that concurrently represent multiple precision levels. As parameters for different bit-widths, such as INT8, INT4, and INT2, are nested within the same parameter space, the system can significantly reduce the amount of memory needed to store the model parameters, i.e. as compared to storing separate models for each precision level.

[0017]

[0015] The nested representation of the model parameters allows for extraction of a reduced bit-width model from a larger bit-width model for use during inference. This enables a single trained model to be adapted to various hardware environments with differing computational requirements, whereby deployment can be performed dynamically and flexibly, e.g., based on the real-time availability of computational resources. For instance, a bit-width can be selected in dependence upon the specific memory capacity, processing resources, or power constraints of a computer device, such as an edge device or mobile user equipment. For example, if the selected bit -width is less than the bit-width of the model parameters, the model parameters can be truncated to the selected bit-width without requiring another version of the model to be provided to the computer device.

[0018]

[0016] Beyond global precision settings, the system also supports updating the machine learning model by applying different bit-widths to different sets of layers, allowing for model performance to be traded for computational efficiency and vice versa. Specifically, the bit- width of model parameters belonging to a first set of layers can be a first number of bits, while the parameters of a second set of layers can be reduced to a second number of bits that is less than the first number of bits. This granular optimization can, for example, allow critical layers of the transformer or large language model (LLM) to retain higher precision while less sensitive layers are more aggressively quantized to conserve computational resources.

[0019]

[0017] When training the model, the system can optimize (e.g., minimize) a sum of individual training objective terms associated with multiple different bit- widths supported by the nested structure. For example, by determining, for each iteration, a training objective based on both the first output values determined based on the reduced precision parameter values and the second output values determined based on the updated (truncated) parametervalues, the resulting model can maintain high accuracy across multiple precision levels rather than being optimized for only one. The training objective can, for example, be determined from a sum of first and second objective terms as described above, e.g. a sum in which the training objective terms are weighted differently to prioritize specific precision performance, e.g. based on the computational resources available in a target deployment environment. The training process can adjust the parameter values of the model parameters themselves and / or the quantization parameters — such as clipping thresholds or scaling factors — to minimize this combined training objective.

[0020]

[0018] By training the model to jointly optimize different bit- widths of the model parameters, the accuracy of lower precision representations (e.g.. INT2) can be improved compared to optimizing these lower precision representations individually. That is, a lower precision representation obtained by truncating the model parameters of the updated version of the machine learning model can be more accurate compared to model parameters of the same precision determined by other methods that do not jointly optimize different bit-widths of the model parameters.

[0021] BRIEF DESCRIPTION OF THE DRAWINGS

[0022]

[0019] FIG. 1 shows an example model parameter from which corresponding reduced precision representations can be produced.

[0023]

[0020] FIG. 2 shows an example machine learning system for quantizing model parameters of a machine learning model.

[0024]

[0021] FIG. 3 is a flow diagram of an example process for quantizing the model parameters of a machine learning model.

[0025]

[0022] FIG. 4 shows an example system for using a machine learning model to process input values to determine a result.

[0026]

[0023] FIG. 5 is a flow diagram of an example process for using a machine learning model.

[0027]

[0024] FIG. 6 shows example results comparing performance of different quantized versions of a machine learning model.

[0028]

[0025] Like reference numbers and designations in the various drawings indicate like elements.

[0029] DETAILED DESCRIPTION

[0030]

[0026] FIG. 1 shows an example model parameter 100 having a first number of bits 108 (in this case, 8 bits) arranged as a sequence (11011001). In this example, the modelparameter is represented as an unsigned 8-bit integer (INT8), although other sizes and types of representation can be used instead. As described below, the model parameter 100 represents a final parameter value determined following a training process that jointly optimizes the model parameters 100 across different levels of precision. Such training can, for example, allow the machine learning model to maintain performance across various hardware environments with differing computational requirements.

[0031]

[0027] The model parameter 100 can be processed to produce a reduced precision parameter value by extracting a second number of most significant bits (MSBs) 102, 104. For example, a 4-bit approximation of the model parameter 100 can be produced by setting to zero all bits of the model parameter 100 following its most significant four bits to obtain, in this case, 11010000. Similarly, a 2-bit approximation of the model parameter 100 can be produced by setting to zero all the bits after the two MSBs 102 to obtain 11000000, and so on.

[0032]

[0028] In general, an / --bit approximation of a model parameter can be extracted from a c-bit model parameter (0 < r < c) by "slicing" the most significant r bits from the model parameter 100, which can, for example, be implemented computationally using aright bitshift by c — r, followed by a left bit-shift of the same order.

[0033]

[0029] Thus, a machine learning model that comprises a plurality of model parameters 100 that each comprise a first number of bits 108 can be converted to a reduced precision machine learning model in which one or more (e.g., each) of the model parameters 100 has a reduced precision defined by a second number of bits 102, 104 that is less than the first number of bits 108. Different versions of the machine learning model corresponding to different levels of quantization can therefore be produced without needing to retrain the machine learning model or provide separate storage for multiple sets of model parameters.

[0034]

[0030] FIG. 2 shows an example machine learning system 200 that is configured to quantize model parameters 202 of a machine learning model 201. The machine learning system 200 and the machine learning model 201 are examples of systems that are each implemented as computer programs on one or more computers in one or more locations, in which the systems, components, and techniques described in this specification can be implemented. For example, the machine learning system 200 may be implemented on one or more computers comprising hardware accelerators, such as GPUs or TPUs, that are optimized for machine learning computations.

[0035]

[0031] The model parameters 202 can be represented as floating-point parameter values. The machine learning system 200 comprises a quantization function 204 that is used toconvert each of the model parameters 202 to corresponding reduced precision parameter values 206 that each have a first number of bits. For example, the floating-point parameter values can be converted to 8-bit integers.

[0036]

[0032] Any suitable quantization function 204 can be used. For example, the quantization function 204 can determine a MinMax quantization Q of a model parameter w in c bits according to the following expression, in which clamp(-) restricts an input value to a predetermined range from 0 to 2C— 1, and [■] is a rounding operation that rounds its argument to the nearest integer:

[0037]

[0038]

[0033] In this expression, the scaling factor a, and the zero point z, are determined from the maximum and minimum of the model parameters being quantized:

[0039] max(w) — min(w) min(w)

[0040] a = - - - - - , z = - 2C- 1 a

[0041]

[0034] In some implementations, Quantization Aware Training (QAT) can be used, e.g., as described in Jacob et al., “Quantization and training of neural networks for efficient integer-arithmetic-only inference"' in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 2704-2713, 2018. In such cases, the model parameters 202 are optimized during the training, with the quantization function 204 only varying according to the maximum and the minimum of the model parameters 202.

[0042]

[0035] In some implementations, the quantization function 204 can comprise learnable quantization parameters. As one example, the quantization function 204 can be parameterized according to the Omni-Quant method described in Shao et al., “Omni-quant: Omnidirectionally calibrated quantization for large language models"", arXiv:2308.13137. For example, the scaling factor and the zero point can be parameterized using two learnable scales, y and / ?:

[0043] y • max(w) — p • min(iv) 0 • min(w)

[0044] a = - - - - >z= - 2C— 1 a

[0045]

[0036] Values of y and f can be determined by optimizing the training objective (e g., loss function) described below with respect to these parameters.

[0046]

[0037] The system 200 is configured to produce respective updated reduced precision parameter values 208 by extracting a second number of most significant bits (MSBs) belonging to each of the reduced precision parameter values 206. For example, each respective updated reduced precision parameter value 208 can be produced by setting to zero all bits of the respective reduced precision parameter value following the secondnumber of MSBs of the respective reduced precision parameter value, e.g., as described above in connection with FIG. 1.

[0047]

[0038] The machine learning system 200 is further configured to process training data 210 comprising input data 212 (e.g., respective input data 212 for one or more training iterations) to optimize the model parameters 202 and / or quantization parameters associated with the quantization function. For example, the training data 210 can comprise labels 214 defining target outputs of the machine learning model 201 for respective parts (e.g., training samples) of the input data 212. For example, the training data 210 can comprise a plurality of training samples, with each training sample comprising a respective part of the input data 212 and a corresponding label 214. The machine learning system 200 can then be configured to perform one or more training iterations in which a respective one or more of the training samples is used to train the machine learning model 201.

[0048]

[0039] The machine learning system 200 is configured to supply the input data 212 (e.g. part of the input data 212) to the machine learning model 201 to calculate first output values 216 based on the set of reduced precision parameter values 206 and to calculate second output values 218 based on the updated reduced precision parameter values 208. That is, the machine learning model 201 is configured to calculate the first output values 216 by processing the input data 212 (e.g., training samples) in accordance with the set of reduced precision parameter values 206, i.e., performing a forward pass of the model 201 on the part of the input data 212 using the set of reduced precision parameter values 206 in place of the floating-point parameter values defining the model parameters 202 of the machine learning model 201. Similarly, the machine learning model 201 is configured to calculate the second output values 218 by processing the part of the input data 212 in accordance with the updated reduced precision parameter values 208.

[0049]

[0040] The system 200 further comprises a training objective 220 that, in the present example, is configured to process the first output values 216 to determine a first training objective term 222 and the second output values 218 to determine a second training objective term 224. For example, the training objective 222 can be configured to determine the first training objective term 222 by determining discrepancies between the first output values 216 determined for the part of the input data 212 and the corresponding labels 214. Similarly, the second training objective term 224 can be determined based on discrepancies between the second output values 218 determined for the part of the input data 212 and the corresponding labels 214.

[0041] The system also comprises an optimizer 228 that is configured to generate adjusted parameters 226, which may include the model parameters 202 of the machine learning model 201 and / or quantization parameters associated with the quantization function 204, in order to optimize (e.g., minimize) a sum of training objective terms including the first training objective term 222 and the second training objective term 224.

[0050]

[0042] In general, generating the adjusted parameter values 226 can comprise backpropagating gradients of the sum of training objective terms with respect to the learnable model parameters 202 and / or quantization parameters to update the learnable parameters. The optimizer 228 can use any appropriate gradient descent optimization algorithm, e.g. Adam (see e g., Kingma et al., arXiv: 1412.6980) or another optimization algorithm. Typically, the gradients of the sum of training objective terms are determined for a batch of training samples and accumulated or averaged over the batch before performing an update step to generate the adjusted parameter values 226.

[0051]

[0043] As one example, the optimization of the sum of training objective terms can be defined according to the following expression:

[0052]

[0053]

[0044] In this expression, P denotes the learnable parameters, N is the total number of training samples, i indexes the training samples. R is the set of bit- widths being jointly optimized (e.g., R = (8, 4, 2}), A(. is a loss reweighting factor for bit-width r, £(•) is a loss function, F(-) is the output of the forw ard pass of the machine learning model with model parameters determined using a slicing function S(-, r) that extracts the r most significant bits from the quantization function Q(0, c) that is applied to model parameters 9 at bit- width c (e.g., c = 8), and x and y' are the part of the input data and corresponding label (ground truth), respectively, for the training sample.

[0054]

[0045] Any suitable training objective (loss function) can be used. For example, the training objective can comprise a cross-entropy function, a mean squared error, and so on.

[0055]

[0046] In some implementations, the loss reweighting factors A,. can be determined using a search, e.g., a grid search.

[0056]

[0047] The system 200 is configured to provide an updated version of the machine learning model 201 comprising model parameters represented by final parameter values 230 determined based on the adjusted parameter values 226 following one or more iterations inwhich different respective parts of the input data 212 (e.g., different training samples) are used to train the machine learning model 201 and / or quantization function 204.

[0057]

[0048] FIG. 3 is a flow diagram of an example process 300 for quantizing the model parameters of a machine learning model. For convenience, the process 300 will be described as being performed by a system of one or more computers located in one or more locations. For example, a training system, e.g., the training system 200 depicted in FIG. 2, appropriately programmed in accordance with this specification, can perform the process 300.

[0058]

[0049] The system obtains (step 302) a machine learning model comprising model parameters represented by floating-point parameter values.

[0059]

[0050] The system then performs a training process comprising one or more iterations.

[0060]

[0051] The system obtains (step 304) training data comprising input data for the iteration.

[0061]

[0052] For each iteration, the system applies (step 306) a quantization function to convert the floating-point parameter values to a set of corresponding reduced precision parameter values each having a first number of bits.

[0062]

[0053] The system, for each iteration, supplies (step 308) part of the input data to the machine learning model to calculate first output values for the iteration based on the set of reduced precision parameter values, and based on the first output values, determines a first training objective term.

[0063]

[0054] For each iteration, the system, for each of the set of reduced precision parameter values, produces (step 310) a respective updated number by extracting a second number of most significant bits (MSBs) belonging to the respective one of the reduced precision parameter values. The second number is less than the first number.

[0064]

[0055] The system, for each iteration, supplies (step 312) the part of the input data to the machine learning model to calculate second output values for the iteration based on the updated reduced precision parameter values, and based on the second output values, determines a second training objective term.

[0065]

[0056] For each iteration, the system then updates parameters so as to optimize (e.g. minimize) a sum of training objective terms including the first training objective term for the iteration and the second training objective term of the iteration, the adjusted parameter values of parameters that include (i) the model parameters, (ii) quantization parameters associated with the quantization function, or (iii) both.

[0057] The system provides (step 316) an updated version of the machine learning model comprising model parameters represented by final parameter values determined based on the adjusted parameter values following the one or more iterations. The final parameter values each have the first number of bits.

[0066]

[0058] FIG. 4 shows a system 400 for using a machine learning model 401 that uses quantized model parameters 402 to process input values 406 to provide a result 408, e.g., for performing inference. The system 400 is an example of a system implemented as a computer program on one or more computers in one or more locations, in which the systems, components, and techniques described in this specification can be implemented.

[0067]

[0059] The system 400 comprises a controller 410 and a computer device 412 that implements the machine learning model 401 using model parameters provided by the controller 410.

[0068]

[0060] The controller 410 comprises a set of model parameters 414 that each has a first number of bits and comprise a plurality of nested parameter values having different bitwidths. The model parameters 414 are provided according to a training process that optimizes (e.g., minimizes) a training objective term given by a sum of individual training objective terms associated with each of the different bit-widths. For example, the model parameters 414 can have been obtained using the machine learning system 200 described above in connection with FIG. 2, or using the process 300 described above in connection with FIG. 3, e.g., the model parameters 414 can be the final parameter values determined by the process 300.

[0069]

[0061] The controller further comprises a bit-width selector function 416 that is configured to process a device specification 418 associated with the computer device 412 to select a bit-width 420 for the model parameters 401 of the machine learning model 401.

[0070]

[0062] The device specification 418 can, for example, be indicative of computational resources of the computer device 412 on which the machine learning model 401 is to be implemented, such as one or more of: memory capacity, power constraints, number of processing units, memory transfer rates, bandwidth, and so on. As one example, the device specification 418 can identify the type of computer device, such as edge device or mobile user equipment.

[0071]

[0063] The controller 410 also includes a model adapter 422 that is configured to process the selected bit- width 420 and at least some of the set of model parameters 414 to generate truncated parameters 424 that the controller 410 then provides to the computer device 412.

[0064] The model adapter 422 is configured to determine the truncated parameters 424 by, if the input (selected) bit-width 420 is less than the first number of bits, truncating the at least some of the model parameters to the selected bit-width. However, if the input (selected) bit-width 420 equals the first number of bits then the model adapter 422 can provide the model parameters for use by the machine learning model 401 without truncating them.

[0072]

[0065] The system 400 is therefore able to adapt the machine learning model 401 according to the computer device that processes the input values 406 to determine the result 408.

[0073]

[0066] Although FIG. 4 shows only a single computer device 412, more than one computer device may implement the machine learning model 401 and the controller 410 can provide the same or different model parameters to each computer device, e.g.. depending on respective device specifications for each computer device. In some implementations, the controller can be implemented on the one or more computer device, e.g., each computer device can adapt the model parameters that it uses to implement the machine learning model 401.

[0074]

[0067] FIG. 5 is a flow diagram of an example process 500 for using a machine learning model comprising model parameters. For convenience, the process 500 will be described as being performed by a system of one or more computers located in one or more locations. For example, a system, e.g., the system 400 depicted in FIG. 4, appropriately programmed in accordance with this specification, can perform the process 500.

[0075]

[0068] The machine learning model comprises model parameters that each have a first number of bits and comprise a plurality of nested parameter values having different bitwidths. The model parameters can have been provided according to a training process that minimizes a training objective term given by a sum of individual training objective terms associated with each of the different bit-widths. For example, the model parameters 414 can have been obtained using the machine learning system 200 described above in connection with FIG. 2, or using the process 300 described above in connection with FIG.

[0076] 3, e.g., the model parameters 414 can be the final parameter values determined by the process 300.

[0077]

[0069] The system selects (step 502) a bit-width from one of the different bit-widths for use for at least some of the model parameters.

[0078]

[0070] The system can determine (step 504) if the selected bit-width is equal to the first number of bits.

[0071] If the selected bit-width is equal to the first number of bits, the system uses (step 506) the machine learning model with each of at least some of the model parameters having the first number of bits.

[0079]

[0072] If the selected bit-width is less than the first number of bits, the system can truncate the at least some of the model parameters to the selected bit-width and use the machine learning model with each of the at least some of the model parameters having the selected bit-width, wherein using the machine learning model comprises processing input values to provide a result.

[0080]

[0073] FIG. 6 is a line and scatter plot illustrating the relationship between model accuracy and memory footprint for quantized versions of a Gemma-2 9B model obtained using the process 300 of FIG. 3 with Omni-Quant as the base algorithm, which is referred to in the figure as “MatQuanf ’. The vertical axis (y-axis) represents “Task Average”, which is the average accuracy across a collection of downstream evaluation tasks, while the horizontal axis (x-axis) represents “Effective bits per FFN parameter”, indicating the memory and communication cost associated with the feed-forward network (FFN) blocks of the machine learning model.

[0081]

[0074] The results show the performance for: (i) bit-widths 2, 4 and 8 (INT2, INT4, and INT8 representations) that are explicitly optimized jointly during training (MatQuant: triangles); (ii) interpolated bit-widths, INT3 and INT6, which are obtained by slicing the INT8 model without any additional training (MatQuant-Interp: circles); (iii) models trained independently for a fixed target precision (Baseline: inverted triangles); (iv) models optimized using MinMax quantization (MinMax: squares); and (v) models obtained by slicing an INT8 quantized version of the model parameters (Sliced int8: diamonds).

[0082]

[0075] The results demonstrate that a single model can densely span the accuracy-vs-memory trade-off curve, allowing for the extraction of a performant model tailored to specific hardware or deployment constraints without retraining.

[0083]

[0076] In some implementations, the machine learning model can comprise a neural network. The neural network can. for example, comprise a sequence-to-sequence neural network, e.g. a large language model (LLM) or a vision-language model (VLM) that generates an output sequence in response to an input sequence that defines a “prompt” for the system.

[0084]

[0077] In general, the neural network can have any appropriate architecture for generating an output sequence by processing an input sequence.

[0078] As one example, the neural network, e.g. the sequence-to-sequence neural network, may comprise an auto-regressive generative model (e.g., a Transformer, a recurrent neural network, etc.) that can auto-regressively generate an output sequence as the output sequence based on the input. The neural network can, for example, comprise a large language model (LLM) that can generate tokenized representations of text data; a visionlanguage model (VLM) that can generate tokenized representations of image or video data, e.g. in response to a text input or that can generate tokenized representations of text, e.g. in response to an image input; an audio model that can input or generate tokenized representations of audio data; or a multimodal model that can generate tokens representing any of text, image or audio, e.g. in response to an input comprising any of text, image or audio; and so on.

[0085]

[0079] As a particular example the neural network can comprise a stack of transformer blocks. A transformer block typically includes an attention or self-attention neural network layer, which may be followed by a feedforward neural network layer. In general, a selfattention neural network layer applies a self-attention operation, e.g. a query-key-value (QKV) self-attention operation, to elements of an embedding, to update each element of the embedding.

[0086]

[0080] In some implementations, the input sequence and the output sequence, the training input sequence and the one or more corresponding training output sequences, each comprise a sequence of tokens.

[0087]

[0081] A “token’’ as used in this specification is a vector of numerical values having a specified dimensionality, i.e. the number of numerical values is constant across different tokens. Each token can comprise a respective predetermined or learned embedding (an ordered collection of numerical values having a pre-determined dimensionality).

[0088]

[0082] In some implementations the tokens can represent text, e.g., words, wordpieces or characters, in a natural or computer language. For example, text may be received, e.g., as a series of encoded characters, e.g. UTF-8 encoded characters; such “characters” can include Chinese and other similar characters, as well as logograms, syllabograms and the like. A text encoder, i.e. a tokenizer, can process a sequence of text to represent the text as a series of text tokens from a vocabulary of text tokens, e.g. that each represent words, wordpieces or characters in a natural or computer language. The computer language may be any formal language used to communicate with a computer, e.g. a markup language, or a command or configuration language, or a data exchange language such as JSON, or a programming language. The tokenizer can, e.g., implement BPE (Byte Pair Encoding) orWordpiece tokenization. Optionally the text can be obtained from audio data representing speech; the output tokens may be converted into audio data that represent speech corresponding to the text.

[0089]

[0083] Also or instead the tokens may represent an image. For example, a set (sequence) of input or output tokens can represent an image. Each image token may comprise a block encoding of values of the pixels in a different region of an image that maps a set of values of the pixels to a respective image token. The block encoder may comprise a neural network, e.g. having one or more (self-)attention layers, such as a Transformer neural network.

[0090]

[0084] As used herein an image may be any still or moving image, i.e. the image may be part of a video, in 2D or 3D, and may be a monochrome, color or hyperspectral image, i.e. comprising monochrome or color pixels. As defined herein an ‘’image” includes a point cloud e.g. from a LIDAR system, and a ‘'pixel” includes a point of the point cloud. An image may have been captured by a camera or other image sensor from the real world; and objects in the image may comprise physical objects, represented by the image.

[0091]

[0085] Also or instead the tokens may represent an audio waveform. For example, a set (sequence) of input or output tokens can represent audio data representing a waveform e.g. instantaneous audio amplitude values or time-frequency audio data. Each image token may comprise a block encoding of the audio waveform in a different time segment of the audio that maps a set of values representing the audio waveform to a respective image token. The block encoder may comprise a neural network, e g. having one or more (self-)attention layers, such as a Transformer neural network.

[0092]

[0086] In a multimodal system audio data or an image may be flagged by a start-of-audio token or start-of-image token.

[0093]

[0087] In some implementations, the tokens represent text, pixels of an image, or an audio waveform and the neural network is configured to generate the output sequence of tokens to perform a task represented by the input sequence of tokens.

[0094]

[0088] In some implementations the task comprises an image or audio generation task. The input sequence of tokens can then characterize the image or audio to be generated, and the output sequence of tokens can comprise tokens defining an image or audio waveform characterized by the input sequence of tokens, e.g. text tokens.

[0095]

[0089] In some implementations the task comprises an image or audio processing task. The input sequence of tokens can define an image or audio input, and the output sequence of tokens can comprise tokens defining text that describes the image or audio input. Assome examples, the task can be a speech recognition task, an obj ect or action detection task, a classification task, a captioning task, a question-answering task, or a character or word recognition task.

[0096]

[0090] In some implementations the task comprises a multimodal processing task. One or both of the input sequence of tokens and the output sequence of tokens can comprise multimodal data. For example the input sequence of tokens can characterize both an image or audio input and a text input and the output sequence of tokens can comprise tokens defining a result of an image or audio processing task defined by the text, such as an open vocabulary classification or object detection task.

[0097]

[0091] In general, multi-modal data is a combination of two or more different types of data, e.g., two or more of audio data, image data, text data, or graph data. As one example the multi-modal data may comprise audio-visual data, comprising a combination of pixels of an image or of video and audio data representing values of a digitized audio waveform. As another example the multi-modal data may comprise a combination of i) text data representing text in a natural language and ii) pixels of an image or of video or audio data representing values of an audio waveform.

[0098]

[0092] Some examples of multi-modal tasks include: open-vocabulary image classification (the output can classify the image input based on a text input comprising text descriptions of one or more classes in the image); open-vocabulary object detection (the output can detect one or more objects in the image input based on a text input comprising text descriptions of the one or more objects); image captioning (the output can comprise text that describes the image input); text-based image search (the output can identify from amongst multiple images in the image input one or more images that meet a text description of images to be retrieved, the text description being provided in a text input); image-based retrieval (the output can identify from amongst multiple images in the image input one or more images that match a further image in the image input), and so on. The multi-modal processing task to be performed can be defined by text in the input sequence.

[0099]

[0093] In some implementations the task comprises an agent control task in which the agent interacts with an environment to perform the task. The agent can be a mechanical agent such a robot or (semi-)autonomous vehicle, interacting with a real-world environment to perform the task. The neural network can be trained to control a simulated version of the agent in a simulated version of the environment and then afterwards used to control the real agent in the real-world environment. The input sequence of tokens can comprise tokens that represent an observation of the environment, e.g. an image captured by a camera orother imaging device from a real-world environment. The output sequence of tokens comprises tokens that define one or more actions to be performed by the agent in the environment in response to the observation.

[0100]

[0094] There are many suitable training datasets available, depending on the task to be performed. Just as some examples these include, for text: WebLI (Web Language Image, Chen et al. arXiv:2305.18565vl). Some examples for images include: the Visual Genome dataset for Visual Question Answering (Krishna et al., arXiv: 1602.07332); Objects365 (Shao et al., “Objects365: A large-scale, high-quality dataset for object detection’’, IEEE / CVF international conference on computer vision, pages 8430-8439); Open Images V4 (Kuznetsova et al., arXiv: 1811.00982); the SBU dataset (Ordonez et al. “Im2Text: Describing Images Using 1 Million Captioned Photographs”. NeurlPS 2011); the Conceptual Captions datasets, e.g. VI (2M images) or V2 (10M images) (Sharma et al., “Conceptual Captions: A Cleaned, Hypemymed, Image Alt-text Dataset For Automatic Image Captioning”, ACL 2018); and Kinetics for video (Kay et al., arXiv: 1705.06950). An example for audio data is AudioSet (Gemmeke et al., “Audio set: An ontology’ and human-labeled dataset for audio events,” ICASSP, IEEE, 2017, pp. 776-780). An example training dataset for agent (robot) control is described in Ebert et al., arXiv: 2109.13396.

[0101]

[0095] In general, a training dataset for a particular task may comprise task-specific training examples that have been manually generated by a human being and / or task-specific training data may be generated automatically using existing tools. For example an OCR (Optical Character Recognition) task dataset may be generated by applying an OCR tool to a corpus of images; or an object detection task that requires generating object bounding-box coordinates may be generated by applying an existing object detection tool, such as a trained neural network, to a corpus of images; or a set of aligned image and text representations may be generated using ALIGN (Jia et al., arXiv:2102.05918); or instruction-annotated robot trajectories may be obtained as described in Brohan et al., arXiv:2212.06817, in either the real-world or in simulation.

[0102]

[0096] In some implementations the neural network, e.g. a language model or a visual language model, is stored on a user computing device, i.e. a device local to the user, such as a mobile device e.g. a mobile phone, or a smart speaker.

[0103]

[0097] In some implementations the neural network is implemented on a remote server in communication with a user computing device over a wired or wireless network communications link between the user computing device and the server.

[0098] The user computing device may be provided with an input mechanism, such as a text or voice interface, that enables user input from the user in a natural language. The user computing device may be provided with an output mechanism that provides a system output for the user in the or another natural language e.g. as speech or text; or in some other way, e.g. by displaying an image. The input and output mechanism may comprise, e.g., a keyboard, microphone, speaker, display, and / or camera.

[0104]

[0099] As an example the input mechanism may comprise a system configured to input audio data characterizing a speech waveform of speech representing the input from the user in a natural language, and configured to convert the audio data into tokens representing the speech in the natural language, e.g. representing a transcription of the spoken input. The output mechanism may comprise a system configured to receive tokens representing the output for the user in the or another natural language and a system configured to convert the received tokens into audio data representing a waveform of speech representing the output to the user in the natural language, i.e. representing spoken words.

[0105]

[0100] As a further example, the trained system can be deployed in an environment that enables a user to provide a request for the system, e.g. to process a multimodal input to generate a corresponding output sequence output. A user can provide the request, e.g., by way of a user interface or through an application programming interface (API). The request can be transmitted from a user device, e.g., over a data communications network such as the internet, to one or more computers implementing the system, e.g., in a data center. The system can generate an output sequence and then transmit the output sequence to a user device over a data communications network.

[0106]

[0101] A user computing device may be provided as an interface for the neural network, with an input mechanism that enables user input from the user in a natural language and an output mechanism that provides a system output to the user in the natural language. The input and output mechanism may comprise, e.g., a keyboard and display. Also or instead the input and output mechanism may comprise a speech-based mechanism. For example the input mechanism may comprise a system configured to input audio data characterizing a speech waveform of speech representing the input from the user in the natural language and configured to convert the audio data into tokens representing the speech in the natural language, e.g. representing a transcription of the spoken input. The output mechanism may comprise a system configured to receive tokens representing the output to the user in the natural language and a system configured to convert the received tokens into audio datarepresenting a waveform of speech representing the output to the user in the natural language, i.e. representing spoken words.

[0107]

[0102] In some implementations the input sequence comprises one or more natural language statements relating to an environment, in particular areal-world environment, and includes a natural language request relating to the environment. Similarly, the output sequence may be a natural language reply or natural language output statement that also relates to the environment i.e. it provides information relating to the environment, in some implementations relating to or specifying actions to be taken in the environment.

[0108]

[0103] The (trained) neural network can be used for diagnosing a fault, or for correcting undesired behavior, in a mechanical or computing system operating in the real-world environment. The input may comprise a description and / or image of one or more observations of the mechanical or computing system, e.g. of operation of the system, optionally obtained from one or more sensors sensing a condition or operation of the system. An image observation may be converted into a text description e.g. using an image captioning system or in other ways. The generated output sequence may comprise an image, audio, or text that identifies (describes) a likely cause of the fault or undesired behavior. This may be used to repair the fault or correct the behavior. The reward models can define relatively more useful ty pes of output for repairing the fault or correcting the behavior, and other aspects of the response as previously described.

[0109]

[0104] The (trained) neural network can be used for controlling a mechanical agent such as a robot or vehicle. For example, the input may comprise a description of a task to be performed, and the generated output sequence may comprise a list of sub-tasks to be performed by the mechanical agent (trained to perform such sub-tasks), in order to perform the task. The reward models can define relatively more preferable or useful types of subtask, task safety, efficiency, and so on.

[0110]

[0105] As another example, the environment can be a computer security monitoring environment, e.g., the system can be deployed as part of a system that monitors the security of one or more computers. For example, the environment may be a computer network security monitoring environment, and the system can be deployed as part of a system that monitors the security of one or more computers on a computer network, e.g. a wireless network, a cellular network, a local area network and / or the internet. As another example, the environment may alternatively or additionally be a computer system security monitoring environment and the system can be deployed as part of a system that monitors the system for the presence of computer viruses and / or an unresolved softwarevulnerability, e.g. a zero-day exploit. A software vulnerability may be resolved by updating the software (e.g. patching) and / or removing (e.g. uninstalling) the software from the computer system. In these examples, the natural language request can query whether a computer security incident has been resolved (e.g., “has the incident been resolved?”) and the input sequence may comprise relevant statements from system logs, i.e., that are potentially relevant to the event being queried. A computer security incident can be, e.g., a data breach, an unauthorized log-in or other access of a secured system, a detection of a computer virus or detection of a software vulnerability. The incident can be “resolved” when the underlying incident is no longer a threat to the security of the computer system e.g., the computer virus has been removed, the access to the secured system has been removed, the data breach has been mitigated, or the software having the vulnerability has been updated or removed. The system can use the input sequence to generate a reply to the request that comprises a natural language statement indicating whether the incident has been resolved, optionally displaying evidence used to determine this.

[0111]

[0106] The input sequence may include one or more of: code snippets from the software code, system logs, program logs, or other artifacts that should be left on the computer by running the program, or verification rules that represent requirements for the execution of the software program, or natural language statements describing the computer system on which the software executes. In general, the input sequence may include relevant statements, i.e., statements that are potentially relevant to the event being queried.

[0112]

[0107] In some implementations obtaining the input sequence may comprise obtaining, from the system logs, the data characterizing the computer network, or both, or from other data as described above, one or more observations of the computer network (which here includes computers on the network), and processing the one or more observations to generate a natural language representation of the one or more observations. The natural language request may relate to the computer security incident or to the secure operation of the computer network. The method may include using the natural language representation of the one or more observations to provide one or more of the natural language statements describing the computer network and using the natural language reply or the natural language output statement to identity' a security' status of the computer network or a security flaw in the computer network.

[0113]

[0108] As another example, the environment can be a software testing or evaluation environment, e.g., the system can be deployed as part of a system that tests software before deployment or that evaluates already-deployed software to identify bugs. In theseexamples, when the system tests software before deployment, the natural language request can ask whether the software will execute as intended, and the input sequence can include code snippets from the software code and, optionally, natural language statements describing the computer system on which the software will execute. The system can then use the input sequence to generate a reply that indicates whether the code will execute as intended, optionally displaying evidence used to determine this. When the system monitors the execution of code after deployment, the natural language request can ask whether a software program, or a portion of a software program, has executed as intended, and the input sequence can include one or more of: code snippets from the software code, system logs, program logs, or other artifacts that should be left on the computer by running the program, or verification rules that represent requirements for the execution of the software program, or natural language statements describing the computer system on which the software executes. The system can then use the input sequence to generate a reply that indicates whether the code has executed as intended, optionally displaying evidence used to determine this. As a particular example, the software program can be part of the boot up of a computer, and the system can generate a reply each time that the computer starts up to verify whether the computer will function correctly after start up.

[0114]

[0109] As another example, the environment can be an educational environment, e.g., the system can be deployed as part of an education software program that assists a user in learning or practicing one or more corresponding skills. In these examples, the input sequence can include natural language statements describing or referencing a scenario or scene in a real-world or imagined environment, and the request can be a question about the scenario or scene.

[0115] [HO] As another example, the environment can be an information retrieval environment, e.g., the system can be deployed as part of a search engine or other software that allows a user to search for information in a corpus of documents, e.g., the Internet or another electronic document corpus. In these examples, the request can be any appropriate natural language question, and the reply can optionally include evidence such as relevant statements from the corpus of documents, e.g. as identified by searching the corpus using conventional information retrieval techniques.

[0116] [Hl] In some implementations, the language model neural network is a visual language model (VLM). In general, the VLM may process input sequences comprising tokens that each represent natural language or (a part of) an image or video to generate output tokens that each represent natural language or (a part of) an image or video. For example, the VLMmay be configured to describe an image or video using natural language, e.g., to perform an image or video captioning task. As another example, the VLM may be configured to process input tokens representing an image and text tokens representing a query about the image or a request to modify the image, and to generate output tokens representing an answer to the query or representing a version of the image that has been modified in accordance with the request. The VLM may generate output tokens representing an image or video that is generated in response to input tokens providing a visual and / or audio and / or textual description of a desired image or video.

[0117]

[0112] In some implementations, the “language” of the language model is not a natural language (e.g. English), but may instead be a text-based encoding describing an entity or class of entities, e.g. a chemical or biological entity, such as a chemical structure or molecule. For example, the text-based encoding may be a sequence of tokens that defines a molecule or protein, e.g. a sequence specifying an arrangement of atoms or chemical functional groups in a molecule, or the amino acid residues of a protein. The language model may be referred to as a chemical and / or biological language model in such cases. The input for the language generation neural network may therefore be an input string defining a chemical (e.g. protein) structure and the output may be an output string defining a different chemical structure from the input string. The strings may be in the Simplified Molecular Input Line Entry- System, SMILES, format, for example.

[0118]

[0113] In another example of a computer language text generation task, a task-specific training example may comprise an image or video and a sequence of text in a computer language for performing a task in relation to the image or video, e.g. a data processing task that involves analyzing the content of the image or video to provide a result of the analysis or, e.g.. a search to search for information relating to the content of the image or video. The computer language in the model output may comprise computer language for invoking a function or calling one or more external APIs. Merely as one example, such an output may be formatted as a JSON obj ect. As previously, the sequence of text in the multimodal input may define the task to be performed and the second modality input may comprise, e.g. an image or video in relation to which the task is to be performed, e.g. a task that involves manipulation of particular types of data that may benefit from access to an API such as mathematical data, date / time related data, scientific data, recent data that may post-date training of the model (that may be accessed by a search function or API), and so on. After training, when the model is used in inference, the model output may comprise text in the or another computer language for performing a task, e.g. as described above, in relation toan image or video in the second modality input. The method may then include using the text in the computer language to perform the task.

[0119]

[0114] In some implementations, the language model neural network may be used to interact with a human user of a digital assistant such as a smart speaker, smart display, or other device. For example, information defining a task can be obtained from the digital assistant, and the digital assistant can be used to instruct the user to perform the task. For example, this may comprise receiving, at the digital assistant, a request from the user for assistance and determining, in response to the request, a series of tasks for the user to perform, e.g. steps or sub-tasks of an overall task. Then for one or more tasks of the series of tasks, e.g. for each task, e.g. until a final task of the series the digital assistant can be used to output to the user an indication of the task, e.g. step or sub-task, to be performed. This may be done using natural language, e.g. on a display and / or using a speech synthesis subsystem of the digital assistant. Visual, e.g. video, and / or audio observations of the user performing the task may be captured, e.g. using the digital assistant. A system may then be used to determine whether the user has successfully achieved the task e.g. step or sub-task, i.e. from the answer as previously described. If there are further tasks to be completed the digital assistant may then, in response, progress to the next task (if any) of the series of tasks, e.g. by outputting an indication of the next task to be performed. In this way the user may be led step-by-step through a series of tasks to perform an overall task.

[0120]

[0115] As an illustrative example, a user may be interacting with a digital assistant and ask for help performing an overall task consisting of multiple steps, e.g. cooking a pasta dish. While the user performs the task, the digital assistant receives audio and / or video inputs representative of the user's progress on the task, e.g. images or video or sound clips of the user cooking. The digital assistant uses a system as described above, in particular by providing it with the captured audio and / or video and a question that asks whether the user has completed a particular step, e.g. 'Has the user finished chopping the peppers?', to determine whether the user has successfully completed the step. If the answer confirms that the use has successfully completed the step then the digital assistant progresses to telling the user to perform the next step or, if at the end of the task, or if the overall task is a single-step task, then the digital assistant may indicate this to the user. The digital assistant may then stop receiving or processing audio and / or video inputs to ensure privacy and / or reduce power use.

[0116] In a further aspect there is provided a digital assistant device including a system as described above. The digital assistant can also include a user interface to enable a user to request assistance and to output information. In implementations this is a natural language user interface and may comprise a keyboard, voice input-output subsystem, and / or a display. The digital assistant can further include an assistance subsystem configured to determine, in response to the request, a series of tasks for the user to perform. In implementations this may comprise a generative (large) language model, in particular for dialog. The digital assistant can have an observation capture subsystem to capture visual and / or audio observations of the user performing a task; and an interface for the abovedescribed language model neural network (which may be implemented locally or remotely). The digital assistant can also have an assistance control subsystem configured to assist the user. The assistance control subsystem can be configured to perform the steps described above, for one or more tasks e.g. of a series of tasks, e.g. until a final task of the series. More particularly, the assistance control subsystem can output to the user an indication of the task to be performed, capture, using the observation capture subsystem, visual or audio observations of the user performing the task, determine from the abovedescribed answer whether the user has successfully achieved the task. In response, the digital assistant can progress to a next task of the series of tasks and / or control the digital assistant, e.g. to stop capturing observations.

[0121]

[0117] The neural network may comprise a multimodal machine learning system such as a visual language model (VLM). That is implementations of the neural network can perform a multimodal task in which the input and output sequence, collectively, comprise data of multiple different types. As used herein text can include numbers, punctuation, special symbols, and so on.

[0122]

[0118] In some implementations, after training, a particular task that is to be performed by the neural network can be described by part or all of a sequence of text in the input to the system. For example in an input that includes an image such a prompt might specify “Generate a caption”, “Generate a description”, “Answer the following question: [about the image or video]”, or “Detect a person”. Where the system is used for an agent control task a prompt may define “Take the knife out of the drawer”, or “Q: What action should the robot take to take the knife out of the drawer?”. Also or instead such a prompt may give one or more examples of a task to be performed. The neural network can be trained on multiple natural and / or computer languages and the prompt may then specify a language to use.

[0119] A few further examples of some machine learning tasks that can be performed by a system trained as described herein follow. The tasks described below may be tasks that require spatial awareness or other context from the image or video. For example, a prompt may ask “What is the object in the top left comer?”.

[0123]

[0120] In general for the tasks below the system can have been trained or fine-tuned on examples of the input and output for the task. For example the system can have been trained using still or moving images containing one or more objects or actions, and corresponding sequences of text or other data e.g. describing or classifying the images. However large, “foundation” models can, in general, perform some tasks zero-shot, i.e. without having been specifically trained on those tasks.

[0124]

[0121] As one example the task may comprise an object or action detection task. For example the generated output sequence may comprise or represent text that describes or otherwise labels detected object(s) or action(s) in an input comprising an image or audio, and may include coordinates such as bounding-box coordinates for the detected object(s) or action(s), e.g. "102090 100 cat 2030 100 100 dog”.

[0125]

[0122] As another example the task may compnse a classification task, e.g. an object or action classification task. The generated output sequence may comprise data, e.g. text, that classifies the object(s) or action(s) in represented in the conditioning data, e.g. in an image or audio, into one of a plurality of classes, or that otherwise classify object(s) or action(s) represented in the conditioning data.

[0126]

[0123] As another example the task may comprise a still or moving image describing task, e.g. a captioning task (which, as used here, includes an audio description task to explain what is happening in an image). The generated output sequence may comprise data, e.g. text, describing an image or video in the conditioning data. For example the generated output sequence may provide a caption or description or it may count objects in the image or video, or it may provide some other form of description.

[0127]

[0124] As another example the task may comprise a still or moving image questionanswering task. The generated output sequence may comprise data, e.g. text, that answers a question about the input, e.g. an image or audio, where the question is also specified in the input, e.g. as sequence of text. This may be used, e.g., to answer questions about visual plots and charts or about sounds.

[0128]

[0125] As another example the task may comprise a character or word recognition task, e.g. an OCR (optical character recognition) task. The input may comprise a still or movingimage and the generated output sequence may comprise text that represents characters or words in the input, e.g. in a natural language.

[0129]

[0126] As another example the task may comprise a still or moving image generation task. The generated output sequence may comprise image data defining values for pixels of a still or moving image, and the input, e.g. a sequence of text, may describe or characterize the image to be generated. Merely as an example, an image of a plot or chart may be generated to represent the input, e.g. comprising text.

[0130]

[0127] As another example the task may comprise a computer language text generation task. The conditioning data may comprise a natural language description of a task to be performed, and optionally an image (if the task is to be performed on or in relation to an image), and the generated output sequence may comprise text in a computer language to perform the task, e.g. a task of analyzing the content of the image to provide a result of the analysis or to search for information relating to the content of the image.

[0131]

[0128] As a particular example the computer language in the generated output sequence may comprise computer language for invoking a function or calling one or more external APIs. Merely as one example, such an output sequence may comprise data formatted as a JSON object. As previously, the input may define the task to be performed and may also include an image in relation to which the task is to be performed. In general the task can involve manipulation of particular types of data that may benefit from access to an API such as mathematical data, date / time related data, scientific data, recent data that may postdate training of the system (that may be accessed by a search function or API), and so on; and the generated output sequence may comprise text in a computer language for performing the task. The method may then include using the text in the computer language to perform the task.

[0132]

[0129] In general where the generated output sequence comprises text this may be converted to speech representing the text, and an audio (speech) output provided.

[0133]

[0130] In some implementations the task comprises an agent control task in which the agent interacts with an environment to perform the agent control task. In these implementations the input can include an observation characterizing the environment. For example the input can include a sequence of text that defines the task to be performed by the agent and the image can represent an observation of the environment, e.g. captured by a camera or other imaging device from a real-world environment. The generated output sequence can comprise an action selection output, e.g. including text, that is used to select one or more actions to be performed by the agent in the environment in response to the observation. Asan illustration the generated output sequence may define an action as text such as “A: 132 114 1285 25 156”. that can be converted into a control signal for a mechanical agent, such as a robot, e.g. “AT = [0.1, —0.2,0] A / ? = [10°, 25°, —7°]”. The action selection output may also or instead define one or more low-level skills, e.g. from a vocabulary of previously leamt skills. As before, the sequence of text in the input to the system may describe the task to be performed, e.g. ‘“What action should the robot take to [perform task]”. Examples of systems for controlling an agent that may be fine tuned as described herein can include PaLM-E (Dness etal. arXiv:2303.03378), RT-1 (Brohan et al. arXiv:2212.06817), andRT-2 (Brohan et al. arXiv:2307.15818).

[0134]

[0131] In some agent control implementations, the environment is a real-world environment and the agent is a mechanical agent interacting with the real-world environment, e.g., a robot or an autonomous or semi-autonomous land, air, or sea vehicle operating in or navigating through the environment, and the actions are actions taken by the mechanical agent in the real-world environment to perform the task. For example, the agent may be a robot or other mechanical agent interacting with the environment to accomplish a specific task, e.g., to locate or manipulate an object of interest in the environment or to move an object of interest to a specified location in the environment or to navigate to a specified destination in the environment. In these implementations, the observations may include, e g., one or more of: images, object position data, and sensor data to capture observations as the agent interacts with the environment. The actions may define control signals to control the robot or other mechanical agent, e.g., positions, torques, or other control signals for the parts of the mechanical agent, or higher-level control commands.

[0135]

[0132] In some agent control implementations the agent may be a human agent and the environment may be a real-world environment. For example the agent can be a human user of a digital assistant such as a smart speaker, smart display, or some other device that is used to instruct the user to perform actions. The task may be any real-world task that the user wishes to perform. The observations may be obtained from an observation capture subsystem, e.g. a monitoring system such as a video camera or sound capture system, to capture visual observations of the user performing the task. The actions may comprise instructions in the form of, e.g., text, image, video, or audio data such as speech, that guide the user in performing the task.

[0136]

[0133] The described systems and techniques may be applied to a wide range of different types of input sequence and output sequence. In implementations of the describedtechniques the tokens may represent, characterize, or encode any type of information in a sequence e.g. stream of data. The term '’represent” is used, below, generally to refer to any way in which a token can encode part of a sequence. The tokens may include marker tokens, such as a start of sequence token, an end of sequence token, and a separator token (indicating a separation or break between two distinct parts of a sequence). The tokens may, but need not be, drawn from a defined vocabulary of tokens.

[0137]

[0134] Some of these implementations may be used for natural language tasks such as providing a natural language response to a natural language input, e g. for question answering, or for text completion. In some implementations the input sequence may represent text in a natural language and the output sequence may represent text in the same natural language, e.g. a longer item of text. For example in some implementations the input sequence may represent text in a natural language and the output sequence may represent the same text with a missing portion of the text added or filled in. For example the output sequence may represent a predicted completion of text represented by the input sequence. Such an application may be used, e.g. to provide an auto-completion function e.g. for natural language-based search. In some implementations the input sequence may represent a text in a natural language e.g. posing a question or defining a topic, and the output sequence may represent a text in a natural language which is a response to the question or about the specified topic.

[0138]

[0135] As another example the input sequence may represent a first item of text and the output sequence may represent a second, shorter item of text e g. the second item of text may be a summary7of a passage that is the first item of text. As another example the input sequence may represent a first item of text and the output sequence may represent an aspect of the first item of text e.g. it may represent an entailment task, a paraphrase task, a textual similarity task, a sentiment analysis task, a sentence completion task, a grammaticality task, and in general any natural language understanding task that operates on a sequence of text in some natural language e.g. to generate an output that classifies or predicts some property of the text. For example some implementations may be used to identify a natural language of the first item of text, or of spoken words where the input is audio (as described below).

[0139]

[0136] Some implementations may be used to perform neural machine translation. Thus in some implementations the input tokens represent words, wordpieces, or characters in a first natural language and the output tokens represent words, wordpieces or characters in a second, different natural language. That is, the input sequence may represent input text inthe first language and the output sequence may represent a translation of the input text into the second language.

[0140]

[0137] Some implementations may be used for automatic code generation. For example the input tokens may represent words, wordpieces or characters in a first natural language and the output tokens may represent instructions in a computer programming or markup language, or instructions for controlling an application program to perform a task e.g. build a data item such as an image or web page.

[0141]

[0138] Some implementations may be used for speech recognition. In such applications the input sequence may represent spoken words and the output sequence may represent a conversion of the spoken words to a machine-written representation e.g. text. Then the input tokens may comprise tokens representing an audio data input including the spoken words e.g. characterizing a waveform of the audio in the time domain or in the timefrequency domain. The output tokens may represent words, wordpieces, characters, or graphemes of a machine-written, e.g. text, representation of the spoken input, that is representing a transcription of the spoken input.

[0142]

[0139] Some implementations may be used for handwriting recognition. In such applications the input sequence may represent handwritten words, syllabograms or characters and the output sequence may represent a conversion of the input sequence to a machine-written representation e.g. text. Then the input tokens may comprise tokens representing portions of the handwriting and the output tokens may represent words, wordpieces, characters or graphemes of a machine-written, e.g. text, representation of the spoken input.

[0143]

[0140] Some implementations may be used for text-to-speech conversion. In such applications the input sequence may represent text and the output sequence may represent a conversion of the text to spoken words. Then the input tokens may comprise tokens representing words or wordpieces or graphemes of the text and the output tokens may represent portions of audio data for generating speech corresponding to the text, e.g. tokens characterizing a portion of a waveform of the speech in the time domain or in the time-frequency domain, or phonemes.

[0144]

[0141] Some implementations may be used for a genomics task, where the input sequence represents a fragment of a DNA sequence or other molecule sequence and the output sequence is either an embedding of the fragment for use in a downstream task, e.g., by making use of an unsupervised learning technique on a data set of DNA sequence fragments, or an output for the downstream task. Examples of downstream tasks includepromoter site prediction, methylation analysis, predicting functional effects of non-coding variants, and so on.

[0145]

[0142] In some cases, the machine learning task is a combination of multiple individual machine learning tasks, i.e., the system is configured to perform multiple different individual machine learning tasks, e.g., two or more of the machine learning tasks mentioned above. For example, the system can be configured to perform multiple individual natural language understanding tasks, with the network input including an identifier for the individual natural language understanding task to be performed on the network input.

[0146]

[0143] In some implementations the input sequence and the output sequence represent different modalities of input. For example the input sequence may represent text in a natural language and the output sequence may represent an image or video corresponding to the text; or vice-versa. In general the tokens may represent image or video features and a sequence of such tokens may represent an image or video. There are many ways to represent an image (or video) using tokens. As one example an image (or video) may be represented as a sequence of regions of interest (Rols) in the image, optionally including one or more tokens for global image features. For example an image may be encoded using a neural network to extract Rol features; optionally (but not essentially) a token may also include data, e.g. a position encoding, representing a position of the Rol in the image. As another example, the tokens may encode color or intensity values for pixels of an image. As another example, some image processing neural network systems e.g. autoregressive systems, naturally represent images as sequences of image features. As another example, a transformer-based sequence-to-sequence neural network system as previously described may be used to process images instead of or as well as text (e.g. if trained on images instead of or as well as text).

[0147]

[0144] Thus, in some implementations at least one of the input sequence and the output sequence is a sequence representing an image or video, and the tokens represent the image or video. For example, the input sequence may be a sequence of text, the input tokens may represent words, wordpieces, or characters and the output sequence may comprise output tokens representing an image or video e.g. described by the text, or providing a visual answer to a question posed by the text, or providing a visualization of a topic of the text. In another example the input sequence may comprise a sequence of input tokens representing an image or video, and the output tokens may represent words or wordpieces, or characters representing text e.g. for a description or characterization ofthe image or video, or providing an answer to a question posed visually by the image or video, or providing information on a topic of the image or video.

[0148]

[0145] In some other implementations both the input sequence and the output sequence may represent an image or video, and both the input tokens and the output tokens may represent a respective image or video. In such implementations the method / system may be configured to perform an image or video transformation. For example, the input sequence and the output sequence may represent the same image or video in different styles e.g. one as an image the other as a sketch of the image; or different styles for the same item of clothing.

[0149]

[0146] In some implementations the input sequence represents data to be compressed, e.g. image data, text data, audio data, or any other type of data; and the output sequence is a compressed version of the data. The input and output tokens may each comprise any representation of the data to be compressed / compressed data e.g. symbols or embeddings generated / decoded by a respective neural network.

[0150]

[0147] In some implementations the input sequence represents a sequence of actions to be performed by an agent e.g. a mechanical agent in a real-world environment implementing the actions to perform a mechanical task. The output sequence may comprise a modified sequence of actions e.g. one in which an operating parameter, such as a speed of motion or power consumption, has a limited value; or one in which safety or other boundary is less likely to be crossed. Then both the input tokens and the output tokens may represent the actions to be performed.

[0151]

[0148] In some implementations the input sequence represents a sequence of health data and the output sequence may comprise a sequence of predicted treatment. Then the input tokens may represent any aspect of the health of a patient e.g. data from blood and other medical tests on the patient and / or EHR (Electronic Health Record) data; and the output tokens may represent diagnostic information e.g. relating to a disease status of the patient and / or relating to suggested treatments for the patient, and / or relating to a likelihood of an adverse health event for the patient.

[0152]

[0149] As a particular example the sequence-to-sequence model neural network can comprise a multimodal model in which one or both of the model input (i.e. input sequence) and the model output (i.e. output sequence) comprise an image or audio. For example the multimodal machine learning model may be configured to process an input sequence comprising visual tokens representing pixels of a still or moving image (which here may include a point cloud image), and / or data representing an audio waveform e.g.values or features of the audio waveform such as audio tokens, and / or text tokens representing a sequence of text, to generate an output sequence e.g. comprising text tokens representing the still or moving image or audio waveform, and / or comprising a sequence of intensity value inputs for the pixels of an image or a sequence of values defining an audio waveform. A visual token may, e.g., represent multiple pixels in a region of the image, e.g. as features of the region. Such a multimodal model may perform any of the previously described tasks, e.g. using a multimodal input, or by providing a multimodal output, or by converting between different input and output modes (e.g. text / image / audio). For example it may generate text representing, describing (e.g. captioning), or otherwise characterizing an image or audio input, e.g. by answering a question related to the image or audio input, e.g. relating to a future e.g. physical prediction of a state of objects represented by the image or audio. As another example it may generate an image or audio represented, described, or otherwise characterized by a text input, or otherwise in response to the text input, e.g. representing an image or audio answer to a text question.

[0153]

[0150] As previously mentioned, the sequence-to-sequence model neural network may be a language model or vision language model neural network. In general, a (vision) language model neural network can be a neural network that has been trained so that, given a text prompt that includes a sequence of tokens in a natural language, the neural network can generate the next token in the sequence. This process can be repeated to extend the text prompt one token at a time to generate a natural language output, i.e., to generate the natural language output auto-regressively token by token. At each time (“time step”), the language model neural network processes the current sequence to generate a probability distribution over a vocabulary’ of tokens. The next token can then be selected using the probability distribution, e.g., by sampling from the distribution using nucleus sampling or another sampling technique or by selecting the highest-probability token. The tokens in the vocabulary can include any of a variety of tokens, e.g., some combination of words, sub-words, characters, punctuation and other symbols, and numbers. In general, the language model neural network is trained on a corpus of text made up of tokens from the vocabulary (and optionally other tokens that can be mapped to a designated out-of-vocabulary token), to predict the next token in a sequence of tokens from the training data. The (vision) language model neural netw ork can comprise an autoregressive Transformer neural network.

[0151] A (vision) language model neural netw ork can be made to perform a particular task by providing a natural language description of the desired response as an input or ■‘prompt” (input sequence). In some cases, the prompt may be a few-shot prompt where a few, e.g., 1 to 10, examples of a query and an example output are provided in the text prior to the actual query7.

[0154]

[0152] Instead or in addition, a (vision) language model neural network may be “finetuned” to perform a particular task, by obtaining a pre-trained language model neural network trained on a large corpus of examples as previously described and then further training part of all of the language model neural network on a relatively small number of examples particular to the type of task that is to be performed.

[0155]

[0153] The (vision) language model neural network may be a large language model neural netw ork, e.g., one that has greater than 1 billion, 10 billion or 100 billion trained parameters. The (vision) language model neural network may have been trained on greater than 10 billion, 100 billion or 1000 billion words or tokens representing w ords or other tokens.

[0156]

[0154] In this specification, the term "configured" is used in relation to computing systems and environments, as well as computer program components. A computing system or environment is considered "configured" to perform specific operations or actions when it possesses the necessary software, firmware, hardware, or a combination thereof, enabling it to carry7out those operations or actions during operation. For instance, configuring a system might involve installing a software library' with specific algorithms, updating firmw are w ith new7instructions for handling data, or adding a hardw are component for enhanced processing capabilities. Similarly, one or more computer programs are "configured" to perform particular operations or actions when they contain instructions that, upon execution by a computing device or hardware, cause the device to perform those intended operations or actions.

[0157]

[0155] The embodiments and functional operations described in this specification can be implemented in various forms, including digital electronic circuitry, software, firmware, computer hardware (encompassing the disclosed structures and their structural equivalents), or any combination thereof. The subject matter can be realized as one or more computer programs, essentially modules of computer program instructions encoded on a tangible non-transitory storage medium for execution by or to control the operation of a computing device or hardware. The storage medium can be a storage device such as a hard drive or solid-state drive (SSD), a storage medium, a random or serial accessmemory device, or a combination of these. Additionally or alternatively, the program instructions can be encoded on a transmitted signal, such as a machine-generated electrical, optical, or electromagnetic signal, designed to carry information for transmission to a receiving device or system for execution by a computing device or hardware. Furthermore, implementations may leverage emerging technologies like quantum computing or neuromorphic computing for specific applications, and may be deployed in distributed or cloud-based environments where components reside on different machines or within a cloud infrastructure.

[0158]

[0156] The term "computing device or hardware" refers to the physical components involved in data processing and encompasses all types of devices and machines used for this purpose. Examples include processors or processing units, computers, multiple processors or computers working together, graphics processing units (GPUs), tensor processing units (TPUs), and specialized processing hardware such as field-programmable gate arrays (FPGAs) or application-specific integrated circuits (ASICs). In addition to hardware, a computing device or hardware may also include code that creates an execution environment for computer programs. This code can take the form of processor firmware, a protocol stack, a database management system, an operating system, or a combination of these elements. Embodiments may particularly benefit from utilizing the parallel processing capabilities of GPUs, in a General-Purpose computing on Graphics Processing Units (GPGPU) context, where code specifically designed for GPU execution, often called kernels or shaders, is employed. Similarly, TPUs excel at running optimized tensor operations crucial for many machine learning algorithms. By leveraging these accelerators and their specialized programming models, the system can achieve significant speedups and efficiency gains for tasks involving artificial intelligence and machine learning, particularly in areas such as computer vision, natural language processing, and robotics.

[0159]

[0157] A computer program, also referred to as software, an application, a module, a script, code, or simply a program, can be written in any programming language, including compiled or interpreted languages, and declarative or procedural languages. It can be deployed in various forms, such as a standalone program, a module, a component, a subroutine, or any other unit suitable for use within a computing environment. A program may or may not correspond to a single file in a file system and can be stored in various ways. This includes being embedded within a file containing other programs or data (e.g., scripts within a markup language document), residing in a dedicated file, or distributedacross multiple coordinated files (e.g., files storing modules, subprograms, or code segments). A computer program can be executed on a single computer or across multiple computers, whether located at a single site or distributed across multiple sites and interconnected through a data communication network. The specific implementation of the computer programs may involve a combination of traditional programming languages and specialized languages or libraries designed for GPU programming or TPU utilization, depending on the chosen hardware platform and desired performance characteristics.

[0160]

[0158] In this specification, the term "engine" broadly refers to a software-based system, subsystem, or process designed to perform one or more specific functions. An engine is ty pically implemented as one or more software modules or components installed on one or more computers, which can be located at a single site or distributed across multiple locations. In some instances, one or more dedicated computers may be used for a particular engine, while in other cases, multiple engines may operate concurrently on the same one or more computers. Examples of engine functions within the context of Al and machine learning could include data pre-processing and cleaning, feature engineering and extraction, model training and optimization, inference and prediction generation, and post-processing of results. The specific design and implementation of engines will depend on the overall architecture and the distribution of computational tasks across various hardware components, including CPUs, GPUs, TPUs, and other specialized processors.

[0161]

[0159] The processes and logic flows described in this specification can be executed by one or more programmable computers running one or more computer programs to perform functions by operating on input data and generating output. Additionally, graphics processing units (GPUs) and tensor processing units (TPUs) can be utilized to enable concurrent execution of aspects of these processes and logic flows, significantly accelerating performance. This approach offers significant advantages for computationally intensive tasks often found in Al and machine learning applications, such as matrix multiplications, convolutions, and other operations that exhibit a high degree of parallelism. By leveraging the parallel processing capabilities of GPUs and TPUs, significant speedups and efficiency gains compared to relying solely on CPUs can be achieved. Alternatively or in combination with programmable computers and specialized processors, these processes and logic flows can also be implemented using specialized processing hardware, such as field-programmable gate arrays (FPGAs) or application-specific integrated circuits (ASICs), for even greater performance or energy efficiency in specific use cases.

[0162]

[0160] Computers capable of executing a computer program can be based on general-purpose microprocessors, special-purpose microprocessors, or a combination of both. They can also utilize any other type of central processing unit (CPU). Additionally, graphics processing units (GPUs), tensor processing units (TPUs), and other machine learning accelerators can be employed to enhance performance, particularly for tasks involving artificial intelligence and machine learning. These accelerators often work in conjunction with CPUs, handling specialized computations while the CPU manages overall system operations and other tasks. Typically, a CPU receives instructions and data from read-only memory (ROM), random access memory (RAM), or both. The essential elements of a computer include a CPU for executing instructions and one or more memory devices for storing instructions and data. The specific configuration of processing units and memory will depend on factors like the complexity of the Al model, the volume of data being processed, and the desired performance and latency requirements. Embodiments can be implemented on a wide range of computing platforms, from small embedded devices with limited resources to large-scale data center systems with high-performance computing capabilities. The system may include storage devices like hard drives, SSDs, or flash memory for persistent data storage.

[0163]

[0161] Computer-readable media suitable for storing computer program instructions and data encompass all forms of non-volatile memory, media, and memory devices.

[0164] Examples include semiconductor memory devices such as read-only memory' (ROM), solid-state drives (SSDs), and flash memory devices; hard disk drives (HDDs); optical media; and optical discs such as CDs, DVDs, and Blu-ray discs. The specific type of computer-readable media used will depend on factors such as the size of the data, access speed requirements, cost considerations, and the desired level of portability7or permanence.

[0165]

[0162] To facilitate user interaction, embodiments of the subject matter described in this specification can be implemented on a computing device equipped with a display device, such as a liquid cry stal display (LCD) or an organic light-emitting diode (OLED) display , for presenting information to the user. Input can be provided by the user through various means, including a keyboard, touchscreens, voice commands, gesture recognition, or other input modalities depending on the specific device and application. Additional input methods can include acoustic, speech, or tactile input, while feedback to the user can takethe form of visual, auditory, or tactile feedback. Furthermore, computers can interact with users by exchanging documents with a user's device or application. This can involve sending web content or data in response to requests or sending and receiving text messages or other forms of messages through mobile devices or messaging platforms. The selection of input and output modalities will depend on the specific application and the desired form of user interaction.

[0166]

[0163] Machine learning models can be implemented and deployed using machine learning frameworks, such as TensorFlow or JAX. These frameworks offer comprehensive tools and libraries that facilitate the development, training, and deployment of machine learning models.

[0167]

[0164] Embodiments of the subject matter described in this specification can be implemented within a computing system comprising one or more components, depending on the specific application and requirements. These may include a back-end component, such as a back-end server or cloud-based infrastructure; an optional middleware component, such as a middleware server or application programming interface (API), to facilitate communication and data exchange; and a front-end component, such as a client device with a user interface, a web browser, or an app, through which a user can interact with the implemented subject matter. For instance, the described functionality7could be implemented solely on a client device (e.g., for on-device machine learning) or deployed as a combination of front-end and back-end components for more complex applications. These components, when present, can be interconnected using any form or medium of digital data communication, such as a communication network like a local area network (LAN) or a wide area network (WAN) including the Internet. The specific system architecture and choice of components will depend on factors such as the scale of the application, the need for real-time processing, data security requirements, and the desired user experience.

[0168]

[0165] The computing system can include clients and servers that may be geographically separated and interact through a communication network. The specific type of network, such as a local area network (LAN), a wide area network (WAN), or the Internet, will depend on the reach and scale of the application. The client-server relationship is established through computer programs running on the respective computers and designed to communicate with each other using appropriate protocols. These protocols may include HTTP, TCP / IP. or other specialized protocols depending on the nature of the data being exchanged and the security requirements of the system. In certainembodiments, a server transmits data or instructions to a user's device, such as a computer, smartphone, or tablet, acting as a client. The client device can then process the received information, display results to the user, and potentially send data or feedback back to the server for further processing or storage. This allows for dynamic interactions between the user and the system, enabling a wide range of applications and functionalities.

[0169]

[0166] While this specification contains many specific implementation details, these should not be construed as limitations on the scope of any invention or on the scope of what may be claimed, but rather as descriptions of features that may be specific to particular embodiments of particular inventions. Certain features that are described in this specification in the context of separate embodiments can also be implemented in combination in a single embodiment. Conversely, various features that are described in the context of a single embodiment can also be implemented in multiple embodiments separately or in any suitable subcombination. Moreover, although features may be described above as acting in certain combinations and even initially be claimed as such, one or more features from a claimed combination can in some cases be excised from the combination, and the claimed combination may be directed to a subcombination or variation of a subcombination.

[0170]

[0167] Similarly, while operations are depicted in the drawings and recited in the claims in a particular order, this should not be understood as requiring that such operations be performed in the particular order shown or in sequential order, or that all illustrated operations be performed, to achieve desirable results. In certain circumstances, multitasking and parallel processing may be advantageous. Moreover, the separation of various system modules and components in the embodiments described above should not be understood as requiring such separation in all embodiments, and it should be understood that the described program components and systems can generally be integrated together in a single software product or packaged into multiple software products.

[0171]

[0168] Innovative aspects of the present disclosure are also set out in the following numbered clauses:

[0172] Clause 1. A method performed by one or more computers, the method comprising:

[0173] obtaining a machine learning model comprising model parameters represented by¬ floating-point numbers (which may be termed “parameter values’’);obtaining training data comprising input data;

[0174] performing a training process comprising one or more iterations of:

[0175] applying a quantization function to convert the floating-point numbers to a set of reduced precision numbers (which may be termed “reduced precision parameter values'’) each having a first number of bits;

[0176] supplying part of the input data to the machine learning model to calculate first output values based on the set of reduced precision numbers; and

[0177] based on the first output values, determining a first training obj ective term; for each of the set of reduced precision numbers, producing a respective updated number (which may be termed an “ updated reduced precision parameter value”) by extracting a second number of most significant bits (MSBs) belonging to the respective one of the reduced precision numbers, the second number being less than the first number;

[0178] supplying the part of the input data to the machine learning model to calculate second output values based on the updated numbers; and

[0179] based on the second output values, determining a second training objective term; and

[0180] updating (i.e., adjusting) parameters so as to minimize a sum of training objective terms including the first training objective term and the second training objective term, the updated parameters including at least one of: the model parameters and quantization parameters associated with the quantization function; and

[0181] providing an updated version of the machine learning model comprising model parameters represented by final numbers (which may be termed “final parameter values”) determined based on the updated parameters following the one or more iterations, the final numbers each having the first number of bits.

[0182] Clause 2. The method of clause 1, wherein the reduced precision numbers are integers, the updated numbers are updated integers, and the final numbers are final integers.

[0183] Clause 3. The method of clause 2, wherein for each of the integers, the producing the respective updated integer comprises setting to zero all bits of the respective integer following the second number of MSBs of the respective integer.Clause 4. The method of clause 2 or 3, further comprising:

[0184] obtaining a further machine learning model by reducing the bit-width of each of the model parameters to n bits by setting each of the model parameters equal to the n most significant bits of a corresponding one of the final integers.

[0185] Clause 5. The method of clause 2 or 3. further comprising:

[0186] updating the machine learning model by:

[0187] reducing the bit-width of some of the model parameters belonging to a first set of one or more layers of the machine learning model to n bits by setting each of the model parameters equal to n most significant bits of a corresponding one of the final integers; and

[0188] reducing the bit-width of some of the model parameters belonging to a second set of one or more layers of the machine learning model to m bits by setting each of the model parameters equal to m most significant bits of a corresponding one of the final integers,

[0189] wherein n is different from m.

[0190] Clause 6. The method of clause 4 or clause 5, wherein n is equal to the second number.

[0191] Clause 7. The method of any of clauses 4 to 6, wherein n is not equal to a number of bits extracted from the integers during the training process and used to calculate one of the training objective terms.

[0192] Clause 8. The method of any preceding clause, wherein the machine learning model comprises one or more transformers of a large language model (LLM) or a vision language model (VLM).

[0193] Clause 9. The method of any preceding clause, wherein each of the one or more training iterations further comprises:

[0194] for each of the reduced precision numbers, producing a respective further updated number by extracting a third number of most significant bits (MSB) belonging to therespective reduced precision number, the third number being greater than the second number but less than the first number;

[0195] supplying the part of the input data to the machine learning model to calculate third output values based on the further updated numbers; and

[0196] based on the third output values, determining a third training objective term, wherein the training objective terms of the sum include the third training objective term.

[0197] Clause 10. The method of any preceding clause, comprising calculating the sum by weighting the training objective terms differently.

[0198] Clause 11. The method of any preceding clause, wherein the quantization parameters comprise at least one of:

[0199] an upper clipping threshold for clipping the model parameters when applying the quantization function;

[0200] a lower clipping threshold for clipping the model parameters when applying the quantization function;

[0201] a scaling parameter for scaling the model parameters when applying the quantization function; and

[0202] a shift parameter for shifting input values to the machine learning model.

[0203] Clause 12. The method of any preceding clause, wherein the updated parameters include updated quantization parameters associated with the quantization function, wherein the method further comprises: applying the quantization function using the updated quantization parameters to convert the floating-point numbers to the final numbers.

[0204] Clause 13. The method of clause 12, further comprising:

[0205] supplying the part of the input data to the machine learning model to calculate comparison output values based on the floating-point numbers representing the model parameters,

[0206] wherein determining the first training objective term comprises determining a difference between the first output values and the comparison output values, wherein determining the second training objective term comprises determining a difference between the second output values and the comparison output values.Clause 14. The method of clause 13. wherein the first output values, the second output values and the comparison output values comprise outputs of a hidden layer of the machine learning model.

[0207] Clause 15. The method of any preceding clause, wherein for a final one of the one or more iterations, the updated parameters comprise the model parameters represented by the final numbers.

[0208] Clause 16. The method of any preceding clause, wherein:

[0209] the training data comprises labels;

[0210] determining the first training objective term comprises applying a loss function to the first output values and corresponding ones of the labels; and

[0211] determining the second training objective term comprises applying a loss function to the second output values and the corresponding ones of the labels.

[0212] Clause 17. The method of clause 16, wherein the loss function is a cross-entropy function.

[0213] Clause 18. The method of any preceding clause, wherein the parameters comprise the model parameters, wherein for each of the one or more iterations, updating the parameters comprises:

[0214] updating the floating-point numbers based on the sum of the training objective terms; and

[0215] applying the quantization function to the updated floating-point numbers to convert the updated floating-point numbers to a further set of reduced precision numbers each having the first number of bits.

[0216] Clause 19. A method of using a machine learning model trained by the method of any preceding clause, the method performed by a computer device and comprising:

[0217] selecting a bit-width for use for at least some of the model parameters;

[0218] if the selected bit- width is equal to the first number of bits, using the machine learning model with each of the at least some of the model parameters having the first number of bits; andif the selected bit-width is less than the first number of bits, truncating the at least some of the model parameters to the selected bit-width and using the machine learning model with each of the at least some of the model parameters having the selected bitwidth,

[0219] wherein using the machine learning model comprises processing input values to provide a result.

[0220] Clause 20. A method of using a machine learning model comprising model parameters, each of which has a first number of bits and comprises a plurality of nested numbers having different bit-widths and provided according to a training process that minimizes a training objective term given by a sum of individual training objective terms associated with each of the different bit-widths, the method performed by a computer device and comprising:

[0221] selecting a bit-width from one of the different bit-widths for use for at least some of the model parameters;

[0222] if the selected bit-width is equal to the first number of bits, using the machine learning model with each of at least some of the model parameters having the first number of bits; and

[0223] if the selected bit-width is less than the first number of bits, truncating the at least some of the model parameters to the selected bit-width and using the machine learning model with each of the at least some of the model parameters having the selected bit-width,

[0224] wherein using the machine learning model comprises processing input values to provide a result.

[0225] Clause 21. The method of clause 19 or clause 20, w herein the nested numbers are nested integers.

[0226] Clause 22. The method of clause 20 or clause 21, wherein the computer device comprises one of:

[0227] an edge device; and

[0228] a mobile user equipment.Clause 23. The method of any of clauses 20 to 22, wherein selecting the bit- width is performed in dependence upon a memory capacity’ of the device.

[0229] Clause 24. The method of any of clauses 19 to 23, comprising:

[0230] selecting a further bit-width for use for further ones of the model parameters; if the selected further bit-width is equal to the first number of bits, using the machine learning model with each of the further ones of the model parameters having the first number of bits; and

[0231] if the selected further bit- width is less than the first number of bits, truncating the further ones of the model parameters to the further selected bit-width and using the machine learning model with each of the further ones of the model parameters having the selected bit-width.

[0232] Clause 25. The method of clause 24, wherein the further ones of the model parameters belong to a different layer of the machine learning model than the at least some of the model parameters.

[0233] Clause 26. A system comprising one or more computers and one or more storage devices storing instructions that when executed by the one or more computers cause the one or more computers to perform the operations of the respective method of any one of clauses 1-25.

[0234] Clause 27. One or more computer storage media storing instructions that when executed by one or more computers cause the one or more computers to perform the operations of the respective method of any one of clauses 1-25.

[0235]

[0169] Particular embodiments of the subject matter have been described. Other embodiments are within the scope of the following claims. For example, the actions recited in the claims can be performed in a different order and still achieve desirable results. As one example, the processes depicted in the accompanying figures do not necessarily require the particular order shown, or sequential order, to achieve desirable results. In some cases, multitasking and parallel processing may be advantageous.

[0236]

[0170] What is claimed is:

Claims

CLAIMS1. A method performed by one or more computers, the method comprising:obtaining a machine learning model comprising model parameters represented by floating-point parameter values;performing a training process comprising one or more iterations of:obtaining training data comprising input data for the iteration; applying a quantization function to convert the floating-point parameter values for at least some of the model parameters to corresponding reduced precision parameter values that each have a first number of bits;processing the input data for the iteration using a version of the machine learning model in which the at least some of the model parameters are set to the corresponding reduced precision parameter values to determine first output values for the iteration;for each of the reduced precision parameter values, extracting a second number of most significant bits (MSBs) from each of the reduced precision parameter values to determine a corresponding updated reduced precision parameter value;processing the input data for the iteration using a version of the machine learning model in which the at least some of the model parameters are set to the corresponding updated reduced precision parameter values to determine second output values for the iteration; andoptimizing a training objective determined based on the first output values for the iteration and the second output values for the iteration, the optimizing comprising adjusting parameter values of parameters that include (i) the at least some of the model parameters, (ii) quantization parameters associated with the quantization function, or (iii) both;following the one or more iterations, using the adjusted parameter values to determine final parameter values for the machine learning model that each have the first number of bits; andproviding an updated version of the machine learning model in which the model parameters are set to the corresponding final parameter values.

2. The method as claimed in claim 1, wherein the reduced precision parameter values, the updated reduced precision parameter values, and the final parameter values are integers.

3. The method as claimed in claim 2, wherein for each of the reduced precision parameter values, determining the corresponding updated reduced precision parameter value comprises setting to zero all bits of the reduced precision parameter value following the second number of MSBs of the reduced precision parameter value.

4. The method as claimed in claim 2 or 3, further comprising:obtaining a further machine learning model by reducing the bit-width for each of the model parameters to n bits by setting the parameter value for each of the model parameters to the n most significant bits of a corresponding one of the final parameter values.

5. The method as claimed in claim 2 or 3, further comprising:updating the machine learning model by:reducing the bit-width of the model parameters that belong to a first set of one or more lay ers of the machine learning model to n bits by setting the parameter value for each of the model parameters to the n most significant bits of a corresponding one of the final parameter values; andreducing the bit-width of the model parameters that belong to a second set of one or more layers of the machine learning model to m bits by setting the parameter value for each of the model parameters to m most significant bits of a corresponding one of the final parameter values,wherein n is different from m.

6. The method as claimed in claim 4 or claim 5, wherein n is equal to the second number of MSBs.

7. The method as claimed in any of claims 4 to 6, wherein n is not equal to a number of bits extracted from the reduced precision parameter values during the training process and used to determine the training objective.

8. The method as claimed in any preceding claim, wherein the machine learning model comprises one or more transformers of a large language model (LLM) or a vision language model (VLM).

9. The method as claimed in any preceding claim, further comprising determining the training objective based on a sum of training objective terms comprising: a first training objective term determined based on the first output values for the iteration, and a second training objective term determined based on the second output values for the iteration.

10. The method as claimed in claim 9, wherein each of the one or more iterations further comprises:for each of the reduced precision parameter values, producing a respective further updated reduced precision parameter value by extracting a third number of MSBs belonging to the respective reduced precision parameter value, the third number of MSBs being greater than the second number of MSBs but less than the first number of bits; supplying the input data for the iteration to the machine learning model to calculate third output values based on the further updated reduced precision parameter values; andbased on the third output values, determining a third training objective term, wherein the training objective terms of the sum include the third training objective term.

11. The method as claimed in claim 9 or 10, further comprising:calculating the sum by weighting the training objective terms differently.

12. The method as claimed in any preceding claim, wherein the quantization parameters comprise at least one of:an upper clipping threshold for clipping the model parameters when applying the quantization function;a lower clipping threshold for clipping the model parameters when applying the quantization function;a scaling parameter for scaling the model parameters when applying the quantization function; anda shift parameter for shifting input values to the machine learning model.

13. The method as claimed in any preceding claim, wherein the adjusted parameter values include parameter values of quantization parameters associated with the quantization function, and wherein the method further comprises:applying the quantization function using the adjusted parameter values of the quantization parameters to convert the floating-point parameter values to the final parameter values.

14. The method as claimed in claim 13, further comprising:supplying the input data for the iteration to the machine learning model to calculate comparison output values based on the floating-point parameter values representing the model parameters,wherein determining the training objective comprises determining a difference between the first output values and the comparison output values,wherein determining the training objective comprises determining a difference between the second output values and the comparison output values.

15. The method as claimed in claim 14, wherein the first output values, the second output values and the comparison output values comprise outputs of a hidden layer of the machine learning model.

16. The method as claimed in any preceding claim, wherein for a final one of the one or more iterations, the adjusted parameter values comprise the final parameter values.

17. The method as claimed in any preceding claim, wherein:the training data comprises labels;determining the training objective comprises applying a loss function to the first output values and corresponding ones of the labels; anddetermining the training objective comprises applying a loss function to the second output values and the corresponding ones of the labels.

18. The method as claimed in claim 17, wherein the loss function is a cross-entropy function.

19. The method as claimed in any preceding claim, wherein for each of the one or more iterations, adjusting the parameter values comprises:adjusting the floating-point parameter values based on the training objective determined using the first output values for the iteration and the second output values for the iteration; andapplying the quantization function to the adjusted floating-point parameter values to convert the adjusted floating-point parameter values to corresponding reduced precision parameter values that each have the first number of bits.

20. A method of using a machine learning model trained by the method of any- preceding claim, the method performed by a computer device and comprising:selecting a bit-width for use for at least some of the model parameters;if the selected bit-width is equal to the first number of bits, using the machine learning model with corresponding parameter values of each of the at least some of the model parameters, the corresponding parameter values having the first number of bits; andif the selected bit- width is less than the first number of bits, truncating the corresponding parameter values of the at least some of the model parameters to the selected bit-width and using the machine learning model with the corresponding parameter values of each of the at least some of the model parameters, the corresponding parameter values having the selected bit-width,wherein using the machine learning model comprises processing input values to provide a result.

21. A method of using a machine learning model comprising model parameters, each of which has a first number of bits and comprises a plurality- of nested parameter values having different bit-widths and provided according to a training process that minimizes a training objective given by a combination of individual training objective terms associated with each of the different bit-widths, the method performed by a computer device and comprising:selecting a bit-width from one of the different bit-w idths for use for at least some of the model parameters;if the selected bit-width is equal to the first number of bits, using the machine learning model with corresponding parameter values of each of at leastsome of the model parameters, the corresponding parameter values each having the first number of bits; andif the selected bit- width is less than the first number of bits, truncating the corresponding parameter values of the at least some of the model parameters to the selected bit-width and using the machine learning model with the corresponding parameter values of each of the at least some of the model parameters, the corresponding parameter values each having the selected bitwidth,wherein using the machine learning model comprises processing input values to provide a result.

22. The method as claimed in claim 20 or claim 21 , wherein the nested parameter values are nested integers.

23. The method as claimed in claim 21 or claim 22. wherein the computer device comprises one of:an edge device; anda mobile user equipment.

24. The method as claimed in any of claims 21 to 23. wherein selecting the bit-width is performed in dependence upon a memory capacity of the device.

25. The method as claimed in any of claims 20 to 24, comprising:selecting a further bit-width for use for further ones of the model parameters; if the selected further bit-width is equal to the first number of bits, using the machine learning model with corresponding parameter values of each of the further ones of the model parameters having the first number of bits; andif the selected further bit-width is less than the first number of bits, truncating the corresponding parameter values of the further ones of the model parameters to the further selected bit-width and using the machine learning model with the corresponding parameter values of each of the further ones of the model parameters having the selected bit-width.

26. The method as claimed in claim 25, wherein the further ones of the model parameters belong to a different layer of the machine learning model than the at least some of the model parameters.

27. A system comprising one or more computers and one or more storage devices storing instructions that when executed by the one or more computers cause the one or more computers to perform the operations of the respective method of any one of claims 1-26.

28. One or more computer storage media storing instructions that when executed by one or more computers cause the one or more computers to perform the operations of the respective method of any one of claims 1-26.

Images