Computer system and method for executing a machine learning model

Abstract

A computer system executes a machine learning model having multiple layers, and includes host and work accelerator processors. A window size representing a number of layers to be loaded into accelerator memory is determined. Model data associated with a subset of layers the same size as the window is loaded to the accelerator. The model is iteratively executed by processing a current layer to provide output data and storing this in accelerator memory, offloading the output data to host memory, replacing model data for an already processed layer with model data corresponding to a next layer, and moving to a next layer. The window size is updated during execution based on a transfer duration for loading into accelerator memory, a transfer duration for loading into host memory, or a processing duration of a layer.

Description

COMPUTER SYSTEM AND METHOD FOR EXECUTING A MACHINE LEARNING MODELField

[0001] The present application relates to a computer system for executing a machine learning model comprising a plurality of layers, the system being configured to determine, during the execution of the model, a window size of layers to be stored in an accelerator memory of the system. The application further relates to computer-implemented methods, computer program products and computer-readable storage media for accomplishing the same.Background

[0002] Foundation models based on deep neural networks (DNNs) have become increasingly popular, owing to their impressive performance on a variety of tasks. For example, transformer-based natural language processing models such as the Generative Pretrained Transformer (GPT) models provided by Open Al® are able to provide realistic natural language output, as well as being able to perform tasks including summarisation, classification, translation and so on. Furthermore, such models display emergent characteristics, such as the ability to perform arithmetic, encode data and so on. Similarly impressive models are available for image generation and other modalities.

[0003] Key to the performance of these models is the size of their training data set and the size of their parameter space. GPT-3, for example, has 175 billion parameters and was trained on approximately 500 billion tokens, with GPT-4 rumoured to have over 1 trillion parameters and a training data set of over 10 trillion tokens. The growth in parameter space is trending exponentially.

[0004] As a result of this enormous parameter space and training data set size, training large DNNs requires significant computational resource. Training large DNNs can be accomplished using aggregated computing resources, through parallel and distributed computation that makes use of specialist work accelerator hardwarerather than relying on traditional general-purpose CPUs (central processing units). An example of work accelerator hardware is Graphical Processing Units (GPUs), such as those provided by NVIDIA®. Other example work accelerator hardware includes Tensor Processing Units (TPUs) provided by Google®, Deep Learning Processor Units (DPUs) provided by AMD® and so on.

[0005] The memory capacity of work accelerators has not grown in a similar manner to the size of DNN parameter space. For example, from 2018 to 2021, model size grew by over 1000-fold, but high-end GPU memory increased only three-fold. This lack of memory capacity equates to a requirement for ever larger distributed computing systems, with associated cost. For example, to train a 10 billion parameter model efficiently, a system including 16 GPUs is required at an estimated cost of over 100K USD. This takes the training of such models out of reach of smaller businesses and academic institutions.Summary

[0006] To address these issues, work has been carried out to reduce GPU memory usage in training large DNNs. For example, some techniques rely on using low or mixed precision number representation for model states, but this can slow down model convergence. Other techniques leverage CPU memory, but are based around traditional convolutional neural networks, where memory consumption is dominated by neuron activation states. These techniques are ill-suited to modern Transformerbased architectures, where it is the model and optimiser states that form the memory bottleneck.

[0007] One technique that attempts to address these issues is ZeRO-Offload (J. Ren, S. Rajbhandari, R. Y. Aminabadi, O. Ruwase, S. Yang, M. Zhang, D. Li, and Y. He, "Zerooffload: Democratizing billion-scale model training," in 2021 USENIX Annual Technical Conference) which moves optimiser states from GPU memory to CPU memory while keeping the entire model parameters in GPU memory. It is able to train a model with 13B parameters on a 32GB V100 NVIDIA® GPU. However, as ZeRO-Offload requiresstoring the entire model parameters in the GPU, the trainable model size is limited by the smallest part of the available GPU and CPU memory capacity. Another technique, L2L (B. Pudipeddi et al., "Training large neural networks with constant memory using a new execution algorithm," arXiv, 2020) is specifically designed for transformer-based models, only keeping one transformer block in GPU memory by dynamically offloading the model parameters. Another technique, ZeRO-Infinity (5. Rajbhandari, O. Ruwase, J. Rasley, S. Smith, and Y. He, "Zeroinfinity: Breaking the GPU memory wall for extreme scale deep learning," in Proceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis, 2021, pp. 1-14) adopts a dynamic strategy to leverage additional storage (e.g. NVMe - nonvolatile memory express) and partitions the model parameters and optimiser states across heterogenous memory hierarchy. Whilst these approaches can increase the trainable model size, they come with a significant overhead, slowing down the training process.

[0008] To address the issues outlined above, the disclosure provides an approach for leveraging heterogeneous memory resources to scale up the model size on work accelerators without significantly compromising the execution time. This is accomplished by keeping just a sufficient number of layers of the model and their associated model parameters in accelerator memory to avoid the accelerator stalling during data offloading. This sufficient number of layers is referred to herein as a "window" or "working window" of layers. The size of the working window is determined dynamically during model execution based on any of a number of factors, such as an available capacity of the accelerator memory, a transfer duration for loading model data associated with a layer into the accelerator memory, and a processing time associated with a layer. Accordingly, the number of layers in accelerator memory changes during execution (e.g. during training), to make the best use of available work accelerator memory.

[0009] According to a first aspect of the disclosure, there is provided a computer system for executing a machine learning model, the system comprising: a host processor; a host memory storing a plurality of layers of the machine learning model,each layer having corresponding model data; and a work accelerator processor and an accelerator memory. The system is configured to: determine a window size representing a number of layers to be loaded into accelerator memory; load model data associated with a subset of the plurality of layers, the subset being the same size as the determined window size, from the host memory into accelerator memory; and execute the machine learning model by iteratively: processing, using the work accelerator processor, a current layer of the model based on the corresponding model data loaded into accelerator memory to provide output data for the current layer; storing the output data for the current layer in the accelerator memory; offloading the output data for the current layer to host memory; replacing model data for an already processed layer with model data corresponding to a next layer of the machine learning model outside the subset; moving to a next layer of the model within the subset, such that the next layer within the subset becomes the current layer; during execution of the machine learning model, update the window size and in response cause the accelerator memory to store model data associated with a subset of the plurality layers having the updated window size. The updated window size is determined based on at least one of a first transfer duration for loading model data associated with a layer into the accelerator memory from the host memory, a second transfer duration for loading model data associated with a layer into host memory from the accelerator memory and a processing duration associated with a layer.

[0010] The updated window size may be determined by selecting a window size meeting a plurality of constraints, the constraints based on the first transfer duration, the second transfer duration and the processing duration. The updated window size may be determined by selecting the smallest possible window size that meets the plurality of constraints.

[0011] The plurality of constraints may include a first constraint that a sum of the first transfer durations for the layers in the window is less than or equal to a sum of the processing durations for the layers in the window. The plurality of constraints may include a second constraint that a sum of the second transfer durations for the layersin the window is less than or equal to a sum of the processing durations for the layers in the window. The plurality of constraints may include a third constraint that the sum of the first transfer durations added to the sum of the second transfer durations, minus a sum of an overlapping duration for the layers in the window, the overlapping duration being representative of the duration for which the first and second transfer durations overlap with the processing of a layer, is less than the sum of the processing durations. The plurality of constraints include a fourth constraint that a memory size of the layers in the window does not exceed an available capacity of the accelerator memory. The plurality of constraints include a fifth constraint that a memory size of the layers in the window and a memory size of the subsequent layer to be processed does not exceed an available capacity of the accelerator memory. The plurality of constraints may include a sixth constraint that the maximum of the sum of the first transfer durations and the sum of the second transfer durations is less than or equal to the sum of the processing durations. The constraints defined above and any other constraints discussed herein may be combined in any combination.

[0012] The system may be configured to update the window size every n iterations of the execution of the machine learning model. The system is configured to update the window size in substantially every iteration of execution of the machine learning model, such that n = 1. Alternatively, n may be one of 2, 5, 7, 10, 15, 20, 25, 50 or any other suitable number.

[0013] The system may be configured to determine any of the durations or memory sizes discussed herein during a warm-up iteration, or a plurality of warm-up iterations, of execution of the machine learning model.

[0014] Executing the machine learning model may comprise the forward propagation of input data through the machine learning model. The model data of the current layer may include weights and input data from a preceding layer. The output data for the current layer may include the activations of the current layer. The forward propagation may take place as part of a training process of the machine learningmodel. The forward propagation may take place as part of an inference process of the machine learning model.

[0015] Executing the machine learning model may comprise backward propagation of a loss through the machine learning model. The output data for the current layer may include gradients. The host processor may be configured to update parameters of the machine learning model upon receipt of the output data for the current layer at the host memory, whilst the work accelerator processor processes the next layer. In other words, the host processor may update parameters asynchronously with respect to the processing carried out by the work accelerator processor.

[0016] The constraints and / or durations applied during forward propagation may differ from those applied during backpropagation. In other words, there may be a first set of constraints applied during forward propagation. The first set of constraints may be based on a first set of durations applicable during forward propagation. There may further be a second set of constraints applied during backward propagation. The second set of constraints may be based on a second set of durations applicable during forward propagation.

[0017] The machine learning model may be any layered machine learning model. The machine learning model may be a deep neural network comprising at least one billion parameters. The machine learning model may be a Transformer model. The machine learning model may comprise residual components or gating mechanisms. The machine learning model may be a mixture-of-experts (MoE) model.

[0018] The work accelerator processor may be a graphical processing unit, GPU. The accelerator memory may be built-in memory of the GPU. The accelerator memory may be on-chip memory, suitably of the GPU. That is to say, the work accelerator processor and the accelerator memory may be on a same die.

[0019] According to a second aspect of the disclosure, there is provided a system comprising: a plurality of the computer systems as defined in the first aspect; a coordinating system, comprising a storage storing the machine learning model andplurality of training data items, the coordinating system configured to: distribute a copy of the machine learning model and a respective subset of the training data items to each of the computer systems; cause each computer system to train the received copy of the machine learning model based on the respective subset of the training data to produce training results; receive the training results from each of the computer systems; and combine the training results to produce a trained machine learning model.

[0020] According to a third aspect of the disclosure there is provided a computer-implemented method of executing a machine learning model, the method carried out by a system comprising: a host processor; a host memory storing a plurality of layers of the machine learning model, each layer having corresponding model data; and a work accelerator processor and an accelerator memory; the method comprising: determining a window size representing a number of layers to be loaded into accelerator memory; loading model data associated with a subset of the plurality of layers, the subset being the same size as the determined window size, from the host memory into accelerator memory; executing the machine learning model by iteratively: processing, using the work accelerator processor, a current layer of the model based on the corresponding model data loaded into accelerator memory to provide output data for the current layer; storing the output data for the current layer in the accelerator memory; offloading the output data for the current layer to host memory; replacing model data for an already processed layer with model data corresponding to a next layer of the machine learning model outside the subset; moving to a next layer of the model within the subset, such that the next layer within the subset becomes the current layer; during execution of the machine learning model, update the window size and in response cause the accelerator memory to store model data associated with a subset of the plurality layers having the updated window size; wherein the updated window size is determined based on at least one of a first transfer duration for loading model data associated with a layer into the accelerator memory from the host memory, a second transfer duration for loadingmodel data associated with a layer into host memory from the accelerator memory and a processing duration associated with a layer.

[0021] According to a fourth aspect of the disclosure, there is provided a computer-readable storage medium storing instructions, which when executed, cause a computer system to implement any of the methods disclosed herein.

[0022] According to a fifth aspect of the disclosure, there is provided a computer program product comprising instructions, which when executed, cause a computer system to implement any of the methods disclosed herein.

[0023] The optional features of the first aspect described above apply correspondingly to the second to fifth aspects, and may be combined in any combination.Brief Description of the Drawings

[0024] To assist understanding of the present disclosure and to show how embodiments may be put into effect, reference is made by way of example to the accompanying drawings in which:

[0025] FIG. lisa schematic block diagram of an example system according to aspects of the disclosure.

[0026] FIG. 2 is a schematic block diagram of an example machine learning model.

[0027] FIG. 3A and 3B schematically illustrate an example technique for accelerator memory management.

[0028] FIG. 4A and 4B respectively schematically illustrate the example technique of FIG. 3A-B to forward propagation and backward propagation.

[0029] FIG. 5 is a schematic flowchart of an example method of executing a machine learning model.

[0030] FIG. 6 is a schematic block diagram of an example distributed system for training a machine learning model.

[0031] FIG. 7 is a schematic block diagram of an example computing system.Detailed Description

[0032] In overview, examples of the disclosure provide a means of loading model data associated with layers of a machine learning model into the memory of an accelerator such as a GPU, during model execution. The techniques herein maintain a "window" of layers in accelerator memory, with layers computed by the accelerator being offloaded to host memory and replaced by fresh layers of the model. The size of the window, in terms of the number of layers it encompasses, is varied during execution (e.g. during training or in some circumstances inference). The variance in the size of the window is based on at least one of several factors, including the duration it takes to load a layer, offload a layer and the processing time associated with a layer. For example, the smallest possible window size is allocated that meets a plurality of invariants specifying the relationships between these durations, so as to minimise accelerator memory usage whilst avoiding stalling caused by waiting for layers to be loaded or offloaded.

[0033] FIG. 1 illustrates a computer system 100 according to an example of the disclosure. The computer system 100 comprises a processor 110, volatile memory 120, storage 130 and a work accelerator subsystem 200.

[0034] The processor 110 may take the form of a central processing unit (CPU), which is general purpose in nature. The CPU may be any suitable processor, including a multi-core processor. The volatile memory 120 may be random access memory, and is configured to transiently store any suitable data for operation of the system 100, including instructions for execution by the CPU 110. The storage 130 comprises any suitable storage media for non-transient (i.e. permanent) data storage, and may encompass suitable hard disk drives, solid state drives and so on. In some examples,the storage 130 may encompass secondary storage devices, such as the NVIDIA® NVMe SSD (solid state drive) or the like.

[0035] The components 110, 120, 130 are components of the host computer system 100, as distinct from the components of the work accelerator subsystem 200. Accordingly, they may be referred to herein as the host processor 110, host memory 120 and host storage 130 to distinguish them from corresponding components of the work accelerator 200.

[0036] The work accelerator subsystem 200 is a subsystem which processing can be offloaded to by the host system 100. In the context of the present disclosure, elements of the processing of (e.g. the training of) a machine learning model are offloaded to the work accelerator subsystem 200. The work accelerator subsystem 200 includes specialised (i.e. special purpose) hardware that renders it suitable for carrying out the offloaded processing in an accelerated manner, when compared to the general purpose CPU 110.

[0037] An example of a suitable work accelerator subsystem 200 is a graphics processing unit (GPU). Whilst GPUs were initially designed for accelerating computer graphics and image processing, such GPUs have found great utility in the training of machine learning models. For example, the accelerator subsystem may be an NVIDIA® V100 GPU or an NVIDIA A10 GPU, or any other suitable GPU provided by NVIDIA or any other supplier.

[0038] Other example work accelerator subsystems 200 are possible and contemplated within the scope of the disclosure. Other example work accelerator subsystems 200 include Tensor Processing Units (TPUs) provided by Google®, Deep Learning Processor Units (DPUs) provided by AMD® and so on.

[0039] Whatever the particular accelerator subsystem 200 used, the accelerator subsystem 200 includes processing hardware 210 and a local memory 220, hereinafter referred to as an accelerator processor 210 and accelerator memory 220. In general, the memory capacity of the accelerator memory 220 will be substantially smaller thanthat of the host memory 120. For example, whereas the RAM 120 of the host computer may be as large as 512GB or 1024GB, the size of the available local memory may be 100GB or less, such as 80 GB, 32GB or 16GB. As discussed further hereinbelow, examples of the disclosure are able to make efficient use of the relatively small amount of available accelerator memory 220.

[0040] The host system 100 and its accelerator subsystem 200 may be collocated. That is to say, the host system 100 and its accelerator 200 may effectively form one computer system, located in a single chassis or housing. For example, the accelerator 200 subsystem may be disposed on a suitable expansion card (e.g. a PCIe card) in the chassis, or directly mounted to a motherboard on which the host processor 110 is located. The subsystem 200 and host system 100 may be connected by a suitable bus.

[0041] The host system 100 further includes a prefetching and offloading module 140, which may be a software module (e.g. a low-level runtime library) configured to load data between memory 120 and memory 220 as discussed below.

[0042] The system 100 is a simplified example version of system in accordance with the disclosure to illustrate the concepts discussed herein. It will be understood that in reality, a single host computer 100 may host a plurality of accelerator subsystems 200. Equally, each accelerator subsystem 200 may comprise multiple processors 210 - e.g. a GPU may comprise a plurality of streaming multiprocessors configured to operate in parallel. Parallelism in the context of the present techniques is discussed in more detail later with respect to FIG. 6.

[0043] Turning now to FIG. 2, a brief discussion follows of common concepts in the execution of machine learning models. FIG. 2 schematically illustrates the structure of an example machine learning model 300, of the sort processed by the example techniques discussed herein. The model 300 is a layered model, such as a deep neural network. The model 300 comprises a plurality of layers, generally indicated by reference numeral 301. The layers 301 are arranged in sequence from a first layer 302 or input layer to a final layer 305 or output layer, such that each layer 301 receivesinput from the preceding layer 301 and provides output to the subsequent layer 301. In the context of a neural network, each layer 301, may comprise a plurality of artificial neurons, which receives a plurality of inputs from the preceding layer. Each input has an assigned weight, and the output is calculated as a function of the weighted input, referred to as an activation function. The task of training the neural network involves determining the weights by minimising a suitable loss function based on training data. The weights are an example of model data (also referred to herein as model states or model parameters) of the model 300.

[0044] In the example of FIG. 2, the model 300 is a simplified version of a transformer deep neural network. The first layer 302 is an embedding layer, which generates a vector in an embedding space. The layers 303-1 to 303-n are transformer blocks, which each comprise an attention layer and feedforward layer (both not shown). The layer 304 is a normalisation layer, and layer 305 is a pooling layer.

[0045] It will be understood that this is merely an example of a model architecture to which the present techniques apply, which has a relatively straightforward layer structure in which layers feedforward. The techniques are not limited to transformer neural networks, and may be applied to substantially any layered machine learning model. This includes networks that have residual components or gating mechanisms such as mixture of experts (MoE) models.

[0046] As illustrated in FIG. 2, the training process comprises 3 phases, each of which is typically iterated millions of times when training a large DNN, across multiple training epochs, where in each epoch uses the entirety of the training data set.

[0047] In each iteration, there is a forward propagation (FP) phase, in which a batch of training data is passed in a forward direction through the model 300 (i.e. from the first layer 302 to the last layer 305). A loss is computed which reflects the error in the output of the model 300, using an objective function. This is then followed by a backward propagation (BP) phase, in which the loss value is propagated back through the model 300 (i.e. in the opposite direction to the FP), to compute gradients. Thegradients are vectors representing the derivative of the loss function with respect to the weights. Finally, in the third phase, an optimiser uses the aggregated gradients to update weights of individual model layers. Example optimisers include stochastic gradient descent (SGD) or the well-known Adam optimiser.

[0048] During this process, the memory consumption largely stems from model states and residual states. Examples of model states include the aforementioned weights, as well as gradients and optimiser states. Examples of optimiser states include momentums and variances for use in the update of parameters. Residual states may include activations (i.e. the intermediate tensors saved for the BP phase to produce gradients) and other temporary buffers. For example, when training large DNNs such as model 300, model states in particular dominate memory consumption, which in some example experiments which do not implement the techniques discussed herein account for 87.5% of accelerator memory footprint.

[0049] FIG. 3A-B illustrates in outline a technique in accordance with the disclosure for efficient usage of the accelerator memory 220, which may be carried out by prefetching and offloading module 140.

[0050] In overview, the technique reduces the pressure on accelerator memory 220 by dynamically pre-fetching and loading model states associated with only a subset 310 of the layers 301 into accelerator memory from host memory 120. Once execution of a layer 301 is complete, the model states associated with the complete layer 301 are offloaded back to the host memory 120, and replaced with model states associated with a subsequent layer 301. References to loading or storing layers herein should be understood to be a reference to loading or storing the model states associated with that layer, which are required to carryout the processing of that layer.

[0051] The subset 310 of layers 301 held in accelerator memory 220 is referred to herein as a "window" or "working window". The window is a sliding window, in the sense that it notionally slides across the layers 301 of the model 300 as the model 300 is processed.

[0052] The number of layers contained in the window 310 (i.e. the number of layers having model states held in accelerator memory at any one time), referred to herein as the size of the window, is selected dynamically during execution, as discussed further below. This occurs in a manner that means that the overhead in terms of time taken to load in and out layers forming the working window is at least partially (but in some instances wholly) hidden from the accelerator 200, so that the relevant data for executing a layer 301 is present in the accelerator memory 220 at the time it is required, avoiding any stalling waiting for layers 301 to be loaded and whilst retaining the largest amount of layers 301 possible in the accelerator memory 220 to make best use of it. This means that the size of the model trainable by the accelerator is not bounded by the capacity of accelerator memory 120, but instead by the system's 100 storage capacity.

[0053] Turning in more detail to FIG. 3A-B, the host memory 120 stores the model 300. The model 300 is for example loaded into the host memory 120 from storage 130. Whilst the example shows the whole of model 300 being stored in host memory 120, in some examples only a part ofthe model 300 is stored, eitherto reduce host memory 120 consumption, or because as discussed below the model 300 is distributed over multiple systems 100 and thus only a part ofthe model 300 is assigned to the host 100 and its accelerator 200.

[0054] FIG. 3A shows the contents of the accelerator memory 220 at a first time point. At the first time point, the window size is 4, and thus 4 layers 311a-d are stored in respective regions of the accelerator memory 220, having previously been loaded therein. The layers 311 in the window 310 are being processed in sequence. In the example shown, the processing is part of the FP, and so the layers 311 are being processed from left to right as shown in the figure.

[0055] The layer 311a is shaded to indicate that execution of this layer has already occurred. Consequently, as indicated by the arrow labeled with number 1, the layer 311a is offloaded from accelerator memory 220 to the host memory 120. The layer311a is then replaced with layer 311e, which is the subsequent layer outside of the window 310, as indicated by the arrow labeled with number 2.

[0056] Similarly, layer 311b is shaded to indicate that execution of this layer has already occurred. Consequently, as indicated by the arrow labeled with number 3, the layer 311b is offloaded from accelerator memory 220 to the host memory 120. The layer 311b is then replaced with layer 311f, which is the subsequent layer outside of the window 310, as indicated by the arrow labeled with number 4.

[0057] FIG. 3B shows the contents of the accelerator memory 220 at a later time point. At this time point, layers 311c and 311d have been processed, so they are offloaded to host memory 120, and replaced by the subsequent layers 311g, 311h held in host memory 120, as indicated by arrows 1-4 which represent the same operations as shown in FIG. 3A. The arrows between layers 311 in the accelerator memory 220 represent the order of execution of the layers 311. Accordingly, once the right-most layer 311d is processed, execution returns to layer 311e, which replaced earlier-stored layer 311a.

[0058] This process repeats throughout execution of the machine learning model 300, with subsequent layers replacing executed layers in the accelerator memory 220, until all layers have been processed. It will be understood in this schematic representation, the memory region storing each layer remains in position, with its content being replaced and with processing iterating around the window. This reflects to some extent the view from the perspective of the accelerator, which loops through the storage locations forming the window, whilst the host replaces the layers stored therein.

[0059] The loading in and out of layers from the accelerator memory 220 is carried out asynchronously. That is to say, the movement of data between accelerator memory 220 and host memory 120 is carried out independently of the processing of the layers 311 by the accelerator processor 210. For example, prefetching and offloading module 140 may be executed by the host processor 110, in a manner thatoverlaps with the processing carried out by accelerator processor 210. Accordingly, from the perspective of the accelerator processor 210, it simply processes the layers deposited in the memory 220.

[0060] FIG. 4A and 4B show application of the techniques discussed above to FP and BP respectively. In these figures, the processing of the layers 301 are shown from the perspective of the model as a whole, with the window 310 sliding over the layers of the model. However, the process is the same as that discussed above with relation to FIG 3A-B.

[0061] In FIG. 4A, during FP the window 310 is effectively sliding from left to right, with computed layers (e.g. layer 312) being offloaded from accelerator memory 220 to host memory 120 after computation. In this case the computation during FP includes determining the outputs of neurons in each layer based on the application of weights to outputs from the previous layer, using whatever activation function has been selected for that layer. Later layers (e.g. layer 313) are transferred into accelerator memory 220, thus entering the window 310 to replace the offloaded layers.

[0062] In more detail, before executing a layer, a hook function (“pre_forward” hook) is called which causes an asynchronous load operation to fetch the next layer immediately outside the current window from the host memory 120 to the accelerator memory 220. Then the accelerator processor performs the FP computation on the first layer in the current working window. At the end of computation of the first layer, another hook function, "post_forward” is called to cause the already computed layer to be transferred back to the host memory 120. In addition, the computation result is passed to the next layer in the working window, as input to that next layer for its subsequent computation. The window then moves right, towards the next successive layer of the DNN in the FP direction.

[0063] In FIG. 4B, the window 310 is sliding from right to left, with layers to the left (e.g. layer 316) being loaded in to replace offloaded layers (e.g. layer 314) from theright, after computation of such layers. In this case, the computation carried out by the accelerator is the calculation of gradients for each layer.

[0064] As also illustrated in FIG. 4B, the parameter update, indicated by arrow 315 based on the gradients is carried out at least partially, but preferably entirely, by the host processor 110 based on the data offloaded back into host memory 120. This means that the parameter update can also occur concurrently with gradient computation, increasing efficiency.

[0065] In more detail, before computing a layer in the in the working window, a pre_backward hook function is invoked which fetches the layer that is immediately outside the window in the BP direction (i.e. immediately to the left of the window). This function also causes an operation to asynchronously occur which moves the model parameters of the last computed layer in the working window to the host memory 120. This is followed by a call to the optimiser to update the model parameters by the host processor 110. Finally, the accelerator processor 210 computes gradients for the second-last layer of the working window. The host to accelerator data communications run concurrently with the accelerator computation as the window moves in the BP direction.

[0066] The examples discussed in FIG. 3A-4B are to some extent predicated on knowledge of the order of execution of the layers. For most DNNs that is straightforward - the layers are ordered sequentially in order of execution, which can be extracted from a computational graph of the model. This may form a preprocessing step carried out before the above techniques, to derive the sequence of layers to be computed.

[0067] However, as noted above, there are models with residual components or gating mechanisms, where the execution path may change at execution time. For such models, the module 150 may offload all layers directly connected to a branch of execution, or may delay the layer movement until it determines which of the connected layers will be computed next (i.e. when the execution path is actuallychosen). In other examples, techniques are employed which precompute the activated layers to proactively determine which layers are to be moved to the accelerator memory 120. In these cases, whilst some pre-processing may be carried out, the decision as to which layer is next in the execution order is to some extent calculated dynamically during processing of the model.

[0068] In terms of calculating the window size, a plurality of durations are relevant. In particular, at least four durations are relevant for calculating the window size. Firstly, there is the duration required to load a layer from host memory 120 to accelerator memory 220, TtoAcceierator. Secondly, there is the duration required to perform the computation of the layer at the accelerator 200, TCompAcceierator. Thirdly, there is the duration required to offload the computed layer back to host memory 120, TtoHost. Finally, there is the duration TOveriapping, which represents the extent to which (in terms of time), the data transfer overlaps with the accelerator computation.

[0069] Based on these durations, a plurality of invariants - i.e. rules or requirements - are defined that are used to determine the window size. The invariants are as follows:> > < >

[0070] where m is the size of the window. The window size is selected with the smallest possible m which meets the invariants.

[0071] Invariant (1) enforces the requirement that the time taken to transfer new layers into the accelerator memory 220 for computation must be less than or equal to the time taken to compute the layers in the window. Invariant (2) enforces the requirement that the that the time taken to transfer the computed layers to the host memory 120 must be less than or equal to the time taken to compute the layers. Invariant (3) enforces the requirement that the sum of the time taken to transfer layers in and out of accelerator memory 220, minus the time by which those two operations overlap the computation, must also be less than or equal to the time taken to compute a layer.

[0072] Taken together these requirements effectively mean that the time taken to swap in and out layers from accelerator memory 120 must be less than the time taken to compute a layer, such that a new layer is present in accelerator memory 120 when the computation of a layer has taken place. The transfers therefore take place in the "shadow" of the computation, so that the computation is not stalled or otherwise affected by the transfer of layers.

[0073] In addition, the size of the window may also be constrained by the available memory space in the accelerator memory 220 - i.e. a window size cannot be selected that exceeds the amount of available memory. Available memory in this sense refers to the available memory at runtime. This may form a fourth invariant.

[0074] The durations discussed above may be determined during one or more initial warm-up iterations of the model, in which the data transfer time and GPU consumption for each layer in the model is profiled. In other examples, the durations may be estimated based on the transfer time or computation time associated with other similar layers or similar models.

[0075] Furthermore, the durations discussed above differ during FP and BP. That is to say, TtoAcceierator, T(ompAcceieraf-or, Tf-0i0Sf- and Toveriapping are all likely to have different values in FP and BP. This notation is therefore effectively a shorthand for two separate sets of durations, those occurring during FP and those occurring during BP.Depending on the phase of execution (i.e. whether the model is currently carrying out forward or backward propagation), the relevant durations for that phase are used in the invariants.

[0076] Example constraints that may be applied in FP and BP are now discussed in more detail.

[0077] In some examples, the constraints can be grouped into "hard" constraints and "soft" constraints. The hard constraints are those that, if violated, mean that additional overhead may be incurred, such that the time taken in loading the layers in and out is no longer hidden by the shadow of the computation. In contrast, the soft conditions are those that can be used to find the optimal size of the working window.

[0078] Example hard constraints for FP are as follows:

[0079] tflp> tJc2gi,j E [0, L), i = j - 1 (4a)

[0080] tflp> tgJ2ci,j G [0,L),i =j + l (4b)

[0081] S}p> SfreeVi G [0, L) (4c)

[0082] The terms (4a) and (4b) effectively correspond to the constraints (1) and (2) discussed above. Particularly, term (4a) specifies that the execution time (t / p) of a given layer / must be longer than or equal to the loading time of the parameters of the next layer j from CPU to GPU memory (tC2g). This allows the FP of the subsequent layer j to be performed immediately upon completion of the FP of layer / , without stalling by waiting for the layer to be fetched.

[0083] Term (4b) defines that the execution time of the layer / must be longer than or equal to the time required to offload parameters of the previous layer (tg2c). Note that in term (4b) the layer j is the preceding layer rather than the subsequent layer as in term (4a).

[0084] Term (4c) defines that the current available (i.e. usable) GPU memory Sfreeis greater than or equal to the space occupied for the parameters used in the FP of the current layer (S / p). In each of the terms, L refers to the set of layers.

[0085] These three terms are defined in respect of an individual layer rather than a sum across the window, and together form a robust offloading mechanism where the size of the working window is 1. In some examples these conditions may be too strict to be met in normal scenarios, though may be possible to apply to a working window size of 1 in circumstances where the connectivity between the host and accelerator permits.

[0086] The soft constraints are used to find the optimal working window size. There may be many acceptable window sizesthat meet the soft constraints. In one example, the window size N is selected as the smallest possible window size that meets the soft constraints, i.e. N = min ({m0, mi, ...,mn}) . In effect, this ensures that the most amount of GPU memory is saved whilst avoiding delays caused by the load in and out of layers.

[0087] Example soft constraints for FP may be as follows:

[0088] Let I = {0,1,2, ..., L-l} be an index set.

[0089] N = min(m G I | (5a) (5b) (5c) hold) (5)><> <

[0093] Term (5a) expresses a requirement that the data movement from CPU to GPU occurs without overhead for the given window size. Term (5b) ensures that there is sufficient GPU memory available to be able to fetch the parameters of another layer from CPU memory, in addition to those already in GPU memory, without causing anOOM (out-of-memory) error. Term (5c) ensures that the used GPU memory can be recycled quickly to further reduce the memory allocation. This is done by enforcing the requirement that the processing time should be greater than or equal to the maximum of the time to load in and load out the layers in the window from memory.

[0094] Turning now to BP, similar constraints are imposed to ensure the gradient movement does not become a bottleneck and affect offloading performance. Example hard constraints similar to those discussed above in respect of FP are shown below:

[0095] tblp> tJc2gVi,j e [0,L), i = j - 1 (6a)

[0096] tblp> tgJ2cVi.j G [0, L), i = j + 1 (6b)

[0097] Sbp> SfreeVi G [0, L) (6c)

[0098] Term (6a) ensures that the time for loading the parameters from CPU memory to GPU memory is less than or equal to the time for executing the backpropagation of the layer. This hides the overhead of offloading and ensures the BP does not stall waiting for a layer to be loaded.

[0099] Term (6b) ensures that the layer's parameters can be offloaded from GPU memory to CPU memory once the BP of the layer has finished. The data movement time is shorter than the previous layer's computation.

[0100] Term (6c) ensures that the current available memory can load the entire parameters of the layer.

[0101] Example soft constraints for BP may be as follows:

[0102] Let I = {0,1,2, ..., L-l} be an index set.

[0103] N = min(m G I | (7a) (7b) (7c) hold) (7)

[0105] ™0Sblp+ 5 < SfreeVi G [0,L - m) (7b)>

[0107] These constraints operate similarly to the ones discussed above in respect of FP, with the exception that term (7c) sums to m-1 on the basis that the extra operations are triggered in the pre hook function, before the backward propagation of the layer is executed.

[0108] Figure 5 illustrates an example technique for executing a machine learning model in line with the examples herein.

[0109] In a first step S501, a window size m is determined that represents a number of layers of the model to be loaded into accelerator memory 220.

[0110] In a second step S502, model data associated with a subset of layers of the model is loaded from the host memory 120 into the accelerator memory 220. The subset has the size m. Accordingly, a window of layers is loaded into memory 220. The model data may include model states such as weights, gradients and optimiser states, with the precise model data depending on whether the layers are forming part of FP or BP.

[0111] This may involve allocating an area of accelerator memory 220 based on the size of the subset of layers, into which the window of layers is loaded. This area of accelerator memory - referred to as a memory window as distinct from the window of layers - may be the same size as the subset of layers. In other words, the allocated memory window is of the same size as the model data associated with the layers in the window of layers. However, in some examples it is desirable to allocate a memory window that is larger than the size of the model data of the layers. For example, the expected size of each layer may not be completely accurate, and thus allocating a slightly larger memory window may act as a precaution to prevent Out-Of-Memory (OOM) errors. Equally, the size of a layer may vary at runtime, depending on which parameters are active. Accordingly, allocating a memory window larger than theexpected size of the window of layers provides a margin of error that reduces the likelihood of an OOM error.

[0112] In steps S503-507 the model is iteratively executed layer-by-layer based on the data in accelerator memory governed by the size of the working window.

[0113] Accordingly, in step S503, the current layer is processed to provide output data for that layer, which may be in the form of activations or gradients depending on whether the processing is FP or BP. The output data is stored in the accelerator memory (step S504) before being offloaded to the host memory (S505).

[0114] At this stage, with computation of the current layer being complete, model data currently stored in the accelerator memory associated with a previously processed layer is replaced with the next layer of the machine learning model that is not currently in the accelerator memory (step S506). This previously processed layer will typically be the oldest layer still in the accelerator memory 120. Consequently in step S507, the window slides to the next position, such that the next layer within the window becomes the current layer for processing.

[0115] The process then returns to step S503 and repeats.

[0116] Step S508 indicates that during the execution of the model (i.e. during the steps S503-S507), the window size is updated. The updated window size can be determined based on a transfer duration for loading model data associated with a layer into the accelerator memory from the host memory, a transfer duration for loading model data associated with a layer into host memory from the accelerator memory and a processing time associated with a layer. For example, as discussed above, the updated window size may be determined by determining the minimum window size m that satisfies the invariants (l)-(3). In other examples, other invariants such as those set out in terms (4)-(7) above may be taken into account.

[0117] In other examples, a window size that satisfies the invariants but is not the minimum size may be selected. For example, the maximum size that meets theinvariants may be selected on the basis that it may be desirable to occupy more of the available GPU memory, or some other value between the minimum and maximum sizes.

[0118] As a consequence, the number of layers stored in accelerator memory is updated based on the updated window size. If the updated window size is larger than the current window size, this will involve loading more layers into the accelerator memory 220. It may also involve increasing the memory window allocated in the accelerator memory 220. Conversely, if the updated window size is smaller, this may involve reducing the number of layers stored in accelerator memory 220. This may be accomplished by simply deleting the stored layers or releasing the allocation of memory associated with the excess layers beyond the new smaller window size.

[0119] In some examples, the step S508 of updating the window size is carried out at each iteration of the model. In other words, in each iteration the window size is calculated for the following window. This has the advantage of assuring that the optimal window size is always selected, such that the largest number of layers can be stored in the accelerator memory 220 without stalling the accelerator processor 210.

[0120] However, the calculation comes with some overhead, and in circumstances where the amount of model data and the computation time of the layers is similar (e.g. when processing a series of layers that have similar structure) this level of repeated window size alteration may be unnecessary. Accordingly, in other examples, the window size is updated periodically. This may be every n iterations of the model, where n is for example one of 2, 5, 7, 10, 15, 20, 25, 50 or any other suitable number.

[0121] The discussion above has largely focused on training of machine learning models such as deep neural networks. However, the techniques may also be applied at inference time, where an input is passed through the model to provide an output in the same manner as in FP during training.

[0122] FIG. 6 illustrates the application of the present techniques to distributed machine learning model training. Particularly, the techniques discussed herein meanthat the entire model can effectively be fit in accelerator memory 120. Accordingly, the need for model parallelism techniques, which divide up the layers of the model between different to make the best use of memory across multiple devices, may be reduced or removed. Consequently, a data parallelism approach may be adopted, in which different devices retain a copy of the entire model, but have different subsets of the training data.

[0123] In FIG. 6, a system 500 includes a plurality of hosts 100-1, 100-2, 100-n, each with their respective accelerators 200-1, 200-2, 200-n, which are configured to operate as discussed herein. A central resource 400 (e.g. a coordinating computer system) comprises a storage 401, which stores the model 300 and a plurality of training data items 402. Copies of the model 300 and subsets of the training data 402 are distributed to each host 100, which train their copy of the model 300 on their respective subset of the training data 402. The results of the training (i.e. the model data produced by the training process) are passed back to the central resource 400, which combines them to produce the trained model 300.

[0124] The mechanics of such data parallelism are well known, and suitable communication primitives for enabling this functionality (e.g. allreduce etc) is provided by implementations of the Message Passing Interface (MPI) standard.

[0125] In reality, one or more of the hosts may act as the central resource coordinating the parallel processing. Furthermore, rather than distribution amongst different hosts, the distribution may be amongst different CPU cores of hosts.

[0126] Advantageously, the above-described examples provide an improved means of managing accelerator memory during execution of a machine learning model. The techniques make optimal use of available accelerator memory, enabling the execution of very large neural networks that otherwise would need to be executed over multiple processing nodes. These techniques enable the use of fewer instances of expensive accelerator hardware to train large models with billions of parameters. By dynamicallyadjusting the size of the window during execution, a window of optimal size is maintained, ensuring optimal memory management.

[0127] For completeness, FIG. 7 schematically shows a non-limiting example of a computing system 1200 that can enact one or more of the methods and processes described above. Computing system 1200 is shown in simplified form. Computing system 1200 may take the form of computer system 100 or 400 described above, or any other computer device discussed herein. Computing system 1200 may take the form of one or more personal computers, server computers, and / or other computing devices.

[0128] Computing system 1200 includes a logic processor 1202, volatile memory 1204, and a nonvolatile storage device 1206. Computing system 1200 may optionally include a display subsystem 1208, input subsystem 1210, communication subsystem 1212, and / or other components not shown in FIG. 7.

[0129] Logic processor 1202 includes one or more physical devices configured to execute instructions. For example, the logic processor may be configured to execute instructions that are part of one or more applications, programs, routines, libraries, objects, components, data structures, or other logical constructs. Such instructions may be implemented to perform a task, implement a data type, transform the state of one or more components, achieve a technical effect, or otherwise arrive at a desired result.

[0130] The logic processor may include one or more physical processors (hardware) configured to execute software instructions. Additionally or alternatively, the logic processor may include one or more hardware logic circuits or firmware devices configured to execute hardware-implemented logic or firmware instructions. Processors of the logic processor 1202 may be single-core or multi-core, and the instructions executed thereon may be configured for sequential, parallel, and / or distributed processing. Individual components of the logic processor optionally may be distributed among two or more separate devices, which may be remotely locatedand / or configured for coordinated processing. Aspects of the logic processor may be virtualised and executed by remotely accessible, networked computing devices configured in a cloud-computing configuration. In such a case, these virtualised aspects are run on different physical logic processors of various different machines, it will be understood.

[0131] Nonvolatile storage device 1206 includes one or more physical devices configured to hold instructions executable by the logic processors to implement the methods and processes described herein. When such methods and processes are implemented, the state of nonvolatile storage device 1206 may be transformed — e.g., to hold different data.

[0132] Nonvolatile storage device 1206 may include physical devices that are removable and / or built-in. Nonvolatile storage device 1206 may include semiconductor memory (e g., ROM, EPROM, EEPROM, FLASH memory, etc.), and / or magnetic memory (e.g., hard-disk drive), or other mass storage device technology. Nonvolatile storage device 1206 may include nonvolatile, dynamic, static, read / write, read-only, sequential-access, location-addressable, file-addressable, and / or content-addressable devices. It will be appreciated that nonvolatile storage device 1206 is configured to hold instructions even when power is cut to the nonvolatile storage device 1206.

[0133] Volatile memory 1204 may include physical devices that include random access memory. Volatile memory 1204 is typically utilised by logic processor 1202 to temporarily store information during processing of software instructions. It will be appreciated that volatile memory 1204 typically does not continue to store instructions when power is cut to the volatile memory 1204.

[0134] Aspects of logic processor 1202, volatile memory 1204, and nonvolatile storage device 1206 may be integrated together into one or more hardware-logic components. Such hardware-logic components may include field-programmable gate arrays (FPGAs), program- and application-specific integrated circuits (PASIC / ASICs),program- and application-specific standard products (PSSP / ASSPs), system-on-a-chip (SOC), and complex programmable logic devices (CPLDs), for example.

[0135] The terms "module," "program," and "engine" may be used to describe an aspect of computing system 1200 typically implemented in software by a processor to perform a particular function using portions of volatile memory, which function involves transformative processing that specially configures the processor to perform the function. Thus, a module, program, or engine may be instantiated via logic processor 1202 executing instructions held by nonvolatile storage device 1206, using portions of volatile memory 1204. It will be understood that different modules, programs, and / or engines may be instantiated from the same application, service, code block, object, library, routine, API, function, etc. Likewise, the same module, program, and / or engine may be instantiated by different applications, services, code blocks, objects, routines, APIs, functions, etc. The terms "module," "program," and "engine" may encompass individual or groups of executable files, data files, libraries, drivers, scripts, database records, etc.

[0136] When included, display subsystem 1208 may be used to present a visual representation of data held by nonvolatile storage device 1206. The visual representation may take the form of a graphical user interface (GUI). Because the herein described methods and processes change the data held by the nonvolatile storage device, and thus transform the state of the nonvolatile storage device, the state of display subsystem 1208 may likewise be transformed to visually represent changes in the underlying data. Display subsystem 1208 may include one or more display devices utilising virtually any type of technology. Such display devices may be combined with logic processor 1202, volatile memory 1204, and / or nonvolatile storage device 1206 in a shared enclosure, or such display devices may be peripheral display devices.

[0137] When included, input subsystem 1210 may comprise or interface with one or more user-input devices such as a keyboard, mouse, touch screen, or game controller.

[0138] When included, communication subsystem 1212 may be configured to communicatively couple various computing devices described herein with each other, and with other devices. Communication subsystem 1212 may include wired and / or wireless communication devices compatible with one or more different communication protocols. As non-limiting examples, the communication subsystem may be configured for communication via a wireless telephone network, or a wired or wireless local- or wide-area network. In some embodiments, the communication subsystem may allow computing system 1200 to send and / or receive messages to and / or from other devices via a network such as the internet.

[0139] Although at least some aspects of the embodiments described herein with reference to the drawings comprise computer processes performed in processing systems or processors, aspects of the invention also extend to computer programs, particularly computer programs on or in a carrier, adapted for putting aspects of the invention into practice. The program may be in the form of non-transitory source code, object code, a code intermediate source and object code such as in partially compiled form, or in any other non-transitory form suitable for use in the implementation of processes according to aspects of the invention. The carrier may be any entity or device capable of carrying the program. For example, the carrier may comprise a storage medium, such as a solid-state drive (SSD) or other semiconductorbased RAM; a ROM, for example, a CD ROM or a semiconductor ROM; a magnetic recording medium, for example, a floppy disk or hard disk; optical memory devices in general; etc.

[0140] The examples described herein are to be understood as illustrative examples of embodiments of the invention. Further embodiments and examples are envisaged. Any feature described in relation to any one example or embodiment may be used alone or in combination with other features. In addition, any feature described in relation to any one example or embodiment may also be used in combination with one or more features of any other of the examples or embodiments, or any combination of any other of the examples or embodiments. Furthermore, equivalentsand modifications not described herein may also be employed within the scope of examples of the invention, which is defined in the claims. The labels "first", "second", "third" etc. herein are merely used to distinguish between similar elements and need not imply a particular ordering.

Claims

32CLAIMS1. A computer system for executing a machine learning model, the system comprising:a host processor;a host memory storing a plurality of layers of the machine learning model, each layer having corresponding model data; anda work accelerator processor and an accelerator memory;the system being configured to:determine a window size representing a number of layers to be loaded into accelerator memory;load model data associated with a subset of the plurality of layers, the subset being the same size as the determined window size, from the host memory into accelerator memory;execute the machine learning model by iteratively:processing, using the work accelerator processor, a current layer of the model based on the corresponding model data loaded into accelerator memory to provide output data for the current layer;storing the output data for the current layer in the accelerator memory;offloading the output data for the current layer to host memory;replacing model data for an already processed layer with model data corresponding to a next layer of the machine learning model outside the subset;moving to a next layer of the model within the subset, such that the next layer within the subset becomes the current layer;33during execution of the machine learning model, update the window size and in response cause the accelerator memory to store model data associated with a subset of the plurality layers having the updated window size;wherein the updated window size is determined based on at least one of a first transfer duration for loading model data associated with a layer into the accelerator memory from the host memory, a second transfer duration for loading model data associated with a layer into host memory from the accelerator memory and a processing duration associated with a layer.

2. The computer system of claim 1, wherein the updated window size is determined by:selecting a smallest possible window size meeting a plurality of constraints, the constraints based on the first transfer duration, the second transfer duration and the processing duration.

3. The computer system of claim 2, wherein the plurality of constraints include:a first constraint that a sum of the first transfer durations for the layers in the window is less than or equal to a sum of the processing durations for the layers in the window;a second constraint that a sum of the second transfer durations for the layers in the window is less than or equal to a sum of the processing durations for the layers in the window; anda third constraint that the sum of the first transfer durations added to the sum of the second transfer durations, minus a sum of an overlapping duration forthe layers in the window, the overlapping duration being representative of the duration forwhich the first and second transfer durations overlap with the processing of a layer, is less than the sum of the processing durations.

4. The computer system of claim 2 or 3, wherein the plurality of constraints include a fourth constraint that a memory size of the layers in the window does not exceed an available capacity of the accelerator memory.

5. The computer system of any preceding claim, wherein the system is configured to update the window size every n iterations of the execution of the machine learning model.

6. The computer system of any preceding claim, wherein the system is configured to update the window size in substantially every iteration of execution of the machine learning model.

7. The computer system of any preceding claim, configured to determine the first transfer duration, the second transfer duration and the processing duration during a warm-up iteration of execution of the machine learning model.

8. The computer system of any preceding claim, wherein:executing the machine learning model comprises forward propagation of input data through the machine learning model,the model data of the current layer includes weights and input data from a preceding layer; andthe output data for the current layer includes the activations of the current layer.

9. The computer system of any preceding claim, wherein:executing the machine learning model comprises backward propagation of a loss through the machine learning model, andthe output data for the current layer includes gradients.

10. The computer system of any preceding claim, wherein the host processor is configured to update parameters of the machine learning model upon receipt of the output data for the current layer at the host memory, whilst the work accelerator processor processes the next layer.

11. The computer system of any preceding claim, wherein the machine learning model is a deep neural network comprising at least one billion parameters.

12. The computer system of any preceding claim, wherein the work accelerator processor is a graphical processing unit, GPU, and the accelerator memory is built-in memory of the GPU.

13. A system comprising:a plurality of the computer systems of any preceding claim;36a coordinating system, comprising a storage storing the machine learning model and plurality of training data items, the coordinating system configured to:distribute a copy of the machine learning model and a respective subset of the training data items to each of the computer systems;cause each computer system to train the received copy of the machine learning model based on the respective subset of the training data to produce training results;receive the training results from each of the computer systems; andcombine the training results to produce a trained machine learning model.

14. A computer-implemented method of executing a machine learning model, the method carried out by a system comprising:a host processor;a host memory storing a plurality of layers of the machine learning model, each layer having corresponding model data; anda work accelerator processor and an accelerator memory;the method comprising:determining a window size representing a number of layers to be loaded into accelerator memory;loading model data associated with a subset of the plurality of layers, the subset being the same size as the determined window size, from the host memory into accelerator memory;37executing the machine learning model by iteratively:processing, using the work accelerator processor, a current layer of the model based on the corresponding model data loaded into accelerator memory to provide output data for the current layer;storing the output data for the current layer in the accelerator memory;offloading the output data for the current layer to host memory;replacing model data for an already processed layer with model data corresponding to a next layer of the machine learning model outside the subset;moving to a next layer of the model within the subset, such that the next layer within the subset becomes the current layer;during execution of the machine learning model, update the window size and in response cause the accelerator memory to store model data associated with a subset of the plurality layers having the updated window size;wherein the updated window size is determined based on at least one of a first transfer duration for loading model data associated with a layer into the accelerator memory from the host memory, a second transfer duration for loading model data associated with a layer into host memory from the accelerator memory and a processing duration associated with a layer.

15. A computer-readable storage medium storing instructions, which when executed, cause a computer system to implement the method of claim 14.

Images