Synchronising time across devices

Abstract

Methods and systems for synchronising time between devices of a system. A host device sends, to each of a plurality of target devices via a respective pathway, a message comprising a timestamp indicating a current global time at the host device. Each target device receives and processes the message, and determines whether the current global time indicated by the timestamp is ahead of or behind a current local time at the target device. If the current global time indicated by the timestamp is ahead of the current local time at the target device, the target devices updates the current local time at the target device to be the current global time indicated by the timestamp. If the current global time indicated by the timestamp is behind the current local time at the target device, the target device maintains the current local time at the target device.

Description

[0001] SYNCHRONISING TIME ACROSS DEVICES

[0002] TECHNICAL FIELD

[0003] The present disclosure relates to systems and methods for synchronising time across devices. For instance, the disclosed techniques may be used to enable devices to converge upon a common time.

[0004] BACKGROUND

[0005] In the field of artificial intelligence (Al), recent years and months have seen rapid advances in so-called large machine learning (ML) models, such as large language models (LLMs) and other foundational models. Such models typically take the form of artificial neural networks (neural networks or neural nets for short) having billions of parameters (weights) or more, requiring vast investments in computational resources. However, advances in the technology have relied on very similar hardware. Existing chips and highly developed tools and libraries are well -optimised for training large language models (LLMs), but they are unsuited to inference, which is the process of running live data (input tokens) through a specific model with learned parameters, to produce results (in LLMs, a series of output tokens). A token may take the form of a vector of numerical values.

[0006] As a consequence, Al models are very expensive to provision and run at scale. Issues such as time taken on conventional hardware to move model parameters from memory to processors mean that very expensive hardware is often used at a small fraction of its theoretical capability.

[0007] Moreover, Al performance is inhibited, as ever faster compute cannot make up for the performance lag caused in inference by moving model weights from memory to the processor units, limiting real-time performance and user experience.

[0008] Potential Al performance in the future is also restricted. Continual advancement of conventional computing is limited by the heat generated by these chips. There is a limit to how fast silicon chips can be cooled, and this has become a new constraint on continuing to scale conventional digital processors (the end of Dennard Scaling). With enough data, bigger Al models are predictably better, but without breakthroughs in compute systems, it will notbe possible to continue to scale Al models to be orders of magnitude larger with sufficiently low latency (time per output token, for instance) to be usable.

[0009] Moreover, with Al model developers all building on similar infrastructure and the balance of its use tilting heavily to inference, without novel hardware the opportunity to create long-term differentiation and competitive advantage from faster, cheaper and higher quality token generation in inference will be limited.

[0010] There are broadly two development paths available when building improved hardware for Al inference. The first is specialisation: honing-in on very specific workloads and building chips that are uniquely suited to those specific requirements. Because model architectures evolve rapidly in the world of Al, while designing, verifying, fabricating and testing chips takes considerable time, companies pursuing this approach face the problem of shooting for a moving target whose exact direction is uncertain.

[0011] The second path is to change the way that computational operations themselves are performed, create different chips from novel building blocks, and build scalable systems on top of these.

[0012] SUMMARY

[0013] Any system implementing an Al model will contain a large number of processing units that are required to send and receive data to / from one another. For example, compute units will need to output the result of a computation (e.g. a matrix-vector multiplication (MVM)) to another compute unit for performing another similar computation, and / or to a vector processing unit (VPU) for performing an activation function (e.g. a softmax calculation), or the like.

[0014] In such a system, data (e.g. the tokens mentioned above) may flow through the system via hundreds, if not thousands of different processors. Different processors may take different amounts of time to process data Since every processor may contribute to the performance of the overall system, it is important to be able to understand how the data is flowing through the system, to identify bottlenecks and to identify underutilized areas. In this context, knowing when events occur is just as important as knowing that they occurred when identifying where performance is being lost. Techniques described herein use timestamps generated by the processors to build a picture of the system, which can be used to identifyperformance issues and also for debugging purposes. In order to do so, it is necessary that time is synchronized across the processors, i.e. that they a common view of the time.

[0015] According to an aspect disclosed herein, there is provided a computer-implemented method of synchronising time between devices of a system comprising a host device configured to send data to a respective device via a respective pathway, the respective pathway comprising one or more intermediate components between the host device and the respective device. The host device is configured to maintain a global time and each device is configured to maintain a respective local time. The method comprises the host device sending, to each of a plurality of target devices via a respective pathway, a message comprising a timestamp indicating a current global time at the host device. The method further comprises each target device receiving and processing the message, and determining whether the current global time indicated by the timestamp is ahead of or behind a current local time at the target device. If the current global time indicated by the timestamp is ahead of the current local time at the target device, the method comprises updating the current local time at the target device to be the current global time indicated by the timestamp. If the current global time indicated by the timestamp is behind the current local time at the target device, the method comprises maintaining the current local time at the target device.

[0016] In embodiments, the method may comprise: the host device periodically or randomly sending respective messages to each target device, where each respective message comprises a respective timestamp indicating a current global time at the host device; and for each respective message, each target device: receiving and processing the respective message; determining whether the current global time indicated by the respective timestamp is ahead of or behind a current local time at the target device; and if the current global time indicated by the respective timestamp is ahead of the current local time at the target device, updating the current local time at the target device to be the current global time indicated by the respective timestamp; or if the current global time indicated by the respective timestamp is behind the current local time at the target device, maintaining the current local time at the target device. In embodiments, the system may comprise a plurality of sets of devices, each set of devices comprising a respective target device.In embodiments, the method may comprise, for one or more respective sets of devices: the target device of the respective set sending, to one, some or all other devices in the respective set, a message comprising a timestamp indicating the current local time at the target device. In embodiments, sending of the message may comprise a timestamp indicating the current local time at the target device is in response to the target device updating the current local time at the target device.

[0017] In embodiments, the method may comprise each target device sending, to the host device, a message comprising a timestamp indicating the current local time at the target device.

[0018] In embodiments, each respective pathway may comprise a common set of intermediate components.

[0019] In embodiments, one or more respective pathways may comprise a different set of intermediate components, and wherein the method may comprise: for each respective pathway, determining a minimum delay associated with receiving and processing a respective message; and determining whether the current global time indicated by the timestamp is ahead of or behind a current local time at the target device based on the minimum delay associated with the respective pathway.

[0020] In embodiments, the method may comprise one or more target devices initialising its local time based on a message comprising a timestamp indicating a current global time at the host device.

[0021] In embodiments, each timestamp may be a Unix timestamp.

[0022] In embodiments, the method may comprise one or more target devices updating its local time to ensure its local time is not ahead of the global time at the host device.

[0023] In embodiments, said updating may be performed periodically and / or in response to a request from the host device.

[0024] In embodiments, the system may comprise a plurality of processors, and wherein each device is a respective processor.

[0025] In embodiments, the system may comprise a plurality of tiles, each tile comprising a plurality of processors, and wherein each respective set of devices belongs to a same respective tile.In embodiments, each device may be one of: an xPU, a GPU, a VPU, an in-memory compute unit, a vector-matrix hardware multiplication unit.

[0026] In embodiments, the system may be configured to process data in a neural network.

[0027] According to another aspect disclosed herein, there is provided computer-readable storage media having program instructions stored thereon that, when executed by one or more processors operatively coupled with the one or more computer-readable storage media, direct a computing system comprising a host device and respective devices to perform the following operations. An operation involves the host device sending, to each of a plurality of target devices via a respective pathway, a message comprising a timestamp indicating a current global time at the host device. An operation involves each target device determining whether the current global time indicated by the timestamp is ahead of or behind a current local time at the target device. An operation involves updating the current local time at the target device to be the current global time indicated by the timestamp if the current global time indicated by the timestamp is ahead of the current local time at the target device. An operation involves maintaining the current local time at the target device if the current global time indicated by the timestamp is behind the current local time at the target device.

[0028] According to another aspect disclosed herein, there is provided a system comprising a host device configured to send data to a respective device via a respective pathway, the respective pathway comprising one or more intermediate components between the host device and the respective device. The host device is configured to maintain a global time and each respective device is configured to maintain a respective local time. The host device is configured to send, to each of a plurality of target devices of the system via a respective pathway, a message comprising a timestamp indicating a current global time at the host device. Each target device is configured to determine whether the current global time indicated by the timestamp is ahead of or behind a current local time at the target device. Each device is configured to update the current local time at the target device to be the current global time indicated by the timestamp if the current global time indicated by the timestamp is ahead of the current local time at the target device.

[0029] In some systems, trace analysis is only concerned with the events occurring on an individual device. In these cases the timestamps on the events need only be consistent for that device to ensure that the events are shown in the order that they occurred. However, when analysiscovers more than one device, the messages from the individual tiles are typically combined into one file. The merge process assumes that all the devices are on the same timebase. Some analysis will not give accurate results if the messages are not correctly aligned. This is especially true if a sequence of events should occur in a specific order.

[0030] The present disclosure provides a mechanism for ‘distributing time’ to devices of a system, which the devices can then use to converge upon a common view of the time. A host device, which maintains a global time / clock (i.e. UNIX time), sends timestamps to the devices. If a device receives a timestamp indicating a time ahead of its local clock, the device updates its clock to correspond to that of the future time indicated by the timestamp. On the other hand, if the device receives a timestamp which indicates a time slower than its local clock, the device ignores the timestamp and continues with its own view of the time. As will be described later, if each device operates according to this algorithm, the devices across the system will eventually converge on a common time.

[0031] This mechanism does not require any additional cabling or hardware in the system. Rather, it utilises the fact that the messaging delays in a system are random, and that there are a sufficient number of messages to converge on a common understanding of time.

[0032] The present disclosure also provides a mechanism for correcting timestamps in the time stamps generated by devices. This may also include correcting the local time maintained by those devices. In these embodiments, devices produce trace messages when performing actions (e.g. computations, such as MVM computations). The trace messages include timestamps, which can be used to determine an order of events occurring in the system, debug the system, identify performance issues in the system, etc.

[0033] It is important to note that whilst embodiments will primarily be described in terms of a system of devices processing data for implementing an AI / ML model, the present disclosure can be applied more generally to any system having a (large) number of devices where there is a need for time to be synchronized across those devices and / or to identify issues with those devices.

[0034] BRIEF DESCRIPTION OF FIGURES

[0035] Particular embodiments will now be described, by way of example only, with reference to the following schematic figures, in which:FIG. 1 schematically illustrates an example system, referred to herein as a “tile”, for processing data of a neural network or the like;

[0036] FIG. 2 schematically illustrates an example system comprising a plurality of tiles, referred to herein as a “die”;

[0037] FIG. 3 schematically illustrates an example system comprising a plurality of dies, referred to herein as a “chip package”;

[0038] FIG. 4 schematically illustrates an example network for inter-tile data transfer;

[0039] FIG. 5 schematically illustrates another example network for inter-die data transfer;

[0040] FIG. 6 schematically illustrates an example system for aligning time across devices;

[0041] FIG. 7 shows how messages can experience different communication path delays;

[0042] FIG. 8 schematically illustrates a local time of a device being updated according to some embodiments;

[0043] FIG. 9 shows an example set of trace data for a tile which is running ahead of other tiles; and

[0044] FIG. 10 shows an example set of trace data for a tile which is running behind other tiles.

[0045] DETAILED DESCRIPTION

[0046] One or more of the embodiments of the present disclosure may be used in a system such as those described below alongside FIGS. 1 to 5. For example, each device may be a computing unit 101 or other device of a tile 100, die 200, or chip packages (“chiplet”) 300. These components are described in more detail below.

[0047] FIG. 1 shows an example system 100 comprising a plurality of computing units 101, e.g., processors. In this example, the computing units 101 are arranged in a grid, where each computing unit 101 is connected to each other through a grid interconnect 102. This example system 100 will be referred to hereinafter as a “tile”. It will be appreciated that a tile 100 may comprise any number of computing units, and sixteen computing units are merely shown for illustrative purposes. The grid interconnect 102 is connected to a bus 103. The grid interconnect 102 is also connected to one or more network components 104, such as a network interface (see FIG. 4) and a network switch (see FIG. 4). Each computing unit 101 is configured to receive input data via the grid interconnect 102 and send output data via the grid interconnect 102. The bus 103 is configured to allow for communication between the various parts(components) within the tile, such as the computing units 101, the network components 104, etc. The bus 103 may be an advanced extensible interface (AXI) bus. In other examples, the bus 103 is another type of suitable bus. The network interface and network switch are described in more detail below, with reference to FIG 4. The tile 100 may comprise one or more routing channels 105 for routing data to components of the same tile 100 or elsewhere, e.g., to a different tile.

[0048] In some examples, each computing unit 101 is the same type of computing unit. In other examples, one or more computing units 101 may be a different type of computing unit. Each computing unit 101 may be one of the following: an in-memory compute (IMC) unit, an analogue in-memory compute unit (AIMC), a central processing unit (CPU), a graphics processing unit (GPU), a vector processing unit (VPU), an accelerator processor, or other suitable units. An IMC unit refers to a computing unit that stores data in internal memory and uses that data to perform computations. For instance, the stored data may be weights of a neural network, and the computations may be vector-matrix-multiplications. An AIMC unit is a particular type of IMC unit that performs computations in the analog domain.

[0049] FIG. 2 shows an example system 200 comprising a plurality of tiles 100. This example system 200 will be referred to hereinafter as a “die”. The tiles 100 are arranged in a grid, where each tile 100 is connected to one or more neighbouring tiles 100. Only four tiles 100 are shown in this example system 200 (labelled (0,0), (0,1), (1,0), and (1,1)), but it should be appreciated that a die 200 may contain any number of tiles 100 and will likely contain many more than four. The die 200 comprises a plurality of interfaces 201 (e.g., peripheral component interconnect express (PCIe) interfaces) and a plurality of interfaces 202 (e.g., universal chiplet interconnect express (UCIe) interfaces) configured to connect to (e.g., send and receive data to and from) other dies 200 (see e.g., FIG. 3). Any suitable format may be used to connect dies 200 together. In this example, UCIe is used to connect two dies that are packaged together into a single chip package, and PCIe is used to connect together dies that are within different chip packages. This is because UCIe has lower latency and works over a shorter distance than PCIe. Tiles are configured to send data to neighbouring tiles. In some examples, a tile 100 may send data to any tile on the same die. For instance, tile (1,0) may receive data from tile (0,0) and send data to tile (1,1). All other combinations may be possible. Tiles 100 may communicate by sending data via one or more of their respective grid interconnects 102, their respective routing channels 105, and their respective network components 104. For instance, tile (1,0) may send data to tile (1,1) via its grid interconnect 102, which may be connected directly to the gridinterconnect 102 of tile (1,1). Additionally or alternatively, tiles may be connected via the routing channels 105. The routing channels 105 may form a network-on-chip (NoC) interconnect, allowing data to be sent to any tile connected to the NoC interconnect. The network components 104 (e.g. network interface and network switch) may be used to facilitate inter-tile communication. For instance, the network switch of a given tile may be connected to the NoC routing channels 105 of neighbouring tiles.

[0050] In some examples, not only are tiles 100 configured to send data to neighbouring tiles (or any tile for that matter), but individual computing units 101 may be configured to send data to some or all computing units belonging to some or all computing units on different tiles. In other words, in some examples, a computing unit 101 may be configured to send data to any other computing unit 101.

[0051] FIG. 3 shows an example system 300 comprising a plurality of dies 200. This example system will be referred to hereinafter as a “chip package”, or a collection of connected dies 200. Each die 200 is connected to one or more other dies. A die 200 may be connected to another die via one or more interfaces (e.g., one or more UCIe interfaces). Only four dies 200 are shown in this example, but it should be appreciated that a chip package 300 may contain any number of dies 200. In the example of FIG. 3, a first die (e.g., the die 200 of FIG. 2) comprises four tiles 100 labelled (0,0), (0,1), (1,0), (1,1), a second die comprises four tiles labelled (2,0), (2,1), (3,0), (3,1), a third die comprises four tiles labelled (0,2), (0,3), (1,2), (1,3), and a fourth die comprises four tiles labelled (2,2), (2,3), (3,2), (3,3).

[0052] Dies 200 are configured to send data to other dies connected via a respective interface. For instance, the first die comprising tile (0,0) may send data to the second die comprising tile (2,0) and also to the third die comprising tile (0,2). In some examples, a die may also send data to a die that it is not directly connected to by routing the data via a connected die (or multiple dies, depending on the system configuration). For example, the first die comprising tile (0,0) may send data to the fourth die comprising tile (2,2) via either of the second or third dies that are shown.

[0053] In some examples, any tile 100 may route data to any other tile, either on the same die 200 or a different die. Similarly, in some examples, any computing unit 101 may route data to any other computing unit, whether the computing units be on the same or different tiles 100, or the same or different dies 200.FIG. 4 illustrates an example network 400 for inter-tile communication, i.e. the sending of data between tiles. This example depicts the network components of die 200 shown in FIG. 2, where the die 200 comprises four tiles 100. FIG. 4 is abstraction of FIG. 2, such that only the network components of the die 200 are shown. As described above, each tile 100 of a die 200 may comprise a network switch 401 and a network interface 402. A tile 100 may also comprise a PCIe interface 201 and / or a UCIe interface 202. For a given tile, the network switch 401 may be connected to the PCIe interface 201 via a routing channel 105, e.g. via a NoC interconnect 403. Similarly, the network switch 401 may be connected to the UCIe interface 202 via the routing channel 105, e.g., via a NoC interconnect 403. The network switch 401 is also shown connected to the network interface 402 via a NoC interconnect 403.

[0054] FIG. 5 illustrates another example network 500 for inter-die communication, i.e. the sending of data between dies 200 of a chip package 300. This example shows the network components of chip package 300 shown in FIG. 3 (abstracted to show only the network-related components), wherein the chip package 300 comprises four dies 200, and each die 200 comprises four tiles 100. As explained above, each die 200 of the chip package 300 may be connected to one or more other dies 200 via one or more UCIe interfaces 202, or other suitable interface. Each die 200 may be connected to a given die via multiple UCIe interfaces 202. This may be based on the number of tiles 100 on said die 200, e.g., a die of four tiles arranged in a 2x2 grid may be connected to another similarly arranged die via two interfaces 202.

[0055] Some embodiments described herein relate to the synchronizing of time between devices (e.g. processors) of a system. For example, the system may comprise or be a tile 100, a die 200, a chiplet 300, or any collection of these. A device may be any component of such a system, such as a computing unit 101. The devices may collectively perform operations for processing data as part of a neural network, such as matrix-vector-multiplications.

[0056] FIG. 6 illustrates an example system 600 for implementing these embodiments. In this example, the system comprises a host node 601 (e.g. a Unix host node), a switch 602 (e.g. an Ethernet switch), one or more intermediate nodes 603 (e.g. Unix nodes), one or more intermediate components 604 (e.g. cards). In this particular example, the host node 601 is connected to the switch 602, which is connected to multiple intermediate nodes 603, each of which is connected to an intermediate component 604 (e.g. system card). Each intermediate component 604 is then connected to one or more sub-systems, e.g. one or more packages 605(collections of chiplets), one or more chiplets 300, 606, one or more dies 200, one or more tiles 100. One or more tiles 100 may be referred to as trace tiles 607 (or “target devices”). It will be appreciated that this example is merely illustrative of a system having a large number of devices. As shown, each trace tile is connected to the host node 601 via a respective pathway (i.e. route, data path, etc.). Each pathway is made up of a set of components and is configured such that data can flow between the host node 601 and the trace tile, e.g. via each of the components. Whilst each pathway is shown as being made up of the same components, in some examples one or more pathways may have a different number and / or type of components.

[0057] The host node 601 is configured to maintain a clock (i.e. maintain a time). The clock may be configured to maintain the current Unix time and may be set according to software or according to a GPS installed on or connected to the host node 601. Either way, the time maintained by the host node 601 is considered to be the true time as far as the system is concerned.

[0058] One or more devices (e.g. each trace tile 100) of the system are configured to maintain a clock. The clock may be initialized by software or by a message sent by the host node 601 or a different component of the system. Over time, the local time according to any particular device may deviate from that of the host node 601. This can lead to different devices having a different view of time. In practice, this means that it is possible that at any particular point during operation of the system, some devices will be out-of-sync compared to others.

[0059] The following described techniques for synchronizing (i.e. aligning) the time of the devices throughout the system.

[0060] The host device 601 is configured to send a message to a set of devices, referred to herein as target devices or trace tiles. The host device 601 may send a single message (e.g. by broadcasting the same message to the target devices) or send separate messages, one for each target device. The (or each) message includes a timestamp corresponding to the current time at the host device 601, i.e. the time, according to the host device 601, at which the message is created or sent.

[0061] Each target device is configured to receive the message and extract the timestamp. The message be sent, from the host device 601 to the target device, via a respective pathway and may be transmitted via a large number of intermediate components of the system.Each target device is configured to compare the extracted timestamp with its own clock time and determine whether its own clock time is ahead of (i.e. in the future compared to) the timestamp sent by the host device 601 or behind (i.e. in the past compared to) the timestamp sent by the host device 601. Or put another way, each target device is configured to use the timestamp to detect when the host device 601 sends a timestamp indicating a time ahead of or behind the local time at the target device.

[0062] Each target device is configured to take one of two actions in response. If the target device determines that the timestamp indicates a future time, the target device updates its local time to align with that of the host device 601 (i.e. to match the timestamp). On the other hand, if the target device determines that the timestamp indicates a past time, the target device does not update its local time to match the timestamp and instead maintains its local time.

[0063] FIG. 7 illustrates the sending of messages to devices, and how different devices will receive messages at different times. In this particular example, a message is sent from the host device 601 to a trace tile via a PCIe interface. Initially, there is a delay (“Delay 1”) caused by the host node 601 reading its Unix time and writing a message containing a timestamp to a buffer. There is then a delay (“Delay 2”) caused by the buffer being read by the tile. There is then a delay (“Delay 3”) caused by the message being sent over the PCIe interface. Finally, there is a delay (“Delay 4”) caused by the tile processing the message. The delay in the tile receiving the timestamp from the host device 601 is a combination of a fixed minimum and a random delay. In the example of FIG. 7 the timestamp arrives at the different tiles after various delays.

[0064] Assuming that the delays are uncorrelated then, statistically, there will be times when a tile receives a message with the minimum delay. If the tile only updates its local time when it receives a time that is further in the future (i.e. closer to the real Unix time) then it will converge on the ‘best’ it can get. Updates that would 'push back' time are ignored. As each tiles follow the same process in parallel, over time they will all converge to the same time and therefore be aligned in time. If the minimum delay is known, then this can allow the Unix traces to be time aligned with the tile traces.

[0065] Previous techniques attempt to align time by sending and receiving messages over networks and dividing the loopback in half. However, this requires a lot of traffic (a ping-pong to every processor) as well as having a transmission latency that is consistent. Neither is practical in large systems.In some examples, the host device 601 may periodically send messages containing a current timestamp to the target devices. In other examples, the host device 601 may randomly send such messages. In other examples, every message (or at least every certain type of message) sent by the host device 601 to the target devices may contain a current timestamp.

[0066] In some examples, the system may be split (divided, separated, organised) into multiple sets of devices (e.g. multiple dies 200 or chiplets 300). Each set of devices may contain a target device.

[0067] In these examples, the target device may forward the message received from the host device 601 to other devices in the same set. Additionally or alternatively, the target device may send (e.g. broadcast) a message containing a timestamp indicating the local time at the target device to the other devices in the set. This may be done after updating its local time to align with the host device 601.

[0068] In some examples, each target device may report its current local time to the host device. This may be done periodically and / or upon request. This may be used to alert the host device 601 to issues in the system. This may trigger the host device 601 to send message(s) containing a timestamp indicating the current time at the host device 601.

[0069] As discussed above, the paths connecting the host device 601 to the target devices may be the same, or they may be different. In the case that the paths are different, the delay time to process a message sent by the host device 601 will vary for different target devices. To accommodate for this, a minimum delay caused by the additional devices in the path may be determined (e.g. calculated or estimated) and used to determine whether or not to update the local. For example, the minimum delay may be subtracted added to the timestamp sent by the host device 601.

[0070] FIG. 8 shows an example in which a tile uses the embodiment described above to update its local time. The actual Unix time maintained by the host device 601 is shown at the top of the figure running from left to right. The host device sends timestamps at three points in time: at time 0, at time 6, and at time 10. The timestamps are written to the buffer, processed by the PCIe and eventually read from the buffer by the tile. Random delays may occur. In this example, the tile initialises its time based on the first message sent by the host device 601. At a later time, the tile receives the second message from the host device, indicating a time 6. The local time at the tile is only 3, so the tile updates its local time to 6. The third message isreceived at a later point in time. The third message indicates the time is 10, which is behind the local time at the tile. In this case, the tile ignores this message and maintains the current local time.

[0071] As has been described above, the algorithm for synchronising time across devices ignores timestamps that would push time backwards. In other words, the target devices only update their local clock if a received timestamp is further ahead in time than its local clock. To guarantee that a timestamp that would push back time would only occur due to long delays rather than the clock drift of the target device’s local clock (e.g. real-time clock (RTC)), the local time must not run faster than Unix time. It can run the same or slower, but not faster. In some examples, this can be ensured either by tuning the local clock of the target device, or by executing software that causes the local clock to skip time, e.g. periodically.

[0072] In these examples, the target devices (e.g. tiles) will continue to receive Unix time. When their clocks drift, there will reach a point when a message with the minimum delay is received that is earlier than local time and the tile will skip forward by a ‘tick’ amount. This amount may be as small as 100ns depending on the system. The amount may be set to any amount. Note that as time is always just going forwards, it will remain monotonic.

[0073] In some examples, the host node 601 may detect errors where the local clocks have run faster than its time base by mistake. This may be done by detecting a trace message that has a time further in the future than Unix time. The sending and processing of trace messages is explained below.

[0074] Some embodiments of the present disclosure relate to correcting erroneous trace messages generated by devices. As discussed above, devices may generate trace messages which include timestamps indicating when the message was generated. For example, a device (e.g. a computing unit 101, tile 100, etc.) may generate a trace message each time a computation (e.g. calculation) is performed, each time a message is sent to another device (e.g. sending a result of a computation, or sending a request for data, etc.). The trace message may include both the timestamp and related data (e.g. a result of a computation), or the trace message may include the timestamp and an identifier (or some other indication) of the associated event, where an event may be a calculation, request, action, etc.

[0075] In some embodiments, the host device 601 (or more generally, a controller of the system, which may be implemented in hardware (e.g. as a CPU) or software, or as a combination)may have knowledge of an execution flow (or graph) for the system, or at least for a computation to be performed by the system. The computation may be made up of a series (sequence) of sub -computations to be performed in order. The sub-computations are performed by a plurality of devices, where each device performed one or more subcomputations. Some computations may be performed in parallel by different devices. Some devices may perform the same computation on different input data. The controller may collate trace messages generated by the devices and use this knowledge of the computation to correct situations when the devices are not synchronised in time. That is, the controller may detect when the devices do not have a common view of the time and modify timestamps accordingly, e.g. for debugging purposes.

[0076] Each time a device executes a sub-computation (e.g. an MVM), the device generates a trace message which includes a timestamp indicating the time at which the sub-computation was executed. The timestamp may indicate the time at which the sub-computation was initially executed and / or finished executing. Each trace message may include an identifier of the device that generated the message. Each trace message may include an identifier of the associated sub-computation.

[0077] The trace messages are sent to the controller in any suitable manner. The controller receives the trace messages from the devices and processes the trace messages. This may include extracting the timestamp from the trace message. Other data may be extracted, such as the device and / or sub -computation identifier.

[0078] The controller uses the timestamps to identify one or more devices that have (or at least appear to have) an error in the local time maintained at the device. These devices will be referred to hereinafter as “target devices”. For example, the controller may use knowledge of the execution flow / graph to identify timestamps that appear erroneous. For example, the controller may determine that a timestamp associated with sub-computation A was performed before sub -computation B whereas, according to its knowledge, sub-computation A should have (or even must have) been performed after sub-computation B. The device that generated the timestamp associated with sub -computation A may be identified as a target device. As another example, the controller may determine that a timestamp associated with subcomputation A was performed after sub -computation B whereas, according to its knowledge, sub -computation A should have (or even must have) been performed before sub-computation B. The device that generated the timestamp associated with sub-computation A may beidentified as a target device. As another example, the controller may determine that a timestamp associated with sub -computation A was performed before or after subcomputation B whereas, according to its knowledge, sub -computation A should have (or even must have) been performed at the same time as (i.e. in parallel with) sub-computation B. The device that generated the timestamp associated with sub -computation A may be identified as a target device.

[0079] In some examples, the controller may determine that a device has an error in its local time by detecting a timestamp associated with a sub -computation differs from one or more timestamps generated by other devices (e.g. the majority of other devices) for a subcomputation that was (or at least should have been) performed at the same time. In these examples, the controller may allow a threshold amount of difference between timestamps issued by the different devices to account for minor differences in processing / executing a sub -computation and / or differences in data and / or instructions required to execute the subcomputation reaching the different devices. That is, the controller may only determine that the device has an error if the timestamp differs from that of the other devices by more than a threshold amount.

[0080] Having identified a target device having an error in its local time, the controller may take one or more of several actions.

[0081] The controller may adjust (i.e. correct) one or more erroneous timestamps. To do this, the controller may determine a minimum time offset by which the target device is incorrect, and adjust the erroneous timestamps accordingly, e.g. by subtracting or adding the time offset to the timestamp. The minimum offset may be determined based on the difference between the timestamp and one or more timestamps generated by other devices. For example, in the case that the controller determines that a timestamp associated with sub -computation A was performed before sub-computation B whereas, according to its knowledge, sub -computation A should have (or even must have) been performed after sub-computation B, the minimum time offset may be determined as the difference between the two timestamps (i.e. the timestamps for sub-computations A and B). The same may apply for the case where the controller determines that a timestamp associated with sub-computation A was performed after sub-computation B whereas, according to its knowledge, sub -computation A should have (or even must have) been performed before sub -computation B. Similarly, in the case where a timestamp associated with sub-computation A should have (or even must have) beenperformed at the same time as (i.e. in parallel with) sub-computation B, the time offset may be determined as the difference between the timestamp associated with sub -computation A and the timestamp associated with sub-computation B.

[0082] The controller may adjust one or more erroneous timestamps by aligning a timestamp to be the same as that of one or more other timestamps, where it is known based on knowledge of the execution flow (i.e. sequence of sub-computations), that the timestamps should be the same. For example, if five devices are meant to execute a sub -computation at the same time and one of the timestamps issued by a device differs from the other four, the controller may align the outlier (i.e. the identified target device) with that of the majority.

[0083] As an additional or alternative action, having an identified an erroneous timestamp, the controller may send a message (e.g. signal) to the target device that issued the erroneous timestamp to cause the target device to update its local time. The message may contain a time that the local time of the target should be aligned to. This may be time that is maintained by the controller (e.g. a Unix host node 601). In other examples, the message may include the determined minimum offset, which the target device may use to adjust its local time.

[0084] FIG. 9 and FIG. 10 are provided to illustrate these embodiments.

[0085] FIG. 9 shows a simulated scenario in which Tile 5 is 250ns ahead of the other Tiles (Tiles 1 to 4). FIG. 9a shows the perfect case where all the Tiles are aligned to 0ns. This is indicated by the timestamps being aligned (these figures show outputs from Perfetto). The execution graph is (Tile 1) —> (Tiles 2-5) —> (Tile 6). Each tile performs a MVM sub-computation followed by a VPU sub-computation. At a session level, Tiles 2-5 all execute immediately after Tile l’s VPU executes. They all complete their VPU executions before Tile 6 starts its MVM execution. FIG. 9b shows the result when Tile 5 is running 250ns ahead of the other Tiles. The timestamps make it appear that Tile 5 is execution later than it should be because its clock is more advanced in time. In this case, it is easy to visually see that Tile 5 has an offset. The output from Perfetto struggles to display the timestamps correctly. There are two features that can be used to identify that Tile 5 is the one with an offset. First, majority voting in this graph makes it easy to compare Tiles 2-5 and identify Tile 5 (the controller does this based on the generated timestamps- the Perfetto diagrams are shown merely to aid understandings). Secondly, though it is difficult to see from the figures, the timestamp logs clearly show the VPU activity on Tile 5 is still executing after the next step in the flow (MVM on Tile 6) has started, which cannot be the case. This can be seen clearer on the Tiletimeline view, shown in FIG. 9c. The other Tiles all complete before Tile 6’s MVM but Tile 5 finishes after.

[0086] FIG. 10 is a similar example except now Tile 5 is running slower than the other Tiles. Tile 5 is shown as starting its MVM before the previous step has completed, which cannot be the case.

[0087] One or more of the examples described herein may vary within the scope of the attached claims. In general, some examples may be implemented in hardware or special purpose circuits, software, logic or any combination thereof. For example, some parts may be implemented in hardware, while other aspects may be implemented in firmware or software which may be executed by a controller, microprocessor or other computing device, although embodiments are not limited thereto. While various embodiments may be illustrated and described as block diagrams, flow charts, or using some other pictorial representation, it is well understood that these blocks, apparatus, systems, techniques or methods described herein may be implemented in, as non-limiting examples, hardware, software, firmware, special purpose circuits or logic, general purpose hardware or controller or other computing devices, or some combination thereof.

[0088] The memory may be of any type suitable to the local technical environment and may be implemented using any suitable data storage technology, such as semiconductor-based memory devices, magnetic memory devices and systems, optical memory devices and systems, fixed memory and removable memory.

[0089] Alternatively, or additionally some examples may be implemented using circuitry, or logic circuitry. The circuitry / logic circuitry may be configured to perform one or more of the functions and / or method steps previously described. As a further example, as used in this disclosure, the term circuitry also covers an implementation of merely a hardware circuit or processor (or multiple processors) or portion of a hardware circuit or processor and its (or their) accompanying software and / or firmware. The term circuitry also covers, for example integrated device.

[0090] The foregoing description has provided by way of exemplary and non-limiting examples a full and informative description of some embodiments. However, various modifications and adaptations may become apparent to those skilled in the relevant arts in view of the foregoing description, when read in conjunction with the accompanying drawings and the appendedclaims. However, all such and similar modifications of the teachings will still fall within the scope as defined in the appended claims.

Claims

CLAIMS1. A computer-implemented method of synchronising time between devices of a system, wherein the system comprises a host device configured to send data to a respective device via a respective pathway, the respective pathway comprising one or more intermediate components between the host device and the respective device, wherein the host device is configured to maintain a global time, wherein each device is configured to maintain a respective local time, and wherein the method comprises:the host device sending, to each of a plurality of target devices via a respective pathway, a message comprising a timestamp indicating a current global time at the host device; andeach target device:receiving and processing the message;determining whether the current global time indicated by the timestamp is ahead of or behind a current local time at the target device; andif the current global time indicated by the timestamp is ahead of the current local time at the target device, updating the current local time at the target device to be the current global time indicated by the timestamp; orif the current global time indicated by the timestamp is behind the current local time at the target device, maintaining the current local time at the target device.

2. The method of claim 1, comprising:the host device periodically or randomly sending respective messages to each target device, where each respective message comprises a respective timestamp indicating a current global time at the host device; andfor each respective message, each target device:receiving and processing the respective message;determining whether the current global time indicated by the respective timestamp is ahead of or behind a current local time at the target device; andif the current global time indicated by the respective timestamp is ahead of the current local time at the target device, updating the current local time at the target device to be the current global time indicated by the respective timestamp; orif the current global time indicated by the respective timestamp is behind the current local time at the target device, maintaining the current local time at the target device.

3. The method of any preceding claim, wherein the system comprises a plurality of sets of devices, each set of devices comprising a respective target device.

4. The method of claim 3, comprising, for one or more respective sets of devices:the target device of the respective set sending, to one, some or all other devices in the respective set, a message comprising a timestamp indicating the current local time at the target device.

5. The method of claim 4, wherein said sending of the message comprising a timestamp indicating the current local time at the target device is in response to the target device updating the current local time at the target device.

6. The method of any preceding claim, comprising:each target device sending, to the host device, a message comprising a timestamp indicating the current local time at the target device.

7. The method of any preceding claim, wherein each respective pathway comprises a common set of intermediate components.

8. The method of claims 1 to 6, wherein one or more respective pathways comprise a different set of intermediate components, and wherein the method comprises:for each respective pathway, determining a minimum delay associated with receiving and processing a respective message; anddetermining whether the current global time indicated by the timestamp is ahead of or behind a current local time at the target device based on the minimum delay associated with the respective pathway.

9. The method of any preceding claim, comprising one or more target devices initialising its local time based on a message comprising a timestamp indicating a current global time at the host device.

10. The method of any preceding claim, wherein each timestamp is a Unix timestamp.

11. The method of any preceding claim, comprising one or more target devices updating its local time to ensure its local time is not ahead of the global time at the host device.

12. The method of claim 11, wherein said updating is performed periodically and / or in response to a request from the host device.

13. The method of any preceding claim, wherein the system comprises a plurality of processors, and wherein each device is a respective processor.

14. The method of claim 3 and claim 13, wherein the system comprises a plurality of tiles, each tile comprising a plurality of processors, and wherein each respective set of devices belongs to a same respective tile.

15. The method of claim 13 or claim 14, wherein each device is one of: an xPU, a GPU, a VPU, an in-memory compute unit, a vector-matrix hardware multiplication unit.

16. The method of any preceding claim, wherein the system is configured to process data in a neural network.

17. Computer-readable storage media having program instructions stored thereon that, when executed by one or more processors operatively coupled with the one or more computer-readable storage media, direct a computing system comprising a host device and respective devices to perform operations of:the host device sending, to each of a plurality of target devices via a respective pathway, a message comprising a timestamp indicating a current global time at the host device; andeach target device:determining whether the current global time indicated by the timestamp is ahead of or behind a current local time at the target device; andupdating the current local time at the target device to be the current global time indicated by the timestamp if the current global time indicated by the timestamp is ahead of the current local time at the target device; ormaintaining the current local time at the target device if the current global time indicated by the timestamp is behind the current local time at the target device.

18. A system comprising a host device configured to send data to a respective device via a respective pathway, the respective pathway comprising one or more intermediate components between the host device and the respective device, wherein:the host device is configured to maintain a global time;each respective device is configured to maintain a respective local time;the host device is configured to send, to each of a plurality of target devices of the system via a respective pathway, a message comprising a timestamp indicating a current global time at the host device; andeach target device is configured to:determine whether the current global time indicated by the timestamp is ahead of or behind a current local time at the target device; andupdate the current local time at the target device to be the current global time indicated by the timestamp if the current global time indicated by the timestamp is ahead of the current local time at the target device.

Images