Method and system for evaluating communication delay between devices

Abstract

Provided in the present application are a method and system for evaluating a communication delay between devices. When a message is transmitted between a first device and a second device, a plurality of mailbox apparatuses distributed on a message transmission path between the first device and the second device can be used to acquire and record a plurality of time stamps in the message transmission path, such that communication delays of different message transmission path ranges between devices can be dynamically evaluated on the basis of actual requirements.

Description

Methods and systems for assessing communication latency between devices Technical Field

[0001] This application generally relates to the field of computing devices, and more specifically to methods and systems for evaluating communication delays between devices. Background Technology

[0002] In recent years, hybrid computing systems based on the collaborative operation of multiple computing devices have become a hot research topic in the field of high-performance computing. Especially in neural networks that process artificial intelligence (AI) tasks, many computing devices (e.g., AI accelerators) are needed to work together to handle complex AI tasks. This collaborative operation typically involves data exchange and task allocation between multiple computing devices to ensure load balancing and efficient processing. These computing devices need to be interconnected through intermediate interface devices, which inevitably leads to communication latency between device components.

[0003] To improve the overall performance of a device cluster, it is usually necessary to evaluate the communication latency between components of multiple computing devices so that task allocation can be adjusted and network paths optimized in a timely manner based on the evaluation results. Summary of the Invention

[0004] This application provides a method and system for evaluating communication latency between devices, which can dynamically evaluate the communication latency of different message transmission path ranges between devices according to actual needs.

[0005] According to one aspect of this application, a method is provided for evaluating communication delay between a first component of a first device and a second component of a second device coupled through a first interface subsystem and a second interface subsystem. The first interface subsystem is configured with a first interface mailbox device for recording timestamps when the first interface subsystem sends or receives messages. The second interface subsystem is configured with a second interface mailbox device for recording timestamps when the second interface subsystem sends or receives messages. The first component is configured with a first component mailbox device for recording timestamps when the first component sends or receives messages. The second component is configured with a second component mailbox device for recording timestamps when the second component sends or receives messages. The method includes: sending a first mailbox message from the first component to the second component; obtaining and recording a first timestamp when the first component completes sending the first mailbox message; wherein the first mailbox message includes a delay evaluation command field for instructing the second component to send one or more timestamps to the first component for evaluating the communication delay, the one or more timestamps being selected from timestamps recorded in the second component mailbox device or the second interface mailbox device; receiving the one or more timestamps from the second component; and performing a communication delay evaluation based on the first timestamp and the one or more timestamps.

[0006] According to another aspect of this application, a system is provided, including a first component, a first interface subsystem, a second interface subsystem, and a second component. The first component is coupled to the second component via the first and second interface subsystems. The first interface subsystem is configured with a first interface mailbox device for recording timestamps when the first interface subsystem sends or receives messages. The second interface subsystem is configured with a second interface mailbox device for recording timestamps when the second interface subsystem sends or receives messages. The first component is configured with a first component mailbox device for recording timestamps when the first component sends or receives messages, and the second component is configured with a second component mailbox device for recording timestamps when the second component sends or receives messages. The first component is configured to: send a first mailbox message to the second component; obtain and record a first timestamp when the first component completes sending the first mailbox message; wherein the first mailbox message includes a delay assessment command field for instructing the second component to send one or more timestamps to the first component for assessing communication delay, the one or more timestamps being selected from timestamps recorded in the second component mailbox device or the second interface mailbox device; receive the one or more timestamps from the second component; and perform a communication delay assessment based on the first timestamp and the one or more timestamps. Attached Figure Description

[0007] Various other advantages and benefits will become apparent to those skilled in the art upon reading the following detailed description of preferred embodiments. The accompanying drawings are for illustrative purposes only and are not intended to limit the scope of this application. In the drawings:

[0008] Figure 1 illustrates an exemplary process for measuring communication latency between device components;

[0009] Figure 2 illustrates an exemplary process for evaluating communication latency between device components according to an embodiment of this application;

[0010] Figure 3 illustrates an exemplary timestamp queue and message queue in a mailbox device according to an embodiment of this application;

[0011] Figure 4 illustrates an exemplary process for evaluating the communication delay between a first component of a first device and a second component of a second device according to an embodiment of this application;

[0012] Figure 5 shows a system architecture diagram of an example AI accelerator in which embodiments of this application can be implemented;

[0013] Figure 6 shows a schematic block diagram of an example computing device in which embodiments of the present application may be implemented. Detailed Implementation

[0014] The embodiments of the technical solution of this application will now be described in detail with reference to the accompanying drawings. These embodiments are only used to more clearly illustrate the technical solution of this application and are therefore merely examples, and should not be used to limit the scope of protection of this application.

[0015] It should be noted that, unless otherwise stated, the technical or scientific terms used in the embodiments of this application should have the ordinary meaning understood by those skilled in the art to which the embodiments of this application pertain.

[0016] Figure 1 illustrates an exemplary procedure for measuring communication latency between device components. As shown in Figure 1, a physical boundary exists between the first device and the second device; in other words, the first device and the second device are coupled through intermediate physical devices such as an interconnect bus (within the device), a first interface (e.g., a PCIe or Ethernet interface, which is used as an example in this document) subsystem, and a second interface subsystem. Furthermore, a boundary also exists between a first component of the first device (e.g., a first central processing unit (CPU)) and the intermediate physical devices on the first device side. This boundary is represented by a CPU boundary in Figure 1, but it is clear that if the first component of the first device is another device, this boundary could be represented by other boundaries. For example, if the first component is a graphics processing unit (GPU), the boundary could be represented by a GPU boundary. Similarly, a boundary also exists between a second component of the second device and the intermediate physical devices on the second device side, which is also represented by a CPU boundary in Figure 1.

[0017] When the first device wants to measure the communication latency between its first component and the second component of the second device, the first device can obtain the local time (timestamp t0) of the first component using get_time(), and then send a first mailbox message to the second component of the second device using mbox_send(). This first mailbox message can carry a first timestamp t1. The first timestamp t1 is equal to t0 + Δt, which indicates the local time when the first component completes sending the first mailbox message. Here, Δt includes the hardware interrupt response latency of the first component, the processing overhead of the software interrupt service routine, and the time required to send the message, and can be obtained through simulation. Then, the first mailbox message is transmitted to the second component of the second device through the intermediate physical device between the first and second devices. After receiving the mailbox interrupt, the second component obtains its local time (second timestamp t2) using get_time(), and then sends a second mailbox message to the first component of the first device using mbox_send(). This second mailbox message can carry a second timestamp t2 and a third timestamp t3 (t3 = t2 + Δt'). Similarly, this time Δt' includes the hardware interrupt response delay of the second component, the processing overhead of the software interrupt service routine, and the time required to send the message, which can also be obtained through simulation. Next, the second mailbox message is transmitted to the first component of the first device through the intermediate physical device between the second device and the first device. After receiving the mailbox interrupt, the first component obtains its local time (fourth timestamp t4) at this time through get_time(), and then obtains the second timestamp t2 and the third timestamp t3 carried in the second mailbox message through mbox_recv(). Based on timestamps t1 to t4, the first device can obtain the one-way communication delay between the first component and the second component of the second device: delay = ((t4-t1)-(t3-t2)) / 2.

[0018] For simplicity, this paper assumes that the link delay between the first device and the second device is the same in both directions, and that the clock frequencies of the first and second devices are the same. This assumption is also feasible in AI accelerator clusters for neural networks used to process AI tasks, for example, where the first and second devices are first and second AI accelerators with the same structure, the first component is the first top-level RISC-V vector unit (TopRVV) within the first AI accelerator, and the second component is the second TopRVV within the second AI accelerator.

[0019] The communication latency measurement process shown in Figure 1 can yield a single measurement of the overall link latency between two devices. However, this result cannot dynamically reflect the performance of the device cluster under various task loads. To guide hardware and software optimization and dynamic task allocation, an online dynamic latency evaluation method may be needed. In other words, a method is required to dynamically evaluate the communication latency across different message transmission path ranges between devices based on actual needs.

[0020] Figure 2 illustrates an exemplary process for evaluating communication latency between device components according to an embodiment of this application. As shown in Figure 2, similar to Figure 1, there is a physical boundary between the first device and the second device. That is, the first device and the second device are coupled through intermediate physical devices such as an interconnect bus (within the device), a first PCIe interface (which can be an interface such as PCIe or Ethernet, but is exemplified by PCIe in this document), and a second PCIe interface subsystem. Furthermore, there is a boundary (e.g., represented by a CPU boundary) between the first component of the first device (e.g., TopRVV of the AI ​​accelerator in Figure 2) and the intermediate physical devices on the first device side, and a boundary also exists between the second component of the second device (e.g., TopRVV) and the intermediate physical devices on the second device side. It should be noted that Figure 2 uses TopRVV of the AI ​​accelerator as an example of the first and second components to illustrate the communication latency evaluation process according to this application. However, it is readily understood that the first component of the first device and the second component of the second device can be other components in the AI ​​accelerator or other components in other devices; this application is not limited in this respect.

[0021] Unlike Figure 1, as shown in Figure 2, the first PCIe interface subsystem is configured with a first interface mailbox device (e.g., a first PCIe mailbox), which can be used to record timestamps when the first PCIe interface subsystem sends or receives messages; the second PCIe interface subsystem is configured with a second interface mailbox device (e.g., a second PCIe mailbox), which can be used to record timestamps when the second PCIe interface subsystem sends or receives messages; the first component is configured with a first component mailbox device (e.g., a first TopRVV mailbox), which can be used to record timestamps when the first component sends or receives messages; and the second component is configured with a second component mailbox device (e.g., a second TopRVV mailbox), which can be used to record timestamps when the second component sends or receives messages.

[0022] According to an embodiment of this application, when transmitting messages between a first device and a second device, multiple mailboxes distributed along the message transmission path between the first device and the second device as described above can be used to obtain and record multiple timestamps along the message transmission path, thereby dynamically evaluating the communication latency of different message transmission path ranges between devices according to actual needs.

[0023] Specifically, as shown in Figure 2, when the first component of the first device sends a first mailbox message to the second component of the second device, for example via mbox_send(), similar to that shown in Figure 1, the first timestamp t1 when the first component completes sending the first mailbox message is first obtained and recorded. Then, multiple timestamps in the message transmission path from the second component of the second device to the first component of the first device, where the first mailbox message is sent to the second component of the second device, and in the message transmission path from the second component of the second device to the first component of the first device, can be obtained and recorded.

[0024] For example, these timestamps may include a second timestamp t2 recorded at the second interface mailbox device when the sending of the first mailbox message from the second interface mailbox device to the second component mailbox device is completed; a third timestamp t3 recorded at the second component mailbox device when the first mailbox message is received; a fourth timestamp t4 obtained at the second component when a mailbox interruption is received from the second component mailbox device; a fifth timestamp t5 when the sending of the second mailbox message from the second component to the first component is completed; a sixth timestamp t6 recorded at the first interface mailbox device when the sending of the second mailbox message from the first interface mailbox device is completed; a seventh timestamp t7 recorded at the first component mailbox device when the second mailbox message is received; an eighth timestamp t8 obtained at the first component when a mailbox interruption is received from the first component mailbox device, and so on. It should be noted that Figure 2 only exemplarily shows the first to eighth timestamps t8 in the message transmission path between the first and second devices. However, in practical applications, more or fewer timestamps can be obtained as needed to meet different communication latency assessment requirements.

[0025] Based on the timestamp information described above, the communication latency across different message transmission path ranges between the first component of the first device and the second component of the second device can be evaluated as needed. For example, based on the first timestamp t1 and the second timestamp t2, the latency of the PCIe interface link in the transmission direction from the first component to the second component can be evaluated; based on the first timestamp t1 and the third timestamp t3, the latency of the PCIe interface link and the interface-to-component path in the transmission direction from the first component to the second component can be evaluated; based on the first timestamp t1 and the fourth timestamp t4 or the fifth timestamp t5, the latency of the entire message transmission path in the transmission direction from the first component to the second component can be evaluated. Furthermore, the first component of the first device can also evaluate the latency in the reception direction from the second component to the first component based on the fourth timestamp t4 or the fifth timestamp t5 and the sixth timestamp t6, the seventh timestamp t7, or the eighth timestamp t8.

[0026] The selection of which timestamps to use for communication latency assessment can be indicated by the latency assessment command field in the mailbox message. Specifically, the mailbox message width can be 128 bits, of which 24 bits are hardware-dependent, and the remaining bits can be freely used by the software. According to embodiments of this application, the 24 hardware-dependent bits in the mailbox message may include a destination identifier field, a source identifier field, and a command field. The destination identifier field can be used to identify the destination component of the mailbox message. For example, the destination identifier field in a first mailbox message sent by a first component of a first device to a second component of a second device can be used to indicate the mailbox device of the second component. The source identifier field can be used to identify the source component of the mailbox message. For example, the source identifier field in a first mailbox message sent by a first component of a first device to a second component of a second device can be used to indicate the mailbox device of the first component. The command field may include a latency assessment command field. For example, the latency assessment command field in a first mailbox message sent by a first component of a first device to a second component of a second device can be used to instruct the second component to send one or more timestamps to the first component for assessing communication latency, such as one or more of the second timestamp t2 to the fifth timestamp t5.

[0027] Besides the 24 bits related to hardware mentioned above, the remaining bits in the mailbox message can be freely used by software. For example, these bits may include a delay evaluation flag and a timestamp to be sent. When the delay evaluation flag is set, delay evaluation is enabled. According to embodiments of this application, when the mailbox device finishes sending or receiving a mailbox message, it can check the delay evaluation flag in the mailbox message. If the flag is set, the mailbox device will record the corresponding sending or receiving timestamp; otherwise, the mailbox device will not need to record the timestamp. The mailbox message may carry a timestamp, for example, the bit width of which may be 40 bits or more. For example, when the first device is the initiator of the communication delay evaluation, the second device may, in response to the request of the first device, carry one or more timestamps required by the first device in the second mailbox message it sends to the first device for the first device to perform the communication delay evaluation. The first device, as the initiator of the communication delay evaluation, generally does not need to carry the first timestamp t1 in the first mailbox message it sends to the second device. However, in some embodiments, the first device may also carry the first timestamp t1 in the first mailbox message so that the second device can also know its communication delay with the first device. Regarding this choice, an asterisk is used to indicate the optional operation of sending t1 in Figure 2. Similarly, an asterisk is also used to indicate the selection of sending one or more of t2, t3, t4, and t5 in Figure 2.

[0028] The mailbox messages mentioned above generally refer to messages initiated by the CPU of the first device when the first device and the second device communicate in Programmable Input / Output (PIO) mode. As shown in Figure 2, when the first component of the first device completes sending the first mailbox message in PIO mode, it can acquire and record a first timestamp t1. The first timestamp t1 is equal to t0 + Δt, which indicates the local time when the first component completes sending the first mailbox message. This time Δt includes the hardware interrupt response delay of the first component, the processing overhead of the software interrupt service routine, and the time required to send the message, and can be obtained through simulation. In addition to PIO mode, communication between the first device and the second device may also be carried out in Direct Memory Access (DMA) mode, for example, when a large amount of data needs to be transferred. In DMA mode, the mailbox message at the first component of the first device is initiated by the DMA controller. The timestamp when the mailbox message is completed can be referred to as the ninth timestamp t1' in this application, representing the system time when the first component receives the DMA completion (dma_done) interrupt. Thus, in some embodiments, the first device can evaluate the communication delay between the first device and the second device based on this timestamp t1' and other timestamps in the message transmission path.

[0029] As described above, according to the embodiments of this application, when transmitting messages between the first device and the second device, multiple mailbox devices distributed along the message transmission path between the first and second devices can be used to acquire and record multiple timestamps along the message transmission path, thereby dynamically evaluating the communication latency across different message transmission path ranges between the devices. Therefore, each mailbox device needs to be configured with its own counter to acquire the sending timestamp when sending a mailbox message is completed and the receiving timestamp when receiving a mailbox message. Correspondingly, each mailbox device can be configured with a Control Status Register (CSR) to acquire the current counter value of that mailbox device. In the embodiments of this application, to simplify the software flow, the clocks used by the counters of the associated mailbox devices across the entire device preferably come from the same clock source, such as the cfg_noc clock source. In this way, only a simple time offset is needed between the timestamps collected by these mailbox devices to achieve time base synchronization.

[0030] In addition to the send message queue and receive message queue, each mailbox device also needs to be configured with a send timestamp queue to record the timestamp when the mailbox device completes sending a message, and a receive timestamp queue to record the timestamp when the mailbox device receives a message. Accordingly, each mailbox device can be configured with two CSRs, used to read the send timestamp and receive timestamp recorded in the mailbox device, respectively.

[0031] Figure 3 illustrates exemplary timestamp queues and message queues in a mailbox device according to an embodiment of this application. The left side of Figure 3 shows the sending message queues and sending timestamp queues of two sending mailbox devices (mailbox A and mailbox B), and the right side shows the receiving message queue and receiving timestamp queue of a receiving mailbox device (mailbox C). For example, as shown in Figure 3, in the sending message queue of mailbox A, the delay evaluation flag is set in two mailbox messages (represented by long bars filled with diagonal lines), and therefore the sending timestamps (also represented by long bars filled with diagonal lines) corresponding to these two mailbox messages will be recorded in the sending timestamp queue of mailbox A. Similarly, the delay evaluation flag is set in the sending message queue of mailbox B in two mailbox messages (represented by long bars filled with cross lines), and therefore the sending timestamps (also represented by long bars filled with cross lines) corresponding to these two mailbox messages will be recorded in the sending timestamp queue of mailbox B. On the receiving end, email messages in the sending message queue of mailbox A and email messages in the sending message queue of mailbox B are recorded in the receiving message queue of mailbox C in turn, and the corresponding receiving timestamps are also recorded in the receiving timestamp queue of mailbox C.

[0032] Note that blank-filled bars in the send and receive message queues represent mailbox messages whose delay evaluation flag is not set; the timestamps for these mailbox messages do not need to be acquired or recorded. Furthermore, the send message queue and the send timestamp queue are decoupled, as are the receive message queue and the receive timestamp queue. In other words, the storage of sent messages in the send message queue and the storage of send timestamps in the send timestamp queue can be performed independently, and the correspondence between sent messages and their send timestamps can be set by software. Similarly, the storage of received messages in the receive message queue and the storage of received timestamps in the receive timestamp queue can be performed independently, and the correspondence between received messages and their receive timestamps can also be set by software. Here, as shown in Figure 3, the send message queue, send timestamp queue, receive message queue, and receive timestamp queue are all First-In-First-Out (FIFO) queues.

[0033] The method and system for evaluating communication latency between device components according to embodiments of this application have been described above with reference to Figures 2 and 3. An exemplary process for evaluating communication latency between a first component of a first device and a second component of a second device, for example, will be further described below with reference to Figure 4. Figure 4 illustrates an exemplary process 400 for evaluating communication latency between a first component of a first device and a second component of a second device according to embodiments of this application.

[0034] As previously described, the first device and the second device are coupled through a first interface subsystem and a second interface subsystem. The first interface subsystem is configured with a first interface mailbox device for recording timestamps when the first interface subsystem sends or receives messages. The second interface subsystem is configured with a second interface mailbox device for recording timestamps when the second interface subsystem sends or receives messages. The first component is configured with a first component mailbox device for recording timestamps when the first component sends or receives messages. The second component is configured with a second component mailbox device for recording timestamps when the second component sends or receives messages.

[0035] Process 400 may include operations 410 to 430. At operation 410, the first component sends a first mailbox message to the second component, and acquires and records a first timestamp when the first component completes sending the first mailbox message. The first mailbox message includes a latency assessment command field, instructing the second component to send one or more timestamps to the first component for assessing communication latency. These one or more timestamps are selected from timestamps recorded in the second component's mailbox device or the second interface mailbox device. At operation 420, the first component receives one or more timestamps from the second component, and at operation 430, the first component performs a communication latency assessment based on the first timestamp and the one or more timestamps received from the second component.

[0036] According to some embodiments of this application, one or more timestamps received from the second component may include one or more of the following: a second timestamp recorded at the second interface mailbox device when the sending of the first mailbox message from the second interface mailbox device to the second component mailbox device is completed; a third timestamp recorded at the second component mailbox device when the first mailbox message is received; a fourth timestamp obtained at the second component when a mailbox interruption is received from the second component mailbox device; and a fifth timestamp when the sending of the second mailbox message from the second component to the first component is completed. The second mailbox message may include one or more timestamps to be sent to the first component.

[0037] According to some embodiments of this application, communication delay assessment can also be performed based on at least one of the following timestamps: a sixth timestamp recorded in the first interface mailbox device when the sending of the second mailbox message from the first interface mailbox device is completed; a seventh timestamp recorded in the first component mailbox device when the second mailbox message is received; and an eighth timestamp obtained when the mailbox interruption from the first component mailbox device is received at the first component.

[0038] According to some embodiments of this application, the first component may also send a first timestamp to the second component via a first email message.

[0039] According to some embodiments of this application, the first mailbox message may include a delay evaluation flag, which is set to indicate that delay evaluation is enabled. The second mailbox message may also include a delay evaluation flag, which is set to indicate that delay evaluation is enabled.

[0040] According to some embodiments of this application, each of the first interface mailbox device, the second interface mailbox device, the first component mailbox device, and the second component mailbox device records a timestamp when a message is sent or received in response to a delay evaluation flag being set in the sent or received message.

[0041] According to some embodiments of this application, each mailbox device is configured with: a sending timestamp queue for recording the timestamp when the mailbox device finishes sending a message; and a receiving timestamp queue for recording the timestamp when the mailbox device receives a message.

[0042] According to some embodiments of this application, each mailbox device is further configured with a sending message queue and a receiving message queue, wherein the sending message queue is decoupled from the sending timestamp queue, and the receiving message queue is also decoupled from the receiving timestamp queue.

[0043] According to some embodiments of this application, the first email message may include a destination identifier field and a source identifier field, wherein the destination identifier field is used to indicate a second component email device and the source identifier field is used to indicate a first component email device.

[0044] According to some embodiments of this application, when the first component communicates with the second component in DMA mode, the first component can also acquire and record the ninth timestamp when the first component receives the DMA completion interrupt, and perform communication delay evaluation based on the ninth timestamp.

[0045] According to some embodiments of this application, the first interface mailbox device and the first component mailbox device are configured with their respective counters based on the same clock source on the first device, and the second interface mailbox device and the second component mailbox device are configured with their respective counters based on the same clock source on the second device.

[0046] According to some embodiments of this application, the first device is a first AI accelerator, the first component is a first TopRVV inside the first AI accelerator, and the second device is a second AI accelerator, the second component is a second TopRVV inside the second AI accelerator.

[0047] Figure 5 illustrates a system architecture diagram of an example AI accelerator 500 in which embodiments of this application may be implemented. The AI ​​accelerator 500 in Figure 5 may refer to a neural network accelerator or neural network processing unit (NPU), for example, a dedicated microprocessor designed to accelerate machine learning and AI tasks. Unlike traditional CPUs and GPUs, accelerators or NPUs are specifically optimized for neural network operations such as convolution computations and vector operations.

[0048] The AI ​​accelerator 500 shown in Figure 5 includes multiple physical components (PEs) designed to provide maximum parallelism to accelerate neural network operations. These PEs are organized in a two-dimensional mesh network and interconnected via a network of routers (represented as "R" in Figure 5). Furthermore, the AI ​​accelerator 500 includes double data rate (DDR) memory modules and caches, such as last-level caches (LLCs), to support efficient data storage and retrieval. The AI ​​accelerator 500 in Figure 5 is illustrative only and may include more, fewer, or alternative components. The AI ​​accelerator 500 can be designed as a reconfigurable device, such as a field-programmable gate array (FPGA) or an application-specific integrated circuit (ASIC).

[0049] In some embodiments, to optimize resource utilization and enhance the parallel processing capabilities of the AI ​​accelerator 500, the Processing Entities (PEs) within the two-dimensional mesh network can be divided into multiple blocks, each called a core. As shown in Figure 5, each group of 16 PEs (arranged in a 4x4 grid) constitutes a core, so the 64 PEs in the AI ​​accelerator 500 correspond to four cores. Each core is equipped with dedicated DDR memory, LLC, and control circuitry, including a RISC-V Vector Unit (RVV) responsible for vector processing, a Core-Level Scheduler (CoLS) managing the execution and synchronization of multiple PEs, and an Instruction Dispatch Unit (IDU) that distributes instructions to the various execution units within the accelerator. This architecture enables all four cores 510 (i.e., four PE groups) to operate concurrently, ensuring efficient parallel processing.

[0050] In some embodiments, these cores 510 are also organized into a network-on-chip (NoC) for inter-core communication. For example, in Figure 5, four groups of PEs are arranged in a ring NoC, facilitating seamless communication between cores and improving overall computational throughput. The ring NoC architecture includes a circular arrangement of cores 510, where data packets travel along a unidirectional or bidirectional ring, passing through each core 510 until reaching their destination. The ring NoC can be used for data communication between DDR memory and Fast Peripheral Component Interconnect (PCIe), or for PEs to read data from DDR memory belonging to other cores. In addition to the ring NoC, PEs can also be arranged in a two-dimensional mesh NoC to manage data communication between PEs. Furthermore, the ring NoC architecture in Figure 5 is scalable, allowing additional cores or PEs to be easily added to the ring without significantly increasing network complexity. This flexibility supports the scaling of AI accelerators to accommodate larger neural network models or additional computational tasks required. Furthermore, the AI ​​accelerator 500 shown in Figure 5 can interact with external hosts or other AI accelerators via PCIe. The AI ​​accelerator 500 shown in Figure 5 may also include a top-level processor (e.g., TopRVV) for orchestrating cores with instructions and a chip-level scheduler (ChLS). As previously described, the first and second devices according to some embodiments of this application may be AI accelerators, and their components may be components in an AI accelerator, such as TopRVV, or core-level RVV, etc.

[0051] Figure 6 shows a schematic block diagram of an example computing device 600 in which embodiments of this application may be implemented. The computing device 600 may be the first device or the second device described in the foregoing embodiments of this application. The computing device 600 may include a bus 602 or other communication mechanism for transmitting information and one or more hardware processors 604 coupled to the bus 602 for processing information. The hardware processors 604 may be, for example, one or more general-purpose microprocessors. The processor 604 may be, for example, a first component in the first device or a second component in the second device described in the foregoing embodiments of this application.

[0052] The computing device 600 may also include a main memory 607, such as random access memory (RAM), cache, and / or other dynamic storage devices, coupled to a bus 602 for storing information and instructions executed by one or more processors 604. The main memory 607 may also be used to store temporary variables or other intermediate information during the execution of instructions executed by one or more processors 604. These instructions, when stored in a storage medium accessible to one or more processors 604, can cause the computing device 600 to become a dedicated machine customized to perform the operations specified in the instructions. The main memory 607 may contain non-volatile media and / or volatile media. Non-volatile media may include, for example, optical discs or magnetic disks. Volatile media may include dynamic memory. Common forms of media may include, for example, floppy disks, flexible disks, hard disks, solid-state drives, magnetic tape or any other magnetic data storage media, CD-ROMs, any other optical data storage media, any physical media with a perforated pattern, RAM, DRAM, PROM and EPROM, FLASH-EPROM, NVRAM, any other memory chip or cartridge, or their networked versions.

[0053] The computing device 600 may implement the techniques described herein using customized hardwired logic, one or more ASICs or FPGAs, firmware, and / or program logic (which, when combined with the computing device, enable the computing device 600 to become or program the computing device 600 as a special-purpose machine). According to one embodiment, the techniques described herein are executed by the computing device 600 in response to one or more sequences of one or more instructions contained in main memory 607 executed by processor(s) 604. These instructions may be read into main memory 607 from another storage medium (such as storage device 609). Execution of the sequence of instructions contained in main memory 607 causes processor(s) 604 to perform the process steps described herein. For example, the processes / methods disclosed herein may be implemented by computer program instructions stored in main memory 607. When these instructions are executed by processor(s) 604, they may perform the steps shown in the corresponding figures and described above. In alternative embodiments, hardwired circuitry may be used in place of or in combination with software instructions.

[0054] The computing device 600 may also include a communication interface 610 coupled to the bus 602. The communication interface 610 provides bidirectional data communication coupling to one or more network links connected to one or more networks. As another example, the communication interface 610 may be a local area network (LAN) card, providing data communication connectivity to a LAN-compatible (or WAN component communicating with a WAN) device. A wireless link may also be implemented.

[0055] The execution of certain operations can be distributed across processors rather than residing within a single machine, or deployed across multiple machines. In some example embodiments, the processor or processor-implemented engine may reside in a single geographic location (e.g., in a home environment, office environment, or server farm). In other example embodiments, the processor or processor-implemented engine may be distributed across multiple geographic locations.

[0056] Each of the processes, methods, and algorithms described in the preceding sections may be fully or partially embodied automatically in code modules executed by one or more computer systems or computer processors including computer hardware. The processes and algorithms may be implemented, partially or fully, in dedicated circuit systems.

[0057] When the functions disclosed herein are implemented as software functional units and sold or used as standalone products, they may be stored in a processor-executable, non-volatile, computer-readable storage medium. Specific technical solutions (all or part) disclosed herein, or aspects contributing to the prior art, may be embodied in the form of a software product. The software product may be stored in a storage medium and includes several instructions that cause a computing device (which may be a personal computer, server, network device, etc.) to perform all or some steps of the methods of the embodiments of this application. The storage medium may include a flash drive, portable hard disk drive, ROM, RAM, magnetic disk, optical disk, other media operable to store program code, or any combination thereof.

[0058] Specific embodiments further provide a system including a processor and a non-transitory computer-readable storage medium storing instructions executable by the processor to cause the system to perform operations corresponding to steps in any method of the embodiments disclosed above. Specific embodiments further provide a non-transitory computer-readable storage medium storing instructions executable by one or more processors to cause the one or more processors to perform operations corresponding to steps in any method of the embodiments disclosed above.

[0059] The embodiments disclosed herein can be implemented via a cloud platform, server, or server cluster (collectively referred to below as the "Service System") that interacts with a client. The client can be a terminal device or a client registered by a user at the platform, wherein the terminal device can be a mobile terminal, a personal computer (PC), or any device that can have the platform application installed.

[0060] The various features and processes described above can be used independently of each other or combined in various ways. All possible combinations and sub-combinations are intended to fall within the scope of this disclosure. Additionally, certain method or process blocks may be omitted in some embodiments. The methods and processes described herein are not limited to any particular order, and their associated blocks or states may be executed in other suitable orders. For example, described blocks or states may be executed in an order other than that specifically disclosed, or multiple blocks or states may be combined into a single block or state. Example blocks or states may be executed sequentially, in parallel, or in some other manner. Blocks or states may be added to or removed from the disclosed example embodiments. The exemplary systems and components described herein may be configured differently than described. For example, components may be added to, removed from, or rearranged compared to the disclosed example embodiments.

[0061] The various operations of the exemplary methods described herein can be performed at least in part by an algorithm. The algorithm may be included in program code or instructions stored in memory (e.g., the aforementioned non-transitory computer-readable storage medium). This algorithm may include a machine learning algorithm. In some embodiments, the machine learning algorithm may not explicitly refer to the computer as performing the function but may learn from training data to generate a predictive model of the function.

[0062] The various operations of the exemplary methods described herein can be performed, at least in part, by one or more processors that are temporarily configured (e.g., by software) or permanently configured to perform the relevant operations. Whether temporarily or permanently configured, these processors can constitute an engine of processor implementation that operates to perform one or more of the operations or functions described herein.

[0063] Similarly, the methods described herein may be implemented at least in part by a processor, wherein one or more specific processors are instances of hardware. For example, at least some operations of the methods may be performed by one or more processors or an engine implemented by a processor. Furthermore, one or more processors may also be operable to support the execution of related operations in a “cloud computing” environment or as the execution of related operations in a “Software as a Service” (SaaS) context. For example, at least some operations may be performed by a group of computers (as an example of a machine containing processors), wherein these operations are accessible via a network (e.g., the Internet) and via one or more appropriate interfaces (e.g., application programming interfaces (APIs)).

[0064] The execution of certain operations can be distributed across processors rather than residing within a single machine, and can be deployed across multiple machines. In some example embodiments, the processor or processor-implemented engine may reside in a single geographic location (e.g., in a home environment, office environment, or server farm). In other example embodiments, the processor or processor-implemented engine may be distributed across multiple geographic locations.

[0065] Throughout this specification, multiple instances may be implemented as components, operations, or structures described as a single instance. Although individual operations of one or more methods are illustrated and described as separate operations, one or more of these individual operations may be performed simultaneously, and not necessarily in the order illustrated. Structures and functions presented as separate components in the example configuration may be implemented as composite structures or components. Similarly, structures and functions presented as single components may be implemented as single components. These and other variations, modifications, additions, and improvements fall within the scope of this document.

[0066] As used herein, "or" is inclusive rather than exclusive unless explicitly indicated by the context. Therefore, in this document, "A, B, or C" means "A, B, A and B, A and C, B and C, or A, B, and C" unless explicitly indicated by the context. Furthermore, "and" is combined and separate unless explicitly indicated by the context. Therefore, in this document, "A and B" means "A and B, combined or separate" unless explicitly indicated by the context. Additionally, multiple instances of resources, operations, or structures described herein may be provided as a single instance. Furthermore, the boundaries between various resources, operations, engines, and data storage devices are somewhat arbitrary and specific operations are illustrated within the context of a particular illustrative configuration. Other functional assignments are foreseeable and fall within the scope of various embodiments of this disclosure. Generally, structures and functions presented as individual resources in example configurations may be implemented as combined structures or resources. Similarly, structures and functions presented as single resources may be implemented as single resources. These and other changes, modifications, additions, and improvements fall within the scope of the embodiments of this disclosure as expressed in the appended claims. Therefore, this specification and drawings should be considered illustrative rather than restrictive.

[0067] The terms “comprising” or “including” are used to indicate the presence of a subsequently claimed feature, but do not preclude the addition of other features. Unless otherwise specifically stated or otherwise understood in the context in which they are used, conditional language such as “may,” “can,” “may,” and “can” is generally intended to convey that certain embodiments include certain features, components, and / or steps that are not included in other embodiments. Therefore, this conditional language is generally not intended to imply that one or more embodiments require features, components, and / or steps in any way, or that one or more embodiments must include logic for determining whether such features, components, and / or steps are included in or performed in any particular embodiment, with or without user input or prompts.

[0068] Although the general overview of the subject matter has been described with reference to specific exemplary embodiments, various modifications and changes can be made to these embodiments without departing from the broad scope of embodiments of this disclosure. The embodiments described herein have been described in sufficient detail to enable those skilled in the art to practice the teachings disclosed. Other embodiments may be used and derived therefrom, such that structural and logical substitutions and changes can be made without departing from the scope of this disclosure. Therefore, "embodiments" are not intended to be limiting, and the scope of the various embodiments is defined only by the appended claims and their full scope.

Claims

1. A method for evaluating communication latency between a first component of a first device and a second component of a second device coupled via a first interface subsystem and a second interface subsystem, wherein the first interface subsystem is configured with a first interface mailbox device for recording timestamps when the first interface subsystem sends or receives messages, the second interface subsystem is configured with a second interface mailbox device for recording timestamps when the second interface subsystem sends or receives messages, the first component is configured with a first component mailbox device for recording timestamps when the first component sends or receives messages, and the second component is configured with a second component mailbox device for recording timestamps when the second component sends or receives messages, the method comprising: The first component sends a first email message to the second component, and obtains and records a first timestamp when the first component completes sending the first email message. The first email message includes a delay evaluation command field, which instructs the second component to send one or more timestamps to the first component for evaluating the communication delay. The one or more timestamps are selected from the timestamps recorded in the second component's email device or the second interface email device. Receive the one or more timestamps from the second component; and Communication delay is assessed based on the first timestamp and the one or more timestamps.

2. The method according to claim 1, wherein, The one or more timestamps include one or more of the following: a second timestamp recorded at the second interface mailbox device when the sending of the first mailbox message from the second interface mailbox device to the second component mailbox device is completed; a third timestamp recorded at the second component mailbox device when the first mailbox message is received; a fourth timestamp obtained at the second component when a mailbox interruption is received from the second component mailbox device; and a fifth timestamp when the sending of the second mailbox message from the second component to the first component is completed.

3. The method according to claim 2, wherein, The second email message includes the one or more timestamps.

4. The method according to claim 2, further comprising: Communication delay is evaluated based on at least one of the following timestamps: a sixth timestamp recorded in the first interface mailbox device when the sending of the second mailbox message from the first interface mailbox device is completed; a seventh timestamp recorded in the first component mailbox device when the second mailbox message is received; and an eighth timestamp obtained when a mailbox interruption is received at the first component from the first component mailbox device.

5. The method according to claim 1, further comprising: The first timestamp is sent to the second component via the first email message.

6. The method according to claim 1, wherein, The first email message includes a delay evaluation flag, which is set to indicate that delay evaluation is enabled.

7. The method according to claim 2, wherein, The second email message includes a delay evaluation flag, which is set to indicate that delay evaluation is enabled.

8. The method according to claim 1, wherein, Each of the first interface mailbox device, the second interface mailbox device, the first component mailbox device, and the second component mailbox device records a timestamp when the message is sent or received in response to a delay evaluation flag being set in the sent or received message.

9. The method according to claim 8, wherein, Each mailbox device is configured with: a send timestamp queue for recording the timestamp when the mailbox device finishes sending a message; and a receive timestamp queue for recording the timestamp when the mailbox device receives a message.

10. The method according to claim 9, wherein, Each mailbox device is also configured with a send message queue and a receive message queue, wherein the send message queue is decoupled from the send timestamp queue, and the receive message queue is also decoupled from the receive timestamp queue.

11. The method according to any one of claims 1 to 10, wherein, The first email message includes a destination identifier field and a source identifier field, wherein the destination identifier field is used to indicate the second component email device, and the source identifier field is used to indicate the first component email device.

12. The method according to any one of claims 1 to 10, wherein, When the first component communicates with the second component in direct memory access (DMA) mode, the method further includes: Obtain and record the ninth timestamp when the first component receives the DMA completion interrupt, and perform a communication delay assessment based on the ninth timestamp.

13. The method according to any one of claims 1 to 10, wherein, The first interface mailbox device and the first component mailbox device are configured with their respective counters based on the same clock source on the first device, and the second interface mailbox device and the second component mailbox device are configured with their respective counters based on the same clock source on the second device.

14. The method according to any one of claims 1 to 10, wherein, The first device is a first artificial intelligence (AI) accelerator, the first component is a first top-level RISC-V vector unit (TopRVV) inside the first AI accelerator, the second device is a second AI accelerator, and the second component is a second TopRVV inside the second AI accelerator.

15. A system comprising a first component, a first interface subsystem, a second interface subsystem, and a second component, wherein the first component is coupled to the second component via the first interface subsystem and the second interface subsystem, the first interface subsystem is configured with a first interface mailbox device for recording timestamps when the first interface subsystem sends or receives messages, the second interface subsystem is configured with a second interface mailbox device for recording timestamps when the second interface subsystem sends or receives messages, the first component is configured with a first component mailbox device for recording timestamps when the first component sends or receives messages, and the second component is configured with a second component mailbox device for recording timestamps when the second component sends or receives messages, wherein the first component is configured to: Send a first email message to the second component, obtain and record the first timestamp when the first component completes sending the first email message, wherein, The first mailbox message includes a latency assessment command field, which instructs the second component to send one or more timestamps to the first component for assessing the communication latency, wherein the one or more timestamps are selected from timestamps recorded in the second component mailbox device or the second interface mailbox device; Receive the one or more timestamps from the second component; and Communication delay is assessed based on the first timestamp and the one or more timestamps.

16. The system according to claim 15, wherein, The one or more timestamps include one or more of the following: a second timestamp recorded at the second interface mailbox device when the sending of the first mailbox message from the second interface mailbox device to the second component mailbox device is completed; a third timestamp recorded at the second component mailbox device when the first mailbox message is received; a fourth timestamp obtained at the second component when a mailbox interruption is received from the second component mailbox device; and a fifth timestamp when the sending of the second mailbox message from the second component to the first component is completed.

17. The system according to claim 16, wherein, The second email message includes the one or more timestamps.

18. The system according to claim 16, wherein, The first component is also configured to: perform a communication delay assessment based on at least one of the following timestamps: a sixth timestamp recorded in the first interface mailbox device when the sending of the second mailbox message from the first interface mailbox device is completed; a seventh timestamp recorded in the first component mailbox device when the second mailbox message is received; and an eighth timestamp obtained when a mailbox interruption is received at the first component mailbox device.

19. The system according to claim 15, wherein, The first component is also configured to send the first timestamp to the second component via the first email message.

20. The system according to claim 15, wherein, The first email message includes a delay evaluation flag, which is set to indicate that delay evaluation is enabled.

21. The system according to claim 16, wherein, The second email message includes a delay evaluation flag, which is set to indicate that delay evaluation is enabled.

22. The system according to claim 15, wherein, Each of the first interface mailbox device, the second interface mailbox device, the first component mailbox device, and the second component mailbox device records a timestamp when the message is sent or received in response to a delay evaluation flag being set in the sent or received message.

23. The system according to claim 22, wherein, Each mailbox device is configured with: a send timestamp queue for recording the timestamp when the mailbox device finishes sending a message; and a receive timestamp queue for recording the timestamp when the mailbox device receives a message.

24. The system according to claim 23, wherein, Each mailbox device is also configured with a send message queue and a receive message queue, wherein the send message queue is decoupled from the send timestamp queue, and the receive message queue is also decoupled from the receive timestamp queue.

25. The system according to any one of claims 15 to 24, wherein, The first email message includes a destination identifier field and a source identifier field, wherein the destination identifier field is used to indicate the second component email device, and the source identifier field is used to indicate the first component email device.

26. The system according to any one of claims 15 to 24, wherein, When the first component communicates with the second component in Direct Memory Access (DMA) mode, the first component is also configured to: Obtain and record the ninth timestamp when the first component receives the DMA completion interrupt, and perform a communication delay assessment based on the ninth timestamp.

27. The system according to any one of claims 15 to 24, wherein, The first interface mailbox device and the first component mailbox device are configured with their respective counters based on the same clock source on the first device, and the second interface mailbox device and the second component mailbox device are configured with their respective counters based on the same clock source on the second device.

28. The system according to any one of claims 15 to 24, wherein, The first component is the first top-level RISC-V vector unit (TopRVV) inside the first artificial intelligence (AI) accelerator, and the second component is the second TopRVV inside the second AI accelerator.

Images