Task execution method, apparatus, heterogeneous computing system, and computing device

Abstract

The present application relates to the technical field of computers. Provided are a task execution method, an apparatus, a heterogeneous computing system, and a computing device. The method is applied to a heterogeneous computing system comprising a processor and an accelerator, the processor and the accelerator being used for jointly executing a computing procedure, the processor acquiring a first task and a second task in the computing procedure that have a dependency relationship, and each task comprising at least one processing task executed by the processor and a computing task executed by the accelerator. On this basis, with the processing tasks in each task as the granularity, the processor can concurrently execute processing tasks that have no dependency relationship and sequentially execute processing tasks that have a dependency relationship, thereby preventing the accelerator from being in a waiting state for a long time, and improving the utilization rate of the accelerator.

Description

Task execution methods, devices, heterogeneous computing systems, and computing equipment

[0001] This application claims priority to Chinese Patent Application No. 202510192515.4, filed on February 20, 2025, entitled “Task Execution Method, Apparatus, Heterogeneous Computing System and Computing Device”, the entire contents of which are incorporated herein by reference. Technical Field

[0002] This application relates to the field of computer technology, and in particular to a task execution method, apparatus, heterogeneous computing system, and computing device. Background Technology

[0003] With the rapid development of technologies such as artificial intelligence (AI) and big data, heterogeneous computing systems containing central processing units (CPUs) and accelerators have emerged, playing an increasingly important role in improving computing efficiency and meeting diverse computing needs.

[0004] In related technologies, taking the execution of AI model inference tasks by a heterogeneous computing system as an example, the CPU breaks down the entire computation process of the inference task into multiple tasks and then assigns the tasks to appropriate accelerators for execution. The CPU will send the task to the accelerator after performing the preprocessing process of the task (such as preparing input data and parameters) so that the accelerator can perform the core computation process of the task (such as matrix calculation).

[0005] However, since the accelerator needs to wait for the CPU to perform the preprocessing process before it can execute the core calculation process based on the received task, the accelerator will be in a waiting state for a long time when the CPU takes a long time to perform the preprocessing process, resulting in low utilization of the accelerator. Summary of the Invention

[0006] This application provides a task execution method, apparatus, heterogeneous computing system, and computing device, which can reduce the processor overhead in heterogeneous computing systems, prevent accelerators from being in a waiting state for a long time, and improve the utilization rate of accelerators.

[0007] Firstly, a task execution method is provided, applied to a processor in a heterogeneous computing system, which also includes an accelerator. The processor and accelerator are used to jointly execute a computational process. The method includes:

[0008] The first task and the second task in the computation process are obtained. The first task and the second task are dependent on each other. The first task includes at least one first processing task executed by the processor and a first computation task executed by the accelerator. The second task includes at least one second processing task executed by the processor and a second computation task executed by the accelerator.

[0009] Concurrently execute at least one first processing task and at least one second processing task that have no dependency relationship, and sequentially execute at least one first processing task and at least one second processing task that have a dependency relationship.

[0010] Wherein, at least one first processing task includes a processing task that instructs the accelerator to perform a first computing task, and at least one second processing task includes a processing task that instructs the accelerator to perform a second computing task.

[0011] In the above method, the processor obtains a first task and a second task that have a dependency relationship in the computing process. Each task includes at least one processing task executed by the processor and a computing task executed by the accelerator. Based on this, the processor can concurrently execute processing tasks that do not have a dependency relationship at the granularity of the processing tasks in each task, and sequentially execute processing tasks that have a dependency relationship, thereby reducing the processor's processing latency, avoiding the accelerator from being in a waiting state for a long time, and improving the utilization rate of the accelerator.

[0012] In some embodiments, concurrently executing at least one first processing task and at least one second processing task where there is no dependency between them, and sequentially executing at least one first processing task and at least one second processing task where there is a dependency between them, includes:

[0013] Create multiple threads that run concurrently;

[0014] The first thread of a plurality of threads sequentially executes at least one third processing task in a first processing task and at least one fourth processing task in a second processing task.

[0015] The second thread among multiple threads sequentially executes at least one fifth processing task in the first processing task and at least one sixth processing task in the second processing task;

[0016] The fifth processing task is executed after the third processing task, and the sixth processing task is executed after the fourth processing task.

[0017] By using the above method and multi-threading technology, the processing tasks in the first and second tasks are concurrently scheduled. While ensuring that the dependencies between the original tasks are not affected, processor overhead is saved, thereby avoiding the accelerator from being in a waiting state for a long time and improving the utilization rate of the accelerator.

[0018] In some embodiments, the method further includes:

[0019] Obtain the total time taken by the heterogeneous computing system to execute the first and second tasks during a historical time period, as well as the computation time taken by the accelerator to execute the first and second computing tasks during a historical time period;

[0020] The bubble time of the accelerator in a historical time period is determined based on the difference between the total time and the computation time.

[0021] Concurrently executing at least one first processing task and at least one second processing task where there is no dependency between them, and sequentially executing at least one first processing task and at least one second processing task where there is a dependency between them, including:

[0022] If the ratio of bubble processing time to total processing time is greater than a threshold, then at least one of the first processing tasks and at least one of the second processing tasks that do not have a dependency relationship will be executed concurrently, and at least one of the first processing tasks and at least one of the second processing tasks that have a dependency relationship will be executed sequentially.

[0023] By using the above method, when a large number of bubbles are detected in the accelerators of a heterogeneous computing system, tasks can be broken down into finer-grained parts in a timely manner to improve the utilization rate of the accelerators.

[0024] In some embodiments, the computation process refers to the inference process or training process of an artificial intelligence (AI) model. Obtaining the first task and the second task in the computation process includes: in response to an inference request or training request of the AI ​​model, obtaining the first task and the second task; wherein at least one of the first processing tasks includes at least one of shape derivation and data segmentation of the input data of the first task.

[0025] In some embodiments, the method further includes: during the execution of the at least one first processing task in a first time period, storing the execution result of at least one of shape derivation and data segmentation of the input data;

[0026] After concurrently executing at least one first processing task and at least one second processing task that have no dependencies on each other, and sequentially executing at least one first processing task and at least one second processing task that have dependencies on each other, the method further includes:

[0027] During the execution of at least one first processing task in the second time period, if the shape of the input data of the first task is the same as the shape of the input data of the first task when at least one first processing task is executed in the first time period, then at least one of shape derivation and data segmentation is skipped, and the accelerator is instructed to execute the first computation task based on the execution result stored in the first time period, after the first time period.

[0028] The above method means that the processor can store the execution results of shape derivation and / or data segmentation each time it executes a task. When the processor executes the same task in a later time, if the shape of the input data of the task has not changed, the previously stored execution results can be reused, saving processor overhead.

[0029] In a second aspect, a task execution apparatus is provided for use in a processor in a heterogeneous computing system, the heterogeneous computing system further including an accelerator, the processor and the accelerator being used to jointly execute a computing process, the apparatus including at least one functional module for implementing the task execution method provided as described in the first aspect or any possible implementation of the first aspect.

[0030] Thirdly, a heterogeneous computing system is provided, the heterogeneous computing system including a processor and an accelerator;

[0031] The processor is used for:

[0032] Obtain a first task and a second task in the computing process, wherein the first task and the second task are dependent on each other, the first task includes at least one first processing task executed by the processor and a first computing task executed by the accelerator, and the second task includes at least one second processing task executed by the processor and a second computing task executed by the accelerator.

[0033] Concurrently execute processing tasks that are not dependent on the at least one first processing task and the at least one second processing task; sequentially execute processing tasks that are dependent on the at least one first processing task and the at least one second processing task; wherein, the at least one first processing task includes a processing task that instructs the accelerator to execute the first computing task, and the at least one second processing task includes a processing task that instructs the accelerator to execute the second computing task.

[0034] The accelerator is used to execute the first computing task and the second computing task according to the instructions of the processor.

[0035] In some embodiments, the processor is configured to: create a plurality of concurrently running threads; sequentially execute a third processing task in the at least one first processing task and a fourth processing task in the at least one second processing task via a first thread among the plurality of threads; and sequentially execute a fifth processing task in the at least one first processing task and a sixth processing task in the at least one second processing task via a second thread among the plurality of threads; wherein the execution order of the fifth processing task is after the execution order of the third processing task, and the execution order of the sixth processing task is after the execution order of the fourth processing task.

[0036] In some embodiments, the processor is further configured to: obtain the total time taken by the heterogeneous computing system to execute the first task and the second task in a historical time period and the computation time taken by the accelerator to execute the first computing task and the second computing task in the historical time period; and determine the bubble time taken by the accelerator in the historical time period based on the difference between the total time taken and the computation time taken.

[0037] The concurrent execution of processing tasks that have no dependency relationship among the at least one first processing task and the at least one second processing task, and the sequential execution of processing tasks that have dependency relationship among the at least one first processing task and the at least one second processing task, includes: if the ratio between the bubble time and the total time is greater than a threshold, then concurrently execute processing tasks that have no dependency relationship among the at least one first processing task and the at least one second processing task, and sequentially execute processing tasks that have dependency relationship among the at least one first processing task and the at least one second processing task.

[0038] In some embodiments, the computational process refers to the inference or training process of an artificial intelligence (AI) model, and the processor is configured to: in response to an inference or training request from the AI ​​model, acquire the first task and the second task; wherein the at least one first processing task includes at least one of shape derivation and data segmentation of the input data of the first task.

[0039] In some embodiments, the processor is further configured to: during the execution of the at least one first processing task in a first time period, store the execution result of at least one of shape derivation and data segmentation of the input data;

[0040] After concurrently executing the at least one first processing task and the at least one second processing task that have no dependency relationship, and sequentially executing the at least one first processing task and the at least one second processing task that have a dependency relationship, the processor is further configured to: during the execution of the at least one first processing task in the second time period, if the shape of the input data of the first task is the same as the shape of the input data of the first task when the at least one first processing task was executed in the first time period, skip at least one of shape derivation and data segmentation, and instruct the accelerator to execute the first computing task based on the execution result stored in the first time period, wherein the second time period is after the first time period.

[0041] Fourthly, a processor is provided that is communicatively connected to an accelerator, the processor is used to control the accelerator, and the processor is used to implement the task execution method provided by the first aspect or any possible implementation thereof.

[0042] Fifthly, a computing device is provided, comprising a processor and a memory, wherein the processor is configured to execute at least a segment of program code stored in the memory, so that the computing device implements the task execution method provided by the first aspect or any possible implementation thereof.

[0043] Sixthly, a computer-readable storage medium is provided for storing at least one piece of program code, which, when executed by a computing device, causes the computing device to implement the task execution method provided by the first aspect or any possible implementation thereof. The storage medium includes, but is not limited to, volatile memory, such as random access memory, and non-volatile memory, such as flash memory, hard disk drive (HDD), and solid-state drive (SSD).

[0044] In a seventh aspect, a computer program product is provided that, when run on a computing device, causes the computing device to implement the task execution method provided by the first aspect or any possible implementation thereof. The computer program product may be a software installation package, which can be downloaded and executed on the computing device when the aforementioned task execution method needs to be implemented. Attached Figure Description

[0045] Figure 1 is a schematic diagram of the computation process executed by a heterogeneous computing system;

[0046] Figure 2 is a schematic diagram of the computation process performed by a related technology;

[0047] Figure 3 is a schematic diagram of an implementation environment provided in an embodiment of this application;

[0048] Figure 4 is a schematic diagram of a heterogeneous computing system provided in an embodiment of this application;

[0049] Figure 5 is a schematic diagram of the structure of a computing device provided in an embodiment of this application;

[0050] Figure 6 is a schematic diagram of the computation process of a heterogeneous computing system provided in an embodiment of this application;

[0051] Figure 7 is a schematic diagram of splitting a task into multiple sub-tasks according to an embodiment of this application;

[0052] Figure 8 is a schematic diagram of the functional architecture of a heterogeneous computing system provided in an embodiment of this application;

[0053] Figure 9 is a flowchart of a task execution method provided in an embodiment of this application;

[0054] Figure 10 is a schematic diagram of the dependency relationship between various processing tasks provided in an embodiment of this application;

[0055] Figure 11 is a schematic diagram of another dependency relationship between various processing tasks provided in an embodiment of this application;

[0056] Figure 12 is a schematic diagram of a multi-threaded scheduling and execution of a processing task provided in an embodiment of this application;

[0057] Figure 13 is a flowchart of a task execution method provided in an embodiment of this application;

[0058] Figure 14 is a schematic diagram of the structure of a task execution device provided in an embodiment of this application. Detailed Implementation

[0059] To make the objectives, technical solutions, and advantages of this application clearer, the embodiments of this application will be further described in detail below with reference to the accompanying drawings. It should be noted that all information (including but not limited to user device information, user personal information, etc.), data (including but not limited to data used for analysis, stored data, displayed data, etc.), and signals involved in this application are authorized by the user or fully authorized by all parties, and the collection, use, and processing of related data must comply with the relevant laws, regulations, and standards of the relevant countries and regions. For example, the calculation processes, AI model training requests, and AI model inference requests involved in this application are all obtained under fully authorized conditions.

[0060] To facilitate understanding, the key terms and concepts involved in this application will be explained below.

[0061] Artificial intelligence (AI) models are a class of mathematical algorithm models that use machine learning concepts to solve practical problems. Typically, AI models include a large number of parameters and calculation formulas (or calculation rules).

[0062] An accelerator, also known as an acceleration chip, acceleration device, acceleration card, or computing card, is a type of specialized hardware device or computer system designed to accelerate computation in AI scenarios. In this application, an accelerator may include, for example, a graphics processing unit (GPU), a neural network processing unit (NPU), an intelligent processing unit (IPU), a tensor processing unit (TPU), or a domain-specific architecture (DSA) chip, and is not limited to these.

[0063] An operator (OP) is a computational unit or function that runs on a computing device. In the field of deep learning, neural network layers and even the entire model are composed of operators, which correspond to the computational logic within the neural network layers. For example, a convolutional layer is an operator; the weight summation process in a fully-connected layer (FC layer) is also an operator.

[0064] A heterogeneous computing system is a computing system that includes different types of processing units. For example, a computing system containing a central processing unit (CPU) and an accelerator is a heterogeneous computing system. The CPU excels at handling complex decision-making and process control, while the accelerator excels at performing matrix and vector calculations. In a heterogeneous computing system, the CPU's flexibility and multitasking capabilities are leveraged to schedule the high-speed computing power of the accelerator, allowing the CPU and accelerator to collaborate efficiently and effectively complete the system's tasks.

[0065] In this application, a compute kernel is a code segment specifically designed for the hardware characteristics of an accelerator to efficiently process specific types of tasks. The compute kernel defines the core computational operations that need to be performed on the accelerator in a task. For example, in an AI model's operator, the compute kernel defines matrix computation operations other than shape derivation and parameter preparation.

[0066] The application scenarios of this application are described below.

[0067] This application applies to scenarios where computational processes are executed using heterogeneous computing systems. A computational process refers to an end-to-end computation that completes a specific function in one go, such as a single inference operation of an AI model, or an iteration during the training of an AI model, but is not limited to these examples.

[0068] Referring to Figure 1, which illustrates a computational process executed by a heterogeneous computing system, the CPU is responsible for breaking down the computational process into multiple tasks, for example, abstracted as a directed acyclic graph. A task refers to a key computational step in the process, such as an operator in an AI model, or at least two operators in an AI model, but is not limited to these. Then, based on the multiple tasks obtained after decomposition, the CPU schedules accelerators to execute the tasks according to the dependencies between them. The accelerators can execute the received tasks serially or in parallel based on these dependencies and return the execution results to the CPU.

[0069] In related technologies, taking the execution of AI model inference requests in a heterogeneous computing system as an example, referring to Figure 2, Figure 2 is a schematic diagram of the computation process of a related technology. As shown in Figure 2, after the CPU breaks down the entire computation process of the inference request into multiple tasks (as shown in Task 1, Task 2, and Task 3 in the figure), it assigns the tasks to the accelerator for execution. The CPU sends the task to the accelerator after performing the preprocessing process (such as preparing input data and parameters), so that the accelerator can execute the core computation process of the task (such as matrix calculation, i.e., executing the computation kernel). However, since the accelerator needs to wait for the CPU to perform the preprocessing process before it can execute the core computation process based on the received task, when the CPU takes a long time to perform the preprocessing process, the accelerator will be in a waiting state for a long time, resulting in wasted computational bubbles and low accelerator utilization.

[0070] It should be understood that in related technologies, taking a task in the computational process as an operator as an example, the task is a whole, exposing only a single call interface for the computational kernel. Taking a GPU as an example of an accelerator, the GPU's computational kernel implementation uses CUDA-C (Unified Computing Device Architecture - C Language Extension) code. When compiling CUDA-C code, the CPU and GPU code are usually compiled separately and linked together, exposing only a single call interface for the computational kernel. When a user calls the CUDA-C entry function, the function first performs a CPU preparation phase, and then the GPU's computational kernel executes. However, how these two parts execute on the CPU and GPU is not visible to the outside world. As another example, taking an NPU as an accelerator, the NPU's computational kernel includes CPU-side functions such as tiling and shape inference, as well as an NPU-side startup kernel interface. When a computational kernel is to be executed, the CPU part of this computational kernel needs to be executed first to prepare parameters for the NPU part, and then the NPU part is executed. Therefore, when executing an operator, if the CPU takes a long time to execute, the GPU or NPU will be in a waiting state for a long time, resulting in low utilization.

[0071] Based on this, this application provides a task execution method. For multiple tasks with dependencies in a heterogeneous computing system (or multiple tasks belonging to the same computing process), each task is finely divided into multiple subtasks, including at least one processing task executed by the processor and a computing task executed by the accelerator. Based on this, the processor can concurrently execute processing tasks without dependencies at the processing task granularity and sequentially execute processing tasks with dependencies, thereby reducing the processor overhead, avoiding the accelerator from being in a waiting state for a long time, and improving the utilization rate of the accelerator.

[0072] The implementation environment of this application is described below.

[0073] Figure 3 is a schematic diagram of an implementation environment provided in an embodiment of this application. As shown in Figure 3, the implementation environment includes a heterogeneous computing system 300, which includes a processor 301 and an accelerator 302, and the processor 301 and the accelerator 302 are communicatively connected. In some embodiments, the accelerator 302 is also referred to as an accelerator card, accelerator device, accelerator chip, computing card, training card, etc., and this application is not limited thereto.

[0074] In this embodiment, the heterogeneous computing system 300 can access a wired or wireless network. The processor 301 and accelerator 302 are used to collaboratively execute multiple tasks (or multiple tasks belonging to the same computational process) that have dependencies within the heterogeneous computing system 300. This application does not limit the type of computational process executed by the heterogeneous computing system 300. A task in a computational process may be an operator, or at least two operators. For example, in response to an inference request from an AI model, the processor 301 breaks down the AI ​​model's inference process into multiple tasks. For any given task, the processor 301 and accelerator 302 collaboratively execute that task. The accelerator 302 executes the computational process of the task, i.e., the computational kernel, while the processor 301 executes processing procedures other than the computational kernel, such as shape derivation, data partitioning, memory allocation, memory release, etc., and is not limited to these. In addition, for any given task, there may be multiple subtasks, including at least one processing task executed by the processor and a computation task executed by the accelerator. The at least one processing task includes a processing task that instructs the accelerator to execute the computation task. This processing task can also be understood as an execution task used to trigger the accelerator to execute the computation process.

[0075] Processor 301 is, for example, a CPU, and accelerator 302 is, for example, a GPU, XPU, IPU, TPU, DSA chip, etc., and this application is not limited to these. The number of processors 301 and accelerators 302 shown in the figure is only for illustrative purposes. The number of accelerators 302 and processors 301 can be one or more. When there are multiple accelerators 302, the heterogeneous computing system 300 has distributed computing capabilities and can use various distributed parallel strategies to execute tasks in the computing process. Among them, distributed parallel strategies are, for example, data parallelism, tensor parallelism, pipelined parallelism, or optimizer parallelism, etc. Pipeline parallelism includes, but is not limited to, naive pipelined parallelism, virtual pipelined parallelism, micro-batch pipelined parallelism, etc.

[0076] In some embodiments, the heterogeneous computing system 300 is deployed on a general-purpose physical server, or a computing device such as a desktop computer or terminal device. For example, referring to FIG4, FIG4 is a schematic diagram of a heterogeneous computing system provided in an embodiment of the present application. As shown in FIG4, taking the heterogeneous computing system 300 deployed in a server as an example, the processor 301 is the CPU of the host in the server. It should be understood that the host is an overall device or module including CPU, memory and other related components, which works with the accelerator 302 in the server to realize various complex computing functions. Among them, the processor 301 and the accelerator 302 are connected for communication through a peripheral component interconnect express (PCIe) link, and the processor 301 and the accelerator 302 interact with each other through the PCIe link. When there are multiple accelerators 302 in the server, the accelerators 302 are connected to each other through high-speed interconnect links. For example, the high-speed interconnect links are Huawei cache coherent system (HCCS), RDMA over converged Ethernet (RoCE), NVIDIA Link, compute express link (CXL), universal chiplet interconnect express (UCIe), cache coherent interconnect for accelerators (CCIX), etc., and this application is not limited to these.

[0077] In other embodiments, the heterogeneous computing system 300 can also be deployed on a cloud platform. A cloud platform, short for cloud computing platform, refers to a service that provides computing, networking, and storage capabilities based on hardware and software resources. By processing and analyzing massive amounts of data remotely via the network "cloud" and then returning it to the user, it features large scale, distribution, virtualization, high availability, scalability, on-demand service, and security. Cloud platforms can achieve rapid provisioning and release of configurable computing resources with relatively low management costs or low interaction complexity between users and service providers.

[0078] Furthermore, the aforementioned wireless or wired networks utilize standard communication technologies and / or protocols. These networks are typically Transmission Control Protocol / Internet Protocol (TCP / IP) networks and RDMA networks, such as RoCE networks and InfiniBand (IB) networks, and are not limited to these. In other embodiments, customized and / or dedicated data communication technologies can be used to replace or supplement the aforementioned data communication technologies.

[0079] Based on the implementation environment shown in Figures 3 and 4 above, this application provides a computing device capable of being configured as the aforementioned heterogeneous computing system 300. The aforementioned implementation environment can be implemented by one or more computing devices. For example, a heterogeneous computing system can be deployed on multiple computing devices, each computing device being configured with a processor and at least one accelerator. Alternatively, a heterogeneous computing system can be deployed on a single computing device, which is configured with a processor and at least one accelerator. The structure of the computing device will now be described with reference to Figure 5.

[0080] Figure 5 is a schematic diagram of a computing device provided in an embodiment of this application. As shown in Figure 5, the computing device 500 includes a memory 501, a processor 502, a communication interface 503, a bus 504, and an accelerator 505. The memory 501, processor 502, communication interface 503, and accelerator 505 are interconnected via the bus 504.

[0081] The memory 501 may be a read-only memory (ROM) or other type of static storage device capable of storing static information and instructions, random access memory (RAM) or other type of dynamic storage device capable of storing information and instructions, or it may be an electrically erasable programmable read-only memory (EEPROM), a compact disc read-only memory (CD-ROM) or other optical disc storage, optical disc storage (including compressed optical discs, laser discs, optical discs, digital universal optical discs, Blu-ray discs, etc.), magnetic disk storage media or other magnetic storage devices, or any other medium capable of carrying or storing desired program code in the form of instructions or data structures and accessible by a computer, but is not limited thereto. In this application, the memory 501 is used to store at least one piece of program code. When the program code stored in the memory 501 is executed by the processor 502, the processor 502 is used to execute the task execution method provided in this application.

[0082] The processor 502 may be a network processor (NP), a central processing unit (CPU), an application-specific integrated circuit (ASIC), or an integrated circuit used to control the execution of the program of the present application. The processor 502 may be a single-core (single-CPU) processor or a multi-core (multi-CPU) processor. The number of processors 502 may be one or more.

[0083] The communication interface 503 uses a transceiver module, such as a transceiver, to enable communication between the computing device 500 and other devices or communication networks. For example, data can be acquired through the communication interface 503.

[0084] The memory 501 and the processor 502 can be set separately or integrated together.

[0085] Bus 504 may include a pathway for transmitting information between various components of computing device 500 (e.g., memory 501, processor 502, communication interface 503, accelerator 505).

[0086] Accelerator 505 may be, for example, a GPU, NPU, IPU, TPU, DSA chip, etc., and this application does not limit it to this.

[0087] This application also provides a computing device cluster, which includes multiple computing devices connected to each other via a network. The network can be a wide area network (WAN) or a local area network (LAN), etc. This computing device cluster is used to implement the task execution method provided in this application.

[0088] The task execution method provided in this application is described below.

[0089] For ease of understanding, the principle of this application will be introduced first with reference to Figure 6. Figure 6 is a schematic diagram of the computation process of a heterogeneous computing system provided in an embodiment of this application. As shown in Figure 6, after the processor decomposes the entire computation process into multiple tasks, the process of scheduling the accelerator to execute the tasks includes: the processor executing the preprocessing process of the task (such as preparing input data and parameters), the processor sending the computation process in the task (including the computing kernel, input data and parameters, etc.) to the accelerator, and the accelerator executing the computation process of the task. Based on this, this application decomposes the tasks in the computation process into multiple sub-tasks with fine granularity (or abstracts the tasks into stages, clarifying the CPU preparation stage, accelerator computation stage, and CPU post-processing stage in the task), and schedules and executes the sub-tasks obtained after decomposition, thereby saving processor-side overhead.

[0090] For example, referring to Figure 7, which is a schematic diagram of splitting a task into multiple subtasks according to an embodiment of this application. As shown in Figure 7, taking the collaborative execution of task 1, task 2, and task 3 by a processor and accelerator in a heterogeneous computing system as an example. In related technologies, since the tasks in the computing process are a whole, when the processor takes a long time to execute the preprocessing process, the accelerator will be in a waiting state for a long time, that is, there are wasted computing bubbles, resulting in low accelerator utilization. In this application, the tasks in the computing process are split into multiple subtasks, including at least one processing task executed by the processor and a computing task executed by the accelerator. The figure illustrates each task as being split into 3 processing tasks and one computing task. For example, the 3 processing tasks of task 1 are processing task A, processing task B, and processing task C, where processing task B is used to trigger the accelerator to execute computing task 1, that is, to execute computing kernel 1. Because each task is split into multiple processing tasks to be executed by the processor, the dependencies between these processing tasks can be reconstructed, and multi-threading technology can be used to schedule these processing tasks concurrently. This saves processor overhead while ensuring that the dependencies between the original tasks are not affected, thereby avoiding the accelerator from being in a waiting state for a long time and improving the utilization of the accelerator.

[0091] Referring to Figure 8 and the heterogeneous computing system described in the aforementioned implementation environment, the principles of this application will be further explained below. Figure 8 is a functional architecture diagram of a heterogeneous computing system provided in an embodiment of this application. As shown in Figure 8, the heterogeneous computing system is used to provide task acquisition function 801 and task execution function 802. In some embodiments, the heterogeneous computing system is also used to provide performance analysis function 803 and dependency analysis function 804.

[0092] The task acquisition function 801 is implemented by the processor in the heterogeneous computing system. Taking the execution of an AI model inference request by the heterogeneous computing system as an example, the processor responds to the AI ​​model's inference request and, according to the AI ​​model's inference flow, acquires multiple tasks in the inference flow, where these tasks have dependencies. A dependency means that the execution of one task depends on the execution result of another task. This could mean that the execution of one task requires the execution result of another task as input, or that the execution of one task can only proceed after obtaining the execution result of another task. For example, multiple tasks include a first task and a second task. The first task is an addition operator, and the second task is a matrix multiplication operator. The second task depending on the execution result of the first task means that after the heterogeneous computing system executes the addition operator and obtains its result, it uses that result as input data for the matrix multiplication operator; or, the heterogeneous computing system needs to execute the addition operator and obtain its result before executing the matrix multiplication operator.

[0093] In this application, the tasks in a heterogeneous computing system include at least one processing task executed by a processor and a computational task executed by an accelerator, wherein each processing task can be implemented by a function. Taking a task as an operator as an example, the function of the operator can be implemented by at least one processing task and a computational task. For example, at least one processing task and a computational task are defined during the implementation of the operator backend and enabled as needed. Each operator backend can define different processing tasks and computational tasks based on its characteristics. This implementation method will be described in detail in subsequent embodiments.

[0094] The task execution function 802 is implemented collaboratively by processors and accelerators in a heterogeneous computing system. For any given task, at least one processing task corresponds to that task, including a processing task that instructs the accelerator to execute the computation task; that is, the processing task is used to trigger the accelerator to execute the computation process, or in other words, to execute the computation kernel. In this application, since each task can correspond to at least one processing task, the processor can concurrently execute processing tasks that do not have dependencies on each other, and sequentially execute processing tasks that do have dependencies on each other, thereby saving processor overhead.

[0095] Schematic illustration: For any given task, at least one processing task further includes at least one of shape derivation and data segmentation of the task's input data. The processor determines the parameters required for the accelerator to perform the computation task by performing shape derivation and data segmentation on the task's input data. Shape derivation refers to determining information such as the dimensions and size of the input data, providing a basis for subsequent data segmentation and obtaining the accelerator's computational kernel parameters. For example, when processing image data, it is necessary to determine the image's dimensions (length, width, height, number of channels, etc.), as well as the size and batch information of the entire dataset. Data segmentation refers to dividing the task's input data into appropriate parts according to the task requirements and the accelerator's characteristics. For example, the input matrix of a matrix multiplication operator can be divided according to the number of computational units and storage capacity on the accelerator. After obtaining the execution result of at least one of the shape derivation and data segmentation, the processor executes the processing task to instruct the accelerator to perform the computation task; that is, it packages and sends the execution result of at least one of the shape derivation and data segmentation along with the computational kernel to the accelerator, or sends a task instruction packet to the accelerator to tell it how to perform the computation. The accelerator executes computational tasks using its own computing resources based on the received task instruction packets. For example, the accelerator reads data according to the starting position and size specified by parameters, and performs calculations using algorithms defined by the computing kernel, such as convolution, pooling, and fully connected operations in deep learning models, to complete the actual computation. Simultaneously, the processor can continue executing other processing tasks related to that task, such as memory release. Alternatively, if the task instructing the accelerator to execute the computation task is the last one among multiple processing tasks, the processor executes the next task's processing task without waiting for the accelerator to complete the current task's processing task.

[0096] In some embodiments, the computing system further provides a performance analysis function 803, implemented by the processor. Illustratively, the processor obtains the total time taken by the heterogeneous computing system to execute multiple tasks over a historical time period, and the computation time taken by the accelerator to execute the computational tasks of these multiple tasks during the same historical time period. Based on the difference between the total time and the computational time, the processor determines the bubble time of the accelerator during the historical time period, i.e., the duration the accelerator is in a waiting state. If the ratio between the bubble time and the total time is greater than a threshold, the processing and computational tasks of each task are executed through the task execution function 802 to save processor overhead. The threshold is a preset threshold, such as 80%, which can be set according to business needs; this application does not limit this setting. Through the performance analysis function 803, when a large number of bubbles are detected in the accelerators of the heterogeneous computing system, tasks are promptly split into fine-grained parts to improve accelerator utilization.

[0097] In some embodiments, the heterogeneous computing system is further configured to provide a dependency analysis function 804, implemented by the processor. Illustratively, after the processor acquires multiple tasks with dependencies through the aforementioned task acquisition function 801, it constructs the dependencies between the processing tasks of these tasks based on the dependencies between these tasks. Based on this, the processor can execute the processing tasks of these tasks according to the dependencies between different processing tasks through the task execution function 802, saving processor overhead while ensuring that the original dependencies between tasks are not affected.

[0098] Schematic illustration: the functionality provided by the aforementioned heterogeneous computing system can be installed as a software toolkit component within a computing device and run by the processor and accelerator within that device. For example, when the hardware architecture of a computing device employs a heterogeneous computing architecture (including computing architectures using computing units with different types of instruction sets), users can install heterogeneous computing frameworks on the device. One such framework is the Compute Architecture for NeuroNet (CANN), a heterogeneous computing framework for neural networks. CANN can support users in quickly building AI applications by providing multi-layered programming interfaces. Additionally, users can install deep learning frameworks on computing devices to compile methods for implementing models, construct large-scale computational graphs, and automatically perform gradient calculations within the computational graph. The functionality provided by the aforementioned computing system can be interfaced and adapted with deep learning frameworks and heterogeneous computing frameworks.

[0099] It should be noted that the functional division of a heterogeneous computing system is not limited to what is shown in Figure 8. In practical applications, more functions can be set according to user needs. For example, a heterogeneous computing system can also be used to provide storage functions, etc. This application does not limit this.

[0100] The task execution method provided in this application will be described below through several method implementation examples.

[0101] Figure 9 is a flowchart of a task execution method provided in an embodiment of this application. As shown in Figure 9, the method is applied to the heterogeneous computing system described in the aforementioned implementation environment. The heterogeneous computing system includes a processor and an accelerator. The method is executed by the processor and includes the following steps 901 to 904.

[0102] 901. In response to the task execution request, the processor obtains the first and second tasks in the computation process.

[0103] In this embodiment, the heterogeneous computing system can access wired or wireless networks to provide users with various computing services, such as AI services. Illustratively, a task execution request is, for example, an inference or training request for an AI model. The processor responds to the inference or training request by acquiring a first task and a second task. The first and second tasks are tasks generated based on the task execution request and are dependent on each other; the second task depends on the execution result of the first task. For example, the first task is an addition operator, and the second task is a matrix multiplication operator. The second task depending on the execution result of the first task means that after the heterogeneous computing system executes the addition operator and obtains the result, it executes the matrix multiplication operator based on that result. It should be understood that the number of tasks generated based on the task execution request can be greater. In this embodiment, for ease of understanding, the first and second tasks are used as examples.

[0104] The AI ​​model involved in this application is used to perform reasoning on input data to output reasoning results. The application areas of the AI ​​model include text reasoning, image reasoning, video reasoning, audio reasoning, and so on. For example, when applied to text reasoning, the input data of the AI ​​model is text, and the AI ​​model is used to analyze the text to output a text summary; or, the AI ​​model is used to reason on question text to output the answer to the question, and so on. As another example, when applied to image reasoning scenarios, the input data of the AI ​​model is an image, and the AI ​​model is used to recognize the image to output an image recognition result. Furthermore, the input data can also be audio or video, and is not limited to these. The AI ​​model training scenario involved in this application can also employ a distributed parallel strategy to implement AI model training.

[0105] 902. The processor determines at least one first processing task executed by the processor and a first computing task executed by the accelerator in the first task, and determines at least one second processing task executed by the processor and a second computing task executed by the accelerator in the second task.

[0106] In the embodiments of this application, both the first task and the second task correspond to at least one processing task, and each processing task can be implemented by a function. Specifically, at least one first processing task of the first task includes a processing task that instructs the accelerator to execute a first computation task, and at least one second processing task of the second task includes a processing task that instructs the accelerator to execute a second computation task.

[0107] In some embodiments, the first task and the second task are determined based on an inference request or a training request from the AI ​​model. At least one first processing task of the first task further includes at least one of shape derivation and data segmentation of the input data of the first task. The processing task instructing the accelerator to perform the first computation task depends on the execution result of at least one of shape derivation and data segmentation of the input data of the first task. Similarly, at least one second processing task of the second task further includes at least one of shape derivation and data segmentation of the input data of the second task. The processing task instructing the accelerator to perform the second computation task depends on the execution result of at least one of shape derivation and data segmentation of the input data of the second task.

[0108] It should be noted that this application does not limit the number of at least one first processing task and at least one second processing task. Furthermore, the number of at least one first processing task and at least one second processing task can be the same, or the number of at least one first processing task and at least one second processing task can be different. In practical applications, tasks can be finely divided according to their functions.

[0109] Taking an example where the number of at least one first processing task and at least one second processing task are the same, for instance, the first task is an addition operator and the second task is a matrix multiplication operator. The first task includes three processing tasks: processing task A (for shape derivation of the input data, also known as the infershape task), processing task B (for segmenting the input data, also known as the tiling task), and processing task C (also known as the execute task, used to instruct the accelerator to perform the first computation task). Similarly, the second task includes three processing tasks: processing task D (for shape derivation of the input data, also known as the infershape task), processing task E (for segmenting the input data, also known as the tiling task), and processing task F (also known as the execute task, used to instruct the accelerator to perform the second computation task).

[0110] Taking an example where the number of at least one first processing task and the number of at least one second processing task differ, for instance, the first task is an addition operator and the second task is a matrix multiplication operator. The first task includes three processing tasks: processing task A (used for shape derivation of the input data, also known as the infershape task), processing task B (used for segmenting the input data, also known as the tiling task), and processing task C (also known as the execute task, used to instruct the accelerator to perform the first computation task). The second task includes two processing tasks: processing task H (used for preprocessing the input data such as shape derivation and data segmentation, also known as the setup task) and processing task F (also known as the execute task, used to instruct the accelerator to perform the second computation task).

[0111] It should be understood that the above description of at least one first processing task and at least one second processing task is merely illustrative. In practical applications, tasks can be divided into more granular categories according to requirements. For example, a task can be split into 5 processing tasks and 1 computation task, or a task can be split into 8 processing tasks and 1 computation task, etc. This application does not limit this.

[0112] After completing steps 901 and 902, the processor, having obtained the first and second tasks with dependencies, determines the first processing task to be executed by the processor in the first task and the second processing task to be executed by the processor in the second task. In some embodiments, the processor can determine whether execution needs to be performed at the processing task level based on the performance analysis results of the heterogeneous computing system executing the first and second tasks over a historical time period.

[0113] Schematic, the processor obtains the total time spent by the heterogeneous computing system in executing the first and second tasks during a historical time period, as well as the computation time spent by the accelerator in executing the first and second computation tasks during the same historical time period. Based on the difference between the total time and the computation time, the processor determines the bubble time spent by the accelerator during the historical time period. If the ratio between the bubble time spent and the total time spent is greater than a threshold, step 902 is executed. The threshold is a preset threshold, such as 80%, which can be set according to business needs, and this application does not limit it. For example, after obtaining the first and second tasks in step 901, the processor first executes a computation process at the task level to obtain the total time spent and the computation time spent by the accelerator. Then, based on the difference between the total time spent and the computation time spent, the processor determines the bubble time spent by the accelerator. If the ratio between the bubble time spent and the total time spent is greater than the threshold, it indicates that the processor's overhead is large, and step 902 is executed to determine the processing tasks executed by the processor in each task.

[0114] Furthermore, in the training scenario of the AI ​​model, the aforementioned historical time period can be the time consumed by the first N iterations of the AI ​​model, where N is a positive integer, and this application does not impose any limitations on this. In this way, during the training process of the AI ​​model, the bubble time consumption of the accelerator is detected in real time. When a large number of bubbles are detected in the accelerators of the heterogeneous computing system, the task is promptly broken down into finer-grained parts to improve the utilization rate of the accelerator.

[0115] The following steps, 903 and 904, describe the process by which the processor executes tasks at the task-processing level.

[0116] 903. The processor determines the dependencies between the processing tasks in at least one first processing task and at least one second processing task.

[0117] In this embodiment, the processor determines the dependencies between at least one first processing task and at least one second processing task based on the dependencies between the first task and the second task, the dependencies between processing tasks in at least one first processing task, and the dependencies between processing tasks in at least one second processing task.

[0118] Taking the example where the number of first and second processing tasks is the same, refer to Figure 10, which is a schematic diagram of the dependency relationship between various processing tasks provided in an embodiment of this application. As shown in Figure 10, the first task is an addition operator, and the second task is a matrix multiplication operator. The first task includes three processing tasks: processing task A (also called the infershape task), processing task B (also called the tiling task), and processing task C (also called the execute task). The second task includes three processing tasks: processing task D (also called the infershape task), processing task E (also called the tiling task), and processing task F (also called the execute task). Based on this, the infershape task of the addition operator is connected to the infershape task of the matrix multiplication operator, meaning there is a dependency relationship between the infershape task of the addition operator and the infershape task of the matrix multiplication operator; and the execute task of the addition operator is connected to the execute task of the matrix multiplication operator, meaning there is a dependency relationship between the execute task of the addition operator and the execute task of the matrix multiplication operator. In addition, the infershape, tiling, and execute tasks of each operator are kept in sequence to ensure that the computation flow when the processor executes at the task-level granularity is logically consistent with the computation flow when the processor executes at the task-level granularity.

[0119] Taking the example of a difference in the number of first and second processing tasks, refer to Figure 11, which is a schematic diagram of another dependency relationship between processing tasks provided in an embodiment of this application. As shown in Figure 11, the first task is an addition operator, and the second task is a matrix multiplication operator. The first task includes three processing tasks: processing task A (used for shape derivation of the input data, also known as the infershape task), processing task B (used for segmenting the input data, also known as the tiling task), and processing task C (also known as the execute task, used to instruct the accelerator to execute the first computation task). The second task includes two processing tasks: processing task H (used for preprocessing the input data such as shape derivation and data segmentation, also known as the setup task) and processing task F (also known as the execute task, used to instruct the accelerator to execute the second computation task). Based on this, the infershape task of the addition operator is connected to the setup task of the matrix multiplication operator, that is, there is a dependency between the infershape task of the addition operator and the setup task of the matrix multiplication operator; and the execute task of the addition operator is connected to the execute task of the matrix multiplication operator, that is, there is a dependency between the execute task of the addition operator and the execute task of the matrix multiplication operator, thereby realizing the differential fine-grained splitting across the back-end of the operator.

[0120] 904. The processor concurrently executes at least one first processing task and at least one second processing task in which there is no dependency between them, and sequentially executes at least one first processing task and at least one second processing task in which there is a dependency between them.

[0121] In the embodiments of this application, for processing tasks that do not have a dependency relationship among at least one first processing task and at least one second processing task, the processor can execute these processing tasks concurrently based on multi-threaded scheduling technology, while for processing tasks that have a dependency relationship among at least one first processing task and at least one second processing task, the processor executes these processing tasks sequentially.

[0122] Schematic, the processor creates multiple threads that run concurrently; through the first thread of the multiple threads, the third processing task of the multiple first processing tasks and the fourth processing task of the multiple second processing tasks are executed sequentially; through the second thread of the multiple threads, the fifth processing task of the multiple first processing tasks and the sixth processing task of the multiple second processing tasks are executed sequentially, wherein the execution order of the fifth processing task is after the execution order of the third processing task, and the execution order of the sixth processing task is after the execution order of the fourth processing task.

[0123] Referring to Figure 12, taking the example of the same number of first and second processing tasks, the implementation of step 904 is described below. Figure 12 is a schematic diagram of multi-threaded scheduling and execution of processing tasks provided by an embodiment of this application. As shown in Figure 12, taking the first and second tasks shown in Figure 10 above as examples, the processor creates multiple threads that run concurrently: thread 1, thread 2, and thread 3. Thread 1 is used to sequentially execute the infershape task of the addition operator and the infershape task of the matrix multiplication operator; thread 2 is used to sequentially execute the tilting task of the addition operator and the tilting task of the matrix multiplication operator; and thread 3 is used to sequentially execute the execute task of the addition operator and the execute task of the matrix multiplication operator. It can be seen that for processing tasks without dependencies, such as the infershape task of the matrix multiplication operator and the tilting task of the addition operator, the processor executes them concurrently through thread 1 and thread 2; for processing tasks with dependencies, such as the infershape task of the addition operator and the infershape task of the matrix multiplication operator, or the infershape task of the addition operator and the tilting task of the addition operator, the processor executes these processing tasks sequentially through each thread.

[0124] In some embodiments, if there is a target processing task in at least one first processing task and at least one second processing task, the target processing task refers to a processing task that does not need to be executed in every calculation, then after the processor executes the target processing task, it executes step 904. That is, these target processing tasks are executed in advance during the initialization phase, further reducing the processor overhead.

[0125] In other embodiments, the first and second tasks are determined based on the inference or training requests of the AI ​​model. At least one first processing task of the first task includes at least one of shape derivation and data segmentation of the input data of the first task, and at least one second processing task of the second task includes at least one of shape derivation and data segmentation of the input data of the second task. Based on this, during the execution of step 904, the processor can store the execution results of shape derivation and / or data segmentation. Thus, when the accelerator is subsequently instructed to execute a computation task, if the shape of the input data has not changed, the stored execution results of shape derivation and / or data segmentation can be reused, further saving processor overhead.

[0126] Schematic example, taking the first task as an example, during the execution of at least one first processing task in the first time period, the processor stores the execution result of at least one of shape derivation and data segmentation on the input data of the first task; then, during the execution of at least one first processing task in the second time period, if the input data of the first task has the same shape as the input data when the first task was executed in the first time period, at least one of the shape derivation and data segmentation is skipped, and based on the execution result stored in the first time period, the accelerator is instructed to execute the first computation task, and the second time period is after the first time period. It should be noted that the second task can be executed in the same way, which will not be described in detail here.

[0127] In some scenarios, if the first and second tasks need to be performed multiple times, step 904 above can be replaced by performing the following steps:

[0128] Based on the dependencies between the processing tasks in at least one first processing task and at least one second processing task, during the i-th execution of at least one first processing task and at least one second processing task (where i is a positive integer), the processor stores the execution results of shape derivation and / or data segmentation corresponding to the first task and the execution results of shape derivation and / or data segmentation corresponding to the second task. During the (i+m)-th execution of at least one first processing task and at least one second processing task (where m is a positive integer), if the shape of the input data of the first task is the same as the shape of the input data of the first task during the i-th execution of at least one first processing task, then the shape derivation and / or data segmentation is skipped, and the processor instructs the accelerator to execute the first computation task based on the execution results stored in the i-th execution; or, if the shape of the input data of the second task is the same as the shape of the input data of the second task during the i-th execution of at least one second processing task, then the processor skips the shape derivation and / or data segmentation, and the processor instructs the accelerator to execute the second computation task based on the execution results stored in the i-th execution.

[0129] The above process is also a kind of intermediate result caching technology. When the first and second tasks need to be executed multiple times (such as multiple iterations in the AI ​​model training scenario), the processor can store the execution results of shape derivation and / or data segmentation when each task is executed. When the processor subsequently executes the same task, if the shape of the input data of the task has not changed, the previously stored execution results of shape derivation and / or data segmentation can be reused, thereby saving processor overhead.

[0130] In summary, in the task execution method provided in this application, the processor obtains a first task and a second task that have a dependency relationship in the computation process. Each task includes at least one processing task executed by the processor and a computation task executed by the accelerator. Based on this, the processor can concurrently execute processing tasks that do not have a dependency relationship at the granularity of the processing tasks in each task, and sequentially execute processing tasks that have a dependency relationship, thereby reducing the processor overhead, avoiding the accelerator from being in a waiting state for a long time, and improving the utilization rate of the accelerator.

[0131] Based on the method shown in Figure 9 above, the task execution method provided in this application will be illustrated below with reference to Figure 13, which is a flowchart of a task execution method provided in an embodiment of this application. As shown in Figure 13, this method is executed by a processor in a heterogeneous computing system. For example, on a server with a 32-core CPU, the CPU is connected to an accelerator card (NPU) via a PCIe bus, and the CPU and NPU together form a heterogeneous computing system. Taking the computational process executed by the heterogeneous computing system as 10 tasks of an AI model as an example, each task needs to perform shape derivation and data segmentation on the CPU first to obtain the parameters of the NPU computing kernel. After obtaining these parameters, these parameters and the NPU computing kernel are packaged and sent to the NPU to enable the heterogeneous computing task of the NPU. When the shape derivation and data segmentation calculation of the current task are completed, the shape derivation and data segmentation calculation of the next task can begin, without waiting for the NPU computing kernel of the current task to be fully executed before starting the execution of the next task.

[0132] Indicatively, the task execution method provided in this application includes the following steps:

[0133] Step 1: The processor responds to the task execution request and obtains multiple tasks with dependencies. That is, it obtains the computing process to be executed in the heterogeneous computing system.

[0134] Step 2: After the heterogeneous computing system executes multiple tasks for the i-th time, the processor obtains the total time spent executing multiple tasks for the i-th time and the computation time spent by the accelerator in executing the computation task. Based on the difference between the total time spent and the computation time spent, the bubble time of the accelerator in the i-th time is determined.

[0135] Step 3: The processor determines whether the ratio between the bubble time of the i-th accelerator and the total time is greater than the threshold. If it is greater than the threshold, the processor executes Step 4 below and enters the process of executing multiple tasks for the (i+1)th time. If it is less than or equal to the threshold, the processor enters the process of executing multiple tasks for the (i+1)th time in the same way as Step 2.

[0136] Step 4: The processor determines the processing tasks to be executed by the processor and the computational tasks to be executed by the accelerator in each task. It should be understood that the number of processing tasks in each task can be the same or different, and can be set according to business needs. For example, in 10 tasks, each task includes 3 processing tasks. Or, in 10 tasks, the first 5 tasks include 3 processing tasks, the last 5 tasks include 2 processing tasks, and so on; this application does not limit this.

[0137] Step 5: The processor determines the dependencies between each processing task in each processing task based on the dependencies between each task.

[0138] Step 6: The processor distributes the processing tasks of each task to the concurrent scheduling engine according to the dependencies between the processing tasks, and schedules and executes the processing tasks in a reasonable manner.

[0139] The processor can employ multi-threading concurrency technology to concurrently execute tasks that have no dependencies, and sequentially execute tasks that have dependencies. Alternatively, the processor can use intermediate result caching technology to store the execution results of shape derivation and / or data segmentation during the (i+1)th execution of each task. When the processor executes each task for the (i+2)th time (or i+3 and beyond), if the shape of the input data for the task has not changed, the previously stored execution results of shape derivation and / or data segmentation can be reused, thereby saving processor overhead.

[0140] Furthermore, based on the embodiment shown in Figure 9 above, it can be seen that the processor may also skip steps 2 and 3, that is, execute step 4 after executing step 1. This application does not limit this.

[0141] In summary, in the task execution method provided in this application, the processor obtains multiple tasks with dependencies. Each task can be broken down into processing tasks executed by the processor and computing tasks executed by the accelerator. Based on this, the processor can reasonably schedule the execution of processing tasks in each task at the granularity of processing tasks, thereby reducing the processor overhead, avoiding the accelerator from being in a waiting state for a long time, and improving the utilization rate of the accelerator.

[0142] Figure 14 is a schematic diagram of a task execution device provided in an embodiment of this application. As shown in Figure 14, the device is configured in a processor of a heterogeneous computing system, and the device includes a task acquisition module 1401 and a task execution module 1402.

[0143] The task acquisition module 1401 is used for at least one first processing task executed by the processor and a first computing task executed by the accelerator, and the second task includes at least one second processing task executed by the processor and a second computing task executed by the accelerator.

[0144] The task execution module 1402 is used to concurrently execute at least one first processing task and at least one second processing task that do not have a dependency relationship, and to sequentially execute at least one first processing task and at least one second processing task that have a dependency relationship.

[0145] Wherein, at least one first processing task includes a processing task that instructs the accelerator to perform a first computing task, and at least one second processing task includes a processing task that instructs the accelerator to perform a second computing task.

[0146] In some embodiments, the task execution module 1402 is configured to:

[0147] Create multiple threads that run concurrently;

[0148] The first thread of a plurality of threads sequentially executes at least one third processing task in a first processing task and at least one fourth processing task in a second processing task.

[0149] The second thread among multiple threads sequentially executes at least one fifth processing task in the first processing task and at least one sixth processing task in the second processing task;

[0150] The fifth processing task is executed after the third processing task, and the sixth processing task is executed after the fourth processing task.

[0151] In some embodiments, the apparatus further includes:

[0152] The time acquisition module is used to acquire the total time taken by the heterogeneous computing system to execute the first and second tasks in a historical time period, as well as the computing time taken by the accelerator to execute the first and second computing tasks in a historical time period.

[0153] The determination module is used to determine the bubble time of the accelerator in a historical time period based on the difference between the total time and the computation time.

[0154] Task execution module 1402 is used for:

[0155] If the ratio of bubble processing time to total processing time is greater than a threshold, then at least one of the first processing tasks and at least one of the second processing tasks that do not have a dependency relationship will be executed concurrently, and at least one of the first processing tasks and at least one of the second processing tasks that have a dependency relationship will be executed sequentially.

[0156] In some embodiments, the computation process refers to the inference process or training process of an artificial intelligence (AI) model. The task acquisition module 1401 is used to: acquire a first task and a second task in response to an inference request or training request of the AI ​​model; wherein at least one of the first processing tasks includes at least one of shape derivation and data segmentation of the input data of the first task.

[0157] In some embodiments, the apparatus further includes:

[0158] A storage module is used to store the execution result of at least one of shape derivation and data segmentation of the input data during the execution of at least one first processing task in the first time period.

[0159] After concurrently executing at least one first processing task and at least one second processing task that have no dependencies on each other, and sequentially executing at least one first processing task and at least one second processing task that have dependencies on each other, the task execution module 1402 is further configured to:

[0160] During the execution of at least one first processing task in the second time period, if the shape of the input data of the first task is the same as the shape of the input data of the first task when at least one first processing task is executed in the first time period, then at least one of shape derivation and data segmentation is skipped, and the accelerator is instructed to execute the first computation task based on the execution result stored in the first time period, after the first time period.

[0161] Through the aforementioned device, the processor acquires a first task and a second task that are dependent on each other in the computation process. Each task includes at least one processing task executed by the processor and a computation task executed by the accelerator. Based on this, the processor can concurrently execute processing tasks that are not dependent on each other, and sequentially execute processing tasks that are dependent on each other, thereby reducing the processor's overhead, preventing the accelerator from being in a waiting state for a long time, and improving the utilization rate of the accelerator.

[0162] Of course, the device can also include other functional units to implement other functions involved in the processor in the above method embodiments. In practical applications, the above functions can be assigned to different functional units as needed, that is, the internal structure of the device can be divided into different functional units to complete all or part of the functions described above. In addition, the device provided in the above embodiments and the above method embodiments belong to the same concept, and its specific implementation process is detailed in the method embodiments, which will not be repeated here.

[0163] In this application, the terms "first," "second," etc., are used to distinguish identical or similar items with substantially the same function. It should be understood that there is no logical or temporal dependency between "first," "second," and "nth," nor does it limit the quantity or execution order. It should also be understood that although the following description uses the terms "first," "second," etc., to describe various elements, these elements should not be limited by the terms. These terms are merely used to distinguish one element from another. For example, without departing from the various examples described, a first operator can be referred to as a second operator, and similarly, a second operator can be referred to as a first operator. Both the first and second operators can be operators, and in some cases, they can be separate and distinct operators.

[0164] In this application, the term "at least one" means one or more, and the term "multiple" means two or more. For example, multiple operators means two or more operators.

[0165] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any person skilled in the art can easily conceive of various equivalent modifications or substitutions within the technical scope disclosed in this application, and these modifications or substitutions should all be covered within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

[0166] In the above embodiments, implementation can be achieved, in whole or in part, through software, hardware, firmware, or any combination thereof. When implemented in software, it can be implemented, in whole or in part, in the form of program structure information. This program structure information includes one or more program instructions. When these program instructions are loaded and executed on a computing device, the processes or functions according to the embodiments of this application are generated, in whole or in part.

[0167] Those skilled in the art will understand that all or part of the steps of the above embodiments can be implemented by hardware or by a program instructing related hardware. The program can be stored in a computer-readable storage medium, such as a read-only memory, a disk, or an optical disk.

[0168] The above-described embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit it. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of this application.

Claims

1. A task execution method, characterized in that, A processor applied in a heterogeneous computing system, the heterogeneous computing system further including an accelerator, the processor and the accelerator being used to jointly execute a computing process, the method comprising: Obtain a first task and a second task in the computing process, wherein the first task and the second task are dependent on each other, the first task includes at least one first processing task executed by the processor and a first computing task executed by the accelerator, and the second task includes at least one second processing task executed by the processor and a second computing task executed by the accelerator. Concurrently execute processing tasks that are not dependent on each other among the at least one first processing task and the at least one second processing task; sequentially execute processing tasks that are dependent on each other among the at least one first processing task and the at least one second processing task. The at least one first processing task includes a processing task that instructs the accelerator to perform the first computing task, and the at least one second processing task includes a processing task that instructs the accelerator to perform the second computing task.

2. The method according to claim 1, characterized in that, The concurrent execution of processing tasks that have no dependency among the at least one first processing task and the at least one second processing task, and the sequential execution of processing tasks that have dependency among the at least one first processing task and the at least one second processing task, includes: Create multiple threads that run concurrently; The third processing task in the at least one first processing task and the fourth processing task in the at least one second processing task are executed sequentially through the first thread of the plurality of threads; The fifth processing task in the at least one first processing task and the sixth processing task in the at least one second processing task are executed sequentially through the second thread among the plurality of threads; The fifth processing task is executed after the third processing task, and the sixth processing task is executed after the fourth processing task.

3. The method according to claim 1 or 2, characterized in that, The method further includes: Obtain the total time taken by the heterogeneous computing system to execute the first task and the second task during a historical time period, and the computation time taken by the accelerator to execute the first computing task and the second computing task during the historical time period; Based on the difference between the total time and the calculation time, the bubble time of the accelerator in the historical time period is determined; The concurrent execution of processing tasks that have no dependency among the at least one first processing task and the at least one second processing task, and the sequential execution of processing tasks that have dependency among the at least one first processing task and the at least one second processing task, includes: If the ratio between the bubble processing time and the total processing time is greater than a threshold, then concurrently execute the processing tasks that do not have a dependency relationship among the at least one first processing task and the at least one second processing task, and sequentially execute the processing tasks that have a dependency relationship among the at least one first processing task and the at least one second processing task.

4. The method according to any one of claims 1 to 3, characterized in that, The computational process refers to the inference or training process of an artificial intelligence (AI) model. Obtaining the first and second tasks within the computational process includes: In response to the inference or training request of the AI ​​model, the first task and the second task are obtained; The at least one first processing task includes at least one of shape derivation and data segmentation of the input data of the first task.

5. The method according to claim 4, characterized in that, The method further includes: During the execution of the at least one first processing task in the first time period, the execution result of at least one of shape derivation and data segmentation of the input data is stored; After concurrently executing the processing tasks that have no dependency among the at least one first processing task and the at least one second processing task, and sequentially executing the processing tasks that have dependency among the at least one first processing task and the at least one second processing task, the method further includes: During the execution of the at least one first processing task in the second time period, if the shape of the input data of the first task is the same as the shape of the input data of the first task when the at least one first processing task is executed in the first time period, then at least one of shape derivation and data segmentation is skipped, and the accelerator is instructed to execute the first computing task based on the execution result stored in the first time period, wherein the second time period is after the first time period.

6. A task execution device, characterized in that, A processor for use in a heterogeneous computing system, the heterogeneous computing system further including an accelerator, the processor and the accelerator being used to jointly execute a computing process, the device comprising: The task acquisition module is used to acquire the first task and the second task in the computing process. The first task and the second task are dependent on each other. The first task includes at least one first processing task executed by the processor and a first computing task executed by the accelerator. The second task includes at least one second processing task executed by the processor and a second computing task executed by the accelerator. The task execution module is used to concurrently execute processing tasks that do not have a dependency relationship among the at least one first processing task and the at least one second processing task, and to sequentially execute processing tasks that have a dependency relationship among the at least one first processing task and the at least one second processing task. The at least one first processing task includes a processing task that instructs the accelerator to perform the first computing task, and the at least one second processing task includes a processing task that instructs the accelerator to perform the second computing task.

7. The apparatus according to claim 6, characterized in that, The task execution module is used for: Create multiple threads that run concurrently; The third processing task in the at least one first processing task and the fourth processing task in the at least one second processing task are executed sequentially through the first thread of the plurality of threads; The fifth processing task in the at least one first processing task and the sixth processing task in the at least one second processing task are executed sequentially through the second thread among the plurality of threads; The fifth processing task is executed after the third processing task, and the sixth processing task is executed after the fourth processing task.

8. The apparatus according to claim 6 or 7, characterized in that, The device further includes: The time consumption acquisition module is used to acquire the total time consumed by the heterogeneous computing system in executing the first task and the second task during a historical time period, as well as the computing time consumed by the accelerator in executing the first computing task and the second computing task during the historical time period. A determination module is used to determine the bubble consumption time of the accelerator in the historical time period based on the difference between the total consumption time and the calculation consumption time; The task execution module is used for: If the ratio between the bubble processing time and the total processing time is greater than a threshold, then concurrently execute the processing tasks that do not have a dependency relationship among the at least one first processing task and the at least one second processing task, and sequentially execute the processing tasks that have a dependency relationship among the at least one first processing task and the at least one second processing task.

9. The apparatus according to any one of claims 6 to 8, characterized in that, The computation process refers to the inference or training process of an artificial intelligence (AI) model. The task acquisition module is used for: In response to the inference or training request of the AI ​​model, the first task and the second task are obtained; The at least one first processing task includes at least one of shape derivation and data segmentation of the input data of the first task.

10. The apparatus according to claim 9, characterized in that, The device further includes: A storage module is configured to store the execution result of at least one of shape derivation and data segmentation of the input data during the execution of the at least one first processing task in the first time period. After concurrently executing the processing tasks that have no dependency among the at least one first processing task and the at least one second processing task, and sequentially executing the processing tasks that have dependency among the at least one first processing task and the at least one second processing task, the task execution module is further configured to: During the execution of the at least one first processing task in the second time period, if the shape of the input data of the first task is the same as the shape of the input data of the first task when the at least one first processing task is executed in the first time period, then at least one of shape derivation and data segmentation is skipped, and the accelerator is instructed to execute the first computing task based on the execution result stored in the first time period, wherein the second time period is after the first time period.

11. A heterogeneous computing system, characterized in that, The heterogeneous computing system includes processors and accelerators; The processor is used for: Obtain a first task and a second task in the computing process, wherein the first task and the second task are dependent on each other, the first task includes at least one first processing task executed by the processor and a first computing task executed by the accelerator, and the second task includes at least one second processing task executed by the processor and a second computing task executed by the accelerator. Concurrently execute processing tasks that are not dependent on the at least one first processing task and the at least one second processing task; sequentially execute processing tasks that are dependent on the at least one first processing task and the at least one second processing task; wherein, the at least one first processing task includes a processing task that instructs the accelerator to execute the first computing task, and the at least one second processing task includes a processing task that instructs the accelerator to execute the second computing task. The accelerator is used to execute the first computing task and the second computing task according to the instructions of the processor.

12. The system according to claim 11, characterized in that, The processor is used for: Create multiple threads that run concurrently; The third processing task in the at least one first processing task and the fourth processing task in the at least one second processing task are executed sequentially through the first thread of the plurality of threads; The fifth processing task in the at least one first processing task and the sixth processing task in the at least one second processing task are executed sequentially through the second thread among the plurality of threads; The fifth processing task is executed after the third processing task, and the sixth processing task is executed after the fourth processing task.

13. The system according to claim 11 or 12, characterized in that, The processor is also used for: Obtain the total time taken by the heterogeneous computing system to execute the first task and the second task during a historical time period, and the computation time taken by the accelerator to execute the first computing task and the second computing task during the historical time period; Based on the difference between the total time and the calculation time, the bubble time of the accelerator in the historical time period is determined; The concurrent execution of processing tasks that have no dependency relationship among the at least one first processing task and the at least one second processing task, and the sequential execution of processing tasks that have dependency relationship among the at least one first processing task and the at least one second processing task, includes: if the ratio between the bubble time and the total time is greater than a threshold, then concurrently execute processing tasks that have no dependency relationship among the at least one first processing task and the at least one second processing task, and sequentially execute processing tasks that have dependency relationship among the at least one first processing task and the at least one second processing task.

14. The system according to any one of claims 11 to 13, characterized in that, The computational process refers to the inference or training process of an artificial intelligence (AI) model, and the processor is used for: In response to the inference or training request of the AI ​​model, the first task and the second task are obtained; The at least one first processing task includes at least one of shape derivation and data segmentation of the input data of the first task.

15. The system according to claim 14, characterized in that, The processor is also used for: During the execution of the at least one first processing task in the first time period, the execution result of at least one of shape derivation and data segmentation of the input data is stored; After concurrently executing the processing tasks that have no dependency among the at least one first processing task and the at least one second processing task, and sequentially executing the processing tasks that have dependency among the at least one first processing task and the at least one second processing task, the processor is further configured to: During the execution of the at least one first processing task in the second time period, if the shape of the input data of the first task is the same as the shape of the input data of the first task when the at least one first processing task is executed in the first time period, then at least one of shape derivation and data segmentation is skipped, and the accelerator is instructed to execute the first computing task based on the execution result stored in the first time period, wherein the second time period is after the first time period.

16. A processor, characterized in that, The processor is communicatively connected to the accelerator, and the processor is used to control the accelerator and to implement the task execution method as described in any one of claims 1 to 5.

17. A computing device, characterized in that, The computing device includes a processor and a memory, the processor being configured to execute at least one piece of program code stored in the memory to enable the computing device to implement the task execution method as described in any one of claims 1 to 5.

18. A computer-readable storage medium, characterized in that, The computer-readable storage medium is used to store at least one piece of program code, which, when executed by a computing device, causes the computing device to implement the task execution method as described in any one of claims 1 to 5.

19. A computer program product, characterized in that, When the computer program product is run on a computing device, the computing device performs the task execution method as described in any one of claims 1 to 5.

Images