Task assignment method and apparatus, and device and computer program product

Abstract

The embodiments of the present application relate to a task assignment method and apparatus, and a device and a computer program product. The method comprises: acquiring a distribution of a plurality of data shards on a node cluster, wherein the plurality of data shards are to be processed by a first task of an application job, and the node cluster comprises a plurality of node groups, each of the plurality of node groups comprising a group of nodes interconnected by means of a bus. The method further comprises: on the basis of the distribution of the plurality of data shards on the node cluster and the plurality of node groups, assigning the first task to a plurality of execution units executing the application job, wherein the plurality of execution units run on the node cluster. In this way, when a task is assigned, the characteristic of bus interconnection within a node group and the location of data to be processed are considered in combination, such that it is possible to select execution units for the purpose of minimizing cross-supernode data reading, thereby reducing network overheads and improving the execution efficiency.

Description

Task allocation methods, apparatus, equipment and computer program products

[0001] This application claims priority to Chinese Patent Application No. 202411993218.1, filed with the Chinese Patent Office on December 30, 2024, entitled “Task Allocation Method, Apparatus, Device and Computer Program Product”, the entire contents of which are incorporated herein by reference. Technical Field

[0002] The embodiments of this application relate to the field of computer technology, and more specifically to task allocation methods, apparatus, devices and computer program products. Background Technology

[0003] In a big data processing platform, a cluster of multiple nodes collectively provides system resources and collaboratively handles various processing tasks of application jobs, such as computation and querying. The platform's resource management system provides unified resource management and scheduling for upper-layer applications, allocating system resources to jobs across various applications. The resource management system can manage how node resources are shared and allocated among multiple users, as well as the priority scheduling of tasks.

[0004] For example, multiple execution units (e.g., containerized execution units) can be launched on multiple nodes to execute a job. During job execution, the resource management system can distribute multiple tasks of the job to different execution units for parallel execution. When allocating tasks, the resource management system can consider factors such as the capacity and utilization of each node to achieve load balancing. Summary of the Invention

[0005] Embodiments of this application provide a task allocation scheme. This scheme can reduce network overhead and communication latency during system task execution, thereby improving execution efficiency.

[0006] In a first aspect, a task allocation method is provided. The method includes: acquiring the distribution of multiple data shards across a node cluster, the multiple data shards to be processed by a first task of an application job, wherein the node cluster comprises multiple node groups, each of the multiple node groups comprising a group of nodes interconnected by a bus; and, based on the distribution and the multiple node groups, allocating the first task to multiple execution units executing the application job, wherein the multiple execution units run on the node cluster. Thus, by considering the characteristics of the bus interconnection within the supernode and the location of the data to be processed when allocating tasks, it becomes possible to select execution units with the aim of minimizing cross-supernode data reads.

[0007] In some embodiments of the first aspect, the method further includes: in response to receiving a request to execute an application job, selecting at least one node group from a node cluster; and running multiple execution units on the at least one node group. This allows resources for the application job to be allocated on a per-bus interconnected node-group basis, such that the execution units executing the job reside within as few node groups as possible, thereby reducing data reads across node groups during execution.

[0008] In some embodiments of the first aspect, the method further includes: in response to receiving a registration request from a first node, determining the node group to which the first node belongs among multiple node groups based on the bus connection status between the first node and other nodes; and storing the association information between the first node and the node group. Thus, the grouping information of the node group can be stored and used for subsequent resource allocation and task scheduling.

[0009] In some embodiments of the first aspect, assigning a first task to multiple execution units includes: determining the amount of data in each of multiple node groups for multiple data shards; selecting a first node group from the multiple node groups based on the data amount; and assigning the first task to a first execution unit running on the first node group among the multiple execution units. Thus, scheduling tasks to a particular node group based on the amount of data to be processed on the node group makes it possible to adjust the amount of data read across node groups during execution.

[0010] In some embodiments of the first aspect, selecting a first node group from multiple node groups based on data volume includes: determining the node group containing the largest data shard among multiple data shards as the first node group. This avoids the execution unit reading the largest data shard to be processed across node groups, thereby improving read efficiency and reducing network overhead.

[0011] In some embodiments of the first aspect, selecting the first node group based on data volume includes: determining the node group containing the largest amount of data among multiple data shards as the first node group. This minimizes the amount of data the execution unit needs to read across node groups when performing the first task.

[0012] In some embodiments of the first aspect, the first execution unit reads at least a portion of a plurality of data fragments from the shared memory of the first node group via a bus. Thus, data reading within the node group can be performed efficiently via a high-speed bus.

[0013] In some embodiments of the first aspect, the first task belongs to the first stage of an application job, and multiple data shards are generated by executing the second stage of the application job, the second stage being the stage preceding the first stage, wherein: the first data shard among the multiple data shards is written to a first shared memory by a first execution unit among multiple execution units, wherein the first data shard is generated by the first execution unit, the first shared memory corresponds to a first node group in a node cluster, and the first node group includes a first node running the first execution unit. Thus, the data shards generated by the execution unit can be shared by subsequent tasks, particularly through high-speed sharing within the node group via a bus.

[0014] In some embodiments of the first aspect, a second data shard among multiple data shards is written to a second shared memory by a second execution unit among multiple execution units. The second data shard is generated by the second execution unit, and the second shared memory corresponds to a second node group in a node cluster, and the second node group includes a second node running the second execution unit. Thus, multiple execution units can execute tasks in parallel and generate corresponding data shards.

[0015] In a second aspect, a task allocation apparatus is provided. The apparatus includes: an acquisition module configured to acquire a distribution of multiple data shards across a node cluster, the multiple data shards being processed by a first task of an application job, wherein the node cluster includes multiple node groups, each of the multiple node groups including a group of nodes interconnected by a bus; and an allocation module configured to allocate the first task to multiple execution units executing the application job based on the distribution and the multiple node groups, wherein the multiple execution units run on the node cluster.

[0016] In some embodiments of the second aspect, the apparatus further includes: a selection module configured to select at least one group of nodes from a node cluster in response to receiving a request to execute an application job; and a running module configured to run a plurality of execution units on at least one group of nodes.

[0017] In some embodiments of the second aspect, the apparatus further includes: a second determining module configured to, in response to receiving a registration request from a first node, determine the node group to which the first node belongs among a plurality of node groups based on the bus connection status between the first node and other nodes; and a storage module configured to store association information between the first node and the node group.

[0018] In some embodiments of the second aspect, the allocation module includes: a third determining module configured to determine the amount of data in each of the plurality of data shards in the plurality of node groups; a second selecting module configured to select a first node group from the plurality of node groups based on the amount of data; and a second allocation module configured to allocate a first task to a first execution unit running in the first node group among the plurality of execution units.

[0019] In some embodiments of the second aspect, the second selection module includes: a fourth determination module configured to determine the node group containing the largest data shard among a plurality of data shards as the first node group.

[0020] In some embodiments of the second aspect, the second selection module includes: a fifth determining module configured to determine the node group containing the largest amount of data among a plurality of data shards as the first node group.

[0021] In some embodiments of the second aspect, the first execution unit reads at least a portion of a plurality of data shards from the shared memory of the first node group via a bus.

[0022] In some embodiments of the second aspect, the first task belongs to the first stage of the application job, and multiple data shards are generated by executing the second stage of the application job, the second stage being the previous stage of the first stage, and wherein: the first data shard among the multiple data shards is written to the first shared memory by the first execution unit among the multiple execution units, wherein the first data shard is generated by the first execution unit, the first shared memory corresponds to the first node group in the node cluster, and the first node group includes the first node running the first execution unit.

[0023] In some embodiments of the second aspect, a second data shard among a plurality of data shards is written to a second shared memory by a second execution unit among a plurality of execution units, the second data shard is generated by the second execution unit, the second shared memory corresponds to a second node group in a node cluster, and the second node group includes a second node running the second execution unit.

[0024] In a third aspect, an electronic device is provided. The electronic device includes a processor and a memory storing computer instructions that, when executed by the processor, cause the electronic device to operate according to the method described in the first aspect or any of its embodiments.

[0025] In a fourth aspect, a computer program product is provided. The computer program product includes instructions that, when executed by an electronic device, cause the electronic device to perform actions according to the method of the first aspect or any embodiment thereof.

[0026] In a fifth aspect, a computer-readable storage medium is provided. The computer-readable storage medium stores instructions that, when executed by an electronic device, cause the electronic device to perform actions according to the method of the first aspect or any embodiment thereof. Attached Figure Description

[0027] Figure 1 shows a schematic diagram of an example environment in which various embodiments of this application can be implemented;

[0028] Figure 2 shows a flowchart of an example task allocation method according to some embodiments of this application;

[0029] Figure 3 shows an overall architecture diagram of an example computing system according to some embodiments of this application;

[0030] Figure 4 shows a schematic diagram of the logical components of an example computing system according to some embodiments of this application;

[0031] Figure 5A shows a schematic diagram of an example implementation of the computing system in Figure 4 according to some embodiments of this application;

[0032] Figure 5B illustrates a schematic diagram of scheduling tasks and reading / writing task data in the example implementation of Figure 5A according to some embodiments of this application;

[0033] Figure 6 shows a schematic block diagram of an example task allocation apparatus according to some embodiments of this application; and

[0034] Figure 7 shows a schematic block diagram of an example device that can be used to implement embodiments of this application. Detailed Implementation

[0035] Embodiments of this application will now be described in more detail with reference to the accompanying drawings. While some embodiments of this application are shown in the drawings, it should be understood that this application can be implemented in various forms and should not be construed as limited to the embodiments set forth herein. Rather, these embodiments are provided to provide a more thorough and complete understanding of this application. It should be understood that the drawings and embodiments of this application are for illustrative purposes only and are not intended to limit the scope of protection of this application.

[0036] In the description of embodiments of this application, the term "comprising" and similar terms should be understood as open-ended inclusion, i.e., "including but not limited to". The term "based on" should be understood as "at least partially based on". The term "one embodiment" or "the embodiment" should be understood as "at least one embodiment". The terms "first", "second", etc., may refer to different or the same objects. Other explicit and implicit definitions may also be included below.

[0037] In big data platforms, resource management systems can provide unified resource allocation and execution scheduling (e.g., computing, storage) for upper-layer applications. In conventional resource management systems, resource scheduling primarily considers resource sharing and allocation among multiple users, as well as task priority scheduling. Take YARN (Yet Another Resource Negotiator), the general resource management system of the open-source framework Hadoop, as an example. YARN can employ a first-come, first-served (FFS) strategy, meaning that resources are allocated and scheduled for the next application only after the application that requested resources first has had its needs met. Furthermore, for multiple user queues requiring resource allocation, YARN can set a minimum guaranteed resource and usage limit for each queue, and supports resource sharing. Moreover, a strategy for allocating fair resources to all applications can also be used, where fairness can be defined through parameters.

[0038] However, due to the distributed nature of big data platforms, the data to be processed may be distributed across different nodes. Retrieving data from these nodes incurs communication overhead and latency. Furthermore, the connectivity between nodes may vary, resulting in different data transmission efficiencies. Given the complexity of distributed platforms, traditional allocation schemes that only consider load balancing and fairness still require further optimization in terms of communication efficiency.

[0039] For example, with the emergence of new bus technologies, computing clusters have taken on a new form. This type of cluster includes multiple super nodes, each of which is itself a group of interconnected nodes (or a small cluster of nodes). That is, in addition to standard network interconnections, bus interconnections are added between nodes within a super node, such as Unified Bus (UB), Compute Express Link (CXL), UALink, and / or NVLink interconnects. These interconnect buses offer high bandwidth and low latency, enabling efficient communication within a super node. On the other hand, two servers between super nodes can communicate via standard network interface cards (NICs). This difference in connectivity results in varying communication overhead and latency when assigning a task to different nodes.

[0040] To address the aforementioned and other issues, embodiments of this application provide a task allocation scheme. This scheme is applicable to node clusters comprising multiple supernodes, each supernode including a group of nodes interconnected by a bus, and multiple execution units executing application jobs can run on the node cluster. When allocating a task of a job to these execution units, this scheme can obtain the distribution of multiple data shards to be processed by the task across the cluster. Then, execution units can be selected to execute the task based on this distribution and the grouping of cluster nodes to multiple supernodes.

[0041] In this way, by combining the characteristics of the bus interconnection within a supernode with the location of the data to be processed when allocating tasks, it becomes possible to select execution units with the aim of minimizing cross-supernode data reads. Furthermore, compared to traditional task allocation methods, network overhead and communication latency during the execution of assigned tasks can be reduced, thereby improving execution efficiency and system performance.

[0042] Referring below to Figure 1, a schematic diagram of an example environment 100 in which various embodiments of this application can be implemented is shown. Example environment 100 includes a distributed node cluster 110. Nodes in node cluster 110 can be servers in a distributed computing platform, each possessing certain hardware computing resources (e.g., processing resources, storage resources, etc.). Nodes in node cluster 110 can be implemented as any device with the required computing capabilities. Examples include, but are not limited to, workstations, rack servers, mainframes, or any combination thereof. These nodes can execute various computing tasks of the distributed computing platform in parallel. For example, multiple nodes in the node cluster can execute multiple tasks of an application job in parallel, such as jobs submitted to the platform in the form of packages or requests containing parameters. Some of these nodes can be interconnected by a bus to form a node group. In this document, such a node group is referred to as a supernode.

[0043] As shown in the figure, the node cluster 110 includes multiple supernodes 120-1, 120-2, ..., 120-M (individually or collectively referred to as supernode 120). For example, supernode 120-1 includes nodes 130-1, 130-2, ..., 130-N. These nodes are interconnected via bus 140. In this document, supernode 120 may also be referred to as a type of bus domain. For example, when bus 140 is a UB bus, supernode 120-1 can be considered a UB domain. It should be understood that bus 140 can be any type of high-speed bus, and the embodiments of this application are not limited to a specific bus type.

[0044] The interconnect bus provides high-speed bandwidth and low latency. Nodes belonging to the same supernode 120 can preferentially use the bus to communicate with each other, while nodes in different supernodes 120 communicate through the network system. It should be understood that the supernodes shown in Figure 1 and the number of nodes contained in a supernode are merely examples; a node cluster can include any appropriate number of supernodes, a supernode can include any appropriate number of nodes, and / or different supernodes can include different numbers of nodes. Furthermore, the node cluster 110 may also include nodes that are not connected to other nodes via the bus, i.e., nodes that do not belong to any supernode.

[0045] For ease of explanation, computing device 150 is also shown separately in Figure 1. Computing device 150 can be configured to perform various actions according to embodiments of this application, such as resource allocation and task scheduling. The following description will be given in the context of computing device 150 performing these actions. It should be understood that although shown as a separate entity, computing device 150 can exist in various forms. In some embodiments, computing device 150 may be a dedicated management node in computing cluster 110. In other embodiments, in addition to performing management node-related actions, computing device 150 may also perform application tasks. In some embodiments, computing device 150 itself may belong to a supernode 120 in node cluster 110. Furthermore, in some embodiments, the functionality of computing device 150 may actually be performed by more than one physical device; that is, computing device 150 itself may be distributed.

[0046] It should be understood that the environment 100 shown in Figure 1 is merely exemplary and should not constitute any limitation on the functionality and scope of the implementation described in this application. Other devices, systems, or components, not shown, may also be present in the example environment 100.

[0047] Referring now to FIG2. FIG2 shows a flowchart of an example task assignment method 200 according to some embodiments of the present application. The example method 200 can be performed, for example, by a computing device 150 as shown in FIG1. ​​It should be understood that method 200 may also include additional actions not shown, and the scope of the present application is not limited in this respect. The method 200 will now be described in detail with reference to the example setting 100 of FIG1.

[0048] At 210, the distribution of multiple data shards across a node cluster is acquired. These multiple data shards are to be processed by the first task of an application job. The node cluster comprises multiple node groups, each of which includes a group of nodes interconnected by a bus. For example, multiple execution units executing an application job can run on node cluster 110. Each supernode 120 is a node group, which includes a group of nodes interconnected by a bus. The multiple data shards to be processed by the first task of the application job can be distributed across node cluster 110. Computing device 150 can acquire the distribution of these multiple data shards across node cluster 110 for use in allocating the first task at 220.

[0049] In some embodiments, computing device 150 may allocate system resources for executing application jobs on a per-node-group basis. In such an embodiment, in response to receiving a request to execute an application job, computing device 150 may select at least one supernode from node cluster 110 and run multiple execution units for executing tasks on the selected supernode.

[0050] In some embodiments, each node may register with computing device 150. Upon receiving a registration request from a node (e.g., the first node), computing device 150 may determine the supernode to which the node belongs based on the bus connection status between the node and other nodes in the cluster, and store the association information between the node and the supernode. This association information will be described in more detail later with reference to Figure 5A.

[0051] At 220, based on the distribution and the multiple node groups, a first task is assigned to multiple execution units that execute application jobs, which run on the node cluster. For example, computing device 150 can select execution units from the multiple execution units to assign a first task to them based on the distribution of multiple data shards in node cluster 110 and the grouping of the nodes connected by a bus into multiple supernodes 120.

[0052] In some embodiments, computing device 150 may determine the amount of data in each of the plurality of data shards on each of the plurality of supernodes 120, and select a first supernode, such as supernode 120-1, based on the amount of data. Computing device 150 may then assign the first task to execution units running on the selected supernode 120-1.

[0053] In some embodiments, computing device 150 may determine the supernode containing the largest data shard among multiple data shards as the first supernode. In other embodiments, computing device 150 may determine the supernode 120 containing the largest amount of data among multiple data shards as the first supernode. For example, a supernode 120 may contain one or more data shards from multiple data shards. Computing device 150 may determine the total data volume of each data shard on the supernode 120, that is, the sum of the sizes of all data shards on the supernode. Furthermore, computing device 150 may select the supernode 120 corresponding to the largest total data volume as the first supernode. In other words, the selected supernode includes the largest portion of the data shards among the multiple data shards.

[0054] Method 200, when allocating tasks, takes into account both the characteristics of the high-speed bus interconnection between nodes within a supernode and the location of the data to be processed, making it possible to select execution units with the aim of minimizing cross-supernode data reads. Selecting execution units in this way reduces network overhead and communication latency during task execution, thereby improving execution efficiency and system performance.

[0055] In some embodiments, the data shards in the plurality of data shards may reside in the shared memory of the plurality of supernodes 120. In some embodiments, the application job includes multiple stages, and the first task belongs to the current stage of the application job, while the plurality of data shards to be processed are generated from the previous stage in which the current stage was executed.

[0056] For example, when multiple execution units are executing the previous stage for an application job, the execution unit running on a node in supernode 120-1 can generate a first data shard and write it to the shared memory of supernode 120-1. Then, another execution unit running on a node in supernode 120-2 can generate a second data shard and write it to the shared memory of supernode 120-2. In this way, the multiple execution units for the next stage can be distributed across the shared memory of multiple supernodes.

[0057] Execution units within supernode 120 can read data from the shared memory of their supernode 120 via the bus without standard network communication. Therefore, at least a portion of multiple data shards can be distributed across the selected supernode 120-1, and the execution unit assigned the first task can read this portion from the shared memory of supernode 120-1 via the bus, thereby avoiding network communication. The reading and writing of multiple data shards will be described in more detail later with reference to Figures 5A and 5B.

[0058] Referring now to Figure 3. Figure 3 shows an overall architecture diagram of an example computing system 300 according to some embodiments of this application. The computing system 300 can be implemented in, for example, the node cluster 110 of Figure 1. The architecture of the computing system 300 mainly includes three parts: an upper-layer application 310, a resource management component 320, and an lower-layer hardware resource 330.

[0059] Hardware resource 330 may include a cluster of nodes for performing computational tasks, including multiple supernodes 320-1, 320-2, ... 320-N. As previously described, each supernode includes a group of nodes interconnected by a bus. Application 310 may be a batch processing application suitable for multi-task parallel processing, which requests and releases resources on hardware resource 330 through resource management component 320 for various tasks to perform its job.

[0060] Resource management component 320 has management functions for node clusters. This function can be implemented, for example, on computing device 150, which is responsible for resource management and task scheduling of the entire cluster. Upon receiving an application job submitted by a user, resource management component 320 can process the client's request and allocate the necessary hardware resources for the job. The resource management component can select at least one node from hardware resources 330 and launch a group of execution units allocated certain resources (e.g., processor resources and memory resources) on the selected node to execute the application job. For example, resource management component 320 can use containerization technology to run execution units, where each execution unit can run as a separate process on the node. Furthermore, the execution unit can have multiple virtual cores to support the parallel execution of multiple task threads.

[0061] In some embodiments, in response to receiving a request to execute an application job, the resource management component 320 can select at least one supernode from supernodes 320-1 to 320-M, and run an execution unit for the job on a node in the selected supernode. In this way, the resource management component 320 can allocate computing resources for the application to be located in as few supernodes as possible. Furthermore, data transfer across supernodes can be minimized during the subsequent execution of various tasks of the application job.

[0062] Resource management component 320 can also have management functions for individual applications, responsible for managing the corresponding application, such as requesting resources for the application and further allocating resources to internal tasks. Application-level management functions can be initiated and monitored by cluster-level management functions. Furthermore, resource management component 320 can also have management functions for individual nodes. Node-level functions can be implemented on the corresponding node, processing commands from cluster management and application management functions, and correspondingly responsible for resource management on the node.

[0063] As a non-limiting example, in a computing system implementation based on the open-source cluster framework Apache Spark, the resource management component 320 can be implemented based on YARN. Taking such an implementation as an example, the method operations according to some embodiments of this application will be described in more detail later with reference to Figures 5A and 5B.

[0064] Referring now to FIG4, FIG4 illustrates a schematic diagram of the logical components of an example computing system 400 according to some embodiments of the present application. For example, computing system 400 may be an example of computing system 300 of FIG3 and may be implemented in computing cluster 110 of FIG1. ​​In this example, the functionality of the embodiments according to the present application is mainly described in conjunction with supernode scheduler 461, node cluster manager 470, and task scheduler 481.

[0065] As shown in the figure, the underlying hardware of the computing system 400 includes a node cluster comprising multiple supernodes 420-1 to 420-M, and each supernode includes a group of computing nodes interconnected by a bus. During system initialization, a resource manager 460 for the entire system is started on the management node (e.g., computing device 150 in Figure 1), and the corresponding node managers are started on each node, and the node managers of the nodes register with the resource manager 460.

[0066] For example, node manager 440 can be started on node 430, and node manager 440 can register node 430 with resource manager 460. In response to receiving a registration request from node 430, resource manager 460 can include node 430 in the system's node cluster. Computing system 400 also includes node cluster manager 470, which maintains the cluster topology. After receiving a node's registration request, resource manager 460 can send a request to node cluster manager 470 to resolve the supernode to which the node resides.

[0067] Furthermore, based on the bus connection status between a node and other nodes in the cluster, the node cluster manager 470 can determine the supernode to which a node belongs. For example, node 430 can be determined to belong to supernode 420-1 because node 430 and other nodes in supernode 420-1 are interconnected by the bus. The resource manager 460 can locally store the association information between the node and the supernode, such as information indicating that node 430 belongs to supernode 420-1. As an example, the resource manager 460 can store the supernode and each node belonging to the supernode in the form of a tree diagram. The resource manager 460 can also record information such as the resource utilization rate of each node, consistent with the native structure. Subsequently, the resource manager 460 can perform global resource allocation and task scheduling based on the above information.

[0068] When an application job is received from a client (e.g., client 410), resource manager 460 can launch the application management module for that application (e.g., on the management node). It should be understood that resource manager 460 can launch different application management modules for multiple applications. For clarity, Figure 4 only shows the application management module 480 for a single application.

[0069] The application management module 480 can request resources for the application from the resource manager 460. The resource manager 460 can allocate the required resources for the application within the system's node cluster. Specifically, the resource manager 460 can launch a certain number of execution units for the application on one or more nodes, each execution unit being allocated corresponding processor and memory resources, etc. In some embodiments, the resource manager 480 can use container technology to allocate resources, allowing execution units to run in their respective containers. For example, the resource manager 480 can allocate execution units 450-1, 450-2, and 450-3 to the aforementioned application. The number of execution units used to execute the application and the amount of resources for each execution unit are determined based on the specific implementation and application requirements.

[0070] In some embodiments, the resource manager 460 includes a supernode scheduler 461, which can allocate resources on a supernode basis. In response to receiving a request to execute an application job, the supernode scheduler 461 can select at least one supernode from the node cluster and run multiple execution units on the selected supernode to execute the application job. In this way, the execution units used to execute the job can be located within a single supernode or as few supernodes as possible. Thus, during job execution, data communication between nodes can utilize the high-speed bus more extensively, thereby reducing network communication and latency compared to traditional solutions.

[0071] For clarity, only one execution unit is shown in each node in Figure 4. However, it should be understood that multiple execution units can run on a single node, and execution units for multiple different applications can also run. The application management module 480 can manage jobs for the corresponding applications. A job can include multiple tasks, and multiple execution units executing an application job can execute multiple tasks of the job in parallel.

[0072] The application management module 480 may include a task scheduler 481, which can assign tasks from application jobs to corresponding application jobs. When assigning tasks, the task scheduler 481 can obtain the distribution of multiple data shards to be processed by the task across the node cluster. Then, the task scheduler 481 can assign tasks on a supernode basis. In other words, the task scheduler 481 can select the execution unit for executing the task based on the distribution of these multiple data shards across various supernodes, as described above in conjunction with method 200.

[0073] In some embodiments, the task scheduler 481 can determine the amount of data in multiple data shards on each supernode. Based on the amount of data on each node, the task scheduler 481 can select a supernode from the supernodes and assign the task to the execution unit running on the selected supernode. For example, the task scheduler 481 can select the supernode that has the largest portion of the multiple data shards. For example, the task scheduler can also additionally consider other appropriate factors such as node occupancy when making the selection. In this way, the execution unit and the data it needs to process can be located on the same supernode as much as possible, thereby enabling the use of high-speed bus communication to read data and reducing network communication overhead and latency across supernodes.

[0074] In some embodiments, an application job can be divided into multiple stages, each stage including a set of tasks. Tasks in each stage can be executed in parallel and produce corresponding outputs. Tasks in the next stage can retrieve the outputs of the previous stage and continue processing. This process involves data shuffling to redistribute data across processes and / or nodes. In some embodiments, each execution unit writes the data it generates to shared memory shared by its supernode. When retrieving data to be processed, the execution unit can read the portion of that data located in the shared memory of its supernode via a bus, as will be described in more detail later with reference to Figures 5A and 5B.

[0075] Figure 4 illustrates an example structure that can be implemented for a system executing embodiments of this application. It should be understood that, in a particular implementation, those skilled in the art can make various adjustments to the component arrangement in the example, such as merging some components, further splitting components, or implementing different components for executing embodiments of this application and their variations, without departing from the scope of this application.

[0076] Referring now to Figure 5A, a schematic diagram of an example implementation 500 of the computing system in Figure 4 according to some embodiments of this application is shown. As a non-limiting example, this example is implemented based on various components of the Spark system managed by YARN. Spark is an open-source distributed computing system that provides a fast, general-purpose, and scalable big data processing platform suitable for handling various data sources and data types. Furthermore, for illustrative purposes, this example uses a UB bus to connect the nodes within a supernode, so each supernode can also be referred to as a UB domain. In the following description of Figures 5A and 5B, the terms "supernode" and "UB domain" will be used interchangeably. It should be understood that in other examples, the nodes within a supernode may also be connected to each other by other types of high-speed buses.

[0077] As shown in the figure, the underlying hardware of example implementation 500 includes a node cluster comprising multiple supernodes 520-1 to 520-M. These supernodes can be the example implementation of supernodes 420-1 to 420-M in Figure 4. Furthermore, in example implementation 500, the resource manager and node manager are implemented based on YARN's ResourceManager and NodeManager (NM) components, respectively.

[0078] During system initialization, ResourceManager 560, used for global resource management, is started on the cluster's management node, and NodeManagers are started on each node. Each NodeManager registers the corresponding node with ResourceManager 560. For example, as shown by arrow 505, NM 540 can be started on node 530, and NM 540 can register node 530 with ResourceManager 560.

[0079] As indicated by arrow 515, upon receiving a registration request from NM 540, ResourceManager 560 can, for example, use the IP address of NM 540 to send a request to RackServerManager 570 to resolve the supernode to which NM 540 belongs (i.e., the supernode to which node 530 belongs). RackServerManager 570 can be an example implementation of the node cluster manager 470 in Figure 4, which is used to maintain the cluster topology. In this example, node 530 can be identified as belonging to supernode 520-1, which, along with the other nodes in supernode 520-1, is interconnected by the UB bus.

[0080] ResourceManager 560 can maintain global node information locally, such as the topology of each supernode and resource utilization, which remains consistent with the system's native state. Based on this information, ResourceManager 560 can perform global resource allocation and task scheduling. As an example, ResourceManager 560 can store the relationships between supernodes and the nodes belonging to those supernodes in the form of a tree graph.

[0081] For example, Table 1 shows an example supernode topology that ResourceManager 560 can store, where supernodes are recorded as sub-entries under the root entry / root of the node cluster, and information about each node is recorded as a sub-entry of its supernode. The example cluster in Table 1 includes supernodes UBC-Domain 1 and UBC-Domain 2. Supernode UBC-Domain 1 includes registered nodes NM1, NM2, NM3, and NM4, and supernodes NM5, NM6, NM7, and NM8. In this example, nodes are identified by their corresponding NodeManager. It should be understood that the number of supernodes and nodes in Table 1 is only an example, and the content can be updated as system states change (such as the addition or removal of nodes, changes in resource usage, etc.).

[0082] Table 1: Example Supernode Topology

[0083] As shown by arrow 525, a user can submit application jobs (in this example, Spark Jobs) through a client (which in this example could be Spark Client) 510. Upon receiving a submitted job, ResourceManager 560 can launch the application management module 580 for that application, as shown by arrow 535. Specifically, client 510 can request ResourceManager 560 to launch ApplicationMaster 580. ApplicationMaster is the application management component of YARN, and it can launch the corresponding Driver component upon startup. As shown in the figure, ApplicationMaster 580 can launch the corresponding Driver 582 after startup. As described above in conjunction with Figure 4, ResourceManager 560 can launch multiple application management modules for multiple applications.

[0084] ApplicationMaster 580 can request resources from ResourceManager 560 for executing application jobs. As mentioned earlier, ResourceManager 560 can allocate resources on a unit UB basis, so that the execution unit of the task is within one UB or as few UBs as possible.

[0085] ResourceManager 580 instructs the appropriate nodes to start Executor processes based on resource allocation. As an example, execution units 550-1, 550-2, 550-3, and 550-4 can be started to execute the job. In example implementation 500, each execution unit is implemented as a Spark Executor component. Although only one execution unit is shown on each node in Figure 5, it should be understood that multiple execution units can run on each node, and execution units can run to execute multiple different application jobs.

[0086] Spark can transform application jobs into Directed Acyclic Graphs (DAGs) and schedule tasks and perform distributed parallel processing in stages. The DAGScheduler (DAG scheduler) 583 can divide the job into multiple stages based on the job's DAG, each stage containing a set of tasks that can be executed in parallel. Tasks in a stage can be scheduled by the task scheduler to be executed in parallel on multiple execution units. The task scheduler is implemented as TaskScheduler 581 in example implementation 500, which can assign tasks to appropriate execution units for execution, for example, as shown by arrow 545. Similar to allocating resources for application jobs, TaskScheduler 581 can schedule tasks based on UB domains to minimize data exchange over standard networks; this functionality is implemented as UBScheduler 565 in the example of Figure 5A.

[0087] The allocation and execution of tasks to execution units will now be described in more detail with reference to Figure 5B. Figure 5B illustrates a schematic diagram of task scheduling and reading / writing task data in the example implementation of Figure 5A according to some embodiments of this application. For emphasis, Figure 5B shows the network interface card (NIC) used for communication by each node, such as the NIC 590. As mentioned above, nodes within a supernode can preferentially transmit data to each other via the bus, while nodes within different supernodes need to communicate via standard network interface cards.

[0088] In the example of Figure 5B, shared memory corresponding to each supernode is also shown. For example, supernode 520-1 has shared memory 591. The shared memory of a supernode resides on one or more nodes within the supernode and can be shared and accessed by the nodes. When executing a task, the execution unit can output the generated data to the shared memory in its supernode through a shuffle write operation. For example, a node within the supernode can write the output of its task execution to shared memory 591. Furthermore, the execution unit also sends shuffle metadata of this data to the driver 582 for subsequent task allocation; this metadata indicates the data size and its position in the node cluster, etc.

[0089] As mentioned earlier, an application job can be divided into multiple stages, and each stage can include a set of tasks. Tasks in each stage can be executed in parallel and produce corresponding outputs, and tasks in the next stage can retrieve the data output from the previous stage to continue processing. Example implementation 500 demonstrates that when executing an application job, the data to be processed in a certain stage can be divided into multiple partitions, each to be executed by a corresponding task. Each partition can be assigned to an execution unit for processing, and data from multiple partitions can be processed in parallel by multiple execution units. Furthermore, an execution unit can have multiple threads. Therefore, each partition can include multiple data shards distributed across multiple different nodes.

[0090] For example, a partition may include multiple data shards generated in the previous stage of the job. Among these multiple data shards, the first data shard may be generated by the first execution unit and written to the first shared memory of its supernode, while the second data shard may be generated by the second execution unit and written to the second shared memory of its supernode.

[0091] Table 2 shows a simplified example of data shards and their corresponding partitions generated at a certain stage of an application job. In this example, four execution units are assigned to execute the application job: Executor1, Executor2, Executor5, and Executor6. As shown in the corresponding row for each execution unit, during the current stage of executing the job task, it can generate multiple data shards and write them to the shared memory of its supernode. These data shards are divided into eight partitions, P1 to P8. These eight partitions will be processed by eight tasks in the next stage, respectively. As shown in the corresponding column for each partition, partitions can include shards of different sizes. It should be understood that the number of execution units, data shards, and partitions in Table 2 are merely illustrative and do not constitute a limitation on the scope of this application.

[0092] Table 2: Example Data Sharding and Partitioning

[0093] When executing the next stage of the task, the execution unit retrieves the partition data to be processed by its assigned task from shared memory through a shuffle read. To reduce data transfer, TaskScheduler 581 may take into account the ability of data to be transferred via a high-speed bus within the supernode when assigning tasks to the Executors executing the job. In such an embodiment, TaskScheduler 581 may determine the amount of data in a task's data fragment across the various supernodes and select the supernodes accordingly, scheduling the execution units on the selected supernodes to execute the task. For example, the task may be scheduled to the supernode with the largest data fragment of the partition. For example, the task may be scheduled to the supernode with the largest amount of data in the partition.

[0094] Taking P1 in Table 2 as an example, its largest 4MB fragment is located in the shared memory corresponding to Executor 6. Therefore, TaskScheduler 581 can schedule the task processing P1 to be executed in the supernode where Executor 6 resides. Let Executor 1, Executor 2, Executor 5, and Executor 6 correspond to execution units 550-1, 550-2, 550-3, and 550-4 in Figure 5B, respectively. Based on the data distribution of each partition, TaskScheduler 581 can ultimately assign the tasks processing P3, P4, P5, and P8 to the execution unit in supernode 520-1 that executes the job, and assign the tasks processing P1, P2, P6, and P7 to the execution unit in supernode 520-2 that executes the job. It should be understood that in practice, TaskScheduler 581 can also additionally combine other appropriate factors such as node occupancy rate to specifically select the execution unit for executing the corresponding task.

[0095] In this way, the execution unit and the data it needs to process can be located in the same supernode as much as possible, allowing the execution unit to use high-speed bus communication to obtain the required data, thereby reducing network communication overhead and latency across supernodes. The execution unit performing the corresponding task can obtain metadata from the driver 582 (written by the execution unit during the execution of the previous stage task) and read the data to be processed from the shared memory accordingly. For data fragments to be processed that are located in the same supernode as the execution unit, the execution unit can read the data fragment from the corresponding shared memory via the bus. For example, execution unit 550-2 can determine that a portion of the data it needs to process is located in shared memory 591 based on the obtained task metadata and read that portion of the data from shared memory 591 via the UB bus.

[0096] The embodiments described above in this application take into account the advantages of efficient communication within supernode interconnections and provide an improved resource and task allocation scheme based on this. In the above embodiments, the computing system can schedule resources for application execution according to bus-interconnected node groups. Furthermore, the system can schedule application tasks to node groups that minimize data movement by reducing the amount of data reorganized across node groups, thereby reducing network communication during job execution. Within a node group, by employing memory sharing, the above embodiments can avoid network communication of data within the node group.

[0097] Based on example tests, for small tasks, the solutions according to some embodiments of this application can completely avoid network overhead, enabling 100% data transmission through high-speed bus communication, thereby improving network transmission efficiency by several times. For large tasks (taking 20,000 cores as an example), the solutions according to some embodiments of this application can reduce network data transmission overhead by 20% and improve transmission efficiency by more than 10%.

[0098] Figure 6 shows a schematic block diagram of an example task allocation apparatus 600 according to some embodiments of this application. Apparatus 600 may be implemented as, for example, or included in the computing device 150 of Figure 1. Apparatus 600 may include multiple modules for performing corresponding actions, such as those in method 200.

[0099] As shown in Figure 6, the device 600 includes an acquisition module 610 and an allocation module 620. The acquisition module 610 is configured to acquire the distribution of multiple data shards across a node cluster, the multiple data shards to be processed by the first task of an application job, wherein the node cluster includes multiple node groups, and each node group includes a set of nodes interconnected by a bus. The allocation module 620 is configured to allocate the first task to multiple execution units executing the application job based on the distribution and the multiple node groups, wherein the multiple execution units run on the node cluster.

[0100] In some embodiments, the apparatus 600 further includes: a selection module configured to select at least one node group from a node cluster in response to receiving a request to execute an application job; and a running module configured to run a plurality of execution units on at least one node group.

[0101] In some embodiments, the apparatus 600 further includes: a second determining module configured to, in response to receiving a registration request from a first node, determine the node group to which the first node belongs among a plurality of node groups based on the bus connection status between the first node and other nodes; and a storage module configured to store association information between the first node and the node group.

[0102] In some embodiments, the allocation module includes: a third determining module configured to determine the amount of data in each of the plurality of data shards in the plurality of node groups; a second selecting module configured to select a first node group from the plurality of node groups based on the amount of data; and a second allocation module configured to allocate a first task to a first execution unit running in the first node group among the plurality of execution units.

[0103] In some embodiments, the second selection module includes a fourth determination module, configured to determine the node group containing the largest data shard among a plurality of data shards as the first node group.

[0104] In some embodiments, the second selection module includes a fifth determination module, configured to determine the node group containing the largest amount of data among multiple data shards as the first node group.

[0105] In some embodiments, the first execution unit reads at least a portion of a plurality of data shards from the shared memory of the first node group via a bus.

[0106] In some embodiments, the first task belongs to the first stage of the application job, and multiple data shards are generated by executing the second stage of the application job, the second stage being the previous stage of the first stage, and wherein: the first data shard among the multiple data shards is written to the first shared memory by the first execution unit among the multiple execution units, wherein the first data shard is generated by the first execution unit, the first shared memory corresponds to the first node group in the node cluster, and the first node group includes the first node running the first execution unit.

[0107] In some embodiments, a second data shard among a plurality of data shards is written to a second shared memory by a second execution unit among a plurality of execution units, the second data shard is generated by the second execution unit, the second shared memory corresponds to a second node group in a node cluster, and the second node group includes a second node running the second execution unit.

[0108] Figure 7 shows a schematic block diagram of an example device 700 that can be used to implement embodiments of the present application. Device 700 can be used to implement the functions of the computing device 150 shown in Figure 1. As shown, device 700 includes a processing unit 701, which can perform various appropriate actions and processes according to computer program instructions stored in random access memory (RAM) 703 and / or read-only memory (ROM) 702, or computer program instructions loaded from storage unit 708 into RAM 703 and / or ROM 702. Various programs and data required for the operation of device 700 may also be stored in RAM 703 and / or ROM 702. The processing unit 701 and RAM 703 and / or ROM 702 are connected to each other via bus 704. Input / output (I / O) interface 705 is also connected to bus 704.

[0109] Multiple components in device 700 are connected to I / O interface 705, including: input unit 706, such as keyboard, mouse, etc.; output unit 707, such as various types of monitors, speakers, etc.; storage unit 708, such as disk, optical disk, etc.; and communication unit 709, such as network card, modem, wireless transceiver, etc. Communication unit 709 allows device 700 to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks.

[0110] Processing unit 701 can be various general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of processing unit 701 include, but are not limited to, CPUs, GPUs, various special-purpose computing chips, digital signal processors (DSPs), and any suitable processor, controller, microcontroller, etc. Processing unit 701 performs the various methods and processes described above, such as method 200. For example, in some embodiments, the methods and processes such as method 200 may be implemented as computer software programs tangibly contained in a machine-readable medium, such as storage unit 708. In some embodiments, part or all of the computer program may be loaded and / or installed on device 700 via RAM and / or ROM and / or communication unit 709. When the computer program is loaded into RAM and / or ROM and executed by processing unit 701, one or more actions of the various methods and processes described above may be performed. Alternatively, in other embodiments, processing unit 701 may be configured to perform one or more actions of the methods and processes described above by any other suitable means (e.g., by means of firmware).

[0111] This application may be a method, apparatus, system, and / or computer program product. A computer program product may include a computer-readable storage medium having computer-readable program instructions loaded thereon for performing various aspects of this application.

[0112] Computer-readable storage media can be tangible devices capable of holding and storing instructions for use by an instruction execution device. Computer-readable storage media can be, for example, but not limited to, electrical storage devices, magnetic storage devices, optical storage devices, electromagnetic storage devices, semiconductor storage devices, or any suitable combination thereof. More specific examples (a non-exhaustive list) of computer-readable storage media include: portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), static random access memory (SRAM), portable compact disc read-only memory (CD-ROM), digital multifunction disc (DVD), memory sticks, floppy disks, mechanical encoding devices, such as punch cards or recessed protrusions storing instructions thereon, and any suitable combination thereof. As used herein, computer-readable storage media is not to be construed as transient signals themselves, such as radio waves or other freely propagating electromagnetic waves, electromagnetic waves propagating through waveguides or other transmission media (e.g., light pulses through fiber optic cables), or electrical signals transmitted through wires.

[0113] The computer-readable program instructions described herein can be downloaded from computer-readable storage media to various computing / processing devices, or downloaded via a network, such as the Internet, local area network, wide area network, and / or wireless network, to an external computer or external storage device. The network may include copper cables, fiber optic cables, wireless transmission, routers, firewalls, switches, gateway computers, and / or edge servers. A network adapter card or network interface in each computing / processing device receives the computer-readable program instructions from the network and forwards them to computer-readable storage media within the respective computing / processing device.

[0114] The computer program instructions used to perform the operations of this application may be assembly instructions, instruction set architecture (ISA) instructions, machine instructions, machine-dependent instructions, microcode, firmware instructions, status setting data, or source code or object code written in any combination of one or more programming languages, including object-oriented programming languages ​​and conventional procedural programming languages. The computer-readable program instructions may be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving remote computers, the remote computer may be connected to the user's computer via any type of network—including a local area network (LAN) or a wide area network (WAN)—or may be connected to an external computer (e.g., via the Internet using an Internet service provider). In some exemplary implementations, electronic circuitry, such as programmable logic circuitry, field-programmable gate arrays (FPGAs), or programmable logic arrays (PLAs), is personalized by utilizing the status information of the computer-readable program instructions. This electronic circuitry can execute the computer-readable program instructions to implement various aspects of this application.

[0115] Various aspects of this application are described herein with reference to flowchart illustrations and / or block diagrams of methods, apparatuses, devices, and computer program products according to exemplary implementations of this application. It should be understood that each block of the flowchart illustrations and / or block diagrams, as well as combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer-readable program instructions.

[0116] These computer-readable program instructions can be provided to a processing unit of a general-purpose computer, a special-purpose computer, or other programmable data processing apparatus to produce a machine such that, when executed by the processing unit of the computer or other programmable data processing apparatus, they create means for implementing the functions / actions specified in one or more blocks of the flowchart and / or block diagram. These computer-readable program instructions can also be stored in a computer-readable storage medium that causes a computer, programmable data processing apparatus, and / or other device to operate in a particular manner. Thus, the computer-readable medium storing the instructions comprises an article of manufacture that includes instructions for implementing aspects of the functions / actions specified in one or more blocks of the flowchart and / or block diagram.

[0117] Computer-readable program instructions may also be loaded onto a computer, other programmable data processing apparatus, or other device to cause a series of operational steps to be performed on the computer, other programmable data processing apparatus, or other device to produce a computer-implemented process, thereby causing the instructions that execute on the computer, other programmable data processing apparatus, or other device to perform the functions / actions specified in one or more boxes of a flowchart and / or block diagram.

[0118] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various exemplary embodiments of this application. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of an instruction containing one or more executable instructions for implementing a specified logical function. In some alternative implementations, the functions marked in the blocks may occur in a different order than those marked in the drawings. For example, two consecutive blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, may be implemented using a dedicated hardware-based system that performs the specified function or action, or using a combination of dedicated hardware and computer instructions.

[0119] Embodiments of this application have been described. The above description is exemplary and not exhaustive, and is not limited to the disclosed embodiments. Many modifications and variations will be apparent to those skilled in the art without departing from the scope and spirit of the described embodiments. The terminology used herein is chosen to best explain the principles of the embodiments, their practical application, or improvements to the technology in the market, or to enable others skilled in the art to understand the various embodiments disclosed herein.

Claims

1. A task allocation method, characterized in that, include: The distribution of multiple data shards across a node cluster is obtained. These multiple data shards will be processed by the first task of the application job. The node cluster includes multiple node groups, and each node group includes a set of nodes interconnected by a bus. as well as Based on the distribution and the plurality of node groups, the first task is assigned to a plurality of execution units that execute the application job, wherein the plurality of execution units run on the node cluster.

2. The method according to claim 1, characterized in that, Also includes: In response to receiving a request to execute the application job, at least one node group is selected from the node cluster; as well as The plurality of execution units are run on the at least one node group.

3. The method according to claim 1, characterized in that, Also includes: In response to receiving a registration request from the first node, the node group to which the first node belongs is determined among the plurality of node groups based on the bus connection status between the first node and other nodes; as well as Store the association information between the first node and the node group.

4. The method according to claim 1, characterized in that, Assigning the first task to the plurality of execution units includes: Determine the amount of data in each of the multiple data shards in each of the multiple node groups; Based on the amount of data, a first node group is selected from the plurality of node groups; and The first task is assigned to the first execution unit running on the first node group among the plurality of execution units.

5. The method according to claim 4, characterized in that, Selecting a first node group from the plurality of node groups based on the data volume includes: determining the node group containing the largest data shard among the plurality of data shards as the first node group.

6. The method according to claim 4, characterized in that, Selecting the first node group based on the aforementioned data volume includes: The node group containing the largest amount of data among the multiple data shards is determined as the first node group.

7. The method according to claim 4, characterized in that, The first execution unit reads at least a portion of the plurality of data shards from the shared memory of the first node group via a bus.

8. The method according to claim 1, characterized in that, The first task belongs to the first stage of the application job, and the multiple data shards are generated by executing the second stage of the application job, the second stage being the stage preceding the first stage, and wherein: The first data shard in the plurality of data shards is written to the first shared memory by the first execution unit in the plurality of execution units, wherein the first data shard is generated by the first execution unit, the first shared memory corresponds to the first node group in the node cluster, and the first node group includes the first node running the first execution unit.

9. The method according to claim 8, characterized in that, in: The second data shard in the plurality of data shards is written to the second shared memory by the second execution unit in the plurality of execution units. The second data shard is generated by the second execution unit. The second shared memory corresponds to the second node group in the node cluster, and the second node group includes the second node running the second execution unit.

10. A task allocation device, characterized in that, include: The acquisition module is configured to acquire the distribution of multiple data shards on a node cluster, the multiple data shards being processed by the first task of the application job, wherein the node cluster includes multiple node groups, each of the multiple node groups including a group of nodes interconnected by a bus. as well as The allocation module is configured to allocate the first task to multiple execution units executing the application job based on the distribution and the multiple node groups, wherein the multiple execution units are running on the node cluster.

11. The apparatus according to claim 10, characterized in that, Also includes: The selection module is configured to select at least one group of nodes from the node cluster in response to receiving a request to execute the application job; as well as The execution module is configured to run the plurality of execution units on the at least one node group.

12. The apparatus according to claim 10, characterized in that, Also includes: The second determining module is configured to, in response to receiving a registration request from the first node, determine the node group to which the first node belongs among the plurality of node groups based on the bus connection status between the first node and other nodes. as well as The storage module is configured to store the association information between the first node and the node group.

13. The apparatus according to claim 10, characterized in that, The allocation module includes: The third determining module is configured to determine the amount of data in each of the multiple data shards in each of the multiple node groups; The second selection module is configured to select a first node group from the plurality of node groups based on the amount of data; and The second allocation module is configured to allocate the first task to the first execution unit running on the first node group among the plurality of execution units.

14. The apparatus according to claim 13, characterized in that, The second selection module includes: The fourth determining module is configured to determine the node group containing the largest data shard among the plurality of data shards as the first node group.

15. The apparatus according to claim 13, characterized in that, The second selection module includes: The fifth determining module is configured to determine the node group containing the largest amount of data among the multiple data shards as the first node group.

16. The apparatus according to claim 13, characterized in that, The first execution unit reads at least a portion of the plurality of data shards from the shared memory of the first node group via a bus.

17. The apparatus according to claim 10, characterized in that, The first task belongs to the first stage of the application job, and the multiple data shards are generated by executing the second stage of the application job, the second stage being the stage preceding the first stage, and wherein: The first data shard in the plurality of data shards is written to the first shared memory by the first execution unit in the plurality of execution units, wherein the first data shard is generated by the first execution unit, the first shared memory corresponds to the first node group in the node cluster, and the first node group includes the first node running the first execution unit.

18. The apparatus according to claim 17, characterized in that, in: The second data shard in the plurality of data shards is written to the second shared memory by the second execution unit in the plurality of execution units. The second data shard is generated by the second execution unit. The second shared memory corresponds to the second node group in the node cluster, and the second node group includes a second node running the second execution unit.

19. An electronic device, characterized in that, The device includes a processor and a memory, the memory storing computer instructions that, when executed by the processor, cause the electronic device to perform the method of any one of claims 1 to 9.

20. A computer program product, characterized in that, The computer program product includes instructions that, when executed by an electronic device, cause the electronic device to perform the method according to any one of claims 1 to 9.

Images