Method and apparatus for task planning, device, and computer program product

Abstract

Embodiments of the present disclosure relate to a method and apparatus for task planning, a device, and a computer program product. The method includes: generating, based on a current state of a task, a heterogeneous graph representation of a plurality of objects related to the task, where the heterogeneous graph representation includes object nodes indicating the plurality of objects and a first edge indicating a relationship between the plurality of objects in the current state. In addition, the method further includes generating a task action based on the heterogeneous graph representation by using a graph neural network. Therefore, according to the embodiments of the present disclosure, the task planning can be modeled by using heterogeneous graph data, and a task action of the task planning can be generated by using the graph neural network, thereby avoiding manual design of a learning strategy, reducing complexity of the task planning, and improving an effect of the task planning.

Description

[0001] METHOD AND APPARATUS FOR TASK PLANNING, DEVICE, AND COMPUTER PROGRAM PRODUCT

[0002] TECHNICAL FIELD

[0003] Embodiments of the present disclosure relate to the field of computers, and specifically, to a method and apparatus for task planning, a device, and a computer program product.

[0004] BACKGROUND

[0005] Task planning is a process of automatically generating a series of action or decision sequences according to environment information, and relates to a plurality of steps such as sensing an environment, setting a target, planning a path, and executing a task, to implement a highly efficient and accurate automatic operation. This technology is widely applied to fields such as robotics, industrial automation, and unmanned driving.

[0006] With development of technologies, importance of task planning is increasingly prominent. The task planning not only improves production efficiency and reduces manpower costs, but also can adapt to various complex environments and execute more elaborate and diversified tasks. The task planning is becoming a key technology in pushing progress of a plurality of industries. With improvement of intelligence and automation levels, the task planning plays a more important role in the future society.

[0007] SUMMARY

[0008] Embodiments of the present disclosure provide a method and apparatus for task planning, a device, a computer program product, and a medium.

[0009] According to a first aspect of the present disclosure, a method for task planning is provided. The method includes: generating, based on a current state of a task, a heterogeneous graph representation of a plurality of objects related to the task, where the heterogeneous graph representation includes object nodes indicating the plurality of objects and a first edge indicating a relationship between the plurality of objects in the current state. In addition, the method further includes generating a task action based on the heterogeneous graph representation by using a graph neural network.

[0010] According to a second aspect of the present disclosure, an apparatus for task planning is provided. The apparatus includes a heterogeneous representation generation module, configured to generate, based on a current state of a task, a heterogeneous graph representation of a plurality of objects related to the task, where the heterogeneous graph representation includes object nodes indicating the plurality of objects and a first edge indicating a relationship between the plurality of objects in the current state. In addition, the apparatus further includes a task action generation module, configured to generate a task action based on the heterogeneous graph representation by using a graph neural network.

[0011] According to a third aspect of the present disclosure, an electronic device is provided. The electronic device includes: at least one processor; and a memory, coupled to the at least one processor and having instructions stored therein, where the instructions, when executed by the at least one processor, cause the device to perform the steps of the method according to the first aspect of the present disclosure.

[0012] According to a fourth aspect of the present disclosure, a computer program product is provided. The computer program product is tangibly stored in a non-transitory computer- readable medium and includes computer-executable instructions. The computer-executable instructions, when executed, cause a computer to perform the steps of the method according to the first aspect of the present disclosure.

[0013] According to a fifth aspect of the present disclosure, a machine-readable storage medium is provided. Machine-executable instructions are stored in the machine-readable storage medium, where the machine-executable instructions are executed by a processor to implement the steps of the method according to the first aspect of the present disclosure.

[0014] The summary part is to introduce selected concepts in a simplified form. The concepts are further described in the following detailed description part. The summary part is neither intended to identify a key feature or main feature of a claimed subject matter, nor intended to limit the scope of the claimed subject matter.

[0015] BRIEF DESCRIPTION OF THE DRAWINGS

[0016] The foregoing and other objectives, features, and advantages of the present disclosure will become more apparent from more detailed descriptions of exemplary embodiments of the present disclosure with reference to the accompanying drawings. In the exemplary embodiments of the present disclosure, the same reference numerals generally represent the same components.

[0017] FIG. 1 is a schematic diagram of an exemplary environment in which a device and / or method can be implemented according to an embodiment of the present disclosure;

[0018] FIG. 2 is a flowchart of a method for task planning according to an embodiment of the present disclosure;

[0019] FIG. 3 is a flowchart of a process of performing task planning according to an embodiment of the present disclosure;

[0020] FIG. 4 is a schematic diagram of a process of using a large language model to generate scene graph data according to an embodiment of the present disclosure;

[0021] FIG. 5 is a schematic diagram of a process of converting scene graph data into heterogeneous graph data according to an embodiment of the present disclosure;

[0022] FIG. 6 is a schematic diagram of an interface of a relationship data labeling tool according to an embodiment of the present disclosure;

[0023] FIG. 7A is a schematic diagram of a process of performing edge prediction according to an embodiment of the present disclosure;

[0024] FIG. 7B is a schematic diagram of a process of updating a heterogeneous graph of a task after performing edge prediction according to an embodiment of the present disclosure;

[0025] FIG. 8A is a schematic diagram of a state diagram of a task according to an embodiment of the present disclosure;

[0026] FIG. 8B is a schematic diagram of a process of updating a state diagram of a task by using a large language model according to an embodiment of the present disclosure;

[0027] FIG. 9 is a schematic diagram of a process of performing task planning according to an embodiment of the present disclosure;

[0028] FIG. 10 is a schematic diagram of an apparatus for task planning according to an embodiment of the present disclosure; and

[0029] FIG. 11 is a schematic block diagram of an exemplary device that is suitable for implementing an embodiment of the present disclosure.

[0030] In the accompanying drawings, the same or corresponding reference numerals represent the same or corresponding parts.

[0031] DETAILED DESCRIPTION The following describes embodiments of the present disclosure in more detail with reference to the accompanying drawings. Although some embodiments of the present disclosure are shown in the accompanying drawings, it should be understood that the present disclosure may be implemented in various forms and should not be construed as being limited to the embodiments described herein, but these embodiments are provided for a more thorough and complete understanding of the present disclosure. It should be understood that the accompanying drawings and the embodiments of the present disclosure are merely used as examples, but are not intended to limit the protection scope of the present disclosure.

[0032] In descriptions of the embodiments of the present disclosure, the term "include" and similar terms should be understood as open inclusion, that is, "include but are not limited to". The term "based on" should be understood as "at least partially based on". The term "one embodiment" or "the embodiment" should be understood as "at least one embodiment". The terms "first", "second", and the like may represent different objects or the same object, unless explicitly stated. Other explicit and implicit definitions may be included below.

[0033] As described above, the task planning technology is developing rapidly, and plays an important role in many fields. In a process of task planning, a series of task decisions or task actions are generated to instruct a task to switch from an initial state to an expected state. In the related art, a learning strategy is usually manually designed according to a task constraint, which causes a large workload in task planning, high complexity in task decision, and a nonideal task planning effect.

[0034] Therefore, an embodiment of the present disclosure provides a solution for task planning. The task planning is modeled by using heterogeneous graph data. First, a heterogeneous graph representation corresponding to a current state of a task is generated, and then the heterogeneous graph representation is processed by using a graph neural network, to generate a task decision, thereby completing task planning.

[0035] Therefore, according to the embodiments of the present disclosure, the task planning can be modeled by using heterogeneous graph data, and a task action of the task planning can be generated by using the graph neural network, thereby avoiding manual design of a learning strategy, reducing complexity of the task planning, and improving an effect of the task planning.

[0036] The following describes the basic principles and several exemplary implementations of the present disclosure with reference to FIG. 1 to FIG. 11. It should be understood that these exemplary embodiments are provided only to enable a person skilled in the art to better understand and further implement the embodiments of the present disclosure, and are not intended to limit the scope of the present disclosure in any manner.

[0037] FIG. 1 shows an exemplary environment 100 in which a device and / or method in an embodiment of the present disclosure can be implemented. As shown in FIG. 1, the exemplary environment 100 may include a computing device 110, which may be a user terminal, a mobile device, a computer, or the like, or may be a computing system, a single server, a distributed server, or a cloud-based server. The computing device 110 may generate a corresponding heterogeneous graph representation 130 according to a current state 120 of a task. The heterogeneous graph representation 130 may include a plurality of objects related to the task and a relationship between the plurality of objects in the current state. For example, task planning may be performed for a robot tableware recycling task, to instruct a robot to sequentially place dishes into a dishwasher. A task constraint of the task is that an uppermost dish needs to be placed first. Each time a dish is placed into the dishwasher, a state of the task is updated. It should be understood that the task is only described as an example. The task is not limited in this embodiment of the present disclosure. Then, the current state 120 of the task may be a text describing a position relationship between current dishes, an image of a current task scene, or the like. In addition, the heterogeneous graph representation 130 may include a plurality of object nodes, each object node corresponds to one dish, and a connection (also referred to as an edge) between the object nodes represents a relationship between the dishes. It should be understood that the connection between the object nodes and the edge between the object nodes represent the same meaning.

[0038] In descriptions of some embodiments of the present disclosure, for the robot tableware recycling task, a node "P" represents a plate, a node "B" represents a bowl, a node "C" represents a cup, a node "D" represents a desk, a node "DW" represents a dishwasher, a connection "on" between the nodes represents "on", a connection "in" represents "in", and a connection "nt" represents "next to". For example, the heterogeneous graph representation 130 may indicate that in a current state of the robot tableware recycling task, a plate 131 is in a dishwasher 133 (through a connection 132), a bowl 134 is on a bowl 135, and so on.

[0039] Based on the heterogeneous graph representation 130, a connection between the object nodes may be generated by using a graph neural network 140, and the connection indicates a possibility that objects associated with the connection have a relationship in the current state. It should be understood that the generated connection described herein is a connection that does not exist in the heterogeneous graph representation 130. For example, the generated connection may include dashed-line edges 151 to 155 in a heterogeneous graph representation 150, and the dashed-line edge 151 may indicate a possibility that a relationship exists between the bowl 134 and the dishwasher 133 associated with the dashed-line edge 151. Then, a task action (also referred to as a task decision) may be generated based on the connection, to complete task planning. For example, in the robot tableware recycling task, a task action 160 may be generated according to the connection 151, for example, placing the bowl 134 into the dishwasher 133, and the state of the task may be updated by performing the task action.

[0040] Therefore, according to the embodiments of the present disclosure, the task planning can be modeled by using heterogeneous graph data, and a task action of the task planning can be generated by using the graph neural network, thereby avoiding manual design of a learning strategy, reducing complexity of the task planning, and improving an effect of the task planning.

[0041] It should be understood that an architecture and a function of the exemplary environment 100 are described for illustrative purposes only, and do not imply any limitation on the scope of the present disclosure. The embodiments of the present disclosure may further be applied to another environment having a different structure and / or function.

[0042] A process according to an embodiment of the present disclosure is described in detail below with reference to FIG. 2 to FIG. 11. For ease of understanding, specific data mentioned in the following descriptions is exemplary, and is not intended to limit the protection scope of the present disclosure. It may be understood that the embodiments described below may further include additional actions that are not shown and / or may omit actions that are shown, and the scope of the present disclosure is not limited in this aspect.

[0043] FIG. 2 is a flowchart of a method 200 for task planning according to an embodiment of the present disclosure. The method may be performed by a computing device, which may be a user terminal, a mobile device, a computer, or the like, or may be a computing system, a single server, a distributed server, or a cloud-based server. At block 202, a heterogeneous graph representation including a plurality of object nodes may be generated based on a current state of a task, where the heterogeneous graph representation indicates a plurality of objects related to the task and a relationship between the plurality of objects in the current state. For example, with reference to FIG. 1, the computing device 110 may generate, based on the current state 120 of the task, the heterogeneous graph representation 130 including the plurality of object nodes. The heterogeneous graph representation indicates the plurality of objects (for example, the plate 131, the dishwasher 133, or the bowl 134) related to the task and the relationship between the plurality of objects in the current state (for example, the connection 132 indicates that the plate 131 is in the dishwasher 133).

[0044] At block 204, a task action may be generated based on the heterogeneous graph representation by using a graph neural network. For example, with reference to FIG. 1, the computing device 110 may generate the task action 160 based on the heterogeneous graph representation 130 by using the graph neural network 140. Therefore, according to the method 200 in this embodiment of the present disclosure, the task planning can be modeled by using heterogeneous graph data, and a task action of the task planning can be generated by using the graph neural network, thereby avoiding manual design of a learning strategy, reducing complexity of the task planning, and improving an effect of the task planning.

[0045] FIG. 3 is a flowchart of a process 300 of performing task planning according to an embodiment of the present disclosure. The process may be performed by a computing device, which may be a user terminal, a mobile device, a computer, or the like, or may be a computing system, a single server, a distributed server, or a cloud-based server. The process 300 according to this embodiment of the present disclosure bridges a gap between a scene graph representation and task planning, making it possible to implement efficient and automatic task execution in a complex environment. By using NetworkX, a heterogeneous graph transformer (HGT) of Py Torch Geometric, and a supervised learning method, effective task planning is achieved based on heterogeneous graph data obtained from a scene graph, thereby improving adaptability and automation levels of the task planning in a real scene.

[0046] As shown in FIG. 3, at block 302, graph modeling is performed on a task, to generate scene graph data of the task. The scene graph data belongs to data of a homogeneous graph type. The scene graph may describe an object and a spatial relationship (such as a position relationship and a containment relationship) of the object in a scene. For example, in the field of computer graphics, a scene graph may be used for constructing and rendering a graphic scene, and may include information such as an object position, rotation, zoom, material, and illumination. The scene graph is represented as a graph data structure. Nodes represent objects, and edges (that is, connections between the nodes) represent relationships between the objects, such as a spatial relationship and an inclusion relationship. For example, task planning may be performed for a robot tableware recycling task, to instruct the robot to place dishes into a dishwasher. A task constraint of the task is that the dish placed uppermost needs to be operated first. Each time a dish is placed into the dishwasher, a state of the task is updated. It should be understood that the task is only described as an example. The task is not limited in this embodiment of the present disclosure.

[0047] For example, in the robot tableware recycling task, graph data modeling may be performed on the dishes in a current state of the task and a relationship between the dishes, to generate scene graph data of the task. In some embodiments, the scene graph data may be generated based on triple data of the nodes. The triple data includes two objects and a relationship between the two objects. For example, a graph processing tool (for example, NetworkX) may be used to preprocess the triple data, to generate the scene graph data. In some embodiments, a large language model may be used to generate ordered triple data of the nodes, and then the scene graph data is generated according to the ordered triple data. Generation of the scene graph data by using the large language model is described below with reference to FIG. 4.

[0048] FIG. 4 is a schematic diagram of a process 400 of using generated scene graph data according to an embodiment of the present disclosure. As shown in FIG. 4, a text 402 may be a description text of a task scene, and scene graph data 406 may be generated by using a large language model 404. In some embodiments, parameters of the large language model may be as follows: max_tokens=1000, top_p=l, temperature=0, frequency _penalty=0.0, presence_penalty=0.0, and stop=["\n"]. In some embodiments, these parameters may be adjusted according to a specific task. For example, in some usage scenarios, the parameter "max_tokens" may be adjusted to 2000.

[0049] In a process of generating the scene graph data by using the large language model, prompt content (prompt) may be set. In some embodiments, the prompt content may be set as follows: "Given a prompt, deduce relationships as many as possible, and provide an update list. If one update represents one relationship, a format is [entity 1, relationship, entity 2], The relationship is directional, and therefore, an order is very important". By using the prompt content, the large language model may split an input text into a subject-predicate-object triple, and store a result in a list. It should be understood that because the relationship is directional, an order of elements in the triple is very important.

[0050] For example, if the prompt content is "Place A into B", the ordered triple data generated by the large language model is "[["A", "in", "B"]]", where "A" may be referred to as a first object node, and "B" may be referred to as a second object node; if the prompt content is "Place C next to D", the ordered triple data generated by the large language model is "[["C", "nt", "D"]]"; if the prompt content is "Put E on the F", the ordered triple data generated by the large language model is "[["E", "on", "F"]]"; and if the prompt content is "A is on the B, B is on the C, D in the E", the ordered triple data generated by the large language model is "[["A", "on", "B"], ["B", "on", "C"], ["D", "in", "E"]]".

[0051] Descriptions are provided with reference to FIG. 4. The text 402 may be "Plate 1 is on plate 2, bowl 2 is on plate 1, bowl 1 is on bowl 2, plate 3 is in the dishwasher, cup 1 is next to plate 1, and cup 1 is also next to plate 2". Then, the generated ordered triple data is "[["Pl", "on", "P2"], ["B2", "on", "Pl"], ["Bl", "on", "B2"], ["P3", "in", "DW"], ["Cl", "nt", "Pl"], ["Cl", "nt", "P2"]]". Then, a graph processing tool (for example, NetworkX) may be used to preprocess the triple data, to generate the scene graph data 406. According to the foregoing process, the task scene may be described by using a natural language, and then the triple data is generated by using the large language model, to generate the scene graph data of the task. In this way, task scene modeling can be automated, manual processing is avoided, and efficiency and flexibility of scene modeling are improved.

[0052] Refer to FIG. 3 again. At block 304, the scene graph data may be converted into the heterogeneous graph data. The graph processing tool has flexibility in processing graph-based data. However, the generated scene graph data (namely, homogeneous graph data) is not directly applicable to training a graph neural network model. To implement effective task planning, the scene graph data needs to be converted into the heterogeneous graph data. The heterogeneous graph data has multi -type object nodes and multi -type edges, which can better represent complex scenes and relationships, enhance an expression capability and applicability of the model, improve training efficiency and prediction accuracy of the graph neural network model, and help implement more precise and efficient task decision. For example, in a task scene in which a robot recycles dishes, there are usually a plurality of types of objects, such as "a plate", "a bowl", "a cup", "a desk", and "a dishwasher", and there are also a plurality of types of relationships, such as "on", "in", and "next to". In comparison, a homogeneous graph usually has the same type of nodes and the same type of edges. A process of converting graph data is described below with reference to FIG. 5.

[0053] FIG. 5 is a schematic diagram of a process 500 of converting scene graph data into heterogeneous graph data according to an embodiment of the present disclosure. As shown in FIG. 5, scene graph data 502 may be converted into heterogeneous graph data 504. When the graph data is converted, the scene graph data may be traversed to store a node type and an initial node embedding of each object node and an edge type and an initial edge embedding of each edge, and then the heterogeneous graph data is generated based on the node type and the initial node embedding of each object node and the edge type and the initial edge embedding of each edge. For example, a node type mapping and a type index mapping may be created. The node type mapping is used for mapping a node to a node type, and the type index mapping is used for tracking an index of each node type. Then, the object nodes of the scene graph data may be traversed, and a unique index is allocated to each node based on the node type (such as "bowl", "cup", and "plate"). The indexes are stored in the node type mapping, and are classified according to the node type and stored in the "type index mapping". Next, a heterogeneous graph instance is created, to traverse the scene graph data to obtain and store information about the node type, the edge type, and an edge confidence. Then, a node type quantity mapping and an edge type quantity mapping may be created to respectively track a node quantity of each node type and an edge quantity of each edge type.

[0054] Next, the nodes in the scene graph data may be traversed, to generate a feature tensor (which is initialized as a zero tensor) for each node type. The feature tensors are stored in the heterogeneous graph instance. Then, an edge index mapping may be created, to track an edge index of each edge type. Then, the edges of the scene graph data may be traversed, to generate a triple for each edge, including a source node, a target node, and the edge type. The triples may be added to the edge index mapping. For each edge type, a corresponding edge index list is converted into a tensor, and is stored in the heterogeneous graph instance. In addition, the zero tensor may be initialized for each edge type as an edge attribute. In this way, the scene graph data may be converted into the heterogeneous graph data. The heterogeneous graph data includes information such as the node type, the initial node embedding, the edge type, and the initial edge embedding, to lay a foundation for training the graph neural network.

[0055] Refer to FIG. 3 again. At block 306, the graph neural network may be trained by using the heterogeneous graph data. In some embodiments, the graph neural network may be a heterogeneous graph transformer (HGT) model. For example, the heterogeneous graph transformer model may include four main parameters: "hidden channels" (a dimension of a hidden layer), "out channels" (a dimension of an output layer), "num heads" (a number of multi-head attention), and "num layers" (a number of transformer layers), and the parameters may be adjusted according to a scale of a data set. The node embedding of the object node may be continuously adjusted and updated in a process of training the graph neural network model, and the heterogeneous graph transformer model may output an adjusted node embedding. In some embodiments, the heterogeneous graph transformer model may be implemented by using a Py Torch Geometric library. The Py Torch Geometric library is a library based on Py Torch, supports writing and training the graph neural network, and is applicable to processing irregular structural data such as an image, a 3D grid, and a point cloud.

[0056] In some embodiments, a supervised learning method may be used to train the graph neural network. For example, the graph neural network is trained by comparing a predicted output of the graph neural network with a ground truth in heterogeneous graph training data by using a loss function, that is, the supervised learning method. In some embodiments, an object node corresponding to a benchmark task action may be set to a high-score ground truth label. The high score may be a score higher than a threshold. For example, in the robot tableware recycling task, a benchmark ground truth label of an object that can be placed into the dishwasher (that is, an object corresponding to the benchmark task action) may be set to a high value (for example, 0.9) in the heterogeneous graph training data, and a benchmark ground truth label of an object that cannot be put into the dishwasher may be set to a low value. During training of the graph neural network, the heterogeneous graph training data needs to be labeled. FIG. 6 shows a relationship data labeling tool configured to label the heterogeneous graph training data. In this way, the graph neural network may be trained by using the training data.

[0057] FIG. 6 is a schematic diagram of an interface 600 of a relationship data labeling tool according to an embodiment of the present disclosure. As shown in FIG. 6, a relationship preset option 602 lists common relationship types, including "in (In)", "on the left (Left)", "next to (Nt)", "occlude (Occludes)", "on (On)", "part of (Partof)", "on the right (Right)", and "under (Under)". A relationship list 604 may display a currently labeled relationship. For example, "In (9->25)" may represent that an object numbered 9 is in an object numbered 25. An image panel 606 displays a to-be-labeled image, for example, may display a placement status of dishes in a dishwasher. In addition, there are a plurality of lines of different colors on the image, and each line represents a relationship in the relationship preset option 602. In addition, a brightness setting 608 and a contrast setting 610 can display objects and a relationship between the objects in the to-be-labeled image more clearly. In this way, an object relationship in a complex scene can be efficiently labeled by using the relationship data labeling tool, to generate a large amount of accurate scene graph data to train the graph neural network model, thereby improving a training effect of the graph neural network model. Refer to FIG. 3 again. At block 308, edge prediction of a heterogeneous graph is performed by using the graph neural network. For example, the node embedding of each object node may be obtained by using the graph neural network. In some embodiments, a vector dot product may be calculated for node embeddings of two object nodes, and the vector dot product is used as a probability that there is a second edge between the two object nodes, to determine a second edge having a highest probability. The second edge is an edge between each source node (for example, a dish) and a target node (for example, the dishwasher). Descriptions are provided with reference to FIG. 1. An action of calculating the vector dot product may be performed by the computing device 110 in FIG. 1. A process of performing edge prediction by using the graph neural network is described below with reference to FIG. 7A. Then, after the edge prediction is performed once, a state of a task changes. A process of updating a heterogeneous graph representation of the task after performing the edge prediction is described below with reference to FIG. 7B. In some embodiments of the present disclosure, a state diagram of the task may be used to represent a change process of a task state, and each node in the state diagram of the task is a scene graph corresponding to the state. The state diagram of the task is described below with reference to FIG. 8A. In addition, after the edge prediction is performed once, the state of the task changes, so that a scene graph of the task also changes. The scene graph and the state diagram of the task may be updated by using the large language model through a natural language. A process of updating the scene graph and the state diagram by using the large language model is described below with reference to FIG. 8B.

[0058] FIG. 7Ais a schematic diagram of a process 700Aof performing edge prediction according to an embodiment of the present disclosure. As shown in FIG. 7A, in a heterogeneous graph representation of a current state of the task, a dashed line represents a to-be-predicted edge. For example, a bowl 704 is on a bowl 706, and the bowl 704 has not been placed into a dishwasher 708. Therefore, there is no connection between the bowl 704 and the dishwasher 708. An edge 710 exists between a plate 702 and the dishwasher 708, and the edge 710 indicates that the plate is in the dishwasher 708. During the edge prediction, a node embedding of each object node may be obtained, then a vector dot product is calculated for node embeddings of two object nodes, and the vector dot product is used as a probability that an edge exists between the two object nodes.

[0059] For example, a vector dot product of an embedding of the bowl 704 and an embedding of the dishwasher 708 may be calculated and used as a predicted probability (that is, an existence probability) of an edge 712. Similarly, predicted probabilities of an edge 714, an edge 717, an edge 718, and an edge 720 may be calculated. In some embodiments, a task decision (that is, a to-be-performed task action) of the task in the current state may be determined based on the predicted probability of the edge. For example, an edge having a highest predicted probability may be selected from the edge 712, the edge 714, the edge 717, the edge 718, and the edge 720, and an object node of the edge having the highest probability may be placed into the dishwasher 708. For example, if the predicted probability of the edge 712 is the highest, the task decision may be "Place the bowl 704 into the dishwasher 708". After the task decision is performed, the state of the task changes, and the heterogeneous graph of the task is correspondingly updated. FIG. 7B is a schematic diagram of a process 700B of updating a heterogeneous graph of a task after performing edge prediction according to an embodiment of the present disclosure. As shown in FIG. 7B, when the "Place the bowl 704 into the dishwasher 708" is performed, the edge 712 (changing from a dashed line in FIG. 7A to a solid line in FIG. 7B) exists between the bowl 704 and the dishwasher 708, indicating that the bowl 704 is in the dishwasher 708.

[0060] FIG. 8A is a schematic diagram of a state diagram 800A of a task according to an embodiment of the present disclosure. As shown in FIG. 8A, a task scene graph corresponding to a state node 802 (a state 0) is 804. After an action of placing a bowl into a dishwasher is performed, the task is switched from the state node 802 to a state node 806 (a state 1), and a task scene graph corresponding to the state node 806 is 808. It should be understood that as a state of the task changes, the state diagram of the task continues to be updated, from the state 1 to a state 2, and to a state N.

[0061] FIG. 8B is a schematic diagram of a process 800B of updating a state diagram of a task by using a large language model according to an embodiment of the present disclosure. As shown in FIG. 8B, a text 810 (that is, a natural language instruction) may be "Place a bowl into a dishwasher". First, the text 810 may be converted into a triple ["B", "in", "DW"] by using the large language model. Then, a breadth-first search (BFS) may be performed starting from an object node corresponding to the "bowl". In this manner, all objects on the "bowl" may be identified and stored in a list, and then a pop-up operation is performed on the list to sequentially remove the objects from the list. When all the objects in the list are removed, the "bowl" can be directly moved to the "dishwasher", to generate a scene graph to update the state diagram of the task.

[0062] A process of performing the breadth-first search may be described with reference to the following examples. For example, assuming that an initial scene graph is "(['A', 'on 'B'], ['B', 'on', 'C'], ['C, 'on', 'D'], ['E', 'in', 'F'], and ['D', 'on', 'desk'])", and the natural language instruction is "Move C into F", the breadth-first search is performed starting from C. First, an object list on C is recognized as ['B', 'A'], then the pop-up operation is performed, A is moved onto a desk, and a new scene graph is "(['A', 'on', 'desk'], ['B', 'on', 'C'], ['C, 'on', 'D'], ['E', 'in', 'F'], and ['D', 'on', 'desk'])". Then, the pop-up operation is continued to be performed, B is moved onto the desk, and the new scene graph is "(['A', 'on', 'desk'], ['B', 'on', 'desk'], ['C, 'on', 'D'], ['E', 'in', 'F'], and ['D', 'on', 'desk'])". Therefore, the list on the object C is empty. Then, the object C may be directly moved into the object F, and the new scene graph is "(['A', 'on', 'desk'], ['B', 'on', 'desk'], ['C, 'in', 'F'], ['E', 'in', 'F'], and ['D', 'on', 'desk'])". In this way, the scene graph and the state diagram of the task can be conveniently updated by using the large language model and the breadth-first search.

[0063] Refer to FIG. 3 again. At block 310, the task planning may be performed based on the edge prediction. For example, as described above, the edge having the highest probability may be selected based on the edge prediction, to determine the task decision, and update the scene graph, the heterogeneous graph, and the state diagram of the task. Then, the edge prediction may continue to be iteratively performed, until the task ends. A process of performing task planning is described below with reference to FIG. 9.

[0064] FIG. 9 is a schematic diagram of a process 900 of performing task planning according to an embodiment of the present disclosure. For example, an initial state of a task is a state 902 (a state 0), and the state 902 corresponds to a task scene graph 904. As described above, edge prediction may be performed by using a graph neural network based on the task scene graph 904, to generate a task decision (for example, placing a bowl 1 into a dishwasher). By performing the task decision, the task may be switched from the state 902 to a state 912 (a state 1), and the state 912 corresponds to a task scene graph 914. Similarly, the edge prediction may be performed by using the graph neural network based on the task scene graph 914 to generate the task decision, to switch a task state to a state 922 (a state 2), which corresponds to a task scene graph 924. The task planning ends, until the task state reaches a final state 952. In some embodiments, a reference scene graph of an expected state may be generated by using a large language model, and then after state switching is performed on the task, when the task scene graph is consistent with the reference scene graph, it can be determined that the task reaches the expected state. For example, a user may specify that the final state of the task is "All objects except a desk are placed into the dishwasher", and the large language model may generate the reference scene graph of the task in the expected state by using the instruction. Then, when the task scene graph is the same as the reference scene graph, that is, when the state of the task corresponds to the reference scene graph, it can be determined that the task planning reaches the expected state.

[0065] FIG. 10 is a schematic diagram of an apparatus 1000 for task planning according to an embodiment of the present disclosure. The apparatus 1000 includes a heterogeneous representation generation module 1002, configured to generate, based on a current state of a task, a heterogeneous graph representation of a plurality of objects related to the task, where the heterogeneous graph representation includes object nodes indicating the plurality of objects and a first edge indicating a relationship between the plurality of objects in the current state. In addition, the apparatus 1000 further includes a task action generation module 1004, configured to generate a task action based on the heterogeneous graph representation by using a graph neural network.

[0066] In some embodiments, the task action generation module 1004 includes: a second edge prediction module, configured to predict, based on the heterogeneous graph representation by using the graph neural network, a second edge between at least two object nodes in the heterogeneous graph representation, where the second edge indicates a possibility that a task action relationship exists between objects represented by the at least two object nodes; and a task action determining module, configured to determine a task action in the current state based on the prediction result.

[0067] In some embodiments, the apparatus 1000 further includes: a current state update module, configured to update the current state of the task based on the generated task action; and a task state update module, configured to repeatedly perform a step of generating the task action based on the current state of the task and updating the current state of the task based on the generated task action, until the task reaches an expected state.

[0068] In some embodiments, the heterogeneous representation generation module 1002 includes: a homogeneous representation generation module, configured to generate, based on the current state of the task, a homogeneous graph representation corresponding to the current state, where the homogeneous graph representation includes object information of the plurality of objects and connection information between the plurality of objects; and a second heterogeneous representation generation module, configured to generate, based on the homogeneous graph representation, the heterogeneous graph representation including the plurality of object nodes, where the plurality of object nodes have a plurality of object node types.

[0069] In some embodiments, the second heterogeneous representation generation module includes: a node information storage module, configured to store a node type and an initial node embedding of each object node and a connection type and an initial connection embedding of each connection by traversing the homogeneous graph representation; and a third heterogeneous representation generation module, configured to generate the heterogeneous graph representation based on the node type and the initial node embedding of each object node and the connection type and the initial connection embedding of each connection.

[0070] In some embodiments, the homogeneous representation generation module includes: a state text obtaining module, configured to obtain a text corresponding to the current state of the task; a triple generation module, configured to generate an ordered triple based on the text by using a language model, where the ordered triple includes a first object node, a connection between object nodes, and a second object node; and a second homogeneous representation generation module, configured to generate, based on the ordered triple, the homogeneous graph representation corresponding to the current state.

[0071] In some embodiments, the apparatus 1000 further includes: a task state switch module, configured to switch the task from the current state to a next state corresponding to execution of the task action; and a state diagram update module, configured to update a state diagram of the task based on the next state, where the state diagram includes a plurality of state nodes and a task scene graph corresponding to each state node, each state node corresponds to one generated task action of the task, and the task scene graph includes a homogeneous graph representation of the task in a corresponding state node.

[0072] In some embodiments, the state diagram update module includes: an action triple generation module, configured to generate, based on an action representation text corresponding to the task action, an ordered triple of the task action by using the language model; a next homogeneous graph generation module, configured to generate a homogeneous graph representation of the next state through a breadth-first search based on the ordered triple of the task action and the homogeneous graph representation of the current state; and a second state diagram update module, configured to update the state diagram of the task by using the homogeneous graph representation of the next state.

[0073] In some embodiments, the apparatus 1000 further includes: an expected state generation module, configured to generate, based on a text corresponding to the expected state of the task, an expected homogeneous graph representation corresponding to the expected state; and a task state determining module, configured to: in response to that a latest state node in the state diagram of the task corresponds to the expected homogeneous graph representation, determine that the task reaches the expected state.

[0074] In some embodiments, the apparatus 1000 further includes: a graph network training module, configured to train the graph neural network by using heterogeneous graph training data, where an object node corresponding to a benchmark task action in the heterogeneous graph training data is set with a high-score ground truth label.

[0075] In some embodiments, the apparatus 1000 further includes: a training data labeling module, configured to label a connection between object nodes in the heterogeneous graph training data by using a labeling tool, where the labeling tool is configured to perform labeling in a visual manner.

[0076] In some embodiments, the task action determining module includes: a node embedding generation module, configured to generate a node embedding of an object node in the heterogeneous graph representation by using the graph neural network; a high-score connection determining module, configured to determine a second edge having a highest probability by using a vector inner product of the node embedding as a probability that the second edge exists between two object nodes; and a second task action determining module, configured to generate the task action based on the second edge having the highest probability and corresponding object nodes.

[0077] FIG. 11 is a schematic block diagram of an exemplary device 1100 that is suitable for implementing an embodiment of the present disclosure. As shown in the figure, the device 1100 includes a processor 1101, which can perform various appropriate actions and processing according to computer program instructions stored in a read-only memory (ROM) 1102 that are loaded into a random access memory (RAM) 1103. The RAM 1103 may further store various programs and data required for operating the device 1100. The processor 1101, the ROM 1102, and the RAM 1103 are connected to each other through a bus 1104. An input / output (I / O) interface 1105 is also connected to the bus 1104.

[0078] The methods and processes described above may be performed by the processor 1101. For example, in some embodiments, the methods and processes described above may be implemented as a computer software program that is tangibly included in a machine-readable medium. In some embodiments, a part or all of the computer program may be loaded into and / or installed onto the device 1100 through the ROM 1102. When the computer program is loaded into the RAM 1103 and executed by the processor 1101, one or more actions of the methods and processes described above may be performed.

[0079] The present disclosure may be a method, an apparatus, a system, and / or a computer program product. The computer program product may include a computer-readable storage medium, having computer-readable program instructions used for performing various aspects of the present disclosure stored therein.

[0080] The computer-readable storage medium may be a tangible device that may hold and store instructions used by an instruction execution device. The computer-readable storage medium may be, for example, but is not limited to, an electrical storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any appropriate combination of the above. A more specific example (a non- exhaustive list) of the computer-readable storage medium includes, but is not limited to, a random access memory (RAM), a read-only memory (ROM), an erasable programmable readonly memory (EPROM or a flash), a static random access memory (SRAM), and any appropriate combination of the above. The computer-readable storage medium used herein is not explained as a transient signal itself, such as a radio wave or another electromagnetic wave propagated freely, an electromagnetic wave propagated through a waveguide or other transmission media (e.g., a light pulse propagated through an optical fiber cable), or an electrical signal transmitted through a wire.

[0081] The computer-readable program instructions described herein may be downloaded to various computing / processing devices from a computer-readable storage medium or to an external computer or external storage device via a network, for example, the Internet, a local area network, a wide area network and / or a wireless network. The network may include a copper transmission cable, optical fiber transmission, wireless transmission, a router, a firewall, a switch, a gateway computer, and / or an edge server. A network adapter card or network interface in each computing / processing device receives computer-readable program instructions from the network and forwards the computer-readable program instructions for storage in a computer-readable storage medium within the various computing / processing devices.

[0082] The computer program instructions for performing operations of the present disclosure may be assembly instructions, instruction set architecture (ISA) instructions, machine instructions, machine-related instructions, microcode, firmware instructions, status setting data, or source code or object code written in any combination of one or more programming languages. The programming languages include object-oriented programming languages such as Smalltalk and C++, and conventional procedural programming languages such as a "C" language or a similar programming language. The computer-readable program instructions may be executed entirely on a computer of a user, partly on the computer of the user, as a standalone software package, partly on the computer of the user and partly on a remote computer, or entirely on the remote computer or a server. In a situation involving the remote computer, the remote computer may be connected to a computer of a user through any type of network including a local area network (LAN) or wide area network (WAN), or may be connected to an external computer (for example, through the Internet by using an Internet service provider). In some embodiments, an electronic circuit, such as a programmable logic circuit, a field programmable gate array (FPGA), or a programmable logic array (PLA), is customized by using state information of the computer-readable program instructions. The electronic circuit may execute the computer-readable program instructions, to implement various aspects of the present disclosure.

[0083] The aspects of the present disclosure are described herein with reference to the flowcharts and / or the block diagrams of the method, the apparatus (system), and the computer program product in the embodiments of the present disclosure. It should be understood that each block in the flowcharts and / or the block diagrams and a combination of the blocks in the flowcharts and / or the block diagrams may be implemented through the computer-readable program instructions.

[0084] The computer-readable program instructions may be provided to a processing unit of a general-purpose computer, a special-purpose computer, or another programmable data processing apparatus, to produce a machine. In this way, the instructions, when executed by the processing unit of the computer or the another programmable data processing apparatus, generate an apparatus that implements the functions / actions specified in one or more blocks in the flowcharts and / or the block diagrams. The computer-readable program instructions may alternatively be stored in a computer-readable storage medium. The instructions cause a computer, a programmable data processing apparatus, and / or another device to work in a particular manner. In this way, a computer-readable medium storing the instructions includes an artifact that includes instructions for implementing various aspects of the functions / actions specified in one or more blocks of the flowcharts and / or block diagrams.

[0085] The computer-readable program instructions may alternatively be loaded onto a computer, another programmable data processing apparatus, or another device, to cause a series of operational steps to be performed on the computer, the another programmable data processing apparatus, or the another device to produce a computer implemented process, such that the instructions that are executed on the computer, the another programmable data processing apparatus, or the another device implement the functions / actions specified in one or more blocks of the flowcharts and / or block diagrams.

[0086] The flowcharts and block diagrams in the accompanying drawings show a system architecture, functions, and operations that may be implemented by using the system, the method, and the computer program product according to a plurality of embodiments of the present disclosure. In this regard, each block in a flowchart or a block diagram may represent a module, a program segment, or a part of instructions. The module, the program segment, or the part of instructions include one or more executable instructions used for implementing specified logic functions. In some alternative implementations, functions labeled in the blocks may alternatively be performed in an order different from that labeled in the accompanying drawings. For example, two consecutive blocks may actually be executed substantially in parallel, or may sometimes be executed in a reverse order. This depends on the functions involved. It should be further noted that, each block in the block diagram and / or the flowchart, and a combination of blocks in the block diagram and / or the flowchart may be implemented by using a dedicated hardware-based system that performs a specified function or action, or may be implemented by using a combination of dedicated hardware and computer instructions.

[0087] The embodiments of the present disclosure have been described above. The above descriptions are exemplary, not exhaustive and are not limited to the disclosed embodiments. Many modifications and changes are clear to a person of ordinary skill in the art without departing from the scope and spirit of the described embodiments. The selection of the terms used herein is intended to be the best explanation of the principles and practical applications of the various embodiments, or technical improvements of the technology in the market, or to enable others of ordinary skill in the art to understand the embodiments disclosed herein.

Claims

CLAIMSWhat is claimed is:

1. A method (200) for task planning, comprising: generating (202), based on a current state of a task, a heterogeneous graph representation of a plurality of objects related to the task, wherein the heterogeneous graph representation comprises object nodes indicating the plurality of objects and a first edge indicating a relationship between the plurality of objects in the current state; and generating (204) a task action based on the heterogeneous graph representation by using a graph neural network.

2. The method (200) according to claim 1 , characterized in that the generating a task action based on the heterogeneous graph representation by using a graph neural network comprises: predicting, based on the heterogeneous graph representation by using the graph neural network, a second edge between at least two object nodes in the heterogeneous graph representation, wherein the second edge indicates a possibility that a task action relationship exists between objects represented by the at least two object nodes; and determining a task action in the current state based on the prediction result.

3. The method (200) according to claim 1, further comprising: updating the current state of the task based on the generated task action; and repeatedly performing a step of generating the task action based on the current state of the task and updating the current state of the task based on the generated task action, until the task reaches an expected state.

4. The method (200) according to claim 1, characterized in that the generating a heterogeneous graph representation of a plurality of objects related to the task comprises: generating, based on the current state of the task, a homogeneous graph representation corresponding to the current state, wherein the homogeneous graph representation comprises object information of the plurality of objects and connection information between the plurality of objects; and generating, based on the homogeneous graph representation, the heterogeneous graphrepresentation comprising the plurality of object nodes, wherein the plurality of object nodes have a plurality of object node types.

5. The method (200) according to claim 4, characterized in that the generating the heterogeneous graph representation comprising the plurality of object nodes comprises: storing a node type and an initial node embedding of each object node and a connection type and an initial connection embedding of each connection by traversing the homogeneous graph representation; and generating the heterogeneous graph representation based on the node type and the initial node embedding of each object node and the connection type and the initial connection embedding of each connection.

6. The method (200) according to claim 4, characterized in that the generating a homogeneous graph representation comprises: obtaining a text corresponding to the current state of the task; generating an ordered triple based on the text by using a language model, wherein the ordered triple comprises a first object node, a connection between object nodes, and a second object node; and generating, based on the ordered triple, the homogeneous graph representation corresponding to the current state.

7. The method (200) according to claim 6, further comprising: switching the task from the current state to a next state corresponding to execution of the task action; and updating a state diagram of the task based on the next state, wherein the state diagram comprises a plurality of state nodes and a task scene graph corresponding to each state node, each state node corresponds to one generated task action of the task, and the task scene graph comprises a homogeneous graph representation of the task in a corresponding state node.

8. The method (200) according to claim 7, characterized in that the updating a state diagram of the task comprises: generating an ordered triple of the task action by using the language model based on anaction representation text corresponding to the task action; generating a homogeneous graph representation of the next state through a breadth-first search based on the ordered triple of the task action and the homogeneous graph representation of the current state; and updating the state diagram of the task by using the homogeneous graph representation of the next state.

9. The method (200) according to claim 7, further comprising: generating, based on a text corresponding to an expected state of the task, an expected homogeneous graph representation corresponding to the expected state; and determining, in response to that a latest state node in the state diagram of the task corresponds to the expected homogeneous graph representation, that the task reaches the expected state.

10. The method (200) according to claim 1, further comprising: training the graph neural network by using heterogeneous graph training data, wherein an object node corresponding to a benchmark task action in the heterogeneous graph training data is set with a high-score ground truth label.

11. The method (200) according to claim 10, further comprising: labeling a connection between object nodes in the heterogeneous graph training data by using a labeling tool, wherein the labeling tool is configured to perform labeling in a visual manner.

12. The method (200) according to claim 2, characterized in that the generating a task action comprises: generating a node embedding of an object node in the heterogeneous graph representation by using the graph neural network; determining a second edge having a highest probability by using a vector inner product of the node embedding as a probability that the second edge exists between two object nodes; and generating the task action based on the second edge having the highest probability and corresponding object nodes.

13. An apparatus for task planning, comprising: a heterogeneous representation generation module, configured to generate, based on a current state of a task, a heterogeneous graph representation of a plurality of objects related to the task, wherein the heterogeneous graph representation comprises object nodes indicating the plurality of objects and a first edge indicating a relationship between the plurality of objects in the current state; and a task action generation module, configured to generate a task action based on the heterogeneous graph representation by using a graph neural network.

14. An electronic device, comprising: at least one processor; and a memory, coupled to the at least one processor and having instructions stored therein, wherein the instructions, when executed by the at least one processor, cause the device to perform the method according to any one of claims 1 to 12.

15. A computer program product, tangibly stored in a non-transitory computer-readable medium and comprising machine-executable instructions, wherein the machine-executable instructions are used for performing the method according to any one of claims 1 to 12.

Images