Multi-agent collaborative embodied task reasoning and robot scheduling system, and method

Abstract

Disclosed are a multi-agent collaborative embodied task reasoning and robot scheduling system, a method, an electronic device, a computer-readable storage medium, a computer program product, and a computer program. The scheduling system comprises an interaction unit, which is used for acquiring a natural language instruction issued by a user, sending the instruction to an intelligent processing unit and interacting therewith, receiving an execution result of a task and embodied state information of a robot, and when task information is missing, sending a request for supplementation of missing task information; the intelligent processing unit, which is used for performing dynamic perception on environment and task states on the basis of a user instruction, embodied state information and environment information fed back by the robot, and stored memory, and performing dynamic planning and decision-making on the basis of a perception result and a reflection result of a task execution result, so as to implement task reasoning and robot scheduling; and a robot fleet, which is used for executing the task on the basis of an action sequence, and feeding back embodied state information of respective robots and environment information.

Description

Embodied Task Reasoning and Robot Scheduling System and Method for Multi-Agent Collaboration

[0001] Cross-references to related applications

[0002] This application claims priority to Chinese Patent Application No. 2025100650417, filed in China on January 15, 2025, the entire contents of which are incorporated herein by reference. Technical Field

[0003] This disclosure relates to the fields of large language models and intelligent robots, specifically to a multi-agent collaborative embodied task reasoning and robot scheduling system and method, electronic device, computer-readable storage medium, computer program product and computer program, which is particularly suitable for application scenarios in complex office environments where robots and humans collaborate to perform tasks. Background Technology

[0004] Existing service robots typically only execute preset instructions when handling real-world tasks, lacking a deep understanding of complex natural language commands and the ability to make dynamic reasoning, making it difficult to cope with tasks in real-world environments that are full of uncertainty and change. Furthermore, traditional service robots often fail to proactively seek human assistance when faced with task failure or missing information, resulting in low task execution efficiency or even complete failure.

[0005] While large language models have made significant progress in natural language processing in recent years, combining them with embodied robots and applying them to real-world office scenarios still presents numerous challenges. For example, existing service robots, when performing real-world tasks, typically only follow preset instructions, lacking a deep understanding of complex natural language and dynamic reasoning capabilities, making them ill-equipped to handle uncertainty and variability in the environment. Furthermore, these robots cannot proactively seek human assistance when tasks fail or information is missing, leading to low efficiency or even task failure. Despite the significant advancements in large language models in natural language processing, combining them with embodied robots for practical office scenarios still faces multiple challenges, including the perception and planning of embodied tasks, efficient request mechanisms in human-machine collaboration, and issues related to system integration and resource optimization, as well as robot deployment. Overcoming these technical bottlenecks is crucial for realizing intelligent, adaptable, and safe service robots. Therefore, there is an urgent need for a method that combines embodied task reasoning and execution to enhance the task reasoning, decision-making, and scheduling capabilities of service robots in complex environments, meeting practical application needs. Summary of the Invention

[0006] This disclosure aims to at least partially address one of the technical problems existing in the prior art.

[0007] To this end, this disclosure provides a multi-agent collaborative embodied task reasoning and robot scheduling system and method, which introduces a multi-agent collaborative architecture based on a large language model. By simulating the four thinking steps of human problem-solving—perception, planning, execution, and reflection—it endows service robots with natural language understanding, dynamic task reasoning, autonomous decision-making, and human-machine collaboration capabilities in complex environments, enabling them to efficiently handle complex task instructions and dynamic situations.

[0008] To achieve the above objectives, the embodiments disclosed herein employ the following technical solutions.

[0009] The first aspect of this disclosure provides a multi-agent collaborative embodied task reasoning and robot scheduling system, comprising:

[0010] The interaction unit is used to acquire natural language commands issued by the user and send them to the intelligent processing unit, and to interact with the intelligent processing unit to receive the execution results of the task and the robot's physical state information, and to issue a request to supplement the missing task information when the intelligent processing unit determines that the task information is missing.

[0011] The intelligent processing unit is used to dynamically perceive the environment and task status based on user instructions sent by the interaction unit, embodied state information and environmental information fed back by the robot, and the memory stored in the intelligent processing unit itself. Based on the perception results and the reflection results on the task execution results, it performs dynamic planning and decision-making to realize task reasoning and robot scheduling.

[0012] The robot swarm executes tasks according to the action sequence issued by the intelligent processing unit, and feeds back its own status information through the interface, which helps the intelligent processing unit to dynamically perceive the environmental status and update the memory of the intelligent processing unit.

[0013] In some embodiments, the intelligent processing unit adopts a multi-agent collaborative architecture based on a large language model, including a memory module, a perceptual agent, a planning agent, a decision-making agent, and a reflective agent. Each agent interacts with the others through natural language and shares the memory stored in the memory module.

[0014] The sensing agent is used to dynamically perceive the environment and task status based on the user instructions obtained by the interaction unit, combined with the robot's physical state information and historical task execution logs, and generate a task perception package containing task objectives, location information and real-time task execution feedback.

[0015] The planning agent is used to dynamically reason about task decomposition and execution schemes and generate task plans based on the task perception package and the reflection results provided by the reflection agent.

[0016] The decision-making agent is used to parse the task plan, generate action sequences, and perform robot scheduling to guide the robot to complete the task.

[0017] The reflective agent is used to evaluate the task execution results. If the task execution results do not meet expectations, it provides feedback for the next round of task reasoning to replan or adjust the execution plan.

[0018] In some embodiments, the memory module stores long-term memory and short-term memory;

[0019] The long-term memory includes an environmental semantic map, the user's default location, static location information of items, location information of public equipment, user preferences, and historical task execution logs;

[0020] The short-term memory includes real-time status information and dynamic context information during the current task execution process. The real-time status information includes user instructions, current task execution progress, robot's embodied state information, dynamic environment information, reflection results of the reflective agent, and user feedback records during task execution.

[0021] In some embodiments, the intelligent processing unit possesses task reasoning modes including reactive, proactive, and prospective.

[0022] Reactive approach: When the perceptual agent determines that the task information is complete based on the user instructions sent by the interaction unit and the embodied state feedback from the robot, it extracts the task objective and operation requirements from the task information and generates a task perception package containing the task objective, location information, and real-time feedback; the planning agent generates task execution steps based on the task perception package and the reflection results provided by the reflecting agent to form a task plan, which should meet the most basic executable conditions; the decision-making agent generates an action sequence after matching the task plan issued by the planning agent with the embodied capabilities of the robot cluster.

[0023] Active mode: When the perceptual agent determines that the task information is incomplete based on the user instructions sent by the interaction unit and the embodied state feedback from the robot, it sends a request for supplementary task information to the planning agent to fill the information gap. The planning agent forms a corresponding task plan based on the request and the reflection results provided by the reflection agent. The decision agent generates an action sequence based on the task plan and obtains the required supplementary information by searching relevant information in the memory module or by sending a query request to the user through the interaction unit.

[0024] Proactive approach: When the current task cannot be completed after multiple rounds of proactive reasoning, the perceptual agent sends a multi-party assistance request to the planning agent. The planning agent forms a corresponding task plan based on the request and the reflection results provided by the reflection agent. The decision agent generates an action sequence based on the task plan and initiates inquiry requests to other users in the environment through the interaction unit to expand the scope of information acquisition and ensure the smooth execution of the task.

[0025] In some embodiments, the method of initiating a query request to other users in the environment through the interaction unit includes:

[0026] Individual contact: Identify potential assisting users, send requests to obtain necessary information, including personnel not directly involved in the task or personnel not recorded in the memory module;

[0027] Group broadcast: Publish assistance requests in work-related groups to coordinate resources from multiple parties to obtain the information needed for the task.

[0028] In some embodiments, the robot includes:

[0029] A robot knowledge container is used to store environmental semantic maps, target user locations, and item coordinate information, and supports real-time updates;

[0030] A high-level function container for robots, providing functional tools for acquiring and controlling the robot's embodied state, including navigation, object interaction, and real-time status feedback;

[0031] The robot execution module is used to perform corresponding tasks based on the action sequence generated by the decision-making agent, including navigation, object interaction, and sending online messages;

[0032] The embodied state feedback interface is used to feed back the robot's current physical state and task progress to the perceptual agent in order to update the task state information.

[0033] In some embodiments, the action sequence generated by the decision-making agent includes online operations and physical actions. The decision-making agent sends action instructions to the robot execution module through an interface, and dynamically adjusts the execution plan during task execution.

[0034] In some embodiments, the process of the decision-making agent scheduling the robot includes:

[0035] The decision-making agent, based on the available functions and tools for the robot, breaks down the task plan output by the planning agent into multiple sub-tasks and generates a complete sequence of actions. Based on the robot's load, position, and capabilities, the decision-making agent selects one or more sub-tasks to be executed most recently, determines the most suitable robot using a load balancing strategy, and sends specific action instructions to the corresponding robot execution module via a unified communication protocol. During the robot's execution of action instructions, the decision-making agent continuously receives operating status, battery level, execution progress, or error reports from each robot via an interface to monitor the smooth progress of the actions, thus achieving continuous tracking of the execution status. Simultaneously, the decision-making agent identifies abnormal situations based on rules or learning models; once an anomaly is detected, it triggers error correction or interrupts task execution.

[0036] In some embodiments, the reflective agent analyzes the real-time task execution feedback recorded in the task awareness package to determine whether the task has been successfully completed and records the execution status, thereby achieving real-time evaluation of the task execution results. If the task is not completed or fails to execute, the reflective agent diagnoses the cause of failure, provides optimization suggestions to the decision-making agent, and stores the reflection results in the memory module during subsequent information transmission. The reflective agent also optimizes the reasoning and execution process of future tasks based on historical reflection results.

[0037] In some embodiments, the reflection results include a judgment on the success or failure of task execution, an analysis of the causes of anomalies, and suggestions for improvement.

[0038] A second aspect of this disclosure provides a method for embodied task reasoning and robot scheduling, comprising the following steps:

[0039] It acquires natural language commands issued by the user, as well as receives the execution results of the task and the robot's embodied state information, and issues a request to supplement the missing task information when it determines that the task information is missing.

[0040] Based on the natural language instructions, the embodied state information fed back by the robot, the perceived environmental information, and the stored memory, the robot dynamically perceives the environmental and task states. Based on the perception results and the reflection results on the task execution results, the robot performs dynamic planning and decision-making to realize task reasoning and robot scheduling. The embodied state information is fed back by the robot cluster after executing tasks according to the received total work sequence. It is used to dynamically perceive the environmental state and update the stored memory.

[0041] In some embodiments, the method supports three task interaction modes:

[0042] Reactive interaction: Directly executes the complete natural language instructions;

[0043] Proactive interaction: When task information is missing, it sends a user request to the instruction or searches its own stored memory to supplement the missing task information;

[0044] Proactive interaction: When proactive interaction fails to obtain missing task information, requests are sent to other users in the environment to seek help through one-on-one contact or broadcasting in a group, and multiple parties collaborate to complete the supplementation of missing task information.

[0045] A third aspect of this disclosure provides an electronic device, including: at least one processor; and a memory communicatively connected to the at least one processor, wherein the memory stores instructions executable by the at least one processor to implement the embodied task reasoning and robot scheduling method of any embodiment of the second aspect of this disclosure.

[0046] A fourth aspect of this disclosure provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the embodied task reasoning and robot scheduling method of any embodiment of the second aspect of this disclosure.

[0047] A fifth aspect of this disclosure provides a computer program product, including a computer program that, when executed by a processor, implements the embodied task reasoning and robot scheduling method of any embodiment of the second aspect of this disclosure.

[0048] A sixth aspect of this disclosure provides a computer program, including computer program code, which, when run on a computer, causes the computer to execute the embodied task reasoning and robot scheduling method according to any embodiment of the second aspect of this disclosure.

[0049] The features and beneficial effects of the embodiments disclosed herein are as follows:

[0050] 1) Through reactive, proactive, and active interaction mechanisms, service robots can achieve dynamic reasoning and efficient execution of complex tasks;

[0051] 2) By combining the specific status information from the robot's feedback module, the task plan can be optimized in real time to improve task adaptability and execution efficiency;

[0052] 3) The large-scale model, based on a combination of "large-scale training data + high-dimensional parameters + iterative optimization," continuously learns language rules and the relationships between concepts within a layered network structure, thereby gaining deep natural language understanding and reasoning capabilities. Regarding the multi-agent architecture, multiple large-scale model agents with different functions or strategies are assigned roles and work collaboratively under a shared communication mechanism (such as natural language communication or a unified task protocol). Each agent can leverage its designated strengths and obtain more comprehensive task solutions through collaborative reasoning in stages such as "perception-planning-execution-reflection." This division of labor and collaboration helps to "simplify complexity," applying the powerful language understanding and reasoning capabilities of the large-scale model to multi-step, multi-role task scenarios: lower-level functional modules such as perception and execution do not need to master all language and reasoning logic, but are coordinated by higher-level large language model agents. Ultimately, through efficient communication and complementary roles within the entire collaborative architecture, dynamic collaboration and deep reasoning for complex tasks can be achieved, giving the system better adaptability and problem-solving capabilities.

[0053] Based on the above analysis, the multi-agent collaborative framework proposed in this embodiment of the present disclosure has deep natural language understanding and dynamic human-machine collaboration capabilities, and can proactively initiate information requests or collaborate with users to solve problems during the execution process;

[0054] 4) Introduce a reflection mechanism. Introducing a "Reflection Agent" into the multi-agent collaborative framework can be regarded as a kind of meta reasoning mechanism. It is linked with the specific perception, planning and execution processes and is specifically responsible for monitoring, evaluating and dynamically optimizing the overall task execution process of the system, thereby ensuring the accuracy of task results.

[0055] 5) It is suitable for complex and dynamic environments and can provide efficient, intelligent and humanized services in real office scenarios.

[0056] In summary, this disclosure presents a multi-agent collaborative embodied task reasoning and robot scheduling system and method, electronic device, computer-readable storage medium, computer program product, and computer program. Leveraging the powerful natural language understanding and reasoning capabilities of a large language model, it can efficiently handle complex task instructions and dynamic situations. Multi-agent collaboration enhances the system's flexibility and task decomposition capabilities, enabling agents to communicate in human language and collaboratively complete complex tasks. This not only increases the interpretability of the algorithm but also improves overall execution efficiency. By coordinating large language model agents, humans, and robots, this framework achieves seamless information sharing and real-time decision-making, ensuring intelligent and adaptive task execution in environments such as office settings. This integration approach not only improves the system's adaptability and robustness but also significantly enhances the effectiveness of human-machine collaboration, making it suitable for diverse and complex real-world application environments and enabling efficient execution of embodied tasks and robot scheduling. Attached Figure Description

[0057] Figure 1 is a schematic diagram of the overall framework of the embodied task reasoning and robot scheduling system for multi-agent collaboration provided in the first aspect of this disclosure.

[0058] Figure 2 is a schematic diagram of the process of multi-agent collaborative reasoning based on a large model in an embodiment of this disclosure.

[0059] Figure 3 is a flowchart of the task execution of the intelligent agent processing unit in reactive, proactive and prospective modes in a specific embodiment of this disclosure. Detailed Implementation

[0060] To make the objectives, technical solutions, and advantages of this disclosure clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this disclosure.

[0061] Conversely, this disclosure covers any alternatives, modifications, equivalent methods, and schemes made within the spirit and scope of this disclosure as defined by the claims. Furthermore, to provide the public with a better understanding of this disclosure, certain specific details are described in detail below. However, this disclosure will be fully understood by those skilled in the art even without these detailed descriptions.

[0062] This disclosure provides an embodied task reasoning and robot scheduling system and method based on large-scale multi-agent collaboration, an electronic device, a computer-readable storage medium, a computer program product, and a computer program. Through multi-agent collaboration and dynamic reasoning, it achieves efficient execution of complex tasks and real-time scheduling of robot embodied states. The execution process and methods of each module of the system are described in detail below with reference to implementation examples.

[0063] Referring to Figures 1 and 2, a first aspect of this disclosure proposes a multi-agent collaborative embodied task reasoning and robot scheduling system, including an interconnected interaction unit 100, an intelligent processing unit 200, and an embodied robot cluster 300. The embodied robot cluster 300 contains multiple robots 310 performing tasks. The various units in the system work collaboratively, ensuring that the robots can adapt to dynamically changing task requirements through three interaction modes: reactive, proactive, and forward-looking. Wherein:

[0064] The interaction unit 100 is used to acquire natural language commands issued by the user and send them to the intelligent processing unit 200; the interaction module 100 is also used to interact with the intelligent processing unit 200, receive the execution results of the task and the physical status information of the robot 310, and provide measures to send a request to the user to supplement key task information when task information is missing.

[0065] The intelligent processing unit 200 is used to dynamically perceive the environment and task status based on the user instructions sent by the interaction unit 100, the embodied state information and environmental information fed back by the robot, and the memory stored in the intelligent processing unit 100 itself. Based on the perception results and the reflection results on the task execution results, it performs dynamic planning and decision-making to realize task reasoning and robot scheduling.

[0066] The embodied robot cluster 300 executes tasks according to the action sequence issued by the intelligent processing unit 200, and feeds back its own embodied status through the interface, helping the intelligent processing unit 200 to dynamically perceive the environmental status and update the memory of the intelligent processing unit 200.

[0067] In some embodiments, the interaction unit 100 interacts with the user through a common social platform (such as WeChat), receives natural language commands from the user, provides feedback on task execution results, and proactively sends supplementary information requests to the user when task information is missing. The natural language commands issued by the user can be text commands or voice commands, which the interaction unit 100 recognizes and transmits to the intelligent processing unit 200.

[0068] In some embodiments, the intelligent processing unit 200 adopts a multi-agent collaborative architecture based on a large language model, including a memory module 210, a perceptual agent 220, a planning agent 230, a decision-making agent 240, and a reflective agent 250. The multi-agent collaboration enables task reasoning and action sequence generation, and all agents interact with each other through natural language.

[0069] The memory module 210 includes a long-term memory storage 211 for storing semantic maps, user location and item information, and a short-term memory storage 212 for storing real-time state information of task execution. The memory module 210 is shared among the intelligent agents.

[0070] The perceptual agent 220 is used to dynamically perceive the environment and task status based on the natural language instructions given by the user obtained by the interaction unit 100, combined with the robot's embodied state information and historical task execution logs, and generate a task perception package containing task objectives, location information and real-time feedback.

[0071] The planning agent 230 is used to dynamically reason about task decomposition and execution schemes and generate task plans based on the reflection results provided by the task perception package and the reflection agent 250.

[0072] The decision-making agent 240 is used to parse the task plan, generate action sequences including specific online operations and physical actions, and perform robot scheduling to guide the robot 310 to complete the task.

[0073] The reflective agent 250 is used to evaluate the results of task execution (including the reasoning content of each agent and the results of the robot's task execution). If the results of task execution do not meet expectations, it provides feedback for the next round of task reasoning to replan or adjust the execution plan.

[0074] In some embodiments, each robot 310 includes:

[0075] Robot knowledge container 311 is used to store environmental semantic maps, target user locations, and item coordinate information, and supports real-time updates;

[0076] Robot advanced function container 312 is used to provide function tools for robot 310 to acquire and control the embodied state, including navigation, object interaction and real-time state feedback;

[0077] The robot execution module 313 is used to perform corresponding tasks based on the action sequence generated by the decision-making agent 240, such as navigation, object interaction, and sending online messages.

[0078] The embodied state feedback interface 314 is used to feed back the current physical state of the robot 310 (such as current position, battery status, etc.) and task progress to the perceptual agent 220, so as to help the perceptual agent 220 dynamically perceive the embodied state of the robot, provide real-time data support for task reasoning and planning, and thus update the task state information.

[0079] In some embodiments, the intelligent processing unit 200 is the core of this disclosure. This unit generates action sequences and dynamically schedules them based on the collaborative interaction between the perceptual agent 220, the planning agent 230, the decision-making agent 240, and the reflective agent 250. Through multimodal information fusion, multi-agent collaboration, and real-time human-computer interaction, it comprehensively analyzes and infers user natural language commands, robot embodied states, and environmental perception data, thereby providing the system with efficient and flexible task execution capabilities. The components of the intelligent processing unit 200 are described in detail below.

[0080] 1. Memory module

[0081] The memory module 210 is the core of the entire system's data support, consisting of a long-term memory storage unit 211 and a short-term memory storage unit 212. The memory module 210 supports real-time updates. If environmental changes occur during task execution (such as the movement of objects), the sensing agent 220 can perceive the latest environmental information, ensuring that the system has environmental adaptability and memory retention capabilities during task execution, providing information support for task reasoning and decision-making.

[0082] Long-term memory storage 211: Used to store relatively stable environmental semantic maps, user default locations and static location information of items, location information of public equipment, user preferences and task execution logs, etc. Long-term memory storage 211 uses key-value databases (such as Redis) and graph databases (such as Neo4j) for persistent storage of spatial semantic information, and combines spatial indexing algorithms to accelerate retrieval.

[0083] Short-term memory 212: Used to dynamically store real-time state and dynamic context information during the execution of the current task, including user instructions, current task progress, robot's embodied state, dynamic environment information, reflection results of the reflective agent 250, and human feedback records during task execution. Short-term memory 212 employs an in-memory database and memory caching mechanism (such as a key-value memory model) to achieve fast read and write operations. Unlike long-term memory 211, the information stored in short-term memory 212 is cleared or transferred to the history record after the task is completed to maintain available system memory space.

[0084] 2. Perceptual intelligent agent

[0085] The perceptual agent 220 receives natural language commands from the user and the robot's current state information, and parses and integrates these to form a task perception package, which serves as input to the planning agent 230. The perception process of the perceptual agent 220 is divided into three modes: reactive perception, proactive perception, and prospective perception, to respond to different user commands.

[0086] Reactive perception: The perception agent 220 receives natural language commands from the user sent by the interaction unit 100 and the robot's current embodied state fed back by the embodied state feedback interface 314. When the perception agent 220 determines that the task information is complete (the user commands are clear and the environmental information is complete), it directly extracts the task objectives and operation requirements from the task information and generates a task perception package by combining the robot's current embodied state and environmental information.

[0087] Active perception: When the perception agent 220 determines that the task information is incomplete, the perception agent 220 sends a request for supplementary task information to the planning agent 230 based on the missing information. The planning agent 230 generates a corresponding task plan based on the request and the reflection results provided by the reflection agent 250. The decision agent 240 generates a specific action sequence. The perception agent 220 actively checks the memory module 210 and / or actively queries the user who issues the instruction to obtain the required supplementary information based on the obtained task information and combined with the robot's current embodied state and environmental information to generate a task perception package.

[0088] Proactive Perception: In complex tasks, the perceptual agent 220 combines the physical state information from the memory module 210 and the robot's embodied state feedback interface to dynamically perceive the current embodied state. If proactive perception and planning fail, the perceptual agent 230 sends a multi-party assistance request to the planning agent 230. The planning agent 230 generates a corresponding task plan based on the request and the reflection results provided by the reflection agent 250. The decision agent 240 generates a specific action sequence. The perceptual agent 220 seeks assistance from other people or groups in the environment based on the action sequence to obtain auxiliary information to supplement the task information, providing a broader planning space for subsequent decisions. The perceptual agent 220 generates a task perception package based on the acquired task information and combined with the robot's current embodied state and environmental information.

[0089] In this embodiment, the specific steps for the sensing agent 220 to generate the task sensing package include: step 1) input data acquisition; step 2) pattern determination and information gap detection; step 3) information integration; and step 4) output and callback.

[0090] Step 1) Input data collection:

[0091] The input data collected by the perceptual agent 220 includes user natural language commands, the robot's physical state, and environmental information. The user natural language commands are text commands transmitted from the interaction unit 100. These text commands can be original text commands issued by the user or the recognition results of voice commands obtained through a speech-to-text model. The robot's physical state includes real-time robot location information, battery status, and fault reports transmitted through the robot's physical state feedback interface. Environmental information includes sensor data and historical records. Sensor data includes images captured by the robot's onboard camera and real-time radar data collected by the robot's onboard ranging radar (such as lidar). Historical records include long-term and short-term memory stored in the memory module 210, such as user preferences, target object coordinates, task execution logs, task completion status, and abnormal status reports. The perceptual agent 220 stores the collected real-time data in the short-term memory storage 212 of the memory module 210.

[0092] Step 2) Pattern Determination and Information Gap Detection:

[0093] First, based on the pre-trained large language model GPT4o, the input user command is subjected to intent recognition, key element extraction (including location and item name, etc.) and context understanding to generate preliminary command parsing results;

[0094] Subsequently, the perceptual agent 220 determines whether the task information, composed of the instruction parsing result and environmental information, is complete. The completeness of the instruction parsing result is determined by whether it has a clear target object and operational requirements. For example, the instruction parsing result "Please bring me the documents on desk A in office A" has a clear target object, execution action, and target location, thus constituting a complete instruction parsing result. When the perceptual agent 220 determines that the task information is incomplete (e.g., the target location is unclear, the target object attributes are ambiguous, or key elements conflict), it first triggers an active perception mode, sending a request for supplementary task information to the planning agent 230 to address the information gap. The planning agent 230 generates a corresponding task plan based on this request and the reflection results provided by the reflection agent 250. The decision agent 240 then generates a specific action sequence based on this task plan. The perceptual agent 220 responds to this action sequence by actively searching for relevant information in the memory module 210 or by actively sending information to the instruction parsing agent through the interaction unit 100. The system issues a user-initiated query request (the perceptual agent 220 prioritizes searching the memory module 210; if the missing information cannot be obtained after searching, it then issues a user-initiated query request to the instruction through the interaction unit 100) to obtain the necessary supplementary information. If the task decomposition cannot be completed after multiple proactive perceptions or planning corrections, the perceptual agent 220 triggers a prospective perception mode and sends a multi-party assistance request to the planning agent 230. The planning agent 230 generates a corresponding task plan based on the request and the reflection results provided by the reflection agent 250. The decision agent 240 generates a specific action sequence based on the task plan. The perceptual agent 220 responds to this action sequence to further expand information sources, including calling more online resources, contacts, or groups, obtaining richer contextual support, and generating auxiliary information. Once the perceptual agent 220 determines that the task information is complete, it triggers a reactive perception mode, directly extracting the task objectives and operational requirements from the complete task information to generate a preliminary perception package.

[0095] Step 3) Information Integration:

[0096] The perceptual agent 220 integrates the generated preliminary perception package with the robot's embodied state and environmental information into a task perception package, which is used to provide a directly operable structured description for the planning agent 230.

[0097] Step 4) Output and Callback:

[0098] The perceptual agent 220 transmits the task-based perception package to the planning agent 230 and keeps listening in the subsequent process. Once new information or user instruction updates are detected during the task execution phase of the robot 310, the task perception package is supplemented or corrected in real time.

[0099] 3. Planning Intelligent Agents

[0100] The planning agent 230 utilizes the multi-step reasoning capabilities of a large language model, combined with the long-term and short-term memory of the memory module 210, to analyze the user intent and environmental information in the task perception package and generate a specific task plan. The reasoning process of the planning agent 230 includes the following three modes, each addressing different task complexities and execution states.

[0101] Reactive planning: When the planning agent 230 determines that the task perception package meets the most basic executable conditions, the planning agent 230 quickly generates task execution steps based on the task perception package to form a task plan, ensuring that the task objective is clear and the path is well-defined, and then transmits the task plan to the decision agent 240 to directly guide the robot 310 to execute the task.

[0102] Active planning: When the planning agent 230 receives a request for supplementary task information from the perceptual agent 220, the planning agent 230 generates a task plan by spontaneously searching the memory module 210 for relevant information or by sending a request for supplementary information to the instruction. During task execution, the planning agent 230 can dynamically adjust the task plan by combining the reflection results of the reflecting agent 250 and the task perception package fed back by the perceptual agent 220, ensuring the smooth progress of the task.

[0103] Proactive planning: When proactive information gathering fails, the planning agent 230 can foresee that continuing to generate a task plan will lead to task execution failure. When existing information is insufficient to support the generation of a feasible plan, the planning agent 230 proactively initiates a proactive planning mode. Based on the received task awareness package and the reflection results provided by the reflective agent 250, it generates a task plan to request cooperation or assistance from other human team members in the environment, obtaining necessary information to ensure the smooth progress of the task and improve the success rate of user command execution. Specifically, this includes:

[0104] 1) Individual Contact: Precisely locate potential assistants and send requests to obtain necessary information; where potential assistants refer to personnel not directly involved in the task and personnel not recorded in memory module 210; and

[0105] 2) Group broadcast: Publish assistance requests in work-related groups to coordinate resources from multiple parties to obtain the information needed for the task.

[0106] Through real-time human-machine collaboration and physical state feedback, the planning agent 230 has the ability to adaptively correct task plans and dynamically adjust task execution paths, thereby improving the reliability and efficiency of the system in complex task environments.

[0107] In this embodiment, the specific steps for the planning agent 230 to generate a task plan include: step 1) input acquisition and initialization and step 2) mode determination and task plan generation.

[0108] Step 1) Input Acquisition and Initialization:

[0109] The planning agent 230 obtains the task perception package from the perception agent 220, calls the long-term memory and short-term memory stored in the memory module 210 to check, and confirms the currently available resources (for example, for a robot, if it is currently in an idle state, it is an available resource) and prior knowledge (such as the location information of the people and objects involved in the current task).

[0110] Step 2) Pattern Determination and Task Plan Generation:

[0111] When the planning agent 230 receives a task perception packet sent by the perception agent 220 in reactive mode, it triggers a reactive planning mode. The planning agent 230 uses the multi-step reasoning capability of the pre-trained large language model GPT4o to semantically decompose the instruction parsing results in the task perception packet and generate subtasks. The large language model utilizes prompt engineering (PE) and chain-of-thought mechanisms to explain why a certain subtask strategy is adopted, improving interpretability. The planning agent 230 passes the task plan (including task execution flow, personnel and resource requirements, and paths) output by the large language model to the decision agent 240, which generates the next executable action. The task plan generated by the planning agent 230 should meet the most basic executable conditions, namely, completeness, consistency, executability, and unambiguity. Completeness refers to the fact that the generated task plan includes all the necessary elements to complete the task, including the target object, target location, and operation, and each necessary element should be clear and unambiguous. Specifically, for the target object or target location, the task plan clearly defines the target object or target location to be operated on or moved to (e.g., "files in office A"); for the operation, the task plan specifies the core operations that the robot needs to perform (e.g., "pick up", "move to", "give to someone"). Consistency means that the task objectives and environmental information in the task plan are consistent with the current availability of resources, without major conflicts (e.g., no obstacles or robot malfunctions). Executability means that, given the environment (i.e., environmental information perceived by the sensing agent), robot capabilities, and time requirements, the current task can theoretically be completed. Unambiguity means that there are no multiple meanings or ambiguities in the instruction parsing result and context, which would prevent the robot from determining the specific operation object or process; for example, if the instruction parsing result involves prerequisites or preconditions (e.g., permissions, item location), and these are ensured to be met or do not require further confirmation, then unambiguity is satisfied.

[0112] When the planning agent 230 receives a request for supplementary task information from the perception agent 220, it triggers an active planning mode. The planning agent 230 combines the reflection results provided by the reflection agent 250 to generate relevant information in the spontaneous search memory module 210 or to issue a user-initiated query request to the instruction through the interaction unit 100. After supplementing the missing information, the planning agent 230 combines the newly generated task perception package from the perception agent with the reflection results provided by the reflection agent 250 and then uses multi-step reasoning based on a large language model to form a task plan. The task plan is then passed to the decision agent 240.

[0113] When the planning agent 230 receives a multi-party assistance request from the perception agent 220, or when the task perception package still fails to meet the most basic executable conditions after multiple rounds of proactive planning, and it is predicted that continuing to generate a task plan according to the current situation will lead to task execution failure, the planning agent 230 proactively triggers a forward-looking planning mode. Based on the received task perception package and the reflection results provided by the reflection agent 250, the planning agent 230 generates a task plan to request collaboration or assistance from other human team members in the environment (including online groups and dedicated personnel collaboration) to obtain a wider range of information sources. After the task perception package meets the most basic executable conditions, it uses multi-step reasoning based on a large language model to form a task plan and then transmits this task plan to the decision agent 240. Specifically, if any one or more of the following situations recur during the task execution or planning process and cannot be resolved through further collaboration or external assistance, it is determined that continuing to generate a task plan according to the current situation will lead to task execution failure:

[0114] ① Information gaps or conflicts are difficult to fill: The user or memory module actively sends more than 210 requests for information to the instruction, but still cannot fill the critical gaps, or there are contradictory environmental information that cannot be resolved;

[0115] ② Multiple planning attempts fail to be implemented: After trying various alternative task decomposition schemes, it is still impossible to generate an executable action sequence (e.g., the path is completely blocked by obstacles);

[0116] ③ Resource or physical limitations: The robot is out of power and there is no available charging solution; the robotic arm is damaged and there is no way to repair it; or the external equipment required for the task is unavailable.

[0117] ④ Time or priority conflict: The task must be completed within the specified time, but it is found during planning that the time limit cannot be met, or other higher priority tasks are taking up resources, causing the current task to be put on hold and unable to be completed.

[0118] In this embodiment, the planning of interaction and collaboration between intelligent agent 230 and other intelligent agents includes:

[0119] With the perception agent 220: obtain updated task perception packages in real time; if the planning encounters a bottleneck, request the perception agent 220 to query more environmental, sensor or social platform data. The perception agent 220 stores the queried data in the memory module 210 for the planning agent 230 to call, so as to ensure that the planning agent 230 obtains the latest information.

[0120] With decision-making agent 240: The generated task plan is handed over to decision-making agent 240 to determine the specific robot action instructions for the next step, and execution feedback is monitored; and

[0121] With reflective agent 250: When the robot makes a mistake or encounters uncertainty during the execution of a task, reflective agent 250 proposes error correction suggestions or requests to replan, and planning agent 230 will adjust the task plan or replan accordingly.

[0122] 4. Decision-making intelligent agent

[0123] The decision-making agent 240 transforms the task plan generated by the planning agent 230 into a specific sequence of actions that the robot can execute, including both online operations and physical actions. The decision-making agent 240 has three decision-making modes, and selects the appropriate decision-making mode based on the planning mode adopted by the planning agent 230.

[0124] Reactive decision-making: When the planning agent 230 adopts the reactive planning mode, the decision-making agent 240 selects the reactive decision-making mode, directly decomposes the task plan into sub-tasks, generates action sequences, and assigns them to one or more of the most suitable robots 310, guiding the robots 310 to complete the task.

[0125] Proactive Decision Making: When the planning agent 230 adopts the proactive planning mode, the decision agent 240 selects the proactive decision making mode. Based on the task plan generated by the planning agent 230, it generates the search memory module 210 or sends an action sequence to the user through the interaction unit 100 to request supplementary task information in order to obtain the missing task information. The perception agent 220, the planning agent 230 and the decision agent 240 cooperate to perceive, plan and make decisions in sequence, thereby ensuring the smooth execution of the user's instructions.

[0126] Proactive Decision Making: When the planning agent 230 adopts a proactive planning mode, the decision-making agent 240 selects a proactive decision-making mode. Based on the task plan generated by the planning agent 230, the decision-making agent 240 combines the task perception package with proactive planning. It not only generates robot actions or online notifications based on the predetermined plan, but also proactively cooperates with the planning agent 230 by conducting online inquiries through one-on-one or group chats to proactively obtain additional information to optimize subsequent decisions. By dynamically adjusting the task execution order and strategy, the decision-making agent 240 proactively avoids generating erroneous actions, improves the success rate of user command completion, and ensures the efficiency and accuracy of task execution.

[0127] In this embodiment, the specific decision-making process of the decision-making agent 240 includes: step 1) input acquisition and initialization; and step 2) pattern determination and action sequence generation and collaborative allocation.

[0128] Step 1) Input Acquisition and Initialization:

[0129] The decision-making agent 240 obtains the task plan, which includes the task execution process and the personnel and resource requirements, from the planning agent 230.

[0130] Step 2) Pattern determination, action sequence generation, and collaborative allocation:

[0131] The decision-making agent 240 triggers its own decision-making mode based on the attributes of the task plan generated by the planning agent 240.

[0132] Reactive Decision-Making: If the task plan is a complete execution plan for the original instructions, the reactive decision-making mode is triggered to generate an action sequence and assign specific sub-tasks. Specifically, based on the function tools available to the robot 310, the task plan (corresponding to the high-level task steps) output by the planning agent 230 is decomposed into multiple sub-tasks using the reasoning ability of the large language model (specifically, sub-task decomposition can be achieved through thought chains and prompt words plus a few sample decomposition examples). The sub-tasks use action instructions or function interfaces that the robot 310 can directly call, such as "move to coordinates (x,y)", "grab item A", "talk to a user", API calls, etc. Templates are established for basic action instructions (such as "grab" and "move"). By filling parameters (target coordinates, target object, etc.) into the templates, a complete action sequence is generated. Subsequently, for multi-robot scenarios, the decision agent 240 extracts one or more sub-tasks that should be executed in the most recent step from the refined sub-tasks based on indicators such as the load, position, and embodied capabilities of the robot 310. It determines the most suitable robot through a load balancing strategy based on the Hungarian Algorithm and sends specific action instructions to the corresponding robot execution module 313 using a unified ROS Topic communication protocol. During the execution of action instructions by the robot, the decision agent 240 continuously receives the running status, battery level, execution progress, or error reports provided by each robot 310 through the embodied status feedback interface 314 to detect whether the action is proceeding smoothly, thereby achieving continuous tracking of the execution status. At the same time, the decision agent 240 quickly identifies abnormal situations (such as mechanical failures or unexpected path blockages) based on rules or learning models. Once an abnormality is detected, it triggers error correction or interruption processes.

[0133] Proactive decision-making: If the task plan is generated based on proactive planning, that is, when the planning agent 230 discovers that the task information is missing when generating the task, the decision-making agent 250 enters the proactive decision-making mode. In this mode, the robot will not be directly dispatched. Instead, the user will actively inquire about the instructions for the proactive task plan. Specifically, the online operation can be generated by calling the robot (specifically, a smartphone) to send information, or by searching the memory module 210 to obtain the missing information.

[0134] Proactive Decision-Making: If the planning agent 230 enters proactive planning mode, i.e., when proactive information supplementation fails, the planning agent 230 will issue a task plan requiring the initiation of a one-on-one chat or group chat to obtain broader assistance information to facilitate the smooth execution of the instructions. At this time, the decision-making agent 250 enters proactive decision-making mode. In this mode, it will not directly dispatch robots, but will instead, based on the proactive task plan, call the robot's (specifically, a smartphone) message sending method to generate online operations, and initiate corresponding online request assistance messages according to the user or group chat name given in the task plan.

[0135] In this embodiment, the interaction and collaboration between the decision-making agent 240 and other agents includes:

[0136] Interaction with the perceptual agent 220: The latest task perception package is obtained in real time to ensure that the decision is based on the current environmental information for task decomposition and allocation; when the decision-making process encounters a bottleneck, the perceptual agent 220 can be requested to further query environmental, sensor or social platform data, and the query results are stored in the memory module 210 for the decision-making agent 240 to call to optimize the task decomposition and allocation process;

[0137] Collaboration with planning agent 230: Based on the task plan provided by planning agent 230, extract the specific tasks that need to be executed at the current moment; and

[0138] Interaction with the reflective agent 250: After the task is completed, the decision agent 240 submits the decomposition and allocation results to the reflective agent 250, assists the reflective agent 250 in analyzing the execution effect and potential problems, generates reflection results, and integrates optimization suggestions into the decomposition and allocation process of future tasks.

[0139] 5. Reflective intelligent agents

[0140] The reflective agent 250 evaluates the task execution results based on information such as execution logs, task completion status, and abnormal status reports obtained from the task perception package of the perceptual agent 220, and provides reflection results for system optimization to ensure the accuracy and completeness of the task. Specific steps include:

[0141] Task evaluation: The reflective agent 250 analyzes the results of the robot execution module recorded in the task perception package to determine whether the task was successfully completed and records the execution status.

[0142] Problem diagnosis: If the task is not completed or fails to be executed, the reflective agent diagnoses the cause of the failure, provides optimization suggestions to the decision-making agent, and stores the reflection results in the memory unit during subsequent information transmission.

[0143] Process optimization: By reflecting on historical data, optimize the reasoning and execution processes for future tasks to improve the success rate of tasks.

[0144] Step 1) Input Acquisition and Initialization:

[0145] The reflexive agent 250 obtains and integrates the current task execution results, including execution logs, task completion status, and abnormal status reports, from the task perception package of the perceptual agent 220. The reflexive agent 250 also accesses the memory module 210 to call upon long-term and short-term memories to obtain preset task objectives and potentially relevant historical execution records (such as the completion status of similar past tasks). These two parts ultimately form a complete reflexive closed-loop information chain for reflecting on the current step. Through unified timestamp management during the reflection process, the reflexive agent 250 can accurately identify the sequence and causal relationships of each step in task execution, improving diagnostic efficiency.

[0146] Step 2) Task Evaluation:

[0147] The reflective agent 250 compares the current task execution result with the preset task objective. If the current task execution result meets or exceeds the preset task objective, the task is considered successful and execution details (such as time, resource consumption, etc.) are recorded in the task execution log. If part of the current task execution result does not match the preset task objective, such as missing steps, incorrect item coordinates, or unsatisfactory user feedback, the task is considered to have deviated or failed, and the process proceeds to the next step of the problem diagnosis process.

[0148] Step 3) Problem Diagnosis:

[0149] First, the reflective agent 250 retrieves the task perception package generated by the perceptual agent 220, the task plan generated by the planning agent 230, and the action sequence generated by the decision agent 240 as evidence, and comprehensively compares the environmental information before and after the robot executes the action sequence.

[0150] Subsequently, the Reflective Agent 250 utilizes chain-like reasoning from a large language model, such as textually describing the causes of errors and error correction strategies, enabling the Reflective Agent to output judgment processes and improvement plans in natural language, thereby improving interpretability. Alternatively, it can use rule-based reasoning to locate and infer the specific steps in which task deviations or failures occur. The inferred causes of task deviations include any one or more of the following: incomplete perception data, conflicting planning logic, incorrect decision instructions, robot hardware failure, and temporary changes in user requirements.

[0151] Finally, the reflective agent 250 describes the problem diagnosis results in text form, outputs the problem diagnosis results in natural language to improve interpretability, and then proceeds to the next step of the process.

[0152] Step 4) Feedback Generation and Process Optimization:

[0153] If the diagnosed problem can be fixed locally (e.g., by simply re-grasping or replacing the robot), the reflective agent 250 sends a local error correction instruction to the decision agent 240. If the diagnosed problem involves overall task reconstruction, the reflective agent 250 transmits the cause of failure and the need for supplementary information to the planning agent 240. If the diagnosed problem is caused by a lack of key information or requires user decision-making, the reflective agent 250 transmits the information that the interaction unit 100 needs to be called to request user intervention (clarify needs, provide additional resources, etc.) to the planning agent 230. The planning agent 230 generates a corresponding task plan, and the decision agent 240 generates a specific online query action sequence. After the robot responds, perception, planning, and decision-making are performed again.

[0154] In some embodiments, the present disclosure achieves deep collaboration with humans through three task execution modes: reactive, proactive, and active.

[0155] 1) Reactive interaction: When the user's instructions are clear and the information is complete, the system directly executes the task and provides feedback on the results, thereby improving the efficiency of task execution.

[0156] 2) Proactive interaction: When task information is incomplete or execution is hindered, the system initiates inquiries to users or team members through the interaction unit 100 to obtain missing information or request assistance, and dynamically updates the task execution plan.

[0157] 3) Proactive Interaction: In complex tasks, the system combines embodied status feedback information and environmental awareness to proactively discover potential problems or predict future resource needs, and collaborate with users or team members to optimize execution plans to ensure the smooth completion of tasks.

[0158] Referring to Figures 1 and 2, the embodied task reasoning and robot scheduling method based on the above-described system provided in the second aspect of this disclosure includes:

[0159] It acquires natural language commands issued by the user, receives task execution results and robot embodied state information, and issues a request to supplement missing task information when it determines that task information is missing; and

[0160] Based on the acquired natural language instructions, the embodied state information fed back by the robot, the perceived environmental information, and the stored memory, the robot dynamically perceives the environmental and task states. Based on the perception results and the reflection results on the task execution results, the robot performs dynamic planning and decision-making to realize task reasoning and robot scheduling. The embodied state information is fed back by the robot cluster after executing the task according to the received total work sequence. It is used to dynamically perceive the environmental state and update the stored memory.

[0161] In some embodiments, the method of this disclosure specifically includes the following steps:

[0162] a) Receiving user instructions: The interaction unit 100 receives the user's natural language instructions and transmits the instructions to the intelligent processing unit 200;

[0163] b) Generate task perception package: The perceptual agent 220 in the intelligent processing unit 200 parses the user's natural language instructions and combines them with long-term memory, environmental information and the robot's embodied state information to generate a task perception package;

[0164] c) Task reasoning and planning: When receiving instructions from the user for the first time, the planning agent 230 will directly reason from the task perception package, decompose the task, and generate a task plan without considering the reflection results. When receiving instructions from the user for the second time, the planning agent 230 will generate a task plan based on the task perception package and the reflection results provided by the reflection agent 250, and dynamically adjust the task plan according to the environmental information fed back by the perception agent 220 and the robot's embodied state information during the task execution.

[0165] d) Task decision-making and interaction: The decision-making agent 240 generates an executable sequence of actions based on the task plan generated by the planning agent 230, including specific online tasks and physical tasks, and sends them to the corresponding robot execution module. When information is missing, it actively interacts with the user who issues the instruction to supplement the required information.

[0166] e) Task Execution and Status Feedback: The robot execution module receives the action sequence issued by the decision-making agent 240, executes the task through high-level functions, and provides real-time feedback of the current embodied state to the perception agent 220. The perception agent 220 re-perceives the state change information, generates a new task perception package, and updates the memory module 210; and

[0167] f) Reflection and optimization: The reflective agent 250 evaluates the task execution results based on the dynamic environmental information in the task perception package at two different time points. If the results do not meet expectations, it performs problem diagnosis and process optimization to form a reflection result. The planning agent and decision agent re-plan and make decisions based on the reflection result to ensure task completion. After the task is completed, the system updates the memory module information to provide a reference for future task execution.

[0168] A third aspect of this disclosure provides an electronic device, including: at least one processor; and a memory communicatively connected to the at least one processor, wherein the memory stores instructions executable by the at least one processor to implement the embodied task reasoning and robot scheduling method of any embodiment of the second aspect of this disclosure.

[0169] A fourth aspect of this disclosure provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the embodied task reasoning and robot scheduling method of any embodiment of the second aspect of this disclosure.

[0170] A fifth aspect of this disclosure provides a computer program product, including a computer program that, when executed by a processor, implements the embodied task reasoning and robot scheduling method of any embodiment of the second aspect of this disclosure.

[0171] A sixth aspect of this disclosure provides a computer program, including computer program code, which, when run on a computer, causes the computer to execute the embodied task reasoning and robot scheduling method according to any embodiment of the second aspect of this disclosure.

[0172] It should be noted that the explanations and descriptions of the embodied task reasoning and robot scheduling system and method for multi-agent collaboration in the foregoing embodiments also apply to the electronic devices, computer-readable storage media, computer program products and computer programs of the present disclosure embodiments, and will not be repeated here.

[0173] All embodiments disclosed herein can be executed individually or in combination with other embodiments, and are all considered to be within the scope of protection claimed by this disclosure.

[0174] The embodiments of this disclosure are described below with reference to Figure 3 and specific task execution examples.

[0175] Example task: "Zhang San: Take this document to Li Si for signature, and then give it to Wang Wu."

[0176] The deployable embodied robots include: a mobile chassis robot with a smart lock (hereinafter referred to as "Robot No. 1"), a robot dog (hereinafter referred to as "Robot No. 2"), a robotic arm (hereinafter referred to as "Robot No. 3"), and a smartphone (with WeChat and other chat software installed and an account configured).

[0177] For the tasks described above as examples, the task reasoning and robot scheduling processes for each agent in this embodiment of the disclosure are as follows:

[0178] Step 1:

[0179] 1. Perceptual intelligent agent:

[0180] The sensing agent 220 detects a new user command in WeChat, receives the command, and parses the initial information: "The file needs to be obtained from Zhang San and sent to Li Si first, and then to Wang Wu last."

[0181] The sensing agent 220 calls the memory module 210 to query the location information of objects in the environment, Zhang San, Li Si, Wang Wu, and others, as well as the user's intent.

[0182] The perceptual agent 220 uses the queried information, the original user instructions, and the information after perception summary to generate a task perception package, and passes it to the reflective agent 250 (if it is the first execution, it skips the reflective agent 250 and passes it directly to the planning agent 230).

[0183] 2. Planning the intelligent agent:

[0184] The planning agent 230 decomposes the original user instructions in the task-aware package into three steps:

[0185] ① First, obtain the document from Zhang San;

[0186] ②Go to Li Si's location again and give him the signature;

[0187] ③ Finally, go to Wang Wu's position and hand it over to him.

[0188] The planning agent 230 generates a complete task plan based on the above task decomposition results. If important information constituting the above task plan is missing at this time, it starts to generate a backup task plan.

[0189] 3. Decision-making intelligent agent:

[0190] Decision-making agent 240 compares the task plan generated by planning agent 230 with the capabilities of different robots in the embodied robot cluster, and infers and generates specific action sequences:

[0191] ① Physical action: <Robot No. 1: Moves to Zhang San>.

[0192] ② Online operation: <Mobile phone: Send a notification and a QR code for unlocking to Zhang San>.

[0193] The decision-making agent 240 calls the robot's high-level function container 312 according to the generated action sequence, and controls robot No. 1 to navigate to the first target location (i.e., where Zhang San is), and waits for Zhang San to complete the item placement operation.

[0194] Robot No. 1 transmits its embodied feedback information to the perceptual agent 220, enabling the perception of new environmental information and embodied status, such as the placement of items.

[0195] Time step two:

[0196] 1. Perceptual intelligent agent:

[0197] The sensing agent 220 detected that Robot No. 1 had completed the first step of the task and successfully acquired the file.

[0198] The location of Li Si is queried through memory module 210, and the execution status of the previous task is updated in the task awareness package. The updated task awareness package is as follows:

[0199] Current status: File acquired, Robot No. 1 is in Zhang San's position.

[0200] Perception agent 220 passes the updated task perception package to planning agent 230.

[0201] 2. Reflective agents:

[0202] The reflective agent 250 evaluates the task execution result based on the generated plan, the specific task generated by the decision agent 240 at the previous time step, and the task perception package of the state changes before and after the task execution, and confirms that Robot No. 1 has obtained the document; if it fails midway (such as Zhang San not putting in the item), the reflective agent 250 will reflect on the reason for the task execution failure.

[0203] After completing its reflection, the reflective agent 250 will transmit its reflection results to the downstream planning agent 230 and decision-making agent 240 to guide them in making the next planning and decision-making steps, that is, whether to continue to execute the specific task corresponding to the remaining part of the user instruction, or to re-execute the specific task that failed in the previous step.

[0204] 3. Planning the intelligent agent:

[0205] If the execution result of the previous step is correct after reflection by the reflexive agent 250, the planning agent 230 will decompose the unfinished part into steps based on the user instructions and the contents of the task awareness package:

[0206] ① Take the document to Li Si's location and have Li Si sign it;

[0207] ②Go to Wang Wu's location and hand it over to him.

[0208] If the previous step's execution result is found to be erroneous after reflection by the reflecting agent 250, the planning agent 230 will generate a remedial plan based on the reflection result. For example, if the error was due to Zhang San not scanning the code to put the file into Robot One, then the planning agent 230 will update the task plan as follows:

[0209] Send Zhang San a confirmation message to ask if he has put the file in the robot locker;

[0210] 4. Decision-making intelligent agent:

[0211] Based on the task plan issued by the planning agent 230, the decision-making agent 240 generates a specific sequence of actions after matching it with the capabilities of the robot swarm.

[0212] Physical action: <Robot No. 1: Moves to Li Si's position>.

[0213] Online operation: <Smartphone: Send a notification to Li Si, reminding him to prepare to sign>.

[0214] The decision-making agent 240 calls the high-level function container 312 of Robot No. 1 according to the generated action sequence, controls Robot No. 1 to navigate to Li Si's location, and sends a notification to him using a smartphone.

[0215] After arriving at Li Sichu according to the navigation, Robot No. 1 waits for the signing process to be completed, monitors the document status in real time, and transmits the embodied status to the perceptual agent 220 through its own embodied status feedback interface 314.

[0216] Step 3:

[0217] 1. Perceptual intelligent agent:

[0218] The sensing agent 220 received feedback from Robot No. 1, confirming that the document had been signed by Li Sichu.

[0219] The sensing agent 220 calls the memory module 210 to query Wang Wu's location and environmental information.

[0220] Update the execution status of the previous task to the task awareness package. The updated task awareness package contains the following: Current status: The document has been signed, and Robot No. 1 is in Li Si's position.

[0221] Perceptive agent 220 passes the updated task perception package to reflective agent 250.

[0222] 2. Reflective agents:

[0223] Based on the task plan, specific tasks, and task awareness packages of state changes before and after task execution generated in the previous step, the reflective agent 250 evaluates the task execution result: if the document has been signed, the current task is confirmed to be successful and the next stage of the task is executed; if the document has not been signed (e.g., Li Si did not perform the signing operation), the reflective agent 250 will provide feedback on the reason.

[0224] After completing its reflection, the reflecting agent 250 transmits the reflection results to the downstream planning agent 230 and decision-making agent 240, guiding them to continue executing subsequent instructions or generate remedial measures.

[0225] 3. Planning the intelligent agent:

[0226] Based on the task perception package provided by the perception agent 220, the planning agent 230 identifies the unfinished task parts and generates the following task plan: Take the signed document to Wang Wu's location and deliver the document to Wang Wu.

[0227] If the previous task failed, the planning agent 230 generates a remedial plan based on the feedback from the reflective agent 250. For example, if the document is not signed, the planning agent 230 will insert a remedial step (such as having the robot resubmit the document to Li Si).

[0228] 4. Decision-making intelligent agent:

[0229] Based on the task plan issued by the planning agent 230 and the capabilities of the robot swarm, the decision-making agent 240 generates a specific sequence of actions:

[0230] ① Physical action: <Robot No. 1: Moves to Wang Wu's position>.

[0231] ②Online operation: <Smartphone: Send a notification to Wang Wu, reminding him to prepare to receive the file>.

[0232] The decision-making agent 240 calls the high-level function container 312 of Robot No. 1 to control Robot No. 1 to navigate to Wang Wu's location and send a notification to Wang Wu via smartphone.

[0233] After Robot One reaches the target, it checks the delivery status to ensure that the document has been successfully delivered to Wang Wu; Robot One then transmits its embodied feedback information to the perceptual agent 220, reporting the document delivery status.

[0234] 5. Task completed:

[0235] If the file is successfully delivered to Wang Wu, the perceptual agent 220 will update the task perception package with the final task completion status. After passing through the reflection agent 250, the planning agent 230 and the decision agent 240 in sequence, an action sequence is generated to notify the user that the task has been completed. The smartphone responds to the action sequence and notifies the user that the task has been completed.

[0236] If the document is not successfully delivered, the reflective agent 250 will report the reason for the failure and guide the planning agent 230 and the decision-making agent 240 to generate remedial measures to ensure the successful completion of the task.

[0237] The embodiments disclosed herein demonstrate three task processing modes during task execution:

[0238] 1. Reactive task processing: Upon receiving user instructions, it automatically begins to perceive the environment, plan the problem, generate specific decision tasks, reflect on task behavior and results, and finally complete the user instructions, just like a human assistant.

[0239] 2. Proactive task completion: When information is incomplete, proactively interact with the user to complete the information and ensure the task is executed smoothly.

[0240] 3. Proactive Assistance: By using embodied feedback from robots to monitor the execution process in real time, predict the results of instruction execution, and proactively seek human assistance, the correct implementation of tasks can be ensured.

[0241] In summary, the embodied task reasoning and robot scheduling system and method, electronic device, computer-readable storage medium, computer program product, and computer program provided in this disclosure have the following advantages:

[0242] 1. Multi-agent collaboration: Through the collaborative work of perception, planning, decision-making and reflection agents, intelligent reasoning and efficient execution of complex tasks can be achieved.

[0243] 2. Dynamic Reasoning and Interaction: It features reactive, proactive, and forward-looking task interaction modes, effectively solving the problems of incomplete task information and dynamic environmental changes.

[0244] 3. Closed-loop feedback and optimization: The system continuously optimizes system performance and task execution results by dynamically evaluating the execution results through reflective agents.

[0245] Therefore, the embodiments of this disclosure effectively improve the task execution capability of service robots in real-world scenarios, providing an intelligent and dynamic solution for handling complex tasks.

[0246] In the description of this specification, the references to terms such as "one embodiment," "some embodiments," "illustrative embodiment," "example," "specific example," or "some examples," etc., indicate that a specific feature, structure, material, or characteristic described in connection with that embodiment or example is included in at least one embodiment or example of this disclosure. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples.

[0247] Although embodiments of this disclosure have been shown and described, those skilled in the art will understand that various changes, modifications, substitutions and alterations can be made to these embodiments without departing from the principles and spirit of this disclosure, the scope of which is defined by the claims and their equivalents.

Claims

1. A multi-agent collaborative embodied task reasoning and robot scheduling system, characterized in that, include: The interaction unit is used to acquire natural language commands issued by the user and send them to the intelligent processing unit, and to interact with the intelligent processing unit to receive the execution results of the task and the robot's physical state information, and to issue a request to supplement the missing task information when the intelligent processing unit determines that the task information is missing. The intelligent processing unit is used to dynamically perceive the environment and task status based on user instructions sent by the interaction unit, embodied state information and environmental information fed back by the robot, and the memory stored in the intelligent processing unit itself. Based on the perception results and the reflection results on the task execution results, it performs dynamic planning and decision-making to realize task reasoning and robot scheduling. The robot swarm executes tasks according to the action sequence issued by the intelligent processing unit, and feeds back its own status information through the interface, which helps the intelligent processing unit to dynamically perceive the environmental status and update the memory of the intelligent processing unit.

2. The system according to claim 1, characterized in that, The intelligent processing unit adopts a multi-agent collaborative architecture based on a large language model, including a memory module, a perceptual agent, a planning agent, a decision-making agent, and a reflective agent. Each agent interacts with the others through natural language and shares the memory stored in the memory module. The sensing agent is used to dynamically perceive the environment and task status based on the user instructions obtained by the interaction unit, combined with the robot's physical state information and historical task execution logs, and generate a task perception package containing task objectives, location information and real-time task execution feedback. The planning agent is used to dynamically reason about task decomposition and execution schemes and generate task plans based on the task perception package and the reflection results provided by the reflection agent. The decision-making agent is used to parse the task plan, generate action sequences, and perform robot scheduling to guide the robot to complete the task. The reflective agent is used to evaluate the task execution results. If the task execution results do not meet expectations, it provides feedback for the next round of task reasoning to replan or adjust the execution plan.

3. The system according to claim 2, characterized in that, The memory module stores long-term memory and short-term memory; The long-term memory includes an environmental semantic map, the user's default location, static location information of items, location information of public equipment, user preferences, and historical task execution logs; The short-term memory includes real-time status information and dynamic context information during the current task execution process. The real-time status information includes user instructions, current task execution progress, robot's embodied state information, dynamic environment information, reflection results of the reflective agent, and user feedback records during task execution.

4. The system according to claim 2 or 3, characterized in that, The intelligent processing unit possesses task reasoning modes including reactive, proactive, and forward-looking. Reactive approach: When the perceptual agent determines that the task information is complete based on the user instructions sent by the interaction unit and the embodied state feedback from the robot, it extracts the task objective and operation requirements from the task information and generates a task perception package containing the task objective, location information and real-time feedback; the planning agent generates task execution steps based on the task perception package and the reflection results provided by the reflecting agent to form a task plan, which should meet the most basic executable conditions. The decision-making agent generates an action sequence by matching the task plan issued by the planning agent with the specific capabilities of the robot cluster. Active mode: When the perceptual agent determines that the task information is incomplete based on the user instructions sent by the interaction unit and the embodied state feedback from the robot, it sends a request for supplementary task information to the planning agent to fill the information gap. The planning agent forms a corresponding task plan based on the request and the reflection results provided by the reflection agent. The decision agent generates an action sequence based on the task plan and obtains the required supplementary information by searching relevant information in the memory module or by sending a query request to the user through the interaction unit. Proactive approach: When the current task cannot be completed after multiple rounds of proactive reasoning, the perceptual agent sends a multi-party assistance request to the planning agent. The planning agent forms a corresponding task plan based on the request and the reflection results provided by the reflection agent. The decision agent generates an action sequence based on the task plan and initiates inquiry requests to other users in the environment through the interaction unit to expand the scope of information acquisition and ensure the smooth execution of the task.

5. The system according to claim 4, characterized in that, The method of initiating a query request to other users in the environment through the interaction unit includes: Individual contact: Identify potential assisting users, send requests to obtain necessary information, including personnel not directly involved in the task or personnel not recorded in the memory module; Group broadcast: Publish assistance requests in work-related groups to coordinate resources from multiple parties to obtain the information needed for the task.

6. The system according to any one of claims 2 to 5, characterized in that, The robot includes: A robot knowledge container is used to store environmental semantic maps, target user locations, and item coordinate information, and supports real-time updates; A high-level function container for robots, providing functional tools for acquiring and controlling the robot's embodied state, including navigation, object interaction, and real-time status feedback; The robot execution module is used to perform corresponding tasks based on the action sequence generated by the decision-making agent, including navigation, object interaction, and sending online messages; The embodied state feedback interface is used to feed back the robot's current physical state and task progress to the perceptual agent in order to update the task state information.

7. The system according to claim 6, characterized in that, The action sequence generated by the decision-making agent includes online operations and physical actions. The decision-making agent sends action instructions to the robot execution module through an interface, and dynamically adjusts the execution plan during task execution.

8. The system according to claim 6 or 7, characterized in that, The process by which the decision-making agent schedules the robot includes: The decision-making agent, based on the available functions and tools for the robot, breaks down the task plan output by the planning agent into multiple sub-tasks and generates a complete sequence of actions. Based on the robot's load, position, and capabilities, the decision-making agent selects one or more sub-tasks to be executed most recently, determines the most suitable robot using a load balancing strategy, and sends specific action instructions to the corresponding robot execution module via a unified communication protocol. During the robot's execution of action instructions, the decision-making agent continuously receives operating status, battery level, execution progress, or error reports from each robot via an interface to monitor the smooth progress of the actions, thus achieving continuous tracking of the execution status. Simultaneously, the decision-making agent identifies abnormal situations based on rules or learning models; once an anomaly is detected, it triggers error correction or interrupts task execution.

9. The system according to any one of claims 2 to 8, characterized in that, The reflective agent analyzes the real-time task execution feedback recorded in the task perception package to determine whether the task has been successfully completed and records the execution status, thereby achieving real-time evaluation of the task execution results. If the task is not completed or fails to execute, the reflective agent diagnoses the cause of the failure, provides optimization suggestions to the decision-making agent, and stores the reflection results in the memory module during subsequent information transmission. The reflective agent also optimizes the reasoning and execution process of future tasks based on historical reflection results.

10. The system according to any one of claims 2 to 9, characterized in that, The reflection results include assessments of the success or failure of task execution, analysis of the causes of anomalies, and suggestions for improvement.

11. A method for embodied task reasoning and robot scheduling, characterized in that, Includes the following steps: It acquires natural language commands issued by the user, as well as receives the execution results of the task and the robot's embodied state information, and issues a request to supplement the missing task information when it determines that the task information is missing. Based on the natural language instructions, the embodied state information fed back by the robot, the perceived environmental information, and the stored memory, the robot dynamically perceives the environmental and task states. Based on the perception results and the reflection results on the task execution results, the robot performs dynamic planning and decision-making to realize task reasoning and robot scheduling. The embodied state information is fed back by the robot cluster after executing tasks according to the received total work sequence. It is used to dynamically perceive the environmental state and update the stored memory.

12. The method according to claim 11, characterized in that, The method supports three task interaction modes: Reactive interaction: Directly executes the complete natural language instructions; Proactive interaction: When task information is missing, it sends a user request to the instruction or searches its own stored memory to supplement the missing task information; Proactive interaction: When proactive interaction fails to obtain missing task information, requests are sent to other users in the environment to seek help through one-on-one contact or broadcasting in a group, and multiple parties collaborate to complete the supplementation of missing task information.

13. An electronic device, comprising: At least one processor; and The memory that is communicatively connected to the at least one processor The memory stores instructions executable by the at least one processor to implement the embodied task reasoning and robot scheduling method as described in claim 11 or 12.

14. A computer-readable storage medium having a computer program stored thereon, the computer program, when executed by a processor, implementing the embodied task reasoning and robot scheduling method as described in claim 11 or 12.

15. A computer program product comprising a computer program that, when executed by a processor, implements the embodied task reasoning and robot scheduling method as described in claim 11 or 12.

16. A computer program comprising computer program code, which, when run on a computer, causes the computer to perform the embodied task reasoning and robot scheduling method as described in claim 11 or 12.

Images