Multi-agent collaborative embodied task reasoning and robot scheduling system and method

Abstract

A multi-agent collaborative embodied task reasoning and robot scheduling system, comprising a memory module, a sensing agent, a planning agent, a decision-making agent, and a reflection agent, wherein the sensing agent acquires a natural language instruction of a user and environment information to form a task sensing package; the planning agent performs reasoning on the basis of the task sensing package and a long short-term memory to generate a task plan; the decision-making agent parses the task plan, generates an execution action, and guides a robot to synchronously execute a task in both physical and network environments; the system further has a human-machine cooperation mechanism, actively interacts with human team members according to task requirements, and sends an assistance request or acquires supplementary information; and the reflection agent dynamically evaluates a task execution result, provides feedback, and optimizes an execution process. By means of a large model-based multi-agent collaborative framework, the robot is endowed with dynamic reasoning and active interaction capabilities, and the requirements for efficient task reasoning and collaborative execution in a dynamic environment are met.

Description

Embodied Task Reasoning and Robot Scheduling System and Method for Multi-Agent Collaboration

[0001] Cross-reference of related applications

[0002] This application is based on and claims priority to Chinese Patent Application No. 202510065041.7, filed on January 15, 2025, the entire contents of which are incorporated herein by reference. Technical Field

[0003] This disclosure belongs to the field of large language models and intelligent robots, specifically involving a multi-agent collaborative embodied task reasoning and robot scheduling system and method, which is particularly suitable for application scenarios in complex office environments where robots and humans collaborate to perform tasks. Background Technology

[0004] Existing service robots typically only execute preset instructions when handling real-world tasks, lacking a deep understanding of complex natural language commands and the ability to make dynamic reasoning, making it difficult to cope with tasks in real-world environments that are full of uncertainty and change. Furthermore, traditional service robots often fail to proactively seek human assistance when faced with task failure or missing information, resulting in low task execution efficiency or even complete failure.

[0005] While large language models have made significant progress in natural language processing in recent years, combining them with embodied robots and applying them to real-world office scenarios still presents numerous challenges. For example, existing service robots, when performing real-world tasks, typically only follow preset instructions, lacking a deep understanding of complex natural language and dynamic reasoning capabilities, making them ill-equipped to handle uncertainty and variability in the environment. Furthermore, these robots cannot proactively seek human assistance when tasks fail or information is missing, leading to low efficiency or even task failure. Despite the significant advancements in large language models in natural language processing, combining them with embodied robots for practical office scenarios still faces multiple challenges, including the perception and planning of embodied tasks, efficient request mechanisms in human-machine collaboration, and issues related to system integration and resource optimization, as well as robot deployment. Overcoming these technical bottlenecks is crucial for realizing intelligent, adaptable, and safe service robots. Therefore, there is an urgent need for a method that combines embodied task reasoning and execution to enhance the task reasoning, decision-making, and scheduling capabilities of service robots in complex environments, meeting practical application needs. Summary of the Invention

[0006] The first aspect of this disclosure provides an embodied task reasoning and robot scheduling system for multi-agent collaboration, including an interconnected interaction unit, an intelligent processing unit, and an embodied robot cluster, wherein the embodied robot cluster contains multiple robots performing tasks.

[0007] The interaction unit is used to acquire natural language commands issued by the user and send them to the intelligent processing unit; the interaction unit is also used to interact with the intelligent processing unit, receive the execution results of the task and robot status information, and provide measures to send a request to the user to supplement key task information when task information is missing.

[0008] The intelligent processing unit adopts a multi-agent collaborative architecture based on a large language model, including a memory module, a perceptual agent, a planning agent, a decision-making agent, and a reflective agent. It performs task reasoning and action sequence generation through multi-agent collaboration, with all agents interacting via natural language. The memory module stores long-term and short-term memories, which are shared among the agents. The perceptual agent dynamically perceives the environment and task status based on user instructions obtained from the interaction unit, combined with the robot's embodied state information and historical task execution logs, generating a task perception package containing task objectives, location information, and real-time task execution feedback. The planning agent dynamically reasones about task decomposition and execution plans based on the task perception package and the reflection results provided by the reflective agent, generating a task plan. The decision-making agent parses the task plan, generates action sequences, and schedules the robot to guide it in completing the task. The reflective agent evaluates the task execution results; if the results do not meet expectations, it provides feedback for the next round of task reasoning to replan or adjust the execution plan.

[0009] The robot cluster executes tasks according to the action sequence issued by the intelligent processing unit, and feeds back its own embodied state information through the interface to help the perceptual agent dynamically perceive the environmental state in order to update the memory module.

[0010] In some embodiments, the memory module includes a long-term memory and a short-term memory;

[0011] The long-term memory stored in the long-term memory storage device includes environmental semantic map, user default location, static location information of items, location information of public equipment, user preferences, and historical task execution logs;

[0012] The short-term memory dynamically stored in the short-term memory includes real-time status information and dynamic context information during the current task execution process. The real-time status information includes user instructions, current task execution progress, robot's embodied state information, reflection results of the reflective agent, and user feedback records during task execution.

[0013] In some embodiments, the intelligent processing unit possesses task reasoning modes including reactive, proactive, and prospective.

[0014] Reactive approach: When the perceptual agent determines that the task information is complete based on the user instructions sent by the interaction unit and the embodied state feedback from the robot, it extracts the task objective and operation requirements from the task information and generates a task perception package containing the task objective, location information, and real-time feedback; the planning agent generates task execution steps based on the task perception package and the reflection results provided by the reflecting agent to form a task plan, which should meet the most basic executable conditions; the decision-making agent generates an action sequence after matching the task plan issued by the planning agent with the embodied capabilities of the robot cluster.

[0015] Active mode: When the perceptual agent determines that the task information is incomplete based on the user instructions sent by the interaction unit and the embodied state feedback from the robot, it sends a request for supplementary task information to the planning agent to fill the information gap. The planning agent forms a corresponding task plan based on the request and the reflection results provided by the reflection agent. The decision agent generates an action sequence based on the task plan and obtains the required supplementary information by searching relevant information in the memory module or by sending a query request to the user through the interaction unit.

[0016] Proactive approach: When the current task cannot be completed after multiple rounds of proactive reasoning, the perceptual agent sends a multi-party assistance request to the planning agent. The planning agent forms a corresponding task plan based on the request and the reflection results provided by the reflection agent. The decision agent generates an action sequence based on the task plan and initiates inquiry requests to other users in the environment through the interaction unit to expand the scope of information acquisition and ensure the smooth execution of the task.

[0017] In some embodiments, the method of initiating a query request to other users in the environment through the interaction unit includes:

[0018] Individual contact: Identify potential assisting users, send requests to obtain necessary information, including personnel not directly involved in the task or personnel not recorded in the memory module;

[0019] Group broadcast: Publish assistance requests in work-related groups to coordinate resources from multiple parties to obtain the information needed for the task.

[0020] In some embodiments, the robot includes:

[0021] A robot knowledge container is used to store environmental semantic maps, target user locations, and item coordinate information, and supports real-time updates;

[0022] A high-level function container for robots, providing functional tools for acquiring and controlling the robot's embodied state, including navigation, object interaction, and real-time status feedback;

[0023] The robot execution module is used to perform corresponding tasks based on the action sequence generated by the decision-making agent, including navigation, object interaction, and sending online messages;

[0024] The embodied state feedback interface is used to feed back the robot's current physical state and task progress to the perceptual agent in order to update the task state information.

[0025] In some embodiments, the action sequence generated by the decision-making agent includes online operations and physical actions. The decision-making agent sends action instructions to the robot execution module through an interface, and dynamically adjusts the execution plan during task execution.

[0026] In some embodiments, the process of the decision-making agent scheduling the robot includes:

[0027] The decision-making agent, based on the available functions and tools for the robot, breaks down the task plan output by the planning agent into multiple sub-tasks and generates a complete action sequence. Based on the robot's load, position, and capabilities, the decision-making agent selects one or more sub-tasks from the sub-tasks that should be performed at the most recent step. Using a load balancing strategy, it determines the most suitable robot and sends specific action instructions to the corresponding robot execution module via a unified communication protocol. During the robot's execution of action instructions, the decision-making agent continuously receives operating status, battery level, execution progress, or error reports from each robot via an interface to detect whether the actions are proceeding smoothly, thereby achieving continuous tracking of the execution status. Simultaneously, the decision-making agent identifies abnormal situations based on rules or learning models. Once an anomaly is detected, it triggers error correction or interrupts task execution.

[0028] In some embodiments, the reflective agent analyzes the real-time task execution feedback recorded in the task awareness package to determine whether the task has been successfully completed and records the execution status, thereby achieving real-time evaluation of the task execution results. If the task is not completed or fails to execute, the reflective agent diagnoses the cause of failure, provides optimization suggestions to the decision-making agent, and stores the reflection results in the memory module during subsequent information transmission. The reflective agent also optimizes the reasoning and execution process of future tasks based on historical reflection results.

[0029] In some embodiments, the reflection results include a judgment on the success or failure of task execution, an analysis of the causes of anomalies, and suggestions for improvement.

[0030] A second aspect of this disclosure provides a method for embodied task reasoning and robot scheduling based on a system according to any embodiment of the first aspect of this disclosure, comprising the following steps:

[0031] a) Receiving user instructions: The interaction unit receives natural language instructions from the user and transmits the instructions to the intelligent processing unit;

[0032] b) Generate a task perception package: The sensing agent dynamically perceives the environment and task status based on the received user instructions and in combination with the robot's physical state and historical task execution logs, and generates a task perception package that includes the task objective, location information and real-time task execution feedback.

[0033] c) Dynamic planning tasks: For the first time receiving instructions from the user, without considering the reflection results, the planning agent directly reasons from the task perception package, decomposes the task, and generates a task plan; for subsequent times receiving instructions from the user, the planning agent generates a task plan based on the task perception package and the reflection results provided by the reflection agent, and dynamically adjusts the task plan according to the environmental information fed back by the perception agent and the robot's embodied state information during task execution.

[0034] d) Generate execution actions: The decision-making agent generates specific online and physical tasks according to the task plan, sends them to the corresponding robots, and actively interacts with the user who issued the instruction to supplement the required information when task information is missing;

[0035] e) Task execution and status feedback: The robot receives action instructions, executes tasks through advanced functions, and provides real-time feedback of its current embodied state to the perceptual agent. The perceptual agent then re-perceives the state change information, generates a new task perception package, and updates the memory module.

[0036] f) Reflection and optimization: The reflecting agent evaluates the task execution results based on the dynamic environmental information in the task perception package at the previous and next time points. If the results do not meet expectations, the agent performs problem diagnosis and process optimization to form a reflection result. The planning agent and the decision agent re-plan and make decisions based on the reflection result to ensure the task is completed.

[0037] In some embodiments, the method supports three task interaction modes:

[0038] Reactive interaction: Directly executes complete task instructions;

[0039] Proactive interaction: Sending instructions to the user or system to request supplementary information about the missing task;

[0040] Proactive interaction: When proactive interaction fails to obtain information, requests are sent to other users in the environment through the interaction unit to seek help through one-on-one contact or broadcasting in a group, and multiple parties collaborate to complete the task by supplementing the information needed.

[0041] In some embodiments, during task execution, the reflective agent and the planning agent work together to achieve continuous monitoring and dynamic adjustment of the task status.

[0042] In some embodiments, during the robot's task execution, the robot's physical state information is obtained in real time through an interface, providing the sensing agent with position, path, and task execution status data to help the system achieve dynamic perception and task reasoning.

[0043] A third aspect of this disclosure provides an electronic device, including:

[0044] At least one processor; and a memory communicatively connected to said at least one processor;

[0045] The memory stores instructions executable by the at least one processor, the instructions being configured to perform the embodied task reasoning and robot scheduling method according to the second aspect of this disclosure.

[0046] A fourth aspect of this disclosure provides a computer-readable storage medium storing computer instructions for causing the computer to execute the embodied task reasoning and robot scheduling method of the second aspect of this disclosure.

[0047] A fifth aspect of this disclosure provides a computer program product, including a computer program or instructions, which, when executed by a processor, implement the embodied task reasoning and robot scheduling method of the second aspect of this disclosure. Attached Figure Description

[0048] Figure 1 is a schematic diagram of the overall framework of the embodied task reasoning and robot scheduling system for multi-agent collaboration provided in the first aspect of this disclosure.

[0049] Figure 2 is a schematic diagram of the process of multi-agent collaborative reasoning based on a large model in an embodiment of this disclosure.

[0050] Figure 3 is a flowchart of the task execution of the intelligent agent processing unit in reactive, proactive and look-ahead modes in the embodiments of this disclosure.

[0051] Figure 4 is a block diagram of an electronic device according to an embodiment of the present disclosure. Detailed Implementation

[0052] To make the objectives, technical solutions, and advantages of this disclosure clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this disclosure.

[0053] Conversely, this disclosure covers any alternatives, modifications, equivalent methods, and schemes made within the spirit and scope of this disclosure as defined by the claims. Furthermore, to provide the public with a better understanding of this disclosure, certain specific details are described in detail below. However, this disclosure will be fully understood by those skilled in the art even without these detailed descriptions.

[0054] This disclosure aims to at least partially address one of the technical problems existing in the prior art.

[0055] To this end, this disclosure provides a multi-agent collaborative embodied task reasoning and robot scheduling system and method, which introduces a multi-agent collaborative architecture based on a large language model. By simulating the four thinking steps of human problem-solving—perception, planning, execution, and reflection—it endows service robots with natural language understanding, dynamic task reasoning, autonomous decision-making, and human-machine collaboration capabilities in complex environments, enabling them to efficiently handle complex task instructions and dynamic situations.

[0056] This disclosure provides an embodied task reasoning and robot scheduling system and method based on large-scale model multi-agent collaboration. Through multi-agent collaboration and dynamic reasoning, it achieves efficient execution of complex tasks and real-time scheduling of robot embodied states. The following describes in detail the execution process and methods of each module of the system with reference to implementation examples.

[0057] Referring to Figures 1 and 2, a first aspect of this disclosure proposes a multi-agent collaborative embodied task reasoning and robot scheduling system, including an interconnected interaction unit 100, an intelligent processing unit 200, and an embodied robot cluster 300. The embodied robot cluster 300 contains multiple robots 310 performing tasks. The various units in the system work collaboratively, ensuring that the robots can adapt to dynamically changing task requirements through three interaction modes: reactive, proactive, and forward-looking. Wherein:

[0058] The interaction unit 100 is used to acquire natural language commands issued by the user and send them to the intelligent processing unit 200; the interaction module 100 is also used to interact with the intelligent processing unit 200, receive the execution results of the task and the status information of the robot 310, and provide measures to send a request to the user to supplement key task information when task information is missing.

[0059] The intelligent processing unit 200 adopts a multi-agent collaborative architecture based on a large language model, including a memory module 210, a perceptual agent 220, a planning agent 230, a decision-making agent 240, and a reflective agent 250. It performs task reasoning and action sequence generation through multi-agent collaboration, with all agents interacting via natural language. The memory module 210 includes a long-term memory 211 for storing semantic maps, user location, and item information, and a short-term memory 212 for storing real-time state information of task execution. The memory module 210 is shared among all agents. The perceptual agent 220 is used to combine the robot's embodied state information with historical tasks, based on the natural language commands issued by the user obtained by the interaction unit 100. The execution log dynamically perceives the environment and task status, generating a task perception package containing task objectives, location information, and real-time feedback. The planning agent 230 dynamically infers task decomposition and execution plans based on the task perception package and the reflection results provided by the reflecting agent 250, generating a task plan. The decision agent 240 parses the task plan, generates action sequences including specific online operations and physical actions, and performs robot scheduling to guide the robot 310 to complete the task. The reflecting agent 250 evaluates the task execution results (including the inference content of each agent and the results of the robot's task execution). If the task execution results do not meet expectations, it provides feedback for the next round of task inference to replan or adjust the execution plan.

[0060] The embodied robot cluster 300 executes tasks according to the action sequence issued by the intelligent processing unit 200, and feeds back its own embodied state through the interface to help the sensing agent 220 dynamically perceive the environmental state in order to update the memory module 210.

[0061] In some embodiments, the interaction unit 100 interacts with the user through a common social platform (such as WeChat), receives natural language commands from the user, provides feedback on task execution results, and proactively sends supplementary information requests to the user when task information is missing. The natural language commands issued by the user can be text commands or voice commands, which the interaction unit 100 recognizes and transmits to the intelligent processing unit 200.

[0062] In some embodiments, each robot 310 includes:

[0063] Robot knowledge container 311 is used to store environmental semantic maps, target user locations, and item coordinate information, and supports real-time updates;

[0064] Robot advanced function container 312 is used to provide function tools for robot 310 to acquire and control the embodied state, including navigation, object interaction and real-time state feedback;

[0065] The robot execution module 313 is used to perform corresponding tasks based on the action sequence generated by the decision-making agent 240, such as navigation, object interaction, and sending online messages.

[0066] The embodied state feedback interface 314 is used to feed back the current physical state of the robot 310 (such as current position, battery status, etc.) and task progress to the perceptual agent 220, so as to help the perceptual agent 220 dynamically perceive the embodied state of the robot, provide real-time data support for task reasoning and planning, and thus update the task state information.

[0067] In some embodiments, the intelligent processing unit 200 is the core of this disclosure. This unit generates action sequences and dynamically schedules them based on the collaborative interaction between the perceptual agent 220, the planning agent 230, the decision-making agent 240, and the reflective agent 250. Through multimodal information fusion, multi-agent collaboration, and real-time human-computer interaction, it comprehensively analyzes and infers user natural language commands, robot embodied states, and environmental perception data, thereby providing the system with efficient and flexible task execution capabilities. The components of the intelligent processing unit 200 are described in detail below.

[0068] 1. Memory module

[0069] The memory module 210 is the core of the entire system's data support, consisting of a long-term memory storage unit 211 and a short-term memory storage unit 212. The memory module 210 supports real-time updates. If environmental changes occur during task execution (such as the movement of objects), the sensing agent 220 can perceive the latest environmental information, ensuring that the system has environmental adaptability and memory retention capabilities during task execution, providing information support for task reasoning and decision-making.

[0070] Long-term memory storage 211: Used to store relatively stable environmental semantic maps, user default locations and static location information of items, location information of public equipment, user preferences and task execution logs, etc. Long-term memory storage 211 uses key-value databases (such as Redis) and graph databases (such as Neo4j) for persistent storage of spatial semantic information, and combines spatial indexing algorithms to accelerate retrieval.

[0071] Short-term memory 212: Used to dynamically store real-time state and dynamic context information during the execution of the current task, including user instructions, current task progress, robot's embodied state, reflection results of the reflective agent 250, and human feedback records during task execution. Short-term memory 212 employs an in-memory database and memory caching mechanism (such as a key-value memory model) to achieve fast read and write operations. Unlike long-term memory 211, the information stored in short-term memory 212 is cleared or transferred to the history record after the task is completed to maintain available system memory space.

[0072] 2. Perceptual intelligent agent

[0073] The perceptual agent 220 receives natural language commands from the user and the robot's current state information, and parses and integrates these to form a task perception package, which serves as input to the planning agent 230. The perceptual agent 220's perception process operates in three modes: reactive perception, proactive perception, and look-ahead perception, to respond to different user commands.

[0074] Reactive perception: The perception agent 220 receives natural language commands from the user sent by the interaction unit 100 and the robot's current embodied state fed back by the embodied state feedback interface 314. When the perception agent 220 determines that the task information is complete (the user commands are clear and the environmental information is complete), it directly extracts the task objectives and operation requirements from the task information and generates a task perception package by combining the robot's current embodied state and environmental information.

[0075] Active perception: When the perception agent 220 determines that the task information is incomplete, the perception agent 220 sends a request for supplementary task information to the planning agent 230 based on the missing information. The planning agent 230 generates a corresponding task plan based on the request and the reflection results provided by the reflection agent 250. The decision agent 240 generates a specific action sequence. The perception agent 220 actively checks the memory module 210 and / or actively queries the user who issues the instruction to obtain the required supplementary information based on the obtained task information and combined with the robot's current embodied state and environmental information to generate a task perception package.

[0076] Proactive Perception: In complex tasks, the perceptual agent 220 combines the physical state information from the memory module 210 and the robot's embodied state feedback interface to dynamically perceive the current embodied state. If proactive perception and planning fail, the perceptual agent 230 sends a multi-party assistance request to the planning agent 230. The planning agent 230 generates a corresponding task plan based on the request and the reflection results provided by the reflection agent 250. The decision agent 240 generates a specific action sequence. The perceptual agent 220 seeks assistance from other people or groups in the environment based on the action sequence to obtain auxiliary information to supplement the task information, providing a broader planning space for subsequent decisions. The perceptual agent 220 generates a task perception package based on the acquired task information and combined with the robot's current embodied state and environmental information.

[0077] In this embodiment, the specific steps for the sensing agent 220 to generate the task sensing package include:

[0078] Step 1) Input data collection:

[0079] The input data collected by the perceptual agent 220 includes user natural language commands, the robot's physical state, and environmental information. The user natural language commands are text commands transmitted from the interaction unit 100. These text commands can be original text commands issued by the user or the recognition results of voice commands obtained through a speech-to-text model. The robot's physical state includes real-time robot location information, battery status, and fault reports transmitted through the robot's physical state feedback interface. Environmental information includes sensor data and historical records. Sensor data includes images captured by the robot's onboard camera and real-time radar data collected by the robot's onboard ranging radar (such as lidar). Historical records include long-term and short-term memory stored in the memory module 210, such as user preferences, target object coordinates, task execution logs, task completion status, and abnormal status reports. The perceptual agent 220 stores the collected real-time data in the short-term memory storage 212 of the memory module 210.

[0080] Step 2) Pattern Determination and Information Gap Detection:

[0081] First, based on the pre-trained large language model GPT4o, the input user command is subjected to intent recognition, key element extraction (including location and item name, etc.) and context understanding to generate preliminary command parsing results;

[0082] Subsequently, the perceptual agent 220 determines whether the task information composed of the instruction parsing result and environmental information is complete. The completeness determination of the instruction parsing result is to determine whether the instruction parsing result has a clear target object and operation requirements. For example, the instruction parsing result "Please bring me the documents on the desk in office A" has a clear target object, execution action and target location, and is a complete instruction parsing result. When the perceptual agent 220 determines that the task information is incomplete (such as the target location is unknown, the target object attributes are unclear, or there are conflicts in key elements), it first triggers the active perception mode and sends a request for supplementary task information to the planning agent 230 to address the information gap. The planning agent 230 generates a corresponding task plan based on the request and the reflection results provided by the reflection agent 250. The decision agent 240 then generates a specific action sequence based on the task plan. In response to this action sequence, the perceptual agent 220 actively searches for relevant information in the memory module 210 or actively sends a user-initiated query request to the instruction through the interaction unit 100 (the perceptual agent 220 prioritizes searching the memory module 210; if the missing information cannot be obtained after the search, it then sends a user-initiated query request to the instruction through the interaction unit 100). The perceptual agent 220 requests additional information to the planning agent 230. If the task decomposition cannot be completed after multiple proactive perception or planning corrections, the perceptual agent 220 triggers a prospective perception mode and sends a multi-party assistance request to the planning agent 230. The planning agent 230 generates a corresponding task plan based on the request and the reflection results provided by the reflection agent 250. The decision agent 240 generates a specific action sequence based on the task plan. The perceptual agent 220 responds to the action sequence to further expand information sources, including calling more online resources, contacts, or groups, obtaining richer contextual support, and generating auxiliary information. Once the perceptual agent 220 determines that the task information is complete, it triggers a reactive perception mode, directly extracting the task objectives and operational requirements from the complete task information to generate a preliminary perception package.

[0083] Step 3) Information Integration:

[0084] The perceptual agent 220 integrates the generated preliminary perception package with the robot's embodied state and environmental information into a task perception package, which is used to provide a directly operable structured description for the planning agent 230.

[0085] Step 4) Output and Callback:

[0086] The perceptual agent 220 transmits the task-based perception package to the planning agent 230 and keeps listening in the subsequent process. Once new information or user instruction updates are detected during the task execution phase of the robot 310, the task perception package is supplemented or corrected in real time.

[0087] 3. Planning Intelligent Agents

[0088] The planning agent 230 utilizes the multi-step reasoning capabilities of a large language model, combined with the long-term and short-term memory of the memory module 210, to analyze the user intent and environmental information in the task perception package and generate a specific task plan. The reasoning process of the planning agent 230 includes the following three modes, each addressing different task complexities and execution states.

[0089] Reactive planning: When the planning agent 230 determines that the task perception package meets the most basic executable conditions, the planning agent 230 quickly generates task execution steps based on the task perception package to form a task plan, ensuring that the task objective is clear and the path is well-defined, and then transmits the task plan to the decision agent 240 to directly guide the robot 310 to execute the task.

[0090] Active planning: When the planning agent 230 receives a request for supplementary task information from the perceptual agent 220, the planning agent 230 generates a task plan by spontaneously searching the memory module 210 for relevant information or by sending a request for supplementary information to the instruction. During task execution, the planning agent 230 can dynamically adjust the task plan by combining the reflection results of the reflecting agent 250 and the task perception package fed back by the perceptual agent 220, ensuring the smooth progress of the task.

[0091] Proactive planning: When proactive information gathering fails, the planning agent 230 can foresee that continuing to generate a task plan will lead to task execution failure. When existing information is insufficient to support the generation of a feasible plan, the planning agent 230 proactively initiates a proactive planning mode. Based on the received task awareness package and the reflection results provided by the reflective agent 250, it generates a task plan to request cooperation or assistance from other human team members in the environment, obtaining necessary information to ensure the smooth progress of the task and improve the success rate of user command execution. Specifically, this includes:

[0092] 1) Individual contact: Accurately locate potential assistants and send requests to obtain necessary information; where potential assistants refer to personnel who are not directly involved in the task and personnel who are not recorded in memory module 210.

[0093] 2) Group broadcast: Publish assistance requests in work-related groups to coordinate resources from multiple parties to obtain the information needed for the task.

[0094] Through real-time human-machine collaboration and physical state feedback, the planning agent 230 has the ability to adaptively correct task plans and dynamically adjust task execution paths, thereby improving the reliability and efficiency of the system in complex task environments.

[0095] In this embodiment, the specific steps for planning the intelligent agent 230 to generate a task plan include:

[0096] Step 1) Input Acquisition and Initialization:

[0097] The planning agent 230 obtains the task perception package from the perception agent 220, calls the long-term memory and short-term memory stored in the memory module 210 to check, and confirms the currently available resources (for example, for a robot, if it is currently in an idle state, it is an available resource) and prior knowledge (such as the location information of the people and objects involved in the current task).

[0098] Step 2) Pattern Determination and Task Plan Generation:

[0099] When the planning agent 230 receives the task perception packet sent by the perception agent 220 in reactive mode, it triggers the reactive planning mode. The planning agent 230 uses the multi-step reasoning capability of the pre-trained large language model GPT4o to semantically decompose the instruction parsing results in the task perception packet and generate sub-tasks. The large language model utilizes prompt engineering (PE) and chain-of-thought mechanisms to explain why a certain sub-task strategy is adopted, improving interpretability. The planning agent 230 passes the task plan (including task execution flow, personnel and resource requirements, and paths) output by the large language model to the decision agent 240, which generates the next executable action. The task plan generated by the planning agent 230 should meet the most basic executable conditions, namely, completeness, consistency, executability, and unambiguity. Completeness means that the generated task plan includes the necessary elements to complete the task, including the target object, target location, and operation, and each necessary element should be clear and unambiguous. In some embodiments, the task plan clearly defines the target object or location to be operated on or moved to (e.g., "files in office A"); for operations, the task plan specifies the core operations the robot needs to perform (e.g., "pick up", "move to", "hand over to someone"). Consistency means that the task objectives and environmental information in the task plan are consistent with the current availability of resources, without major conflicts (e.g., no obstacles or robot malfunctions). Executability means that, given the environment (i.e., environmental information perceived by the sensing agent), robot capabilities, and time requirements, the current task can theoretically be completed. Unambiguity means that there are no multiple meanings or ambiguities in the instruction parsing result and context, causing the robot to be unable to determine the specific operation object or process; for example, when the instruction parsing result involves preliminary steps or preconditions (e.g., permissions, item locations), and it is ensured that these are met or do not require further confirmation, then unambiguity is satisfied.

[0100] When the planning agent 230 receives a request for supplementary task information from the perception agent 220, it triggers an active planning mode. The planning agent 230 combines the reflection results provided by the reflection agent 250 to generate relevant information in the spontaneous search memory module 210 or to issue a user-initiated query request to the instruction through the interaction unit 100. After supplementing the missing information, the planning agent 230 combines the newly generated task perception package from the perception agent with the reflection results provided by the reflection agent 250 and then uses multi-step reasoning based on a large language model to form a task plan. The task plan is then passed to the decision agent 240.

[0101] When the planning agent 230 receives a multi-party assistance request from the perception agent 220, or when the task perception package still fails to meet the most basic executable conditions after multiple rounds of proactive planning, and it is predicted that continuing to generate a task plan according to the current situation will lead to task execution failure, the planning agent 230 proactively triggers a forward-looking planning mode. Based on the received task perception package and the reflection results provided by the reflection agent 250, the planning agent 230 generates a task plan to request collaboration or assistance from other human team members in the environment (including online groups and dedicated personnel collaboration) to obtain a wider range of information sources. After the task perception package meets the most basic executable conditions, it uses multi-step reasoning based on a large language model to form a task plan and then transmits this task plan to the decision agent 240. Specifically, if any one or more of the following situations recur during the task execution or planning process and cannot be resolved through further collaboration or external assistance, it is determined that continuing to generate a task plan according to the current situation will lead to task execution failure:

[0102] ① Information gaps or conflicts are difficult to fill: The user or memory module actively sends more than 210 requests for information to the instruction, but still cannot fill the critical gaps, or there are contradictory environmental information that cannot be resolved;

[0103] ② Multiple planning attempts fail to be implemented: After trying various alternative task decomposition schemes, it is still impossible to generate an executable action sequence (e.g., the path is completely blocked by obstacles);

[0104] ③ Resource or physical limitations: The robot is out of power and there is no available charging solution; the robotic arm is damaged and there is no way to repair it; or the external equipment required for the task is unavailable.

[0105] ④ Time or priority conflict: The task must be completed within the specified time, but it is found during planning that the time limit cannot be met, or other higher priority tasks are taking up resources, causing the current task to be put on hold and unable to be completed.

[0106] In this embodiment, the interaction and collaboration between the planning agent 230 and other agents includes:

[0107] With the perception agent 220: obtain updated task perception packages in real time; if the planning encounters a bottleneck, request the perception agent 220 to query more environmental, sensor or social platform data. The perception agent 220 stores the queried data in the memory module 210 for the planning agent 230 to call, so as to ensure that the planning agent 230 obtains the latest information.

[0108] With decision-making agent 240: The generated task plan is handed over to decision-making agent 240 to determine the specific robot action instructions for the next step, and the execution feedback is listened to;

[0109] With reflective agent 250: When the robot makes a mistake or encounters uncertainty during the execution of a task, reflective agent 250 proposes error correction suggestions or requests to replan, and planning agent 230 will adjust the task plan or replan accordingly.

[0110] 4. Decision-making intelligent agent

[0111] The decision-making agent 240 transforms the task plan generated by the planning agent 230 into a specific sequence of actions that the robot can execute, including both online operations and physical actions. The decision-making agent 240 has three decision-making modes, selecting the appropriate mode based on the planning mode adopted by the planning agent 230.

[0112] Reactive decision-making: When the planning agent 230 adopts the reactive planning mode, the decision-making agent 240 selects the reactive decision-making mode, directly decomposes the task plan into sub-tasks, generates action sequences, and assigns them to one or more of the most suitable robots 310, guiding the robots 310 to complete the task.

[0113] Proactive Decision Making: When the planning agent 230 adopts the proactive planning mode, the decision agent 240 selects the proactive decision making mode. Based on the task plan generated by the planning agent 230, it generates the search memory module 210 or sends an action sequence to the user through the interaction unit 100 to request supplementary task information in order to obtain the missing task information. The perception agent 220, the planning agent 230 and the decision agent 240 cooperate to perceive, plan and make decisions in sequence, thereby ensuring the smooth execution of the user's instructions.

[0114] Proactive Decision Making: When the planning agent 230 adopts a proactive planning mode, the decision-making agent 240 selects a proactive decision-making mode. Based on the task plan generated by the planning agent 230, the decision-making agent 240 combines the task perception package with proactive planning. It not only generates robot actions or online notifications based on the predetermined plan, but also proactively cooperates with the planning agent 230 by conducting online inquiries through one-on-one or group chats to proactively obtain additional information to optimize subsequent decisions. By dynamically adjusting the task execution order and strategy, the decision-making agent 240 proactively avoids generating erroneous actions, improves the success rate of user command completion, and ensures the efficiency and accuracy of task execution.

[0115] In this embodiment, the specific decision-making process of the decision-making agent 240 includes:

[0116] Step 1) Input Acquisition and Initialization:

[0117] The decision-making agent 240 obtains the task plan, which includes the task execution process and the personnel and resource requirements, from the planning agent 230.

[0118] Step 2) Pattern determination, action sequence generation, and collaborative allocation:

[0119] The decision-making agent 240 triggers its own decision-making mode based on the attributes of the task plan generated by the planning agent 240.

[0120] Reactive Decision Making: If the task plan is a complete execution plan for the original instructions, a reactive decision-making mode is triggered to generate an action sequence and assign specific sub-tasks. In some embodiments, based on the function tools available to the robot 310, the task plan output by the planning agent 230 is decomposed into multiple sub-tasks using the reasoning ability of a large language model (specifically, sub-task decomposition can be achieved through thought chains and prompt words plus a few sample decomposition examples). The sub-tasks use action instructions or function interfaces that the robot 310 can directly call, such as "move to coordinates (x,y)", "grab item A", "talk to a user", API calls, etc. Templates are established for basic action instructions (such as "grab", "move", etc.), and a complete action sequence is generated by filling parameters (target coordinates, target object, etc.) into the templates. Subsequently, for multi-robot scenarios, the decision agent 240 extracts one or more sub-tasks that should be executed in the most recent step from the refined sub-tasks based on indicators such as the load, position, and embodied capabilities of the robot 310. It determines the most suitable robot through a load balancing strategy based on the Hungarian Algorithm and sends specific action instructions to the corresponding robot execution module 313 using a unified ROS Topic communication protocol. During the execution of action instructions by the robot, the decision agent 240 continuously receives the running status, battery level, execution progress, or error reports provided by each robot 310 through the embodied status feedback interface 314 to detect whether the action is proceeding smoothly, thereby achieving continuous tracking of the execution status. At the same time, the decision agent 240 quickly identifies abnormal situations (such as mechanical failures or unexpected path blockages) based on rules or learning models. Once an abnormality is detected, it triggers error correction or interruption processes.

[0121] Proactive decision-making: If the task plan is generated based on proactive planning, that is, when the planning agent 230 discovers that the task information is missing when generating the task, the decision-making agent 250 enters the proactive decision-making mode. In this mode, the robot will not be directly dispatched. Instead, the user will actively inquire about the instructions for the proactive task plan. Specifically, the online operation can be generated by calling the robot (specifically, a smartphone) to send information, or by searching the memory module 210 to obtain the missing information.

[0122] Proactive Decision-Making: If the planning agent 230 enters proactive planning mode, i.e., when proactive information supplementation fails, the planning agent 230 will issue a task plan requiring the initiation of a one-on-one chat or group chat to obtain broader assistance information to facilitate the smooth execution of the instructions. At this time, the decision-making agent 250 enters proactive decision-making mode. In this mode, it will not directly dispatch robots, but will instead, based on the proactive task plan, call the robot's (specifically, a smartphone) message sending method to generate online operations, and initiate corresponding online request assistance messages according to the user or group chat name given in the task plan.

[0123] In this embodiment, the interaction and collaboration between the decision-making agent 240 and other agents includes:

[0124] Interaction with the perceptual agent 220: The latest task perception package is obtained in real time to ensure that the decision is based on the current environmental information for task decomposition and allocation; when the decision-making process encounters a bottleneck, the perceptual agent 220 can be requested to further query environmental, sensor or social platform data, and the query results are stored in the memory module 210 for the decision-making agent 240 to call to optimize the task decomposition and allocation process;

[0125] Collaboration with planning agent 230: Based on the task plan provided by planning agent 230, extract the specific tasks that need to be executed at the current moment;

[0126] Interaction with the reflective agent 250: After the task is completed, the decision agent 240 submits the decomposition and allocation results to the reflective agent 250, assists the reflective agent 250 in analyzing the execution effect and potential problems, generates reflection results, and integrates optimization suggestions into the decomposition and allocation process of future tasks.

[0127] 5. Reflective intelligent agents

[0128] The reflective agent 250 evaluates the task execution results based on information such as execution logs, task completion status, and abnormal status reports obtained from the task perception package of the perceptual agent 220, and provides reflection results for system optimization to ensure the accuracy and completeness of the task. Specific steps include:

[0129] Task evaluation: The reflective agent 250 analyzes the results of the robot execution module recorded in the task perception package to determine whether the task was successfully completed and records the execution status.

[0130] Problem diagnosis: If the task is not completed or fails to be executed, the reflective agent diagnoses the cause of the failure, provides optimization suggestions to the decision-making agent, and stores the reflection results in the memory unit during subsequent information transmission.

[0131] Process optimization: By reflecting on historical data, optimize the reasoning and execution processes for future tasks to improve the success rate of tasks.

[0132] Step 1) Input Acquisition and Initialization:

[0133] The reflexive agent 250 obtains and integrates the current task execution results, including execution logs, task completion status, and abnormal status reports, from the task perception package of the perceptual agent 220. The reflexive agent 250 also accesses the memory module 210 to call upon long-term and short-term memories to obtain preset task objectives and potentially relevant historical execution records (such as the completion status of similar past tasks). These two parts ultimately form a complete reflexive closed-loop information chain for reflecting on the current step. Through unified timestamp management during the reflection process, the reflexive agent 250 can accurately identify the sequence and causal relationships of each step in task execution, improving diagnostic efficiency.

[0134] Step 2) Task Evaluation:

[0135] The reflective agent 250 compares the current task execution result with the preset task objective. If the current task execution result meets or exceeds the preset task objective, the task is considered successful and execution details (such as time, resource consumption, etc.) are recorded in the task execution log. If part of the current task execution result does not match the preset task objective, such as missing steps, incorrect item coordinates, or unsatisfactory user feedback, the task is considered to have deviated or failed, and the process proceeds to the next step of the problem diagnosis process.

[0136] Step 3) Problem Diagnosis:

[0137] First, the reflective agent 250 retrieves the task perception package generated by the perceptual agent 220, the task plan generated by the planning agent 230, and the action sequence generated by the decision agent 240 as evidence, and comprehensively compares the environmental information before and after the robot executes the action sequence.

[0138] Subsequently, the Reflective Agent 250 utilizes chain-like reasoning from a large language model, such as textually describing the causes of errors and error correction strategies, enabling the Reflective Agent to output judgment processes and improvement plans in natural language, thereby improving interpretability. Alternatively, it can use rule-based reasoning to locate and infer the specific steps in which task deviations or failures occur. The inferred causes of task deviations include any one or more of the following: incomplete perception data, conflicting planning logic, incorrect decision instructions, robot hardware failure, and temporary changes in user requirements.

[0139] Finally, the reflective agent 250 describes the problem diagnosis results in text form, outputs the problem diagnosis results in natural language to improve interpretability, and then proceeds to the next step of the process.

[0140] Step 4) Feedback Generation and Process Optimization:

[0141] If the diagnosed problem can be fixed locally (e.g., by simply re-grasping or replacing the robot), the reflective agent 250 sends a local error correction instruction to the decision agent 240. If the diagnosed problem involves overall task reconstruction, the reflective agent 250 transmits the cause of failure and the need for supplementary information to the planning agent 240. If the diagnosed problem is caused by a lack of key information or requires user decision-making, the reflective agent 250 transmits the information that the interaction unit 100 needs to be called to request user intervention (clarify needs, provide additional resources, etc.) to the planning agent 230. The planning agent 230 generates a corresponding task plan, and the decision agent 240 generates a specific online query action sequence. After the robot responds, perception, planning, and decision-making are performed again.

[0142] In some embodiments, this disclosure achieves deep collaboration with humans through three task execution modes: reactive, proactive, and active.

[0143] 1) Reactive interaction: When the user's instructions are clear and the information is complete, the system directly executes the task and provides feedback on the results, thereby improving the efficiency of task execution.

[0144] 2) Proactive interaction: When task information is incomplete or execution is hindered, the system initiates inquiries to users or team members through the interaction unit 100 to obtain missing information or request assistance, and dynamically updates the task execution plan.

[0145] 3) Proactive Interaction: In complex tasks, the system combines embodied status feedback information and environmental awareness to proactively discover potential problems or predict future resource needs, and collaborate with users or team members to optimize execution plans to ensure the smooth completion of tasks.

[0146] Referring to Figures 1 and 2, the embodied task reasoning and robot scheduling method based on the above-described system provided in the second aspect of this disclosure includes the following steps:

[0147] a) Receiving user instructions: The interaction unit 100 receives the user's natural language instructions and transmits the instructions to the intelligent processing unit 200;

[0148] b) Generate task perception package: The perceptual agent 220 in the intelligent processing unit 200 parses the user's natural language instructions and combines them with long-term memory, environmental information and the robot's embodied state information to generate a task perception package;

[0149] c) Task reasoning and planning: When receiving instructions from the user for the first time, the planning agent 230 will directly reason from the task perception package, decompose the task, and generate a task plan without considering the reflection results. When receiving instructions from the user for the second time, the planning agent 230 will generate a task plan based on the task perception package and the reflection results provided by the reflection agent 250, and dynamically adjust the task plan according to the environmental information fed back by the perception agent 220 and the robot's embodied state information during the task execution.

[0150] d) Task decision-making and interaction: The decision-making agent 240 generates an executable sequence of actions based on the task plan generated by the planning agent 230, including specific online tasks and physical tasks, and sends them to the corresponding robot execution module. When information is missing, it actively interacts with the user who issues the instruction to supplement the required information.

[0151] e) Task execution and status feedback: The robot execution module receives the action sequence issued by the decision agent 240, executes the task through the high-level function, and feeds back the current embodied state to the perception agent 220 in real time. The perception agent 220 re-perceives the state change information, generates a new task perception package, and updates the memory module 210.

[0152] f) Reflection and optimization: The reflective agent 250 evaluates the task execution results based on the dynamic environmental information in the task perception package at two different time points. If the results do not meet expectations, it performs problem diagnosis and process optimization to form a reflection result. The planning agent and decision agent re-plan and make decisions based on the reflection result to ensure task completion. After the task is completed, the system updates the memory module information to provide a reference for future task execution.

[0153] The embodiments of this disclosure are described below with reference to Figure 3 and specific task execution examples.

[0154] Example task: "Zhang San: Take this document to Li Si for signature, and then give it to Wang Wu."

[0155] The deployable embodied robots include: a mobile chassis robot with a smart lock (hereinafter referred to as "Robot No. 1"), a robot dog (hereinafter referred to as "Robot No. 2"), a robotic arm (hereinafter referred to as "Robot No. 3"), and a smartphone (with WeChat and other chat software installed and an account configured).

[0156] For the example task described above, the task reasoning and robot scheduling process for each agent in this embodiment of the disclosure is as follows:

[0157] Step 1:

[0158] 1. Perceptual intelligent agent:

[0159] The sensing agent 220 detects a new user command in WeChat, receives the command, and parses the initial information: "The file needs to be obtained from Zhang San and sent to Li Si first, and then to Wang Wu last."

[0160] The sensing agent 220 calls the memory module 210 to query the location information of objects in the environment, Zhang San, Li Si, Wang Wu, and others, as well as the user's intent.

[0161] The perceptual agent 220 uses the queried information, the original user instructions, and the information after perception summary to generate a task perception package, and passes it to the reflective agent 250 (if it is the first execution, it skips the reflective agent 250 and passes it directly to the planning agent 230).

[0162] 2. Planning the intelligent agent:

[0163] The planning agent 230 decomposes the original user instructions in the task-aware package into three steps:

[0164] ① First, obtain the document from Zhang San;

[0165] ②Go to Li Si's location again and give him the signature;

[0166] ③Finally, go to Wang Wu's position and hand it over to him.

[0167] The planning agent 230 generates a complete task plan based on the above task decomposition results. If important information constituting the above task plan is missing at this time, it starts to generate a backup task plan.

[0168] 3. Decision-making intelligent agent:

[0169] Decision-making agent 240 compares the task plan generated by planning agent 230 with the capabilities of different robots in the embodied robot cluster, and infers and generates specific action sequences:

[0170] ① Physical action: <Robot No. 1: Moves to Zhang San>.

[0171] ② Online operation: <Mobile phone: Send a notification and a QR code for unlocking to Zhang San>.

[0172] The decision-making agent 240 calls the robot's high-level function container 312 according to the generated action sequence, and controls robot No. 1 to navigate to the first target location (i.e., where Zhang San is), and waits for Zhang San to complete the item placement operation.

[0173] Robot No. 1 transmits its embodied feedback information to the perceptual agent 220 to perceive new environmental information and embodied state, such as the placement of items.

[0174] Time step two:

[0175] 1. Perceptual intelligent agent:

[0176] The sensing agent 220 detected that Robot No. 1 had completed the first step of the task and successfully acquired the file.

[0177] The location of Li Si is queried through memory module 210, and the execution status of the previous task is updated in the task awareness package. The updated task awareness package is as follows:

[0178] Current status: File acquired, Robot No. 1 is in Zhang San's position.

[0179] Perception agent 220 passes the updated task perception package to planning agent 230.

[0180] 2. Reflective agents:

[0181] The reflective agent 250 evaluates the task execution result based on the generated plan, the specific task generated by the decision agent 240 at the previous time step, and the task perception package of the state changes before and after the task execution, and confirms that Robot No. 1 has obtained the document; if it fails midway (such as Zhang San not putting in the item), the reflective agent 250 will reflect on the reason for the task execution failure.

[0182] After completing its reflection, the reflective agent 250 will transmit its reflection results to the downstream planning agent 230 and decision-making agent 240 to guide them in making the next planning and decision-making steps, that is, whether to continue to execute the specific task corresponding to the remaining part of the user instruction, or to re-execute the specific task that failed in the previous step.

[0183] 3. Planning the intelligent agent:

[0184] If the execution result of the previous step is correct after reflection by the reflexive agent 250, the planning agent 230 will decompose the unfinished part into steps based on the user instructions and the contents of the task awareness package:

[0185] ① Take the document to Li Si's location and have Li Si sign it;

[0186] ②Go to Wang Wu's location and hand it over to him.

[0187] If the previous step's execution result is found to be erroneous after reflection by the reflecting agent 250, the planning agent 230 will generate a remedial plan based on the reflection result. For example, if the error was due to Zhang San not scanning the code to put the file into Robot One, then the planning agent 230 will update the task plan as follows:

[0188] Send a confirmation message to Zhang San, asking him if he has put the file in the robot locker;

[0189] 4. Decision-making intelligent agent:

[0190] Decision-making agent 240 generates a specific sequence of actions based on the task plan issued by planning agent 230 and after matching it with the capabilities of the robot swarm.

[0191] Physical action: <Robot No. 1: Moves to Li Si's position>.

[0192] Online operation: <Smartphone: Send a notification to Li Si, reminding him to prepare to sign>.

[0193] The decision-making agent 240 calls the high-level function container 312 of Robot No. 1 according to the generated action sequence, controls Robot No. 1 to navigate to Li Si's location, and sends a notification to him using a smartphone.

[0194] After arriving at Li Sichu according to the navigation, Robot No. 1 waits for the signing process to be completed, monitors the document status in real time, and transmits the embodied status to the perceptual agent 220 through its own embodied status feedback interface 314.

[0195] Step 3:

[0196] 1. Perceptual intelligent agent:

[0197] The sensing agent 220 received feedback from Robot No. 1, confirming that the document had been signed by Li Sichu.

[0198] The sensing agent 220 calls the memory module 210 to query Wang Wu's location and environmental information.

[0199] Update the execution status of the previous task to the task awareness package. The updated task awareness package contains the following: Current status: The document has been signed, and Robot No. 1 is in Li Si's position.

[0200] Perceptive agent 220 passes the updated task perception package to reflective agent 250.

[0201] 2. Reflective agents:

[0202] Based on the task plan, specific tasks, and task awareness packages of state changes before and after task execution generated in the previous step, the reflective agent 250 evaluates the task execution result: if the document has been signed, the current task is confirmed to be successful and the next stage of the task is executed; if the document has not been signed (e.g., Li Si did not perform the signing operation), the reflective agent 250 will provide feedback on the reason.

[0203] After completing its reflection, the reflecting agent 250 transmits the reflection results to the downstream planning agent 230 and decision-making agent 240, guiding them to continue executing subsequent instructions or generate remedial measures.

[0204] 3. Planning the intelligent agent:

[0205] Based on the task perception package provided by the perception agent 220, the planning agent 230 identifies the unfinished task parts and generates the following task plan: Take the signed document to Wang Wu's location and deliver the document to Wang Wu.

[0206] If the previous task failed, the planning agent 230 generates a remedial plan based on the feedback from the reflective agent 250. For example, if the document is not signed, the planning agent 230 will insert a remedial step (such as having the robot resubmit the document to Li Si).

[0207] 4. Decision-making intelligent agent:

[0208] Based on the task plan issued by the planning agent 230 and the capabilities of the robot swarm, the decision-making agent 240 generates a specific sequence of actions:

[0209] ① Physical action: <Robot No. 1: Moves to Wang Wu's position>.

[0210] ②Online operation: <Smartphone: Send a notification to Wang Wu, reminding him to prepare to receive the file>.

[0211] The decision-making agent 240 calls the high-level function container 312 of Robot No. 1 to control Robot No. 1 to navigate to Wang Wu's location and send a notification to Wang Wu via smartphone.

[0212] After Robot One reaches the target, it checks the delivery status to ensure that the document has been successfully delivered to Wang Wu; Robot One then transmits its embodied feedback information to the perceptual agent 220, reporting the document delivery status.

[0213] 5. Task completed:

[0214] If the file is successfully delivered to Wang Wu, the perceptual agent 220 will update the task perception package with the final task completion status. After passing through the reflection agent 250, the planning agent 230 and the decision agent 240 in sequence, an action sequence is generated to notify the user that the task has been completed. The smartphone responds to the action sequence and notifies the user that the task has been completed.

[0215] If the document is not successfully delivered, the reflective agent 250 will report the reason for the failure and guide the planning agent 230 and the decision-making agent 240 to generate remedial measures to ensure the successful completion of the task.

[0216] The embodiments disclosed herein demonstrate three task processing modes during task execution:

[0217] 1. Reactive task processing: Upon receiving user instructions, it automatically begins to perceive the environment, plan the problem, generate specific decision tasks, reflect on task behavior and results, and finally complete the user instructions, just like a human assistant.

[0218] 2. Proactive task completion: When information is incomplete, proactively interact with the user to complete the information and ensure the task is executed smoothly.

[0219] 3. Proactive Assistance: By using embodied feedback from robots to monitor the execution process in real time, predict the results of instruction execution, and proactively seek human assistance, the correct implementation of tasks can be ensured.

[0220] In summary, the embodied task reasoning and robot scheduling system and method for multi-agent collaboration provided in this disclosure has the following characteristics:

[0221] 1. Multi-agent collaboration: Through the collaborative work of perception, planning, decision-making and reflection agents, intelligent reasoning and efficient execution of complex tasks can be achieved.

[0222] 2. Dynamic Reasoning and Interaction: It features reactive, proactive, and forward-looking task interaction modes, effectively solving the problems of incomplete task information and dynamic environmental changes.

[0223] 3. Closed-loop feedback and optimization: The system continuously optimizes system performance and task execution results by dynamically evaluating the execution results through reflective agents.

[0224] Therefore, this disclosure effectively enhances the task execution capabilities of service robots in real-world scenarios, providing an intelligent and dynamic solution for handling complex tasks.

[0225] To implement the above embodiments, this disclosure also proposes an electronic device. The electronic device includes a processor and a memory for storing processor-executable instructions. The processor is configured to execute the instructions to implement the embodied task reasoning and robot scheduling method according to the second aspect of this disclosure.

[0226] As an example, Figure 4 is a block diagram of an electronic device 900 according to an embodiment of the present disclosure. As shown in Figure 4, the electronic device 900 may further include:

[0227] The memory 910 and processor 920 are connected by a bus 930, which connects the different components (including the memory 910 and the processor 920). The memory 910 stores a computer program, which, when executed by the processor 920, implements the method described in the embodiments of this disclosure.

[0228] Bus 930 represents one or more of several bus architectures, including a memory bus or memory controller, a peripheral bus, a graphics acceleration port, a processor, or a local bus using any of the various bus architectures. Examples of these architectures include, but are not limited to, the Industry Standard Architecture (ISA) bus, the Micro Channel Architecture (MAC) bus, the Enhanced ISA bus, the Video Electronics Standards Association (VESA) local bus, and the Peripheral Component Interconnect (PCI) bus.

[0229] Electronic device 900 typically includes a variety of electronic device readable media. These media can be any available media that can be accessed by electronic device 900, including volatile and non-volatile media, removable and non-removable media.

[0230] The memory 910 may also include computer system readable media in the form of volatile memory, such as random access memory (RAM) 940 and / or cache memory 950. The server 900 may further include other removable / non-removable, volatile / non-volatile computer system storage media. By way of example only, the storage system 960 may be used to read and write non-removable, non-volatile magnetic media (not shown in Figure 4, commonly referred to as a "hard disk drive"). Although not shown in Figure 4, disk drives for reading and writing to removable non-volatile disks (e.g., "floppy disks") and optical disk drives for reading and writing to removable non-volatile optical disks (e.g., CD-ROMs, DVD-ROMs, or other optical media) may be provided. In these cases, each drive may be connected to the bus 930 via one or more data media interfaces. The memory 910 may include at least one program product having a set (e.g., at least one) of program modules configured to perform the functions of the embodiments of this disclosure.

[0231] A program / utility 980 having a set (at least one) of program modules 970 may be stored, for example, in memory 910. Such program modules 970 include—but are not limited to—an operating system, one or more application programs, other program modules, and program data. Each or some combination of these examples may include an implementation of a network environment. Program modules 970 typically perform the functions and / or methods described in the embodiments of this disclosure.

[0232] Electronic device 900 can also communicate with one or more external devices 990 (e.g., keyboard, pointing device, display 991, etc.), and with one or more devices that enable a user to interact with electronic device 900, and / or with any device that enables electronic device 900 to communicate with one or more other computing devices (e.g., network card, modem, etc.). This communication can be performed via input / output (I / O) interface 992. Furthermore, electronic device 900 can also communicate with one or more networks (e.g., local area network (LAN), wide area network (WAN), and / or public networks, such as the Internet) via network adapter 993. As shown, network adapter 993 communicates with other modules of electronic device 900 via bus 930. It should be understood that, although not shown in Figure 4, other hardware and / or software modules can be used in conjunction with electronic device 900, including but not limited to: microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data backup storage systems.

[0233] The processor 920 performs various functional applications and data processing by running programs stored in the memory 910.

[0234] It should be noted that the implementation process and technical principles of the electronic device in this embodiment are explained in the foregoing description of the method of the present disclosure embodiment, and will not be repeated here.

[0235] To implement the above embodiments, this disclosure also proposes a computer-readable storage medium storing a computer program thereon, characterized in that the program, when executed by a processor, implements the embodied task reasoning and robot scheduling method of the second aspect embodiment.

[0236] To implement the above embodiments, this disclosure also proposes a computer program product, including a computer program that, when executed by a processor, implements the embodied task reasoning and robot scheduling method of the second aspect embodiment.

[0237] In this disclosure, a reactive, proactive, and active interaction mechanism is used to enable service robots to perform dynamic reasoning and efficient execution of complex tasks.

[0238] In this disclosure, the task plan is optimized in real time by combining the specific status information of the robot feedback module, thereby improving task adaptability and execution efficiency.

[0239] In this disclosure, the large-scale model, based on a combination of "large-scale training data + high-dimensional parameters + iterative optimization," continuously learns language rules and the relationships between concepts within a layered network structure, thereby acquiring deep natural language understanding and reasoning capabilities. Regarding the multi-agent architecture, multiple large-scale model agents with different functions or strategies are assigned roles and work collaboratively under a shared communication mechanism (such as natural language communication or a unified task protocol). Each agent can leverage its designated strengths and obtain more comprehensive task solutions through collaborative reasoning in stages such as "perception-planning-execution-reflection." This division of labor and collaboration helps to "simplify complexity," applying the powerful language understanding and reasoning capabilities of the large-scale model to multi-step, multi-role task scenarios: lower-level functional modules such as perception and execution do not need to master all language and reasoning logic, but are coordinated by higher-level large-scale language model agents. Ultimately, through efficient communication and complementary roles within the entire collaborative architecture, dynamic collaboration and deep reasoning for complex tasks can be achieved, giving the system better adaptability and problem-solving capabilities.

[0240] Based on the above analysis, the multi-agent collaborative framework proposed in this disclosure has deep natural language understanding and dynamic human-machine collaboration capabilities, and can proactively initiate information requests or collaborate with users to solve problems during execution.

[0241] In this disclosure, a reflection mechanism is introduced. The "Reflection Agent" introduced in the multi-agent collaborative framework can be regarded as a kind of meta-reasoning mechanism. It is linked with the specific perception, planning and execution processes and is specifically responsible for monitoring, evaluating and dynamically optimizing the overall task execution process of the system, thereby ensuring the accuracy of the task results.

[0242] This disclosure is applicable to complex and dynamic environments and can provide efficient, intelligent, and user-friendly services in real-world office scenarios.

[0243] In summary, this disclosure proposes a multi-agent collaborative embodied task reasoning and robot scheduling system and method. Leveraging the powerful natural language understanding and reasoning capabilities of a large language model, it can efficiently handle complex task instructions and dynamic situations. Multi-agent collaboration enhances the system's flexibility and task decomposition capabilities, enabling agents to communicate in human language and collaboratively complete complex tasks. This not only increases the algorithm's interpretability but also improves overall execution efficiency. By coordinating large language model agents, humans, and robots, this framework achieves seamless information sharing and real-time decision-making, ensuring intelligent and adaptive task execution in environments such as office settings. This integration approach not only improves the system's adaptability and robustness but also significantly enhances the effectiveness of human-machine collaboration, making it suitable for diverse and complex real-world application environments, achieving efficient execution of embodied tasks and robot scheduling.

[0244] It should be understood that the various forms of processes shown above can be used to reorder, add, or delete steps. For example, the steps described in this disclosure can be executed in parallel, sequentially, or in different orders, as long as the desired result of the technical solution disclosed in this disclosure can be achieved, and this is not limited herein.

[0245] The specific embodiments described above do not constitute a limitation on the scope of protection of this disclosure. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this disclosure should be included within the scope of protection of this disclosure.

[0246] In the description of this specification, the references to terms such as "one embodiment," "some embodiments," "illustrative embodiment," "example," "specific example," or "some examples," etc., indicate that a specific feature, structure, material, or characteristic described in connection with that embodiment or example is included in at least one embodiment or example of this disclosure. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples.

[0247] Although embodiments of this disclosure have been shown and described, those skilled in the art will understand that various changes, modifications, substitutions and alterations can be made to these embodiments without departing from the principles and spirit of this disclosure, the scope of which is defined by the claims and their equivalents.

Claims

1. A multi-agent collaborative embodied task reasoning and robot scheduling system, comprising an interconnected interaction unit, an intelligent processing unit, and an embodied robot cluster, wherein the embodied robot cluster contains multiple robots performing tasks; The interaction unit is used to acquire natural language commands issued by the user and send them to the intelligent processing unit; the interaction unit is also used to interact with the intelligent processing unit, receive the execution results of the task and robot status information, and provide measures to send a request to the user to supplement key task information when task information is missing. The intelligent processing unit adopts a multi-agent collaborative architecture based on a large language model, including a memory module, a perception agent, a planning agent, a decision-making agent, and a reflection agent. It performs task reasoning and action sequence generation through multi-agent collaboration, with all agents interacting via natural language. The memory module stores both long-term and short-term memory, which is shared among the agents. The perception agent dynamically perceives the environment and task status based on user instructions obtained from the interaction unit, combined with the robot's embodied state information and historical task execution logs, generating a task perception package containing task objectives, location information, and real-time task execution feedback. The planning agent dynamically reasones about task decomposition and execution schemes based on the task perception package and the reflection results provided by the reflection agent, generating a task plan. The decision-making agent is used to parse the task plan, generate action sequences, and schedule the robot to guide it to complete the task. The reflection agent is used to evaluate the task execution results. If the task execution results do not meet expectations, it provides feedback for the next round of task reasoning to replan or adjust the execution plan. The robot cluster executes tasks according to the action sequence issued by the intelligent processing unit, and feeds back its own embodied state information through the interface to help the perceptual agent dynamically perceive the environmental state in order to update the memory module.

2. The system according to claim 1, wherein, The memory module includes a long-term memory and a short-term memory. The long-term memory stored in the long-term memory storage device includes environmental semantic map, user default location, static location information of items, location information of public equipment, user preferences, and historical task execution logs; The short-term memory dynamically stored in the short-term memory includes real-time status information and dynamic context information during the current task execution process. The real-time status information includes user instructions, current task execution progress, robot's embodied state information, reflection results of the reflective agent, and user feedback records during task execution.

3. The system according to claim 1, wherein, The intelligent processing unit possesses task reasoning modes including reactive, proactive, and forward-looking. Reactive approach: When the perceptual agent determines that the task information is complete based on the user instructions sent by the interaction unit and the embodied state feedback from the robot, it extracts the task objective and operation requirements from the task information and generates a task perception package containing the task objective, location information and real-time feedback; the planning agent generates task execution steps based on the task perception package and the reflection results provided by the reflecting agent to form a task plan, which should meet the most basic executable conditions. The decision-making agent generates an action sequence by matching the task plan issued by the planning agent with the specific capabilities of the robot cluster. Active mode: When the perceptual agent determines that the task information is incomplete based on the user instructions sent by the interaction unit and the embodied state feedback from the robot, it sends a request for supplementary task information to the planning agent to fill the information gap. The planning agent forms a corresponding task plan based on the request and the reflection results provided by the reflection agent. The decision agent generates an action sequence based on the task plan and obtains the required supplementary information by searching relevant information in the memory module or by sending a query request to the user through the interaction unit. Proactive approach: When the current task cannot be completed after multiple rounds of proactive reasoning, the perceptual agent sends a multi-party assistance request to the planning agent. The planning agent forms a corresponding task plan based on the request and the reflection results provided by the reflection agent. The decision agent generates an action sequence based on the task plan and initiates inquiry requests to other users in the environment through the interaction unit to expand the scope of information acquisition and ensure the smooth execution of the task.

4. The system according to claim 3, wherein, The method of initiating a query request to other users in the environment through the interaction unit includes: Individual contact: Identify potential assisting users, send requests to obtain necessary information, including personnel not directly involved in the task or personnel not recorded in the memory module; Group broadcast: Publish assistance requests in work-related groups to coordinate resources from multiple parties to obtain the information needed for the task.

5. The system according to claim 1, wherein, The robot includes: A robot knowledge container is used to store environmental semantic maps, target user locations, and item coordinate information, and supports real-time updates; A high-level function container for robots, providing functional tools for acquiring and controlling the robot's embodied state, including navigation, object interaction, and real-time status feedback; The robot execution module is used to perform corresponding tasks based on the action sequence generated by the decision-making agent, including navigation, object interaction, and sending online messages; The embodied state feedback interface is used to feed back the robot's current physical state and task progress to the perceptual agent in order to update the task state information.

6. The system according to claim 5, wherein, The action sequence generated by the decision-making agent includes online operations and physical actions. The decision-making agent sends action instructions to the robot execution module through an interface, and dynamically adjusts the execution plan during task execution.

7. The system according to claim 5, wherein, The process by which the decision-making agent schedules the robot includes: The decision-making agent, based on the available functions and tools for the robot, breaks down the task plan output by the planning agent into multiple sub-tasks and generates a complete action sequence. Based on the robot's load, position, and capabilities, the decision-making agent selects one or more sub-tasks from the sub-tasks that should be performed at the most recent step. Using a load balancing strategy, it determines the most suitable robot and sends specific action instructions to the corresponding robot execution module via a unified communication protocol. During the robot's execution of action instructions, the decision-making agent continuously receives operating status, battery level, execution progress, or error reports from each robot via an interface to detect whether the actions are proceeding smoothly, thereby achieving continuous tracking of the execution status. Simultaneously, the decision-making agent identifies abnormal situations based on rules or learning models. Once an anomaly is detected, it triggers error correction or interrupts task execution.

8. The system according to claim 1, wherein, The reflective agent analyzes the real-time task execution feedback recorded in the task perception package to determine whether the task has been successfully completed and records the execution status, thereby achieving real-time evaluation of the task execution results. If the task is not completed or fails to execute, the reflective agent diagnoses the cause of the failure, provides optimization suggestions to the decision-making agent, and stores the reflection results in the memory module during subsequent information transmission. The reflective agent also optimizes the reasoning and execution process of future tasks based on historical reflection results.

9. The system according to claim 8, wherein, The reflection results include assessments of the success or failure of task execution, analysis of the causes of anomalies, and suggestions for improvement.

10. A method for embodied task reasoning and robot scheduling based on the system described in any one of claims 1 to 9, wherein, Includes the following steps: a) Receiving user instructions: The interaction unit receives natural language instructions from the user and transmits the instructions to the intelligent processing unit; b) Generate a task perception package: The sensing agent dynamically perceives the environment and task status based on the received user instructions and in combination with the robot's physical state and historical task execution logs, and generates a task perception package that includes the task objective, location information and real-time task execution feedback. c) Dynamic planning task: When receiving a user's instruction for the first time, without considering the reflection results, the planning agent directly reasones about the task perception package, decomposes the task, and generates a task plan. For situations where the user's instructions are not received for the first time, the planning agent generates a task plan based on the task perception package and the reflection results provided by the reflection agent, and dynamically adjusts the task plan according to the environmental information fed back by the perception agent and the robot's embodied state information during task execution. d) Generate execution actions: The decision-making agent generates specific online and physical tasks according to the task plan, sends them to the corresponding robots, and actively interacts with the user who issued the instruction to supplement the required information when task information is missing; e) Task execution and status feedback: The robot receives action instructions, executes tasks through advanced functions, and provides real-time feedback of its current embodied state to the perceptual agent. The perceptual agent then re-perceives the state change information, generates a new task perception package, and updates the memory module. f) Reflection and optimization: The reflecting agent evaluates the task execution results based on the dynamic environmental information in the task perception package at the previous and next time points. If the results do not meet expectations, the agent performs problem diagnosis and process optimization to form a reflection result. The planning agent and the decision agent re-plan and make decisions based on the reflection result to ensure the task is completed.

11. The method according to claim 10, wherein, The method supports three task interaction modes: Reactive interaction: Directly executes complete task instructions; Proactive interaction: Sending instructions to the user or system to request supplementary information about the missing task; Proactive interaction: When proactive interaction fails to obtain information, requests are sent to other users in the environment through the interaction unit to seek help through one-on-one contact or broadcasting in a group, and multiple parties collaborate to complete the task by supplementing the information needed.

12. The method according to claim 10, wherein, During task execution, the reflective agent and the planning agent work together to achieve continuous monitoring and dynamic adjustment of the task status.

13. The method according to claim 10, wherein, During the robot's task execution, the robot's physical state information is obtained in real time through an interface, providing the sensing agent with position, path, and task execution status data to help the system achieve dynamic perception and task reasoning.

14. An electronic device, comprising: At least one processor; And, a memory communicatively connected to the at least one processor; The memory stores instructions executable by the at least one processor, the instructions being configured to perform the method described in any one of claims 10 to 13.

15. A computer-readable storage medium storing computer instructions for causing the computer to perform the method of any one of claims 10 to 13.

16. A computer program product comprising a computer program or instructions which, when executed by a processor, implement the method as described in any one of claims 10 to 13.

Images