Robot control method and apparatus

Abstract

Disclosed in the present application are a robot control method and an apparatus. A specific embodiment of the method comprises: by means of a pre-trained multi-modal large model and on the basis of multi-modal environment data, determining pose data of a target object corresponding to a target task; by means of a pre-trained trajectory generation model and on the basis of the pose data, determining a moving trajectory of a robot from the current position to the target object; by means of a pre-trained skill model, determining an operating parameter corresponding to the target task; and controlling the robot to move on the basis of the moving trajectory and the operating parameter.

Description

Robot control methods and devices

[0001] Cross-references to related applications

[0002] This application claims priority and benefit to Chinese patent application No. 202510025199.1 filed with the China National Intellectual Property Administration (CNIPA) on January 7, 2025, which is incorporated herein by reference in its entirety. Technical Field

[0003] This disclosure relates to the field of artificial intelligence technology, specifically to the fields of large models, natural language understanding, and machine learning, and particularly to a robot control method, apparatus, computer-readable medium, electronic device, and computer program product. Background Technology

[0004] Traditional robot control methods have limited capabilities when performing complex or long-sequence tasks, and cannot be applied to most real-world scenarios. For example, in a healthy home setting, when a robot is given the task of "getting a glass of water from the bedroom to the kitchen," it often fails to complete the task completely and accurately. Summary of the Invention

[0005] This disclosure provides a robot control method, apparatus, computer-readable medium, and electronic device.

[0006] In a first aspect, embodiments of this disclosure provide a robot control method, comprising: determining the pose data of a target object corresponding to a target task based on multimodal environment data using a pre-trained multimodal large model; determining the robot's running trajectory from its current position to the target object based on the pose data using a pre-trained trajectory generation model; determining the operation parameters corresponding to the target task using a pre-trained skill model; and controlling the robot to run according to the running trajectory and operation parameters.

[0007] In some examples, the above method also includes: decomposing the target task using a multimodal large model to obtain a task sequence including multiple sub-tasks; and determining the operation parameters corresponding to the target task using a pre-trained skill model, including: generating the sub-task operation parameters corresponding to each of the multiple sub-tasks in the task sequence using the skill model to obtain the operation parameters, wherein the operation parameters include multiple sub-task operation parameters.

[0008] In some examples, the skill model is used to generate the subtask operation parameters corresponding to each of the multiple subtasks in the task sequence. This includes: generating the initial operation parameters of each of the multiple subtasks according to the skill model; and adjusting the initial operation parameters of each of the multiple subtasks according to the operation scenario corresponding to the target task to obtain the subtask operation parameters corresponding to each of the multiple subtasks.

[0009] In some examples, the aforementioned trajectory includes a movement trajectory indicating the robot's mobile device's movement from its current position to the target object, and an obstacle avoidance trajectory indicating the robot's robotic arm's obstacle avoidance during movement. Additionally, the aforementioned trajectory generation model, based on pose data, determines the robot's movement trajectory from its current position to the target object, including: generating the movement trajectory and obstacle avoidance trajectory using the trajectory generation model based on pose data and environmental feature data obtained from multimodal environment data using a multimodal large model.

[0010] In some examples, the trajectory generation model is trained as follows: a first sample set is obtained, wherein the first sample in the first sample set includes first environment data and first trajectory data. The first trajectory data is used to characterize the robot's trajectory from its current position to the target sample object in the 3D environment corresponding to the first environment data. A machine learning algorithm is used to train the trajectory generation model with the start and end positions corresponding to the first environment feature data and the first trajectory data as inputs and the first trajectory data as the desired output. The first environment feature data is obtained by a multimodal large model based on the first environment data in the first sample where the first trajectory data is located.

[0011] In some examples, the first trajectory data is obtained as follows: during the controlled operation of the robot in the three-dimensional environment from its current position to the target sample object, the robot collects the movement trajectory corresponding to the mobile device and the obstacle avoidance trajectory corresponding to the robotic arm through various sensors set on the robot; the movement trajectory and obstacle avoidance trajectory are aligned in the spatial and temporal dimensions to obtain the first trajectory data.

[0012] In some examples, the skill model is trained as follows: a second sample set is obtained, wherein the second sample in the second sample set includes action sequence data corresponding to a skill in the skill library. During the process of the robot being controlled to demonstrate the skill, the action sequence data is collected by various sensors set on the robot; the initial skill model is used as the learning object, and the action sequence data is used for imitation learning to obtain an imitation learning model; the imitation learning model is optimized using a reinforcement learning algorithm to obtain the skill model.

[0013] In some examples, the multimodal large model is trained as follows: a third training set is obtained, wherein the third samples in the third sample set include third environment data and pose labels of target sample objects in the 3D environment represented by the third environment data; a machine learning method is used to train the multimodal large model with the third environment data as input and the pose labels corresponding to the third environment data as the expected output.

[0014] Secondly, embodiments of this disclosure provide a robot control device, including: a pose determination unit configured to determine pose data of a target object corresponding to a target task based on multimodal environment data using a pre-trained multimodal large model; a trajectory determination unit configured to determine the robot's running trajectory from its current position to the target object based on the pose data using a pre-trained trajectory generation model; a parameter determination unit configured to determine the operation parameters corresponding to the target task using a pre-trained skill model; and a control unit configured to control the robot to run according to the running trajectory and operation parameters.

[0015] In some examples, the above apparatus further includes: a task decomposition unit configured to decompose the target task using a multimodal large model to obtain a task sequence including multiple sub-tasks; and a parameter determination unit further configured to: generate sub-task operation parameters corresponding to each of the multiple sub-tasks in the task sequence using a skill model to obtain operation parameters, wherein the operation parameters include multiple sub-task operation parameters.

[0016] In some examples, the parameter determination unit is further configured to: generate initial operation parameters for each of the multiple subtasks based on the skill model; and adjust the initial operation parameters for each of the multiple subtasks according to the operation scenario corresponding to the target task, thereby obtaining the operation parameters for each of the multiple subtasks.

[0017] In some examples, the aforementioned trajectory includes a movement trajectory indicating the movement of the robot's mobile device from its current position to the target object and an obstacle avoidance trajectory indicating the robot's robotic arm's obstacle avoidance during movement. The trajectory determination unit is further configured to generate the movement trajectory and obstacle avoidance trajectory based on pose data and environmental feature data obtained from multimodal environment data using a trajectory generation model.

[0018] In some examples, the trajectory generation model is trained as follows: a first sample set is obtained, wherein the first sample in the first sample set includes first environment data and first trajectory data. The first trajectory data is used to characterize the robot's trajectory from its current position to the target sample object in the 3D environment corresponding to the first environment data. A machine learning algorithm is used to train the trajectory generation model with the start and end positions corresponding to the first environment feature data and the first trajectory data as inputs and the first trajectory data as the desired output. The first environment feature data is obtained by a multimodal large model based on the first environment data in the first sample where the first trajectory data is located.

[0019] In some examples, the first trajectory data is obtained as follows: during the controlled operation of the robot in the three-dimensional environment from its current position to the target sample object, the robot collects the movement trajectory corresponding to the mobile device and the obstacle avoidance trajectory corresponding to the robotic arm through various sensors set on the robot; the movement trajectory and obstacle avoidance trajectory are aligned in the spatial and temporal dimensions to obtain the first trajectory data.

[0020] In some examples, the skill model is trained as follows: a second sample set is obtained, wherein the second sample in the second sample set includes action sequence data corresponding to a skill in the skill library. During the process of the robot being controlled to demonstrate the skill, the action sequence data is collected by various sensors set on the robot; the initial skill model is used as the learning object, and the action sequence data is used for imitation learning to obtain an imitation learning model; the imitation learning model is optimized using a reinforcement learning algorithm to obtain the skill model.

[0021] In some examples, the multimodal large model is trained as follows: a third training set is obtained, wherein the third samples in the third sample set include third environment data and pose labels of target sample objects in the 3D environment represented by the third environment data; a machine learning method is used to train the multimodal large model with the third environment data as input and the pose labels corresponding to the third environment data as the expected output.

[0022] Thirdly, embodiments of this disclosure provide a computer-readable medium having a computer program stored thereon, wherein when the program is executed by a processor, it implements the method as described in any implementation of the first aspect.

[0023] Fourthly, embodiments of this disclosure provide an electronic device, including: one or more processors; and a storage device having one or more programs stored thereon, wherein when the one or more programs are executed by the one or more processors, the one or more processors implement the method described in any implementation of the first aspect.

[0024] Fifthly, a computer program product is provided, comprising: a computer program that, when executed by a processor, implements the method described in any implementation of the first aspect. Attached Figure Description

[0025] Other features, objects, and advantages of this disclosure will become more apparent from the following detailed description of non-limiting embodiments with reference to the accompanying drawings:

[0026] Figure 1 is an exemplary system architecture diagram in which an embodiment of this disclosure can be applied;

[0027] Figure 2 is a flowchart of an embodiment of the robot control method according to the present disclosure;

[0028] Figure 3 is a schematic diagram of an application scenario of the robot control method according to this embodiment;

[0029] Figure 4 is a flowchart of yet another embodiment of the robot control method according to the present disclosure;

[0030] Figure 5 is a structural diagram of an embodiment of the robot control device according to the present disclosure;

[0031] Figure 6 is a schematic diagram of the structure of a computer system adapted to implement embodiments of the present disclosure. Detailed Implementation

[0032] The present disclosure will now be described in further detail with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative of the invention and not intended to limit it. Furthermore, it should be noted that, for ease of description, only the parts relevant to the invention are shown in the accompanying drawings.

[0033] It should be noted that, unless otherwise specified, the embodiments and features described in this disclosure can be combined with each other. This disclosure will now be described in detail with reference to the accompanying drawings and embodiments.

[0034] It should be noted that the collection, gathering, updating, analysis, processing, use, transmission, and storage of user personal information involved in this disclosed technical solution all comply with relevant laws and regulations, are used for legitimate purposes, and do not violate public order and good morals. Necessary measures are taken to prevent unauthorized access to user personal information data and to safeguard user personal information security, network security, and national security.

[0035] The robot control method and apparatus provided in this disclosure determine the pose data of the target object corresponding to the target task based on multimodal environmental data through a pre-trained multimodal large model; determine the robot's running trajectory from the current position to the target object based on the pose data through a pre-trained trajectory generation model; determine the operation parameters corresponding to the target task through a pre-trained skill model; and control the robot to run according to the running trajectory and operation parameters. By combining the multimodal large model, trajectory generation model, and skill model, the robot's running trajectory from the current position to the target object and the operation parameters for the target task are generated to control the robot's operation, thereby improving the robot's ability to execute long sequence tasks and enabling the robot to adapt to complex scenarios.

[0036] Figure 1 illustrates an exemplary architecture 100 to which the robot control methods and apparatus of this disclosure can be applied.

[0037] As shown in Figure 1, the system architecture 100 may include terminal devices 101, 102, and 103, a network 104, and a server 105. The communication connections between terminal devices 101, 102, and 103 form a network topology. Network 104 serves as the medium for providing communication links between terminal devices 101, 102, and 103 and server 105. Network 104 may include various connection types, such as wired or wireless communication links or fiber optic cables, etc.

[0038] Terminal devices 101, 102, and 103 can interact with server 105 via network 104 to receive or send data. Terminal devices 101, 102, and 103 can be hardware or software that supports network connectivity for data interaction and processing. When terminal devices 101, 102, and 103 are hardware, they can be various electronic devices that support network connectivity, information acquisition, interaction, display, and processing functions, including but not limited to robots, smartphones, in-vehicle computers, tablets, e-book readers, laptops, and desktop computers. When terminal devices 101, 102, and 103 are software, they can be installed in the aforementioned electronic devices. They can be implemented as multiple software programs or software modules, for example, to provide distributed services, or as a single software program or software module. No specific limitations are made here.

[0039] Server 105 can be a server that provides various services. For example, it can be a background processing server that generates the robot's running trajectory from its current position to the target object and the operation parameters for the target task based on the multimodal environmental data collected by robots 101, 102, and 103, combined with a multimodal large model, a trajectory generation model, and a skill model, in order to control the robot's operation. As an example, server 105 can be a cloud server.

[0040] It should be noted that a server can be either hardware or software. When the server is hardware, it can be implemented as a distributed server cluster consisting of multiple servers, or as a single server. When the server is software, it can be implemented as multiple software programs or software modules (such as software programs or software modules used to provide distributed services), or as a single software program or software module. No specific limitations are made here.

[0041] It should also be noted that the robot control method provided in the embodiments of this disclosure is generally executed by a server, but the possibility of it being executed by a terminal device, or by the server and the terminal device cooperating with each other, is not excluded. Accordingly, all the parts (e.g., each unit) of the robot control device can be located entirely in the server, entirely in the terminal device, or separately in the server and the terminal device.

[0042] It should be understood that the number of terminal devices, networks, and servers in Figure 1 is merely illustrative. Any number of terminal devices, networks, and servers can be included depending on implementation needs. When the electronic devices on which the robot control method runs do not require data transmission with other electronic devices, the system architecture may only include the electronic devices on which the robot control method runs (e.g., servers or terminal devices).

[0043] Referring again to Figure 2, a flow 200 of an embodiment of the robot control method is shown, including the following steps:

[0044] Step 201: Using a pre-trained multimodal large model, determine the pose data of the target object corresponding to the target task based on the multimodal environment data.

[0045] In this embodiment, the execution entity of the robot control method (such as the server in Figure 1) can obtain multimodal environment data remotely or locally through wired network connection or wireless network connection, and determine the pose data of the target object corresponding to the target task based on the multimodal environment data through a pre-trained multimodal large model.

[0046] Multimodal large models refer to large models capable of handling multimodal data (e.g., multimodal environmental data). Specifically, large models (artificial intelligence large models) refer to pre-trained large AI models with a large or very large number of parameters. Large AI models possess excellent contextual understanding, language generation, learning capabilities, and transferability.

[0047] Multimodal environmental data includes, but is not limited to, environmental data such as images, sounds, and point clouds collected by environmental acquisition devices such as vision sensors, hearing sensors, tactile sensors, and radar on robots.

[0048] In this embodiment, the user can issue target tasks to the robot based on voice, text, actions, etc. The multimodal large model can perform semantic understanding and analysis of the target tasks to determine the target object. Then, using the multimodal environmental data collected by the robot in real time as input, the pose (position and attitude) data of the target object in the three-dimensional real environment represented by the multimodal environmental data is determined. The position data includes the absolute position and relative position of the target object.

[0049] As an example, if the objective task is "get me the beverage from the refrigerator", then the target object is the refrigerator. The multimodal large model needs to determine the pose information of the refrigerator in the 3D scene based on the multimodal environment data.

[0050] It's important to note that the target object is often not the object the robot interacts with during the execution of the task. The target object is often embedded within other objects (the target object). The robot needs to move to the location of these other objects before it can operate on the final object located on or within them. For example, in the task "Get me the drink from the refrigerator," the refrigerator is the target object, and the drink is the object the task is directed to.

[0051] In some implementations of this embodiment, the multimodal large model is trained in the following manner:

[0052] First, obtain the third training set.

[0053] The third sample in the third sample set includes the third environmental data and the pose labels of the target sample objects in the three-dimensional environment represented by the third environmental data.

[0054] In this implementation, a third sample set can be obtained by combining the ScanNet open-source dataset (ScanNet is a widely used 3D scene dataset) and data collected according to specific scenarios (such as home service scenarios or logistics application scenarios). The ScanNet open-source dataset already contains RGB-D images of various indoor scenes and their corresponding 3D models, and provides detailed annotations for the objects, including object classification labels and pose data. Therefore, only the images and 3D models collected for specific scenarios need to be annotated with classification labels and pose information.

[0055] Then, a machine learning method is used to train a multimodal large model with the third environment data as input and the pose label corresponding to the third environment data as the expected output.

[0056] First, select a suitable multimodal large model. Based on the specific scenario in which the trained multimodal large model is applied and the fit between different multimodal large models, choose a suitable one from a variety of multimodal large models, such as LLAVA (Large Language and Vision Assistant) 1.5. LLAVA 1.5 is a multimodal large language model suitable for vision and language tasks. It can understand and generate descriptions about image content. Therefore, LLAVA 1.5 can better handle open-source datasets and scenario-specific datasets.

[0057] Then, the third environment data in the third sample is transformed into a target format suitable for multimodal large model processing, such as being transformed into serialized point cloud data or having the corresponding feature vectors extracted.

[0058] Finally, the aforementioned execution entity can iteratively perform the following training operations until the preset termination condition is met, thereby obtaining a pre-trained multimodal large model:

[0059] Untrained target third samples are selected from the third sample set. The third environment data in the target format from the target third samples is input into the multimodal large model to obtain the pose prediction data of the sample objects output by the multimodal large model. Based on the loss between the pose prediction data and the pose labels in the target third samples, the update gradient of the multimodal large model is determined by methods such as gradient descent and stochastic gradient descent to update the parameters of the multimodal large model.

[0060] Specifically, the loss includes the distance error between the location prediction data and the location label, the angle error between the attitude prediction data and the attitude label, and the assessment of whether the large model correctly understood and completed the task based on the task description.

[0061] The preset termination conditions include, for example, the training loss converging, the training time exceeding a preset time threshold, and the number of training iterations exceeding a preset number of iterations threshold. If the prediction performance of a multimodal large model is poor, it is necessary to optimize the model performance by adjusting hyperparameters such as the learning rate and batch size.

[0062] During the training of a multimodal large model, the point cloud data in the third sample set needs to be normalized, including but not limited to denoising, alignment and feature extraction, to ensure that the label data (pose label) of each scene corresponds accurately with the point cloud data.

[0063] This implementation provides a method for training a multimodal large model based on environmental samples, enabling the multimodal large model to accurately identify objects in a 3D scene.

[0064] Step 202: Using a pre-trained trajectory generation model, determine the robot's trajectory from its current position to the target object based on the pose data.

[0065] In this embodiment, the aforementioned execution entity can determine the robot's trajectory from its current position to the target object based on pose data using a pre-trained trajectory generation model. The trajectory generation model represents the correspondence between the target object's pose data and the robot's trajectory from its current position to the target object.

[0066] Trajectory generation models include, for example, multinomial trajectory models, physics-based trajectory models, machine learning models, and deep learning models. Taking deep learning models as an example, trajectory generation models include recurrent neural networks, attention mechanisms, generative adversarial networks, Transformer models, and diffusion models.

[0067] In some implementations of this embodiment, the running trajectory includes a movement trajectory that indicates the movement process of the robot's mobile device from its current position to the target object, and an obstacle avoidance trajectory that indicates the robot's robotic arm to avoid obstacles during the movement.

[0068] In this implementation, the above-mentioned execution entity can perform step 202 in the following way: generate a movement trajectory and an obstacle avoidance trajectory based on the pose data and the environmental feature data obtained by the multimodal large model based on the multimodal environment data through the trajectory generation model.

[0069] A robot generally consists of a mobile device and a robotic arm. The mobile device is used to change the robot's position, and the robotic arm is used to perform the target task.

[0070] In this implementation, the trajectory generation model generates both a movement trajectory and an obstacle avoidance trajectory based on the pose data of the target object and the environmental feature data obtained by the multimodal large model from the multimodal environmental data. These trajectories guide the operation of the robot's mobile device and the robotic arm, respectively, thereby improving the adaptability between the movement trajectory and the robot and enabling control of the robot's movement and obstacle avoidance capabilities in a 3D scene.

[0071] In some implementations of this embodiment, the aforementioned execution entity can train the trajectory generation model in the following manner:

[0072] First, obtain the first sample set.

[0073] The first sample in the first sample set includes first environmental data and first trajectory data. The first trajectory data is used to characterize the robot's running trajectory from its current position to the target sample object in the three-dimensional environment corresponding to the first environmental data.

[0074] The first environmental data can be obtained by combining the ScanNet open-source dataset (ScanNet is a widely used 3D scene dataset) and environmental data collected according to specific scenarios (such as home service scenarios or logistics application scenarios). The first trajectory data can be obtained by using odometry, LiDAR, depth cameras, and other data acquisition devices on the robot during its operation from its current position to the target sample object. The target sample object is the specified object in the 3D environment corresponding to the first environmental data.

[0075] Then, a machine learning algorithm is used, with the start and end positions corresponding to the first environmental feature data and the first trajectory data as inputs, and the first trajectory data as the expected output, to train a trajectory generation model.

[0076] The first environmental feature data is obtained by the multimodal large model based on the first environmental data in the first sample where the first trajectory data is located.

[0077] As an example, the aforementioned execution entity can iteratively train the trajectory generation model in the following manner until a preset termination condition is met:

[0078] Select an untrained target first sample from the first sample set; obtain the first environmental feature data of the first environmental data in the target first sample through a pre-trained multimodal large model; input the first environmental feature data into the trajectory generation model to obtain the predicted trajectory data; calculate the loss between the predicted trajectory data and the first trajectory data in the target first sample, so as to update the parameters of the trajectory generation model through the loss.

[0079] This implementation provides a specific training method for the trajectory generation model, which improves the accuracy of the trained trajectory generation model.

[0080] In this implementation, data from different scenarios and tasks can be added to improve the model's generalization ability and adaptability; and the trajectory generation effect can be evaluated based on path accuracy, obstacle avoidance ability, trajectory smoothness, and task completion efficiency.

[0081] In some implementations of this embodiment, the execution entity can obtain the first trajectory data in the following way: First, during the controlled operation of the robot in the three-dimensional environment from its current position to the target sample object, the robot collects the movement trajectory corresponding to the mobile device and the obstacle avoidance trajectory corresponding to the robotic arm through various sensors set on the robot; then, the movement trajectory and obstacle avoidance trajectory are aligned in the spatial and temporal dimensions to obtain the first trajectory data.

[0082] As an example, during the process of manually controlling a robot to move from its current position to the target sample object, the robot uses sensors such as odometry, lidar, and depth cameras to collect the movement trajectory, and uses devices such as joint encoders and force sensors to collect the obstacle avoidance trajectory.

[0083] Synchronize the movement trajectory and obstacle avoidance trajectory in time to ensure that the motion data of the mobile device and the robotic arm are synchronized in time, facilitating subsequent analysis and processing. Align the movement trajectory and obstacle avoidance trajectory spatially to ensure that their coordinate systems are consistent.

[0084] In this implementation, during the controlled operation of the robot from its current position to the target sample object, the movement trajectory of the mobile device and the obstacle avoidance trajectory of the robotic arm are collected and spatiotemporally aligned, which improves the accuracy of the training samples of the trajectory generation model and helps to improve the training effect of the trajectory generation model.

[0085] Step 203: Determine the operational parameters corresponding to the target task using the pre-trained skill model.

[0086] In this embodiment, the aforementioned execution entity can determine the operation parameters corresponding to the target task through a pre-trained skill model.

[0087] A skill model represents a basic skill set, which includes various fundamental skills that a robot needs to master, such as a robotic arm opening objects, grasping objects, and closing objects. Based on these basic skills, the operational parameters corresponding to the target task can be determined.

[0088] In some implementations, a pre-trained skill model is used to determine the operational parameters corresponding to the target task based on multimodal environmental data collected in real time by the robot at the target object. Combining the real-time collected multimodal environmental data helps to improve the accuracy of the operational parameters corresponding to the target task.

[0089] Step 204: Control the robot to run according to the running trajectory and operating parameters.

[0090] In this embodiment, the aforementioned execution entity can control the robot to run according to the running trajectory and operating parameters.

[0091] Specifically, the robot is controlled to move along a trajectory to the target object, and then the target object is controlled to perform the target task according to the operating parameters. Continuing with the example of "get me the drink from the refrigerator," the robot is controlled to move along a trajectory to the refrigerator, and then the robot is controlled to perform the target task of retrieving the drink from the refrigerator according to the operating parameters.

[0092] It should be noted that the robot control process can be executed only after the complete running trajectory and operating parameters are obtained. For example, in response to obtaining the robot's running trajectory through step 202, the robot can be controlled to run to the target object according to the running trajectory; during the robot's running according to the running trajectory, or after running to the target object, the operating parameters can be obtained through step 203.

[0093] Referring again to Figure 3, which is a schematic diagram 300 of an application scenario of the robot control method according to this embodiment. In the application scenario of Figure 3, user 301 sends a target task to robot 302: "Get me the beverage from the refrigerator." After receiving the target task, the robot uploads the task to server 303. Server 303 first uses a pre-trained multimodal large model to determine the pose data of the target object (refrigerator) corresponding to the target task based on multimodal environment data; then, using a pre-trained trajectory generation model, it determines the robot's running trajectory from its current position to the target object based on the pose data, and controls the robot to run to the target object according to the running trajectory; finally, using a pre-trained skill model, it determines the operation parameters corresponding to the target task, and controls the robot to run according to the operation parameters to retrieve the beverage from the refrigerator.

[0094] The method provided in the above embodiments of this disclosure determines the pose data of the target object corresponding to the target task based on multimodal environment data through a pre-trained multimodal large model; determines the robot's running trajectory from the current position to the target object based on the pose data through a pre-trained trajectory generation model; determines the operation parameters corresponding to the target task through a pre-trained skill model; and controls the robot to run according to the running trajectory and operation parameters. Thus, by combining the multimodal large model, trajectory generation model, and skill model, the method generates the robot's running trajectory from the current position to the target object and the operation parameters for the target task to control the robot's operation, thereby improving the robot's ability to execute long sequence tasks and enabling the robot to adapt to complex scenarios.

[0095] In some optional implementations of this embodiment, the execution entity may also perform the following operations: decompose the target task through a multimodal large model to obtain a task sequence including multiple sub-tasks.

[0096] The aforementioned execution entity inputs the target task into a multimodal large model, analyzes and understands the target task through the multimodal large model, determines multiple nodes in the entire execution process of the target task, and then decomposes the target task into multiple sub-tasks based on multiple nodes to obtain a task sequence.

[0097] Continuing with the example of the objective task "get me the beverage from the refrigerator", it can be broken down into three sub-tasks that have a temporal relationship: opening the refrigerator door, taking the beverage from the refrigerator, and closing the refrigerator door.

[0098] In this implementation, the aforementioned execution entity can perform step 203 as follows: Using the skill model, generate subtask operation parameters corresponding to each of the multiple subtasks in the task sequence, thus obtaining the operation parameters. These operation parameters include multiple subtask operation parameters.

[0099] For the skill base represented by the skill model, the aforementioned execution entity can determine the correspondence between the basic skills in the skill base and multiple sub-tasks; then, for each sub-task, the sub-task parameters corresponding to the sub-task are determined by the basic skills corresponding to the sub-task.

[0100] For example, the three sub-tasks of opening the refrigerator door, taking out a beverage from the refrigerator, and closing the refrigerator door correspond to basic skills such as the robotic arm opening an object, the robotic arm grasping an object, and the robotic arm closing an object, respectively.

[0101] In this implementation, the target task is broken down into multiple subtasks, and the operation parameters of each subtask are determined to control the robot to execute the target task. This reduces the difficulty of executing the target task and improves the robot's ability to execute the target task.

[0102] In some optional implementations of this embodiment, the execution entity can obtain the subtask operation parameters corresponding to each of the multiple subtasks in the following way:

[0103] First, using the skill model, initial operation parameters for each of the multiple subtasks are generated.

[0104] In this implementation, for the skill library represented by the skill model, the aforementioned execution entity can determine the correspondence between the basic skills in the skill library and multiple sub-tasks; then, for each sub-task, the initial parameters of the sub-task corresponding to the sub-task are determined by the basic skills corresponding to the sub-task.

[0105] Then, based on the operation scenario corresponding to the target task, the initial operation parameters of each of the multiple subtasks are adjusted to obtain the operation parameters of each of the multiple subtasks.

[0106] Subtasks corresponding to the same basic skill may correspond to different operational scenarios. For example, opening a refrigerator door and opening a cabinet door correspond to the basic skill of a robotic arm opening objects, but opening a refrigerator door and opening a cabinet door are for different objects and are different operational scenarios, requiring different amounts of force. Therefore, after obtaining the initial operational parameters of the subtask, it is necessary to adjust the initial operational parameters of the subtask according to the specific operational scenario corresponding to the subtask to obtain the subtask's specific operational parameters.

[0107] In this implementation, after obtaining the initial operation parameters of the subtask, the parameters are adjusted based on the operation scenario, which improves the accuracy of the subtask operation parameters and enables the skill model to adapt to various operation scenarios.

[0108] In some implementations of this embodiment, the aforementioned execution subject can be trained to obtain a skill model in the following manner:

[0109] First, obtain the second sample set.

[0110] The second sample in the second sample set includes motion sequence data corresponding to a skill in the skill library. During the process of the robot being controlled to demonstrate the skill, the motion sequence data is collected by various sensors set on the robot.

[0111] In this implementation, data is collected for each basic skill in a specific operational scenario. Specific scenarios are set up in simulated or real-world environments to ensure the authenticity and diversity of the data collected. For each basic skill, the robot is manually controlled to perform the corresponding operation in the specific scenario, and various data such as motion trajectory, force feedback, and visual information are recorded using sensors and cameras on the robot. Then, data preprocessing is performed to remove noise and abnormal data present during the collection process, ensuring data quality. Finally, key features such as hand trajectory, grasping point position, and object state are extracted from the preprocessed data to obtain motion sequence data.

[0112] Then, using the initial skill model as the learning object, the action sequence data is used for imitation learning to obtain the imitation learning model.

[0113] Imitation learning, also known as behavioral cloning, primarily learns how to perform tasks by observing action sequence data. It can quickly learn from large amounts of data and directly map inputs to outputs, similar to supervised learning.

[0114] Finally, the imitation learning model is optimized using reinforcement learning algorithms to obtain the skill model.

[0115] Reinforcement learning is a method that teaches optimal policies through interaction between an agent and its environment. The agent performs actions within the environment and adjusts its behavioral strategies based on rewards (or punishments) received from the environment, aiming to maximize long-term cumulative rewards. The fundamental principle of reinforcement learning can be summarized as "trial and error learning," where the agent continuously tries, observes the results, and adjusts its strategies to find behavioral patterns that maximize rewards.

[0116] In this implementation, a mimicry learning model is used as the agent to perform reinforcement learning.

[0117] In this implementation, the skill model is obtained by combining imitation learning and reinforcement learning, which improves the accuracy of the skill model and its adaptability to the real 3D environment.

[0118] Referring again to Figure 4, a schematic flow 400 of another embodiment of the robot control method according to the present disclosure is shown, including the following steps:

[0119] Step 401: Using a pre-trained multimodal large model, determine the pose data of the target object corresponding to the target task based on the multimodal environment data.

[0120] Step 402: Using a pre-trained trajectory generation model, determine the robot's trajectory from its current position to the target object based on the pose data.

[0121] Step 403: Control the robot to run to the target object according to the running trajectory.

[0122] Step 404: Decompose the target task using a multimodal large model to obtain a task sequence that includes multiple sub-tasks.

[0123] Step 405: Using the skill model, generate the subtask operation parameters corresponding to each of the multiple subtasks in the task sequence, and obtain the operation parameters including the operation parameters of multiple subtasks.

[0124] Step 406: Control the robot to execute the target task according to the operating parameters.

[0125] As can be seen from this embodiment, compared with the embodiment corresponding to Figure 2, the robot control method flow 400 in this embodiment specifically illustrates the robot control process by combining a multimodal large model, a trajectory generation model, and a skill model, which further improves the robot's ability to execute long sequence tasks and makes the robot more adaptable to complex scenarios.

[0126] Referring again to Figure 5, as an implementation of the methods shown in the above figures, this disclosure provides an embodiment of a robot control device, which corresponds to the method embodiment shown in Figure 2, and the device can be specifically applied to various electronic devices.

[0127] As shown in Figure 5, a robot control device includes: a pose determination unit 501, configured to determine the pose data of a target object corresponding to a target task based on multimodal environment data using a pre-trained multimodal large model; a trajectory determination unit 502, configured to determine the robot's running trajectory from its current position to the target object based on the pose data using a pre-trained trajectory generation model; a parameter determination unit 503, configured to determine the operation parameters corresponding to the target task using a pre-trained skill model; and a control unit 504, configured to control the robot to run according to the running trajectory and operation parameters.

[0128] In some implementations of this embodiment, the above-mentioned apparatus further includes: a task decomposition unit (not shown in the figure) configured to decompose the target task through a multimodal large model to obtain a task sequence including multiple sub-tasks; and a parameter determination unit 503 further configured to: generate sub-task operation parameters corresponding to each of the multiple sub-tasks in the task sequence through a skill model to obtain operation parameters, wherein the operation parameters include multiple sub-task operation parameters.

[0129] In some implementations of this embodiment, the parameter determination unit 503 is further configured to: generate initial operation parameters for each of the multiple sub-tasks through the skill model; and adjust the initial operation parameters for each of the multiple sub-tasks according to the operation scenario corresponding to the target task, thereby obtaining the operation parameters for each of the multiple sub-tasks.

[0130] In some implementations of this embodiment, the above-mentioned running trajectory includes a movement trajectory indicating the movement process of the robot's mobile device from its current position to the target object and an obstacle avoidance trajectory indicating the robot's robotic arm to avoid obstacles during the movement. The trajectory determination unit 502 is further configured to generate the movement trajectory and the obstacle avoidance trajectory by means of the trajectory generation model, based on the pose data and the environmental feature data obtained by the multimodal large model based on the multimodal environmental data.

[0131] In some implementations of this embodiment, the trajectory generation model is trained as follows: a first sample set is obtained, wherein the first sample in the first sample set includes first environment data and first trajectory data. The first trajectory data is used to characterize the robot's running trajectory from its current position to the target sample object in the three-dimensional environment corresponding to the first environment data. A machine learning algorithm is used to train the trajectory generation model with the start and end positions corresponding to the first environment feature data and the first trajectory data as inputs and the first trajectory data as the desired output. The first environment feature data is obtained by a multimodal large model based on the first environment data in the first sample where the first trajectory data is located.

[0132] In some implementations of this embodiment, the first trajectory data is obtained as follows: during the controlled operation of the robot in the three-dimensional environment from its current position to the target sample object, the robot collects the movement trajectory corresponding to the mobile device and the obstacle avoidance trajectory corresponding to the robotic arm through various sensors set on the robot; the movement trajectory and obstacle avoidance trajectory are aligned in the spatial and temporal dimensions to obtain the first trajectory data.

[0133] In some implementations of this embodiment, the skill model is trained as follows: a second sample set is obtained, wherein the second sample in the second sample set includes action sequence data corresponding to a skill in the skill library. During the process of the robot being controlled to demonstrate the skill, the action sequence data is collected by various sensors set on the robot; the initial skill model is used as the learning object, and the action sequence data is used for imitation learning to obtain an imitation learning model; the imitation learning model is optimized using a reinforcement learning algorithm to obtain the skill model.

[0134] In some implementations of this embodiment, the multimodal large model is trained in the following way: a third training set is obtained, wherein the third samples in the third sample set include third environment data and pose labels of target sample objects in the three-dimensional environment represented by the third environment data; a machine learning method is used to train the multimodal large model with the third environment data as input and the pose labels corresponding to the third environment data as the expected output.

[0135] In this embodiment, the pose determination unit in the robot control device determines the pose data of the target object corresponding to the target task based on multimodal environmental data using a pre-trained multimodal large model; the trajectory determination unit determines the robot's running trajectory from its current position to the target object based on the pose data using a pre-trained trajectory generation model; the parameter determination unit determines the operation parameters corresponding to the target task using a pre-trained skill model; and the control unit controls the robot to run according to the running trajectory and operation parameters. Thus, by combining the multimodal large model, the trajectory generation model, and the skill model, the robot's running trajectory from its current position to the target object and the operation parameters for the target task are generated to control the robot's operation, thereby improving the robot's ability to execute long sequence tasks and enabling the robot to adapt to complex scenarios.

[0136] Referring now to FIG6, a schematic diagram of the structure of a computer system 600 suitable for implementing devices (such as devices 101, 102, 103, 105 shown in FIG1) of the embodiments of the present disclosure is shown. The device shown in FIG6 is merely an example and should not impose any limitation on the functionality and scope of use of the embodiments of the present disclosure.

[0137] As shown in Figure 6, the computer system 600 includes a processor (e.g., CPU, Central Processing Unit) 601, which can perform various appropriate actions and processes based on programs stored in read-only memory (ROM) 602 or programs loaded from storage section 608 into random access memory (RAM) 603. RAM 603 also stores various programs and data required for the operation of system 600. The processor 601, ROM 602, and RAM 603 are interconnected via bus 604. Input / output (I / O) interface 605 is also connected to bus 604.

[0138] The following components are connected to I / O interface 605: an input section 606 including a keyboard, mouse, etc.; an output section 607 including a cathode ray tube (CRT), liquid crystal display (LCD), etc., and speakers, etc.; a storage section 608 including a hard disk, etc.; and a communication section 609 including a network interface card such as a LAN card, modem, etc. The communication section 609 performs communication processing via a network such as the Internet. A drive 610 is also connected to I / O interface 605 as needed. A removable medium 611, such as a disk, optical disk, magneto-optical disk, semiconductor memory, etc., is installed on drive 610 as needed so that computer programs read from it can be installed into storage section 608 as needed.

[0139] In particular, according to embodiments of this disclosure, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments of this disclosure include a computer program product comprising a computer program carried on a computer-readable medium, the computer program containing program code for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via communication section 609, and / or installed from removable medium 611. When the computer program is executed by processor 601, it performs the functions defined above in the methods of this disclosure.

[0140] It should be noted that the computer-readable medium of this disclosure may be a computer-readable signal medium or a computer-readable storage medium, or any combination thereof. A computer-readable storage medium may be, for example,—but not limited to—an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of a computer-readable storage medium may include, but are not limited to: an electrical connection having one or more wires, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination thereof. In this disclosure, a computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, apparatus, or device. In this disclosure, a computer-readable signal medium may include a data signal propagated in baseband or as part of a carrier wave, carrying computer-readable program code. Such propagated data signals may take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. Computer-readable signal media can also be any computer-readable medium other than computer-readable storage media, which can send, propagate, or transmit a program for use by or in connection with an instruction execution system, apparatus, or device. The program code contained on the computer-readable medium can be transmitted using any suitable medium, including but not limited to: wireless, wire, optical fiber, RF, etc., or any suitable combination thereof.

[0141] Computer program code for performing the operations of this disclosure can be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, and C++, as well as conventional procedural programming languages ​​such as "C" or similar programming languages. The program code can be executed entirely on the client computer, partially on the client computer, as a standalone software package, partially on the client computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving a remote computer, the remote computer can be connected to the client computer via any type of network—including a local area network (LAN) or a wide area network (WAN)—or can be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0142] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of apparatus, methods, and computer program products according to various embodiments of the present disclosure. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, may be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0143] The units described in the embodiments of this disclosure can be implemented in software or hardware. The described units can also be housed in a processor; for example, it can be described as: a processor including a pose determination unit, a trajectory determination unit, a parameter determination unit, and a control unit. The names of these units do not necessarily limit the unit itself; for example, the pose determination unit can also be described as "a unit that determines the pose data of a target object corresponding to a target task based on multimodal environment data using a pre-trained multimodal large model."

[0144] In another aspect, this disclosure also provides a computer-readable medium, which may be included in the device described in the above embodiments; or it may exist independently and not assembled into the device. The computer-readable medium carries one or more programs that, when executed by the device, cause the computer device to: determine the pose data of the target object corresponding to the target task based on multimodal environment data using a pre-trained multimodal large model; determine the robot's running trajectory from its current position to the target object based on the pose data using a pre-trained trajectory generation model; determine the operation parameters corresponding to the target task using a pre-trained skill model; and control the robot to run according to the running trajectory and operation parameters.

[0145] The above description is merely a preferred embodiment of this disclosure and an explanation of the technical principles employed. Those skilled in the art should understand that the scope of the invention involved in this disclosure is not limited to technical solutions formed by specific combinations of the above-described technical features, but should also cover other technical solutions formed by arbitrary combinations of the above-described technical features or their equivalents without departing from the above-described inventive concept. For example, technical solutions formed by substituting the above-described features with (but not limited to) technical features of this disclosure that have similar functions.

Claims

1. A robot control method, comprising: By using a pre-trained multimodal large model, the pose data of the target object corresponding to the target task can be determined based on multimodal environmental data. Based on the pose data, the pre-trained trajectory generation model determines the robot's trajectory from its current position to the target object. The operational parameters corresponding to the target task are determined by using a pre-trained skill model; The robot is controlled to run according to the stated trajectory and operating parameters.

2. The method according to claim 1, wherein, Also includes: The target task is decomposed using the multimodal large model to obtain a task sequence including multiple sub-tasks; as well as The process of determining the operational parameters corresponding to the target task through a pre-trained skill model includes: Using the skill model, subtask operation parameters corresponding to each of the multiple subtasks in the task sequence are generated, and the operation parameters are obtained, wherein the operation parameters include multiple subtask operation parameters.

3. The method according to claim 2, wherein, The step of generating subtask operation parameters corresponding to each of the multiple subtasks in the task sequence through the skill model includes: The skill model is used to generate the initial operation parameters for each of the multiple sub-tasks. Based on the operation scenario corresponding to the target task, the initial operation parameters of each of the multiple subtasks are adjusted to obtain the operation parameters of each of the multiple subtasks.

4. The method according to claim 1, wherein, The operating trajectory includes a movement trajectory indicating the movement of the robot's mobile device from its current position to the target object, and an obstacle avoidance trajectory indicating that the robot's robotic arm should avoid obstacles during the movement. The pre-trained trajectory generation model determines the robot's trajectory from its current position to the target object based on the pose data, including: The trajectory generation model generates the movement trajectory and the obstacle avoidance trajectory based on the pose data and the environmental feature data obtained by the multimodal large model based on the multimodal environment data.

5. The method according to claim 4, wherein, The trajectory generation model is trained in the following manner: A first sample set is obtained, wherein the first sample in the first sample set includes first environmental data and first trajectory data, and the first trajectory data is used to characterize the robot's running trajectory from its current position to the target sample object in the three-dimensional environment corresponding to the first environmental data. The trajectory generation model is trained using a machine learning algorithm, with the start and end positions corresponding to the first environmental feature data and the first trajectory data as inputs and the first trajectory data as the desired output. The first environmental feature data is obtained by the multimodal large model based on the first environmental data in the first sample where the first trajectory data is located.

6. The method according to claim 5, wherein, The first trajectory data is obtained in the following way: During the controlled operation of the robot in the three-dimensional environment from its current position to the target sample object, the robot collects the movement trajectory of the mobile device and the obstacle avoidance trajectory of the robotic arm through various sensors installed on the robot. The first trajectory data is obtained by aligning the movement trajectory and the obstacle avoidance trajectory in the spatial and temporal dimensions.

7. The method according to any one of claims 1-4, wherein, The skill model was trained in the following manner: A second sample set is obtained, wherein the second sample in the second sample set includes action sequence data corresponding to a skill in the skill library. During the process of the robot being controlled to demonstrate the skill, the action sequence data is collected by multiple sensors set on the robot. Using the initial skill model as the learning object, the action sequence data is subjected to imitation learning to obtain an imitation learning model; The skill model is obtained by optimizing the imitation learning model using a reinforcement learning algorithm.

8. The method according to claim 1, wherein, The multimodal large model is trained in the following way: Obtain a third training set, wherein the third sample in the third sample set includes third environment data and pose labels of target sample objects in the three-dimensional environment represented by the third environment data; The multimodal large model is trained using machine learning methods, with the third environment data as input and the pose label corresponding to the third environment data as the expected output.

9. A robot control device, comprising: The pose determination unit is configured to determine the pose data of the target object corresponding to the target task based on the multimodal environment data through a pre-trained multimodal large model. The trajectory determination unit is configured to determine the robot's running trajectory from its current position to the target object based on the pose data using a pre-trained trajectory generation model. The parameter determination unit is configured to determine the operational parameters corresponding to the target task through a pre-trained skill model; The control unit is configured to control the robot to run according to the running trajectory and the operating parameters.

10. A computer-readable medium having a computer program stored thereon, wherein, When the program is executed by the processor, it implements the method as described in any one of claims 1-8.

11. An electronic device, comprising: One or more processors; Storage device, on which one or more programs are stored, When the one or more programs are executed by the one or more processors, the one or more processors implement the method as described in any one of claims 1-8.

12. A computer program product, comprising: A computer program that, when executed by a processor, implements the method according to any one of claims 1-8.

Images