Control method and training method, control assembly, and robot

Abstract

A control method and a training method, a control assembly, and a robot, the control method comprising: on the basis of an image of an environment in which a robot is located, determining a first interaction pose for an object to be interacted with; on the basis of an output instruction of the previous time step and the first interaction pose, generating a combined feature of a current time step; inputting the combined feature of the current time step and a combined feature of at least one historical time step into a trained attribute prediction module to obtain an attribute feature of the object to be interacted with, the attribute feature being used for representing attribute data affecting interaction with the robot; and inputting the attribute feature and the combined feature of the current time step into a trained reinforcement learning module to obtain an output instruction of the current time step.

Description

Control and training methods, control components and robots

[0001] This application claims priority to Chinese Patent Application No. 202411523655.7, filed on October 29, 2024, entitled "Control Method, Training Method, Control Component and Robot", the entire contents of which are incorporated herein by reference. Technical Field

[0002] This application relates to the field of robot control technology, and in particular to a control and training method, control components and robot. Background Technology

[0003] Manipulating articulated objects with robots is a challenging task, requiring a deep understanding of the object and compliant movement to avoid damage to both the object and the robot. Related technologies utilize point cloud input to predict availability maps, then use reinforcement learning strategies to predict manipulation actions, or use multi-view RGB (Red, Green, Blue, RGB) images / partial point clouds to estimate component poses, and execute planned actions through a heuristic manipulation module. However, encountering unfamiliar objects will affect the robot's generalization ability and execution success rate. Summary of the Invention

[0004] This application provides a control and training method, a control component, and a robot, implemented in the following manner:

[0005] On one hand, embodiments of this application provide a control method, including: determining a first interaction pose for an object to be interacted with based on an image of the environment in which the robot is located; generating a splicing feature for the current time step based on the output command of the previous time step and the first interaction pose; inputting the splicing feature of the current time step and the splicing feature of at least one historical time step into a trained attribute prediction module to obtain attribute features of the object to be interacted with; the attribute features are used to characterize attribute data that affect the interaction with the robot; and inputting the attribute features and the splicing feature of the current time step into a trained reinforcement learning module to obtain the output command for the current time step.

[0006] On the other hand, embodiments of this application provide a training method for a control model, the control model including an attribute prediction module and a reinforcement learning module; the training method includes: generating simulation stitching features for the second time step based on simulation output commands at a first time step and observation data at a second time step; the first time step is the previous time step of the second time step; inputting the simulation stitching features of the second time step and simulation stitching features from at least one historical time step into the attribute prediction module to obtain predicted simulation attribute features; the simulation attribute features are used to characterize attribute data affecting the interaction between the robot and the simulation environment; inputting the simulation output commands and target attribute features of the second time step into the reinforcement learning module to obtain simulation output commands for the second time step; the target attribute features are the predicted simulation attribute features or simulation attribute features generated based on attribute data of sample objects; obtaining motion rewards after controlling the robot's movement based on the simulation output commands of the second time step; training the attribute prediction module based on the predicted simulation attribute features and the generated simulation attribute features, and training the reinforcement learning module through reinforcement learning based on the motion rewards.

[0007] In another aspect, embodiments of this application provide a control component applied to a robot. The control component includes: a determining unit, configured to determine a first interactive pose for an object to be interacted with based on an image of the robot's environment; a generating unit, configured to generate a stitched feature for the current time step based on the output command of the previous time step and the first interactive pose; a first input unit, configured to input the stitched feature of the current time step and stitched features of at least one historical time step into a trained attribute prediction module to obtain attribute features of the object to be interacted with; the attribute features are used to characterize attribute data affecting the interaction with the robot; and a second input unit, configured to input the attribute features and the stitched feature of the current time step into a trained reinforcement learning module to obtain the output command of the current time step.

[0008] In another aspect, embodiments of this application provide a robot including the aforementioned control components.

[0009] In another aspect, embodiments of this application provide a computer-readable storage medium storing a computer program that, when executed by a processor, implements the aforementioned control method or training method for the control model.

[0010] In another aspect, embodiments of this application provide a computer program product containing instructions, which, when run on a computer, causes the computer to execute the aforementioned control method or control model training method.

[0011] It is evident that the reinforcement learning module can consider the attribute data, interaction pose, and historical output commands of the object to be interacted with when outputting instructions, thereby reducing reliance on vision, better understanding the intrinsic characteristics of objects, having high generalization ability when dealing with unseen objects, exhibiting a high success rate in the real world, and reducing the gap between simulation and reality.

[0012] It should be understood that the above description and the detailed description below are merely exemplary and explanatory, and are not intended to limit the technical solutions of this application. Attached Figure Description

[0013] Figure 1 is a schematic diagram of the implementation process of a control method provided in an embodiment of this application;

[0014] Figure 2 is a schematic diagram of the implementation process of a control method provided in an embodiment of this application;

[0015] Figure 3 is a schematic diagram of the implementation process of a control method provided in an embodiment of this application;

[0016] Figure 4 is a schematic diagram of the implementation process of a control method provided in an embodiment of this application;

[0017] Figure 5 is a schematic diagram of the implementation process of a control method provided in an embodiment of this application;

[0018] Figure 6 is a schematic diagram of the implementation process of a training method for a control model provided in an embodiment of this application;

[0019] Figure 7 is a schematic diagram of the implementation process of a training method for a control model provided in an embodiment of this application;

[0020] Figure 8 is a schematic diagram of the application of reinforcement learning strategy provided in an embodiment of this application;

[0021] Figure 9 is a schematic diagram of the application of reinforcement learning strategies provided in the embodiments of this application;

[0022] Figure 10 is a schematic diagram of the experimental setup of an embodiment of this application;

[0023] Figure 11 is a schematic diagram of the experimental results of an embodiment of this application;

[0024] Figure 12 is a schematic diagram of the composition structure of a control component provided in an embodiment of this application;

[0025] Figure 13 is a schematic diagram of the composition structure of a training device provided in an embodiment of this application;

[0026] Figure 14 is a schematic diagram of the hardware entity of a computer device provided in an embodiment of this application. Detailed Implementation

[0027] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. Obviously, the described embodiments are only a part of the embodiments of this invention, and not all of them. All other embodiments obtained by those skilled in the art based on the embodiments of this invention are within the scope of protection of this invention.

[0028] In the following description, references to "some embodiments" refer to a subset of all possible embodiments, specifically the same or different subsets of all possible embodiments, which can be combined with each other without conflict. The terms "first / second / third" are used only to distinguish similar objects and do not represent a specific ordering of objects; that is, the order of "first / second / third" can be interchanged where permitted, so that the technical solutions described herein can be implemented in a sequence other than that illustrated or described.

[0029] Unless otherwise defined, the technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application pertains. The terminology used herein is for descriptive purposes only and is not intended to limit the scope of this application.

[0030] The technical solutions provided in this application can be executed by the processor of a computer device. The computer device can be a server, laptop computer, tablet computer, desktop computer, smart TV, set-top box, mobile device (e.g., mobile phone, portable video player, personal digital assistant, dedicated messaging device, portable gaming device), or any other device with data processing capabilities.

[0031] In the field of robot control, for manipulating articulated objects, the first method uses point cloud input to predict the availability graph and then uses reinforcement learning to predict the operation. However, using point cloud features as policy input leads to a significant gap between simulation and reality, and the availability graph is often occluded by the robot arm during execution. The second method uses multi-view RGB images to estimate the part's pose and executes the planned action through a heuristic operation module. This requires more time for part pose estimation, and the predicted waypoints need adjustment at each local point, resulting in an uneven trajectory. The third method uses partial point cloud data to estimate the part's pose and executes the planned action through a heuristic operation module, but it lacks a feedback mechanism to adjust the planned action, thus ignoring the object's intrinsic characteristics and easily leading to unsafe behaviors. Therefore, when dealing with unseen objects, the above methods lack high generalization ability and are unlikely to achieve high success rates in the real world.

[0032] Taking the control method applied to a robot as an example, the method includes: determining the first interaction pose for the object to be interacted with based on the environmental image of the robot; generating the stitched features of the current time step based on the output command of the previous time step and the first interaction pose; inputting the stitched features of the current time step and the stitched features of at least one historical time step into a trained attribute prediction module to obtain attribute features; the attribute features are used to characterize the attribute data of the object to be interacted that are related to the interaction; and inputting the attribute features and the stitched features of the current time step into a trained reinforcement learning module to obtain the output command of the current time step. In this way, the reinforcement learning module can consider the attribute data of the object to be interacted that are related to the interaction, the interaction pose, and historical output commands when outputting commands, reducing reliance on vision, better understanding the intrinsic characteristics of objects, exhibiting high generalization ability when handling unseen objects, demonstrating a high success rate in the real world, and reducing the gap between simulation and reality.

[0033] A hinged object is an object composed of multiple movable parts connected by hinges, slides, or other joints. Typical hinged objects include doors, drawers, and robotic arms. The connection between the joints allows the object to perform certain movements, such as rotation and sliding. Reinforcement learning is a machine learning method that optimizes action strategies through interaction with the environment and a trial-and-error process to maximize cumulative rewards. A reinforcement learning strategy refers to the action selection rules that the reinforcement learning algorithm continuously adjusts during training to make optimal decisions in different situations. A point cloud is a three-dimensional data representation where each point has its own three-dimensional coordinates (X, Y, Z). Point clouds are typically acquired using devices such as LiDAR or RGB-D cameras and are widely used in tasks such as 3D modeling, object detection, and scene understanding.

[0034] Figure 1 is a schematic diagram of the implementation process of a robot control method provided in an embodiment of this application, including steps S101 to S104:

[0035] Step S101: Determine the first interaction pose for the object to be interacted with based on the environmental image of the robot.

[0036] The environmental image must at least contain the object to be interacted with, and can be obtained by taking pictures of the environment in which the robot is located through image acquisition modules such as cameras carried by the robot.

[0037] The object to be interacted with is the object that the robot operates on. Taking the robot performing the task of opening a door as an example, the object to be interacted with is the door handle that is grabbed during the opening process.

[0038] The first interactive pose is the position and orientation that the robot's end effector (such as a gripper or suction cup) is expected to reach when performing an interactive task. The robot's interactive operation on the object to be interacted with can be a grasping operation, a lifting operation, a translation operation, a pushing and pulling operation, etc., which are not limited here.

[0039] The environmental image includes depth images and RGB images. Machine learning is used to perform object recognition on the RGB images to obtain a mask image of the object to be interacted with. The depth image is then transformed to obtain environmental point cloud data. Multiple second interaction poses are determined using the environmental point cloud data. Finally, the mask image is used to determine the first interaction pose for the object to be interacted with from among these multiple second interaction poses. Specifically, the depth image and RGB image are input into a pre-trained interaction pose determination module to perform object recognition on the RGB image, determine the region where the object to be interacted is located, and then generate a mask image of the object to be interacted with. The interaction pose determination module transforms the depth image to obtain environmental point cloud data. Based on the pre-learned relationship between the environmental point cloud data and the interaction poses, it performs grasping prediction on the environmental point cloud data corresponding to the object to be interacted with, generating multiple second interaction poses. The second interaction pose includes three-dimensional position coordinates and quaternion poses. The position coordinates represent the position where the robot interacts with the object, and the quaternion pose represents the posture of the robot's end effector when interacting with the object. Since a mask image can represent the spatial position of the object to be interacted with in the environment, and the robot's spatial position is relatively close to that of the object when interacting with it, the interaction pose determination module can use the mask image to determine the first interaction pose for the object to be interacted with from multiple second interaction poses. This first interaction pose is close to the spatial position of the object to be interacted with.

[0040] A model is pre-trained to generate the first interactive pose. This model can identify the interactive object in the environment image and generate the first interactive pose for the interactive object. The training samples can be multiple environment images containing the interactive object, and the labels can be the interactive object in the environment image and the corresponding first interactive pose. The trained model can identify the interactive object in the environment image and generate the first interactive pose for the interactive object.

[0041] Step S102: Based on the output command of the previous time step and the first interactive pose, generate the splicing features of the current time step.

[0042] The robot's entire motion execution process is divided into multiple time steps. At each time step, the corresponding action is executed based on the output instructions of the reinforcement learning module. The output instructions contain information related to the robot's action, such as target displacement, target orientation, end effector action, and impedance control parameters. Target displacement is the target position that the robot needs to move to. Target orientation refers to the robot's complete orientation or posture information in three-dimensional space relative to a reference coordinate system (such as the world coordinate system or camera coordinate system). End effector action refers to whether the robot performs a grasping action on an object. Impedance control parameters are used to control the impedance of the output action.

[0043] The stitched features of the current time step include the output instructions from the previous time step and the observation data from the current time step. Observation data refers to the environmental state information of the robot's environment. The stitched features are used to characterize the robot's historical actions and the current environmental state.

[0044] The observed data includes the initial interaction pose, robot joint configuration, relative distance between the robot and the object, end effector pose, and graspability signal. Robot joint configuration refers to the selection and configuration of the robot's joint type, number, arrangement, and drive method during the production process. The relative distance between the robot and the object refers to the distance between the robot's end effector and the object to be interacted with. The end effector pose refers to the position and orientation of the end effector in a specified coordinate system. The graspability signal indicates whether the robot's end effector grasps the object; this is based on distance and contact perception, not direct commands controlling the opening and closing of end effectors such as grippers.

[0045] At each time step, the robot acquires the observation data of the current time step (including the first interaction pose), combines the observation data of the current time step with the output instructions of the previous time step, and obtains the splicing features of the current time step.

[0046] Step S103: Input the splicing features of the current time step and the splicing features of at least one historical time step into the trained attribute prediction module to obtain the attribute features of the object to be interacted with; the attribute features are used to characterize the attribute data that affect the interaction between the robot and the user.

[0047] The attribute prediction module is used to infer the attribute characteristics of the object to be interacted with from historical observation data and output instructions.

[0048] To better understand the environment, it is necessary to obtain attribute data that influences the interaction between the robot and the environment. However, this data is difficult to obtain directly in the real world. Therefore, an attribute prediction module is trained. The concatenated features of the current time step and at least one historical time step are input into the trained attribute prediction module to obtain attribute features. These attribute features can approximately reflect the attribute data that influences the interaction between the robot and the environment. In this way, the attribute prediction module generates attribute features corresponding to the concatenated features of the current time step and at least one historical time step, based on the mapping relationship from multi-time step concatenated features to attribute features learned in advance from the simulation environment.

[0049] At least one historical time step is one or more preceding historical time steps that are adjacent to the current time step.

[0050] Attribute data includes at least one of the following: pivot center, pivot radius, object stiffness, object mass, object joint position, and handle grip signal. The pivot center refers to the center point or axis of rotation of an object or system. The pivot radius is the distance from the pivot center to a point on the rotating object. Object stiffness refers to the object's ability to resist deformation under external forces. Object mass is a physical quantity describing the magnitude of an object's inertia; it is a measure of the amount of matter contained in the object. Object joint position is the current orientation of the object's joints. The handle grip signal indicates whether the handle is currently gripped.

[0051] During simulation, at any time step, the simulation software can directly acquire the attribute data of the object to be interacted with. Based on this attribute data, simulation attribute features can be generated. The attribute prediction module can obtain the predicted simulation attribute features from the concatenated features of the current time step and the concatenated features of at least one historical time step; that is, the predicted simulation attribute features of the sample object to be interacted with. Based on the difference between the generated simulation attribute features and the predicted simulation attribute features, the parameters of the attribute prediction module are adjusted to obtain the trained attribute prediction module. In application, the concatenated features of the current time step and the concatenated features of at least one historical time step are input into the trained attribute prediction module to obtain the attribute features.

[0052] Step S104: Input the attribute features and the concatenated features of the current time step into the trained reinforcement learning module to obtain the output instructions of the current time step.

[0053] For tasks involving robots manipulating objects, a reinforcement learning module is trained. Based on the current environmental state and historical actions, the module outputs the actions the robot should perform at each time step. The robot then executes the actions based on the instructions output by the reinforcement learning module to complete the task.

[0054] At each time step, the reinforcement learning module determines the output command for the current time step based on the attribute features output by the attribute prediction module and the concatenated features of the current time step. The robot can then execute actions based on the output command.

[0055] During the simulation training of the reinforcement learning module, at each time step, a generated simulation attribute feature and a predicted simulation attribute feature are obtained. The generated or predicted simulation attribute feature at the current time step, along with the concatenated feature, is input into the reinforcement learning module to obtain an output command. The robot receives a motion reward after executing the output command. Based on the motion reward, the parameters of the reinforcement learning module are adjusted to obtain the trained reinforcement learning module. In application, the attribute feature and the concatenated feature from the current time step are input into the trained reinforcement learning module to obtain the output command for the current time step.

[0056] For example, when a robot performs a door-opening task, the object to be interacted with is the door handle. Based on the environmental image, the first interaction pose g for the door handle is determined. t Obtain the observation index o at the current time step t. t (including the first interactive pose g) t The output instruction a at time step t-1. t-1 Compared with the observed indicators o t By splicing, the splicing feature p is obtained. t =(o t ⊕a t-1 ), splicing feature p t splicing features p at time step t-1 t-1 Input the trained attribute prediction module to obtain the predicted attribute features. Predicted attribute features The concatenation feature p of the current time step t Input the trained reinforcement learning module to obtain the output instruction 'a' at the current time step. t .

[0057] In this way, the reinforcement learning module can consider the attribute data of the object to be interacted with, the interaction pose, and the historical output instructions when outputting instructions, reducing the dependence on vision, better understanding the intrinsic characteristics of the object, having a high generalization ability when dealing with unseen objects, showing a high success rate in the real world, and reducing the gap between simulation and reality.

[0058] The post-trained attribute prediction module and the post-trained reinforcement learning module are obtained by training with the attribute data of the sample objects to be interacted with in the simulation environment. This attribute data is the attribute data that affects the interaction with the robot.

[0059] At any time step in the training process, the attribute prediction module adjusts its parameters based on the difference between the generated simulated attribute features and the simulated attribute features predicted by the attribute prediction module at that time step; the generated simulated attribute features are obtained based on the attribute data of the sample objects.

[0060] The reinforcement learning module adjusts parameters based on motion rewards, which are obtained after the robot moves in response to simulation output commands. The simulation output commands are output by the reinforcement learning module at any given time step, based on the generated or predicted simulation attribute features.

[0061] The attribute prediction module needs to be trained in the simulation environment. At any time step, the simulation environment can directly acquire attribute data affecting the interaction between the robot and the target object. Based on the acquired attribute data of the sample objects, simulation attribute features are generated. The concatenated features of the current time step and at least one historical time step are input into the attribute prediction module to obtain the predicted simulation attribute features. Based on the difference between the generated simulation attribute features and the simulation attribute features predicted by the attribute prediction module, the parameters of the attribute prediction module are adjusted.

[0062] For example, the attribute data of the sample object The generated simulation attribute features z are obtained by inputting them into the encoding module, which can be a multilayer perceptron (MLP). The concatenated features from the current time step and the concatenated features from at least one historical time step are then input into the attribute prediction module to obtain the predicted simulation attribute features. based on The parameters of the attribute prediction module are adjusted, where sg[.] represents the stopping gradient operator and λ is a preset parameter. This means the attribute prediction module outputs the predicted simulation attribute features. As the true value, the simulated attribute feature z generated by the encoding module is used as the predicted value. The parameters of the encoding module are adjusted by gradient descent to reduce the difference between the predicted value and the true value until the encoding module meets the convergence condition, thus obtaining the trained encoding module.

[0063] The reinforcement learning module needs to be trained in a simulation environment. At any time step, the simulation environment directly acquires attribute data affecting the interaction between the robot and the target object. Based on this acquired attribute data, simulation attribute features can be generated. The simulation attribute features generated at the current time step, along with the concatenated features, are input into the reinforcement learning module, or the simulation attribute features predicted at the current time step are input into the module. The reinforcement learning module then outputs simulation output commands. After the robot responds to the simulation output command, it receives a motion reward. The reinforcement learning module adjusts its parameters based on this reward. In other words, during the training process, the reinforcement learning module learns output commands that yield higher motion rewards. For example, if a certain output command corresponds to a higher motion reward, the reinforcement learning module is more likely to generate this type of output command in similar environmental conditions; conversely, it reduces the probability of generating this type of output command.

[0064] By training the attribute prediction module and the reinforcement learning module with attribute data that affects the interaction between the robot and the simulation environment, the trained attribute prediction module can accurately predict the attribute characteristics of the object to be interacted with in the current time step, and the reinforcement learning module can determine the output command of the current time step based on the intrinsic attributes of the object to be interacted with, thus better understanding the intrinsic characteristics of the object and showing a high success rate in the real world.

[0065] Figure 2 is a schematic flowchart illustrating the implementation of a control method according to an embodiment of this application. This method can be executed by the processor of a computer device. Based on Figure 1, it also includes steps S201 and S202.

[0066] Step S201: Store the splicing features of the current time step in the history cache.

[0067] After acquiring the splicing features at any time step, these features can be stored in the history cache. The history cache stores the splicing features of all historical time steps, containing a preset number of the most recent splicing features, such as the splicing features of the last 10 historical time steps. After obtaining the splicing features of the current time step, the splicing features in the history cache are updated based on these features. Specifically, using a "first-in, first-out" storage method, the splicing features of the current time step are stored in the history cache, and the oldest splicing feature is removed from the history cache.

[0068] Step S202: Obtain the splicing features of at least one historical time step from the historical cache.

[0069] Based on the requirements of the attribute prediction module, at least one historical time step's spliced ​​features can be obtained by reading the historical cache.

[0070] In this way, the splicing features of historical time steps can be effectively stored and accessed through historical caching, thereby improving the access efficiency of splicing features.

[0071] In some embodiments, the method further includes scaling the output command to obtain a target control command.

[0072] The output instructions obtained by the reinforcement learning module may be non-standard action instructions, making it difficult for the robot to execute actions based on them. Therefore, it is necessary to scale the output instructions to obtain target control instructions, enabling the robot to execute actions based on these instructions. The adjusted or scaled target control instructions can be executed safely and effectively by the robot, which may involve adjusting parameters such as the speed, force, and range of the action. The scaled actions are then converted into "robot instructions" that the robot can understand and execute, and sent to the robot's control system to drive the robot to perform the corresponding actions.

[0073] In some embodiments, the output command can be scaled using a formula, i.e., the target control command c t The calculation formula is formula (1): c t =clip(a t ,-1,1)*40+100 (1);

[0074] Among them, a t The `clip` function is a range limiting function for output commands, restricting the output commands to the range [-1, 1]. That is, by using a scaling factor of 40 and an offset of 100, the value of the target control command is limited to between 60 and 140.

[0075] The output command is scaled by a pre-trained scaling module. The output command is then input into the scaling module, and the target control command is output. In this way, target control commands that the robot can accurately recognize can be obtained.

[0076] In some embodiments, the method further includes: controlling the interactive device based on the target control command to execute the interactive action at the current time step.

[0077] The robot interacts with the object to be interacted with through an interactive device, which can be the robot's end effector, such as a gripper.

[0078] The target control instructions include data related to the interactive device performing interactive actions, specifically action instructions that the robot can understand and effectively execute. For example, the target control instructions include at least one of the following: target displacement, target posture, gripper action, and impedance control parameters.

[0079] In this way, the interactive device can accurately execute interactive actions.

[0080] In some embodiments, the method further includes: generating the robot's stiffness matrix based on output instructions; constructing a dynamic model for impedance control based on the robot's mass inertia matrix, damping matrix, and stiffness matrix; and using the dynamic model to control the balance between the robot's motion and contact forces.

[0081] Impedance control of robots is a force-based control method that treats the robot as a programmable spring-mass-damped system. It achieves interactive control between the robot and its environment by adjusting the robot's impedance (i.e., its resistance to external forces). Accordingly, impedance control is a control strategy that regulates the dynamic relationship between the robot and its environment (such as impedance parameters, including at least one of stiffness, damping, and inertia) to control the robot's interaction with the environment. This allows the robot to exhibit better adaptability and flexibility when interacting with its environment.

[0082] Stiffness represents a robot's ability to resist external deformation; adjusting stiffness controls the degree of deformation when the robot is subjected to external forces. Damping represents the friction or energy dissipation capacity between the robot and its environment; adjusting damping controls the robot's velocity changes when subjected to external forces. Inertia represents the robot's response speed to external forces; adjusting inertia controls the robot's acceleration changes when subjected to external forces.

[0083] Impedance-controlled dynamic models describe the dynamic response characteristics of a robot when subjected to external forces, i.e., the balance between the robot's motion state (position, velocity, acceleration) and the contact force. Adjusting the parameters of the matrices used to construct the dynamic model allows control over the robot's behavior in contact environments, such as maintaining stable contact forces and preventing excessive deformation.

[0084] The reinforcement learning module outputs stiffness as a command. Scaling this stiffness yields the target stiffness, and expanding the target stiffness provides the stiffness matrix. A dynamic model is constructed based on the adjusted stiffness matrix, the unadjusted mass-inertia matrix, and the damping matrix. For example, if the stiffness output by the reinforcement learning module is... The stiffness is converted into the target stiffness using formula (2):

[0085] The `clip` function is a range limiting function that limits the scope of a function's output. Constrained within [-1, 1]. Expand target stiffness. The stiffness matrix K is obtained.

[0086] For example, a dynamic model can be constructed based on the mass inertia matrix M, the damping matrix D, and the stiffness matrix K, as shown in formula (3):

[0087] Among them, F ext x is the external force generated by the interaction between the robot and its environment. dFor the expected trajectory, For the expected speed, For the expected acceleration, x c For impedance-controlled output trajectory For impedance-controlled acceleration, The speed is controlled by impedance. Based on the dynamic model, the robot's motion and contact force balance can be controlled by adjusting parameters such as position, speed, and acceleration.

[0088] The expected trajectory serves as a target or reference point to guide the robot's movement, while the output trajectory is the result of adjustments made by the robot during actual operation based on the expected trajectory and real-time environmental information.

[0089] Figure 3 is a schematic flowchart of a control method provided in an embodiment of this application. This method can be executed by the processor of a computer device. Based on Figure 1, the environmental image includes a depth image and a color image, and S101 can be implemented as S301 to S304.

[0090] Step S301: Perform object recognition on the color image to obtain the mask image of the object to be interacted with.

[0091] Depth images primarily represent the distance information of each point in the scene from the image acquisition device. The grayscale value or specific value of each pixel represents the distance from that point to the image acquisition device. Color images primarily represent the color information of objects in the scene. The RGB values ​​of each pixel represent the color components of that point, forming a rich and colorful image, such as a color image being an RGB image.

[0092] A mask image is a special image used to specify the area of ​​operation on the original image. Mask images can be binary (each pixel has only two possible values, such as 0 and 255), grayscale, or multi-channel. In a mask image, the region of interest is white (or a higher grayscale value), and the region of no interest is black (or a lower grayscale value).

[0093] The color image to be processed is input into a pre-trained image segmentation model. The model processes the input image and outputs the segmentation result. Based on the segmentation result, a binary mask image is generated. The object to be interacted with is the region of interest in the mask image. Its pixels can be white (or a specific value), and the remaining pixels can be black (or another specific value).

[0094] Step S302: Convert the depth image to obtain environmental point cloud data.

[0095] Environmental point cloud data is point cloud data collected from the surface of the object to be interacted with. It is a three-dimensional spatial data representation, consisting of a large number of three-dimensional spatial points, which are sampling points on the object's surface or specific locations in space. Each point contains three-dimensional coordinate information (X, Y, Z), and may also include other attribute information such as color, intensity, and reflectivity.

[0096] Acquire depth images and camera parameters, such as camera intrinsics (which describe the camera's internal properties, including focal length, principal point coordinates, image resolution, and distortion parameters) and camera extrinsic parameters (which describe the camera's pose in the world coordinate system). Camera intrinsics can be used to convert each pixel in the depth image into normalized planar coordinates; using the depth value, the corrected normalized planar coordinates, and the depth scaling factor, the coordinates of the point cloud in 3D space can be calculated.

[0097] Step S303: Use environmental point cloud data to determine multiple second interaction poses.

[0098] Environmental point cloud data reflects the surface morphology of objects and can represent multiple objects in the robot's environment. A pre-trained interaction pose determination module can determine the second interaction pose for each object in the environment. Specifically, the environmental point cloud data is input into the interaction pose determination module, which outputs multiple second interaction poses. That is, the interaction pose determination module transforms the depth image to obtain environmental point cloud data. Based on the pre-learned relationship between the environmental point cloud data and the interaction poses, it performs grasping prediction on the environmental point cloud data corresponding to the object to be interacted with, generating multiple second interaction poses.

[0099] Step S304: Determine the first interactive pose from multiple second interactive poses using the mask image.

[0100] A mask image represents the spatial location of an interactive object in the environment. Using the mask image, the first interactive pose corresponding to the interactive object can be determined from multiple second interactive poses. That is, the interactive pose determination module can map multiple second interactive poses onto the mask image, and determine the second interactive pose in the mask image that is closest to the area where the interactive object is located as the first interactive pose, so as to accurately determine the first interactive pose of the interactive object.

[0101] Figure 4 is a schematic flowchart illustrating the implementation of a control method provided in an embodiment of this application. This method can be executed by the processor of a computer device. Based on Figure 1, the training method for the attribute prediction module includes steps S401 to S404.

[0102] Step S401: Based on the simulation output command of the first time step and the observation data of the second time step, generate the simulation splicing feature of the second time step; the first time step is the previous time step of the second time step.

[0103] The simulation stitching feature represents the robot's historical executed actions and current environmental state. Historical executed actions are represented by simulation output commands at the first time step (including at least one of target displacement, target posture, gripper action, and impedance control parameters). The current environmental state is represented by observation data at the second time step (including at least one of first interaction pose, robot joint configuration, relative distance between the robot and the object, end effector pose, and graspability signal). Therefore, the simulation output commands at the first time step and the observation data at the second time step are combined into a data pair, which serves as the simulation stitching feature for the second time step, representing the latest environmental state and the most recent historical executed actions.

[0104] Step S402: Input the simulation stitching features of the second time step and the simulation stitching features of at least one historical time step into the attribute prediction module to obtain the predicted simulation attribute features; the simulation attribute features are used to characterize the attribute data that affect the interaction between the robot and the simulation environment.

[0105] The attribute prediction module is used to predict simulation attribute features from the simulation stitched features of at least two time steps. The aforementioned at least one historical time step includes the first time step.

[0106] Step S403: Input the simulation splicing features and target attribute features of the second time step into the reinforcement learning module to obtain the simulation output command of the second time step; the target attribute features are the predicted simulation attribute features or the simulation attribute features generated based on the attribute data of the sample objects.

[0107] The reinforcement learning module comprehensively considers the current environment state, historical actions, and attribute data of the object to be interacted with to determine the simulation output command. The current environment state and historical actions correspond to simulation splicing features, and the attribute data of the object to be interacted with corresponds to target attribute features. Therefore, the simulation output command for the second time step can be obtained based on step S403.

[0108] The predicted simulation attribute features are the attribute features output by the attribute prediction module, and the generated simulation attribute features are attribute features obtained based on the attribute data of the sample objects. Considering two different methods to determine the target attribute features, and inputting the simulation splicing features from the second time step and the target attribute features into the reinforcement learning module can improve the adaptability and generalization ability of the reinforcement learning module to different attribute features.

[0109] Inputting the attribute data of the sample object into the encoding module will produce the generated simulation attribute features. The encoding module can be trained based on the difference between the predicted simulation attribute features and the generated simulation attribute features.

[0110] Step S404: Train the attribute prediction module based on the predicted simulation attribute features and the generated simulation attribute features.

[0111] The attribute prediction module is based on predicted simulation attribute features. The training is performed based on the difference between the generated simulation attribute features z and the observed features, such as by defining a supervised regularization loss. With loss minimization as the training objective, based on The parameters of the attribute prediction module are adjusted, where sg[.] is the stopping gradient operator and λ is a preset parameter. This involves adjusting the simulated attribute features predicted by the attribute prediction module. As the true value, the simulated attribute feature z generated by the encoding module is used as the predicted value. The parameters of the encoding module are adjusted by gradient descent to reduce the difference between the predicted value and the true value until the encoding module meets the convergence condition, thus obtaining the trained encoding module.

[0112] In this way, an accurate attribute prediction module can be obtained by training based on simulation output instructions and observation data.

[0113] Figure 5 is a schematic flowchart of the implementation of a control method provided in an embodiment of this application. This method can be executed by the processor of a computer device. Based on Figure 4, the training method of the reinforcement learning module includes steps S501 to S502.

[0114] Step S501: After controlling the robot's movement based on the simulation output command of the second time step, obtain the motion reward.

[0115] The robot can execute corresponding actions based on the simulation output instructions of the second time step output by the reinforcement learning module, and obtain motion rewards to evaluate the quality of the robot's action execution. These rewards can include task-aware rewards to encourage the robot to execute tasks in the correct order, and motion-aware rewards to encourage the robot to perform smooth movements when executing tasks.

[0116] Step S502: Train the reinforcement learning module based on motion rewards using reinforcement learning.

[0117] Each time step corresponds to a simulation output command. After the robot executes the corresponding action, it receives a motion reward, and the environmental state changes. Using a reinforcement learning algorithm, based on the environmental state, simulation output command, corresponding motion reward, and the new environmental state after the action execution at each time step, the reinforcement learning module updates its policy at each time step. For example, the reinforcement learning algorithm could be Proximal Policy Optimization (PPO). Thus, at each time step, the reinforcement learning algorithm updates the policy based on the motion reward. Training of the reinforcement learning module stops when the iteration stopping condition is met or the policy reaches the optimization objective.

[0118] Each round consists of multiple time steps. The environment state, simulation output command, corresponding motion reward, and new environment state after the action are executed are saved at each time step. The reinforcement learning policy is updated once per round. When the accumulated number of time steps reaches the corresponding threshold for the round, the policy is updated using a reinforcement learning algorithm based on the motion reward. The reinforcement learning module is then trained using the motion reward corresponding to the simulation output command to obtain an accurate reinforcement learning module.

[0119] Figure 6 is a schematic diagram of the implementation process of a training method for a control model provided in an embodiment of this application. The control model includes an attribute prediction module and a reinforcement learning module; the training method includes steps S601 to S605.

[0120] Step S601: Based on the simulation output command of the first time step and the observation data of the second time step, generate the simulation splicing feature of the second time step; the first time step is the previous time step of the second time step.

[0121] Step S602: Input the simulation stitching features of the second time step and the simulation stitching features of at least one historical time step into the attribute prediction module to obtain the predicted simulation attribute features; the simulation attribute features are used to characterize the attribute data that affect the interaction between the robot and the simulation environment.

[0122] Step S603: Input the simulation output command and target attribute features of the second time step into the reinforcement learning module to obtain the simulation output command of the second time step; the target attribute features are the predicted simulation attribute features or the simulation attribute features generated based on the attribute data of the sample objects.

[0123] Step S604: After controlling the robot's movement based on the simulation output command of the second time step, obtain the motion reward.

[0124] Step S605: Train the attribute prediction module based on the predicted simulation attribute features and the generated simulation attribute features, and train the reinforcement learning module based on motion rewards through reinforcement learning.

[0125] For steps S601 to S605, please refer to the previous description, which will not be repeated here.

[0126] The reinforcement learning module can consider the attribute data of the object to be interacted with, the interaction pose, and the historical output commands when outputting instructions. This reduces the reliance on vision, allows for a better understanding of the object's intrinsic characteristics, and demonstrates high generalization ability when dealing with unseen objects. It also exhibits a high success rate in the real world, reducing the gap between simulation and reality.

[0127] Figure 7 is a schematic diagram of the implementation flow of a control model training method provided in an embodiment of this application. This method can be executed by the processor of a computer device. Based on Figure 6, the control model further includes an encoding module. In order to extract accurate simulation attribute features from the attribute data of the sample object, the training method further includes steps S701 and S702.

[0128] Step S701: Obtain the attribute data of the sample object.

[0129] In the real world, it is difficult to obtain the attribute data of the object to be interacted with. However, during simulation, the attribute data can be directly obtained based on the simulation software. This attribute data is the attribute data of the second time step.

[0130] Step S702: Input the attribute data of the sample object into the encoding module to generate simulation attribute features.

[0131] In some embodiments, in order to accurately train the coding module, the training method further includes training the coding module based on the predicted simulation attribute features and the generated simulation attribute features.

[0132] The encoding module is based on predicted simulation attribute features. The module is trained based on the difference between the generated simulation attribute features z and the actual simulation attribute features. At each time step, the attribute prediction module outputs the predicted simulation attribute features. The encoding module obtains the generated simulation attribute features z, and based on the predicted simulation attribute features... The difference between the generated simulation attribute feature z and the observed feature is determined. For example, a supervised regularization loss is formulated. With loss minimization as the training objective, based on The parameters of the encoding module are adjusted, where sg[.] is the stopping gradient operator and λ is the preset parameter.

[0133] In some embodiments, to ensure the robot executes tasks correctly and generates smooth movements, the system rewards the robot for each action at each time step based on a reward function and the current environmental state. For example, task-aware rewards focus on executing the correct motion sequence, i.e., adhering to the order of A then B, rather than cheating to immediately obtain a reward. Specifically, this includes the state where a door is open and the handle is firmly gripped by the gripper. The reward received was significantly greater than the state of not grabbing the doorknob. Another example is motion-aware rewards, which are used to encourage robots to perform smooth movements, that is, to encourage robots to move smoothly while maintaining a high success rate.

[0134] The task perception reward includes at least one of the following: success reward, used to encourage the robot to complete the interactive task; distance reward, used to encourage the robot to reduce the distance between the interactive device and the object to be interacted with; object state reward, used to encourage the robot to increase the amount of joint position change of the object to be interacted with; and interaction reward, used to encourage the robot to perform the target action according to the expected motion sequence.

[0135] For successful rewards, after the robot executes the interactive action at the current time step, it obtains the distance δ between the robot's interactive device and the object to be interacted with, and determines the first distance index based on the distance δ. First Interaction Metric and time index 1 s The formula for success reward is: When δ≤0.05, It should be 0.05, otherwise =0; when δ≤0.015∧1 contact In this case, It should be 0.5, otherwise 0; 1 s The unit of time is 1 second. contact This is the contact condition, and its value is either 1 or 0. For example, in a door-opening task, δ is the distance between the robot's gripper and the door handle. That is, when the robot makes contact with the object to be interacted with, 1... contact The value is 1, otherwise 1. contact The value is 0.

[0136] For distance rewards, the second distance metric is determined based on distance δ. Second interaction index The formula for distance reward is: When δ≤0.015∧1 contact In this case, It should be 0.8, otherwise It is 0.

[0137] For the object state reward, the change in joint position of the object to be interacted with is q. obj The first interaction index is determined based on the distance δ. Third distance indicator and round length w len The formula for object state reward is: When δ≤0.05, It should be 0.5, otherwise The value is 0. For example, in a door-opening task, the change in the joint position q of the object to be interacted with is 0. obj The angle at which the door opens.

[0138] For interactions, the formula for interaction rewards is 0.2 * 1g When δ≤0.015∧1 contact In the case of 1 g =1, otherwise 1 g It is 0. For example, in the door-opening task, 1 g A value of 1 indicates that the gripper has caught the door handle. g A value of 0 indicates that the gripper did not grab the door handle.

[0139] Each reward corresponds to a weight. Each reward is multiplied by its corresponding weight, and then the results are summed to obtain the task perception reward. For example, the weight of the success reward is 40, the weight of the distance reward is 0.6, the weight of the object state reward is 1, and the weight of the interaction reward is 0.05, so as to accurately classify the task perception rewards.

[0140] In addition, motion perception rewards include at least one of the following: energy rewards to encourage the robot to reduce energy consumption; tracking position rewards to encourage the robot to reduce the difference between the actual position and the desired position; tracking rotation rewards to encourage the robot to reduce the difference between the actual angle and the desired angle; smoothness rewards to encourage the robot to perform smooth movements; and orientation rewards to encourage the robot to reduce movement deviation in the target direction.

[0141] The aforementioned target directions include a first direction and / or a second direction. Accordingly, the aforementioned directional rewards include a first-direction reward for encouraging the robot to reduce motion deviation in the first direction, and a second-direction reward for encouraging the robot to reduce motion deviation in the second direction.

[0142] For energy rewards, the joint torque τ and joint velocity of the robot's interactive devices are obtained. The formula for energy reward is: The weight is -0.05. For example, in a door-opening task, if the gripper grabs the door handle, the smaller the robot's joint torque and joint speed, the higher the product of the energy reward and the weight.

[0143] For tracking position rewards, obtain the robot's desired position c. pos and real location ee pos The formula for the location reward is exp(-4(c pos -ee pos ))*1 d The weight is 0.025. When δ ≤ 0.05, 1 d =1, otherwise 1 d The value is 0. The robot's desired position c is... pos and real location ee pos The smaller the difference between them, the higher the reward for tracking the location.

[0144] For tracking rotation rewards, obtain the robot's expected rotation angle c. ori and the actual rotation angle ee ori The formula for tracking the rotation reward is exp(-4Δ(c)). ori -ee ori ))*1 d The weight is 0.004. When δ ≤ 0.05, 1 d =1, otherwise 1 d The value is 0. The robot's desired rotation angle c is... ori and the actual rotation angle ee ori The smaller the difference between them, the higher the reward for tracking the rotation.

[0145] For smoothness reward, obtain the action 'a' at the current time step. t The action a at the previous time step t-1 The formula for smoothness reward is: The weight is -0.001. sgn(a t ) is a t symbols, In sgn(a t )≠sgn(a t-1 The value is 1 if the condition is met, and 0 otherwise. The action 'a' at the current time step. t The action a at the previous time step t-1 The smaller the difference, the smoother the robot's movements, and the higher the product of the smoothness reward and the weight.

[0146] For the reward in the first direction, obtain the motion component a in the first direction. t [y], the formula for the first direction reward is 1. dy *(a t [y]*15) 2 The weight is -0.005. If the first direction is the y-axis, the motion component a in the first direction... t The smaller [y] is, the higher the product of the reward and weight in the first direction. When 0.02 ≤ δ ≤ 0.08, 1 dy The value is 1, otherwise 1. dy The value is 0. For the reward in the second direction, obtain the motion component a in the second direction. t [z], the formula for the second direction reward is 1 g *(a t [z]*15) 2 The weight is -0.07. If the second direction is the z-axis direction, the motion component a in the second direction... t The smaller [z] is, the higher the product of the reward and weight in the second direction.

[0147] As can be seen, each reward corresponds to a weight. By multiplying each reward by its corresponding weight and then summing them up, we can obtain the task perception reward, which is used to accurately classify motion perception rewards.

[0148] In some embodiments, the above-mentioned simulation output command based on the second time step to control the robot's motion can be implemented by scaling the simulation output command to obtain a simulation control command, and controlling the robot's motion based on the simulation control command.

[0149] The simulation output commands obtained by the reinforcement learning module may be non-standard action commands, making it difficult for the robot to execute actions based on the simulation output commands. Therefore, it is necessary to adjust the simulation output commands, such as scaling, which involves adjusting parameters such as the speed, force, and range of the action, in order to obtain simulation control commands that can be understood and executed by the robot. These commands are then sent to the robot's control system to drive the robot to execute actions safely and effectively based on the simulation control commands.

[0150] For example, the simulation output command can be scaled using formula (4) to obtain the simulation control command c. t c t =clip(a t ,-1,1)*40+100 (4);

[0151] Among them, a t The clip function is a range limiting function that restricts the simulation output command to the range [-1, 1].

[0152] The simulation output command is scaled by a pre-trained scaling module. The simulation output command is input into the scaling module, which then outputs the simulation control command. In other words, the output command is input into a scaling module that has pre-learned the scaling relationship between the output command and the target control command. Based on this pre-learned scaling relationship, the scaling module scales the output command and outputs the target control command.

[0153] Simulation control instructions include data related to the robot's interactive actions. Based on these instructions, the robot can be controlled to perform interactive actions at the current time step. Simulation control instructions are action commands that the robot can understand and effectively execute. For example, simulation control instructions may include at least one of the following: target displacement, target posture, gripper action, and impedance control parameters.

[0154] In this way, we can obtain simulation control commands that the robot can accurately recognize.

[0155] In some embodiments, the above method further includes: generating the stiffness matrix of the robot based on simulation output instructions; constructing a dynamic model for impedance control based on the robot's mass inertia matrix, damping matrix, and stiffness matrix; and using the dynamic model to control the balance between the robot's motion and contact forces.

[0156] The simulation output command of the reinforcement learning module includes stiffness. The stiffness is scaled to obtain the target stiffness, and the target stiffness is expanded to obtain the stiffness matrix. Based on the adjusted stiffness matrix, the unadjusted mass-inertia matrix, and the damping matrix, a dynamic model is constructed to apply impedance control to the robot's motion, ensuring that the robot can perform smooth movements.

[0157] The following describes the application of the control method and control model training method provided in the embodiments of this application in real-world scenarios.

[0158] This application utilizes observation history as input for reinforcement learning, replacing point cloud features, reducing the gap between simulation and reality, and enabling a better understanding of the object's intrinsic properties. Impedance control is employed to achieve a seamless transition from simulation to reality, resulting in smooth and compliant motion.

[0159] Manipulating articulated objects presents a unique challenge compared to rigid object manipulation, as the objects themselves represent a dynamic environment. This application proposes a novel reinforcement learning (RL)-based pipeline equipped with variable impedance control and motion adaptation capabilities, enabling generalized articulated object manipulation using observation history, with a focus on achieving smooth and dexterous motion during zero-shot simulation-to-real-world transfer. To reduce the simulation-to-real-world gap, the pipeline does not directly utilize visual data features (image RGBD / point cloud) as policy input. Instead, it first extracts useful low-dimensional data through off-the-shelf modules, reducing reliance on vision. By inferring object motion and its inherent and intrinsic properties through observation history, impedance control is applied in both simulated and real-world environments to minimize the simulation-to-real-world gap. This application develops a meticulously designed training environment with high randomness and a specialized reward system (task awareness and motion awareness), enabling multi-stage, end-to-end manipulation without heuristic motion planning. Extensive experimentation on various objects achieves an 84% success rate in the real world for the trained policy.

[0160] Figure 8 is a schematic diagram of the application of the reinforcement learning strategy provided in this embodiment. An RL strategy for opening doors and drawers was trained in the simulation. This strategy utilizes historical observation data to adjust its output action based on the object's motion. Directly applying this strategy to closed-loop variable impedance control in the real world, using only a single first frame RGBD image, achieved 80% joint limiting and 84% success rate.

[0161] In diverse training settings, the observation history 801 is input into the adaptation module 802 to output action understanding 803, resulting in a strategy action 804; the strategy action 804 is input into the impedance control module 805 to output robot commands 806; the trained strategy is applied to the real world, and diverse objects 807 can be manipulated using only a single RGBD image.

[0162] Given the ubiquity of articulated objects in both home and industrial environments, it is essential to learn how to manipulate them effectively. Currently, universal articulated object manipulation remains an unsolved problem. The difficulty lies in the fact that true articulated characteristics (such as pivot center, friction, and stiffness) can only be identified after physical contact. Two objects may appear identical, yet their physical properties can be vastly different. Therefore, to achieve a universal articulated object manipulation pipeline, enabling robots to interact seamlessly with unseen objects, a closed-loop pipeline needs to be established to adaptively infer these characteristics during the manipulation phase. Another challenge is the joint constraints of the object, requiring the applied actions to conform to the actual joint motions of the object. If the robot's actions cannot tolerate joint motion and prioritize the completion of given instructions, excessive forces may be generated, damaging both the object and the robot.

[0163] In one technique, articulated object manipulation relies on visual information as the primary input to its pipeline. The operable parts and the sequence of operations or waypoint trajectories are predicted using the visual input of the first frame (in the form of point clouds or RGB images). This sequence or waypoint trajectory is executed directly in an open-loop manner, without considering any physical interactions the robot might have with the object. This ignores the object's intrinsic properties and is prone to unsafe behavior. In another technique, a RL backbone is used to output actions in a closed-loop manner based on visual feedback. Because this pipeline heavily relies on visual feedback at each iteration, a significant gap arises between the visual simulation and reality left over from the vision module. During the manipulation phase, suboptimal operations may be output due to occlusion of the operable parts. Other techniques utilize impedance control as an off-the-shelf low-level controller, adaptively adjusting predicted waypoints based on sample-based heuristics. However, this approach only affects the local trajectory between two predefined set points, resulting in uneven motion.

[0164] In this embodiment, closed-loop RL is combined with learnable impedance control. First, observation history is used to control the object in a closed loop, replacing visual input. Taking how a human opens a door in the dark as an example, intuition is demonstrated: given the position of the doorknob and whether the door has a left or right hinge, a human will estimate the circular motion of the door based on the actions performed and the actual movement of the door. Even without direct visual input, the human will gradually adjust the next action based on feedback information to complete the door-opening task. Based on this intuition, utilizing observation history and reducing reliance on vision has the following advantages: 1) Visual information is used only as auxiliary information, reducing the gap between visual simulation and reality; 2) Observation and action history are used to implicitly learn the object's motion based on the positional error after each execution, achieving a generalizable closed-loop pipeline.

[0165] Secondly, the introduction of variable impedance control into the pipeline solves the problem of compliant motion in articulated object manipulation. Impedance control is suitable for tasks with high tolerance requirements for balance setpoint tracking and object joint motion, fundamentally distinguishing articulated object manipulation from rigid object manipulation. In simulation, a high-frequency variable impedance controller is implemented while its parameters are learned in conjunction with an RL strategy. The inclusion of impedance control in a carefully designed training setup enables the strategy to learn smooth, continuous motions that conform to the object's joint movements, rather than single actions or discrete waypoints, resulting in a higher success rate in real-world scenarios.

[0166] This application proposes a novel articulated object manipulation pipeline based on Reinforcement Learning (RL), using observation and motion history as primary inputs, with vision serving only as auxiliary information. A training environment is designed where each component simulates reality, and a reward function system is implemented to achieve smooth, multi-stage, end-to-end manipulation without any heuristic motion planning. A variable impedance controller is introduced into RL to improve tolerance to object motion and facilitate direct transfer from simulation to reality. Extensive experiments on four tasks and 500 generalizations in the real world demonstrate that zero-point inference achieves success rates of 96% in simulation and 84% in reality, exhibiting high versatility for a wide range of objects.

[0167] Manipulating articulated objects is extremely challenging due to the vast differences in their geometry and physical properties. Existing benchmarks in related technologies still have limited coverage. Research on articulated object manipulation can be broadly categorized into capability-based methods and RL-based methods. On the one hand, capability-based methods rely on visual capability heatmaps, where each point corresponds to a success rate of manipulation, thus selecting contact points and predicting actions. However, heatmaps are often ambiguous and difficult to accurately label, leading to a gap between visual simulation and reality, hindering generalization. On the other hand, while RL-based methods demonstrate better generalization capabilities, their use of point cloud features as policy inputs broadens the exploration space and complicates the task. Furthermore, they use unrealistic entities for training, such as flying grippers or mobile Franka robots, to relax restrictions on inverse kinematics solvers (IK solvers) or collision avoidance conditions, simplifying real-world tasks. This application's embodiments utilize only low-dimensional visual information captured in the first frame, combining historical observations during the manipulation phase to better understand object motion using RL.

[0168] Impedance control belongs to the position-force control family, in which position and force are not decoupled but processed simultaneously, thereby improving tolerance to feedback forces while maintaining good tracking. Many contact-rich robotic tasks, such as object placement or tool assembly, have demonstrated the compatibility of this controller with tasks that simultaneously consider setpoint tracking and object-robot force constraints. In learning-based methods, impedance control is used as a readily available low-level controller to execute downstream instructions under policy guidance. Impedance control parameters are directly incorporated as learnable variables into RL, inverse RL, or analytical optimization methods. Compared to manually adjusting impedance control, variable impedance control is more suitable for different task settings and has lower labor costs. In the embodiments of this application, impedance control parameters are learned during training and directly applied to the real world, enabling the application of impedance control to be extended to the manipulation of articulated objects.

[0169] Given a hinged object O (corresponding to the interactive object mentioned above) and a manipulation task θ, a policy π can be trained to output a dexterous action in a closed-loop manner to complete the task. The manipulation task has been adapted to be more challenging and realistic based on relevant technologies. For pulling tasks (opening doors, drawers), the policy is required to reach and grasp the operable part, then open it until the object's joint position reaches at least 80% of its joint limit, rather than approximately half. Since the robot needs to track the rigid body motion (including translation and rotation) of the object in actual 3D space, this policy requires the robot to perform a large number of dexterous and prolonged movements, especially when applied to rotary joints. In this setup, only robots (such as the Franka robot with a fixed base) are allowed to use realistically feasible inverse kinematics (IK) configurations, and unlike other pathpoint prediction pipelines using flying grippers or suction cups, the absolute feasibility of predicted motion is assumed.

[0170] The framework designed in this application aims to predict one dexterous action at a time, rather than a series of short, primitive actions. At time step t, the output action a of the reinforcement learning policy (corresponding to the reinforcement learning module above) is... t ∈R 11 (Corresponding to the output command above) includes target displacement Target attitude R t ∈R 6 Gripper action G t ∈R 1 and impedance control parameters Use the action scaler to output action a t Convert to robot command c t ∈R 9 (Corresponding to the target control command above).

[0171] At time step t, obtain the observation metric o. t (Corresponding to the observation data above), including the target grasping pose g t ∈R 7 (corresponding to the first interactive pose above), robot joint configuration q t ∈R 7 The relative distance δ between the robot and the object t ∈R 1 End effector pose ee t ∈R 9 Including three-dimensional position and six-dimensional rotation, graspability signals

[0172] In simulations, the target grasping pose is inferred directly from the handle bounding box, while in reality, it needs to be obtained using an off-the-shelf grasping prediction module. In simulations, robot joint configuration can be obtained from the simulation software, while in reality, it needs to be obtained from the robot control system. The relative distance between the robot and the object can be obtained from the simulation software in simulations, but in reality, it needs to be obtained through sensors. The graspability signal is based on distance and contact perception conditions, rather than direct commands controlling the gripper's opening and closing. For graspability signals, they can be obtained from the simulation software in simulations, but in reality, they also need to be obtained through sensors.

[0173] Observation indicator o t It also includes task-related observations, such as the pivot center with added noise in a door-opening task. Pivot radius and right hinge Boolean value Information such as this is used to guide smoother motion execution. Therefore, the observation index o t For formula (5):

[0174] To better understand the environment, privileged observation is used only in the simulation. (Corresponding to the attribute data above), these are values ​​that are difficult to track in the real world. Privileged Observation Including the pivot center Pivot radius object stiffness object mass Joint position of object Handle grab signal Therefore, privileged observation For formula (6):

[0175] Compared to manipulating rigid objects, manipulating articulated objects presents a unique challenge because the object itself is a dynamic environment. The motion of an articulated object can only be observed through physical interaction, or by having its actual ground position hidden within the object. This is similar to motion tasks where environmental parameters (such as terrain friction and slope) are difficult to predict. To address this, an online strategy pipeline is employed, which is widely used in motion tasks and learns two independent modules: an adaptation module σ (corresponding to the attribute prediction module above) and a privileged observation encoder module φ (corresponding to the encoding module above). In simulations, privileged observations (e.g., pivot center, object stiffness, object mass, etc.) are used. These features are extracted by the privileged observation encoder module, and the adaptation module is trained to infer this information from historical observations and action pairs.

[0176] The privileged observation encoder module φ is a shallow MLP used during training to learn the latent representation z of privileged observations. t (Corresponding to the generated simulation attribute features mentioned above). This 20-dimensional vector corresponds to the observation pair or action pair p at the current time step. t =(o t ⊕a t-1 The connections form action inputs. The adaptation module σ is designed as a temporal architecture, starting from H = 10 p... t In the process, potential information about the environment is extracted (corresponding to the predicted simulation attribute features mentioned above). Only a portion of the action history is retained as input to σ: target displacement. Gripping action G t and impedance control parameters

[0177] Since traditional two-stage teacher-student pipelines can lead to feasibility gaps and discrepancies between simulation and reality, this application trains both the adaptation module and the privileged observation encoder module simultaneously in a single training iteration. When jointly training the adaptation module with a reinforcement learning RL backbone, it learns to extract similar privileged information from the historical buffer. The method involves defining a supervised regularization loss on top of the PPO objective. Linear scheduling of λ is used to prevent the policy from taking conservative actions in the initial stage.

[0178] While the proposed framework has been widely used for motion tasks, applying this pipeline to fine manipulation tasks (such as articulated object manipulation) remains a challenge. To efficiently execute multi-stage motion using a single end-to-end policy, a stage-conditional reward (corresponding to the motion reward mentioned above) is introduced, including task-aware rewards and motion-aware rewards. After the robot performs an action at each time step, the system rewards the action based on the reward function and the current environmental state. In this way, a task- and action-related reward system is developed, ensuring the correct sequence of task execution and encouraging the policy to generate smooth actions.

[0179] These clauses are typically activated after the main tasks of policy training have been completed, serving as fine-tuning mechanisms to encourage smoother policy execution. Incorporating this regularization clause helps prevent unnecessary movements or unattainable target postures, bridging the gap between simulation and reality.

[0180] Table 1 shows the reward functions, including: 1 d The corresponding condition is δ≤0.05, 1 dy The corresponding condition is 0.02≤δ≤0.08, 1 g The corresponding condition is δ≤0.015∧1 contact τ is the joint torque. For joint velocity, w lenAs the round length weight, a t [y] represents the motion on the y-axis, a t [z] represents the action on the z-axis. For the reward of task perception, the formula for "success" is: The weight is 40.0; the formula for "distance" is... The weight is 0.6; the formula corresponding to "object state" is... The weight is 1.0; the formula for "scraping" is 0.2 * 1. g The weight is 0.05. For the reward of motion perception, the corresponding formula for "energy" is... The weight is -0.05; the formula for "tracking position" is exp(-4(c pos -ee pos ))*1 d The weight is 0.025; the formula for "tracking rotation" is exp(-4Δ(c ori -ee ori ))*1 d The weight is 0.004; the formula for "smoothness" is... The weight is -0.001; "y regularization" corresponds to formula 1. dy *(a t [y]*15) 2 The weight is -0.005; "z-regularization" corresponds to formula 1. g *(a t [z]*15) 2 The weight is -0.07.

[0181] for When δ≤0.05 When δ > 0.05, =0; for When δ≤0.015∧1 contact hour It should be 0.5, otherwise 0; 1 s This represents 1 second. For 1... d 1 dy 1 g It takes the value 1 if the corresponding condition is met, otherwise it takes the value 0. pos ee is the desired position of the strategy output. pos c is the actual location; ori ee is the desired rotation angle output by the strategy. ori This represents the actual rotation angle.

[0182] For task perception rewards, the formula corresponding to each reward indicator is multiplied by its corresponding weight and then summed to obtain the task perception reward; for action perception rewards, the formula corresponding to each reward indicator is multiplied by its corresponding weight and then summed to obtain the action perception reward.

[0183] Table 1

[0184] Domain-based randomization training strategies facilitate the transition from simulation to reality, primarily addressing the physical gap problem. This requires the strategy to understand object motion through interactions between the robot and the object, while the inherent properties of the object are noisy. In the simulation, training is performed by randomizing parameters such as friction, stiffness, and mass to simulate the noise inherent in the object's properties, and a high-frequency variable impedance controller is used to execute predicted actions. During training, object position and yaw rotation are randomized to cover a reasonable workspace set in the real world. Regarding physical characteristics, joint friction, stiffness, and mass are modified to achieve a more robust simulation-to-realistic transition. For the required grasping posture, after inferring the posture from the part's bounding box, random noise is introduced along the y and z axes, and a randomly rotating target is introduced from a predefined spherical cone.

[0185] The goal of impedance control is to take into account the external force F generated by the interaction between the robot and the environment. ext In this case, control the robot to follow the expected trajectory x d Operation. The impedance control design employs a mass-spring-damped system, which dynamically adjusts the target setpoint based on feedback force and environmental stiffness. The dynamic model of the impedance control is given by formula (7):

[0186] Where M is the robot's mass inertia matrix, D is the damping matrix, and K is the stiffness matrix. This is the impedance trajectory output. Indicates acceleration. x represents velocity. c Indicates location.

[0187] In the pipeline, the stiffness coefficient k of the Cartesian impedance controller is learned and predicted. p This is extended to a six-dimensional diagonal matrix K. M, K, and D are assumed to be positive definite diagonal matrices to ensure system stability. The policy prediction k is then calculated using formula (8). p Scaling:

[0188] In both simulations and actual experiments, this numerical range (60 to 140) consistently produces reasonable motion. Based on the stiffness matrix K, a critical damping condition can be deduced. The damping matrix.

[0189] In strategy output Then, calculate using formula (8). Will Expanding, we get K. Since M and D are known, we can obtain F. ext Based on F ext Impedance control is applied to the robot's movements.

[0190] Figure 9 is a schematic diagram of the application of the reinforcement learning strategy provided in the embodiment of this application. During the simulation, the privileged observation encoder module φ is trained to extract the latent representation z of the privileged observation. t Simultaneously, the adaptive module σ is trained from H = 10 previous (o) t a t-1 Inferring similar privileged information from within. The potential representation z t p with the current time step t These components are connected to form the policy input. In the real world, the trained policy is executed end-to-end based on the adaptation module σ, performing reaching, grasping, and manipulation operations. The required grasping pose is extracted using an RGBD image captured in the first frame through an off-the-shelf vision module. The reinforcement learning policy is trained in simulation and directly transferred to the real-world environment.

[0191] In simulation training, at time step t, the observation index o of the current time step is... t (901) and the action a at the previous time step t-1 t-1 (902) constitutes an observation-action pair p t (903), p t (903) Stored in the history buffer 904. Adaptation module 905, based on the H historical observation-action pairs (p... t-1 to p t-H Extracting similar privileged information (906). Privileged observation encoder module 907 according to privileged observation (908) Latent representation of privileged observations z t (909). Randomize p t and Combination, or p t With z t Combining these elements as input to the reinforcement learning module 910 allows the downstream policy to adapt to z. t and At each time step t, the reinforcement learning module outputs action a. t (911), for a t Scaling, obtaining robot command c t (912), c tThe input impedance control system 913 applies impedance control during the execution of actions. The adaptation module and privileged observation encoder module are based on z... t and The difference training between them is used, and the reinforcement learning module is trained using the PPO algorithm based on the reward corresponding to the action.

[0192] When migrating to real-world applications, an RGB image 914 and a corresponding depth image 915 are acquired through the robot's camera. A point cloud 916 corresponding to the environment is then obtained based on the depth image. The RGB image is input into the first module (SAM) 917, which outputs a mask image of the image to be grasped. The point cloud is input into the second module (GSNet) 918, which outputs multiple grasping poses. The target grasping pose 919 is determined from the multiple grasping poses using the mask image of the image to be grasped. The observation metric o containing the target grasping pose at the current time step is then used. t (920) and the output action a of the previous time step t-1 (921) Forming an observation-action pair p t (922), p t Stored in the history buffer 923. The adaptation module 924 adapts the H historical observation-action pairs (p...) t-1 to p t-H Extracting similar privileged information (925). [The following is a list of items / items] and p t The combined input is fed into reinforcement learning module 926, which outputs the action 'a' at the current time step. t (927), for a t Scaling is performed to obtain robot command c. t (928), c t The input impedance control system 929 applies impedance control during the execution of actions.

[0193] To verify the effectiveness of the proposed method, this application underwent extensive evaluation in simulated and real-world environments. For the data and task settings, in the simulation, experiments were conducted on the IsaacGym simulator and the large-scale PartNet-Mobility dataset, following the settings of relevant techniques. Simulations were performed using a Franka robot with a fixed base and a total of 346 articulated 3D objects, including doors and drawers (modified furniture). In the real-world environment, experiments were conducted using a Franka Emika robotic arm equipped with a handheld RealSense D415 camera to capture RGBD images of various household items. The first frame of the RGBD image was used to extract operable part point clouds using the off-the-shelf modules Segment Anything (SAM) and GSNet for grasping prediction.

[0194] The proposed pipeline was evaluated using the following two tasks: OpenDoor / OpenDoor+ and OpenDrawer / OpenDrawer+. For OpenDoor / OpenDoor+, the robot opens a door initially closed, resulting in a maximum door swing of 15% or 80%. A key requirement of this task setting is that the gripper firmly grasps the door handle when opening the door; cheating by opening the door from the side or using the robot's body is not allowed. For OpenDrawer / OpenDrawer+, the robot opens a drawer initially closed, resulting in a drawer exceeding its maximum opening length by 20% or 80%. The gripper must firmly grasp the handle when opening the drawer. Success rate (SR) was used as the primary evaluation metric in both the simulation and real-world settings.

[0195] The proposed methods are compared with articulated object manipulation pipelines employing a simulation-to-realistic RL paradigm. Comparison Method 1: Directly uses a proximal policy optimization algorithm to learn a state-based policy for each task; the detailed PPO parameters and training strategy are similar to those in the embodiments of this application. Comparison Method 2: An affordability learning framework that uses partial point cloud prediction to assess visual operability affordability, incorporating partial masks as an additional dimension into the task while keeping other aspects unchanged. Comparison Method 3: A vision-based policy learning method that first trains a state-based expert using part-based normalization and part-aware rewards, refining the knowledge into a vision-based student policy. Comparison Method 4: A pure image learning method that utilizes a hand-eye monocular camera to actively perceive connected objects from multiple angles to improve the accuracy of 6D pose estimation. Comparison Method 5: A vision-based method that first performs cross-category part segmentation and pose estimation, then uses the predicted part poses for heuristic manipulation.

[0196] To highlight the contribution and effectiveness of each module, the following four comprehensive ablation studies were conducted: Strategy-free distillation: using only the current time step. t The policy is trained using observational data, omitting the adaptation module and privileged observation encoder module. No variable impedance control: Cartesian position control is used as the low-level controller for the policy. No regularization: Motion-aware rewards are excluded from the reward function. No randomization: All forms of randomization are excluded, including object pose, desired grasping pose, friction, stiffness, mass, and intrinsic noise factors.

[0197] The simulation results are shown in Table 2. It can be seen that while most baseline methods perform well on the training set, their performance drops significantly on the test set. In contrast, the proposed method consistently maintains strong performance on the evaluation set without a sharp decline, highlighting its excellent generalization ability. Even without any direct gain reward, the controller can learn to adapt to different operational phases.

[0198] Table 2

[0199] Figure 10 is a schematic diagram of the experimental setup of an embodiment of this application. The strategy was extensively evaluated in the real world, with test objects including a large number of unseen test items, each differing in appearance 1001, size 1002, hinge direction 1003, hinge stiffness 1004, and 6D orientation 1005. The performance of the method of this application was demonstrated in a reasonable workspace, with the test items facing forward or slightly tilted around the z-axis.

[0200] Figure 11 is a schematic diagram of the experimental results of an embodiment of this application, where the horizontal axis represents the number of time frames, with a larger number of frames indicating a later time; the vertical axis represents the stiffness parameter k. p The value of k. It can be seen that the original k... p When greater than 1, for the original k p Scaling is performed, scaling k p The value is 140; the original k p When less than -1, for the original k p Scaling is performed, scaling k p The initial time stiffness parameter k of the robot is 60. p The stiffness parameter k of the robot is relatively large and changes over time as it approaches the object. p As it shrinks, the robotic arm becomes softer.

[0201] It is evident that the learned impedance control parameters actively adapt to the operational phase, even without direct gain rewards: impedance control hardens as the robot approaches the target and softens as it moves away from it. When the robot moves away from the object, the impedance control parameters are set to a higher k. p The robotic arm becomes more rigid; on the other hand, when the distance is shortened to reduce collision penalty, the robotic arm becomes softer. p It gets smaller.

[0202] Table 3 lists the real-world policy output performance, with 50 experiments conducted on different objects for the pipeline and each ablation model (500 runs in total).

[0203] Table 3

[0204] Based on Figure 10, further research on success rates was conducted by distinguishing between failures in the grasping phase due to grasping pose estimation and failures in the opening phase due to pipeline issues. For OpenDoor+, it was found that 6 / 50 of the inferences failed in the grasping phase, while only 4 / 50 failed in the opening phase, indicating that if a stable grasping pose is initiated, the strategy may achieve a success rate of 40 / 44 = 0.90. For OpenDrawer+, 7 / 8 of the failures were due to unsuccessful grasping.

[0205] As shown in Table 3, in addition to the SR drop in both simulation and real-world scenarios, several other issues were identified, highlighting the non-smooth motion execution in the real world. For control without variable impedance, the primary cause of the failure cases (a 40% drop) was the low flexibility of position control, requiring precise execution of each predicted action, generating significant joint torque to overcome the object's feedback force, leading to the robot arm being triggered to stop. Since the success rate in the simulation was greater than 0.8, this behavior did not significantly impact performance.

[0206] In the real world, high torque is extremely dangerous and can trigger emergency stops, highlighting the necessity of impedance control. Even with manual adjustment of the impedance control baseline, "strategy-free distillation" and "no randomization" often fail halfway through. This is due to the gap between physical simulation and reality caused by non-variable training settings and short-term observations. In the "no regularization" case, the robot's reaching and opening movements are stiff, leading to grasping failures and loss of contact during execution.

[0207] In simulations, the modified StorageFurniture subset from PartNet-Mobility, containing 346 objects, was used for evaluation. The success rate for opening doors was 93%, and for opening drawers, it was 96%. In the real world, the strategy was validated using 10 different articulated objects, varying in appearance, size, hinge direction, hinge stiffness, and 6D pose. The strategy achieved an 80% success rate for opening doors and an 84% success rate for opening drawers. The strategy demonstrated good generalization ability on unseen objects, with a success rate higher than all previous work. Furthermore, it generated smooth and continuous motions not seen in previous work.

[0208] This application introduces a reliable RL strategy that can be seamlessly deployed to various real-world environments. Experiments in both simulation and the real world demonstrate that, in simulation, the operational phase should be learned as a smooth, continuous motion, rather than a discrete waypoint. Thanks to the tolerance of impedance control, closed-loop real-world transmission is more efficient even with slightly less-than-ideal motion prediction.

[0209] This application introduces a novel RL framework equipped with variable impedance control for end-to-end articulated object manipulation. This framework adaptively learns object motion through observation and motion history, rather than simply executing a predicted trajectory before robot contact with the object. It demonstrates strong simulation-to-reality translation capabilities on various real-world test objects, achieving success rates of 80% and 84% in door-opening and drawer-opening tasks, respectively, outperforming related technologies. While achieving quantifiable results, the policy generates smooth and dexterous movements due to carefully designed training settings and reward functions. This application provides an alternative method for utilizing visual information and other potential patterns (such as tactile grasping signals), better bridging the gap between simulation and reality for future RL-based manipulation.

[0210] The current application scenario using this solution is the opening and closing of doors and drawers in a home setting. Through reinforcement learning strategies in both simulation and the real world, the robot is trained to recognize the 6D postures of doors and drawers, and high-frequency variable impedance control ensures smooth movement. Grasping posture input provided by vision modules (such as SAM and GSNet) ensures the robot successfully grasps and opens / closes household items. This allows for high success rates in simulation (93% door opening, 96% drawer opening) and achieves a success rate of 80% or higher in real-world scenarios, improving the automation capabilities of home service robots in handling everyday items and ensuring smooth and continuous movements.

[0211] Future applications of this solution may include assembly line robots in automobile manufacturing. During automobile production, robots will utilize this technology to automatically assemble various articulated components (such as doors and hoods). Based on the posture and 6D pose information of different assembled parts, the robot automatically adjusts its operating strategy to achieve precise motion control. This improves assembly accuracy and efficiency in automobile manufacturing, especially when handling complex mechanical parts, enabling smooth assembly movements, reducing equipment wear, and enhancing the overall automation level of the production line.

[0212] Based on the foregoing embodiments, this application provides a control component, which includes the included units and the modules included in each unit, which can be implemented by a processor in a computer device; of course, it can also be implemented by specific logic circuits; in the implementation process, the processor can be a central processing unit (CPU), a microprocessor unit (MPU), a digital signal processor (DSP), or a field programmable gate array (FPGA), etc.

[0213] Figure 12 is a schematic diagram of the composition structure of a control component provided in an embodiment of this application. The control component 1200 includes a determining unit 1210, a generating unit 1220, a first input unit 1230, and a second input unit 1240.

[0214] The determining unit 1210 is used to determine the first interaction pose for the object to be interacted with based on the environmental image of the robot.

[0215] The generation unit 1220 is used to generate the splicing features of the current time step based on the output command of the previous time step and the first interactive pose.

[0216] The first input unit 1230 is used to input the spliced ​​features of the current time step and the spliced ​​features of at least one historical time step into the trained attribute prediction module to obtain the attribute features of the object to be interacted with; the attribute features are used to characterize the attribute data that affect the interaction between the robot and the robot.

[0217] The second input unit 1240 is used to input the attribute features and the concatenated features of the current time step into the trained reinforcement learning module to obtain the output instructions of the current time step.

[0218] In some embodiments, the trained attribute prediction module and the trained reinforcement learning module are obtained by training on the attribute data of the sample objects to be interacted with in the simulation environment. This attribute data is attribute data that affects the interaction with the robot. In any time step of the training process, the attribute prediction module adjusts its parameters based on the difference between the generated simulation attribute features and the predicted simulation attribute features at that time step. The generated simulation attribute features are obtained based on the attribute data of the sample objects, and the predicted simulation attribute features are the output of the attribute prediction module. The reinforcement learning module adjusts its parameters based on motion rewards. The motion rewards are obtained after the robot responds to the simulation output command and moves. The simulation output command is the output of the reinforcement learning module at that time step based on the generated simulation attribute features or the predicted simulation attribute features.

[0219] In some embodiments, the first input unit 1230 is further configured to store the splicing features of the current time step in a history cache and obtain splicing features of at least one historical time step therefrom.

[0220] In some embodiments, the second input unit 1240 is further configured to scale the output command to obtain a target control command.

[0221] In some embodiments, the second input unit 1240 is further configured to control the interactive device based on the target control command to perform the interactive action at the current time step.

[0222] In some embodiments, the environmental image includes a depth image and a color image; the determining unit 1210 is further configured to perform object recognition on the color image to obtain a mask image of the object to be interacted with; convert the depth image to obtain environmental point cloud data; use the environmental point cloud data to determine a plurality of second interaction poses; and use the mask image to determine a first interaction pose among the plurality of second interaction poses.

[0223] In some embodiments, the training method for the attribute prediction module includes: generating simulation stitching features for a second time step based on simulation output instructions at a first time step and observation data at a second time step; the first time step being the previous time step of the second time step; inputting the simulation stitching features of the second time step and simulation stitching features from at least one historical time step into the attribute prediction module to obtain predicted simulation attribute features; the simulation attribute features are used to characterize attribute data affecting the interaction between the robot and the simulation environment; inputting the simulation stitching features of the second time step and the target attribute features into a reinforcement learning module to obtain simulation output instructions for the second time step; the target attribute features being the predicted simulation attribute features or simulation attribute features generated based on attribute data of sample objects; and training the attribute prediction module based on the predicted simulation attribute features and the generated simulation attribute features.

[0224] In some embodiments, the training method for the reinforcement learning module includes: after controlling the robot's movement with a simulation output command based on a second time step, obtaining a motion reward; and training the reinforcement learning module using reinforcement learning based on the motion reward.

[0225] This application provides a robot that includes the control components described above.

[0226] Figure 13 is a schematic diagram of the composition structure of a training device provided in an embodiment of this application. The training device 1300 includes: a simulation generation module 1310, a first input module 1320, a second input module 1330, an acquisition module 1340, and a training module 1350.

[0227] The simulation generation module 1310 is used to generate simulation splicing features of the second time step based on the simulation output command of the first time step and the observation data of the second time step; the first time step is the previous time step of the second time step.

[0228] The first input module 1320 is used to input the simulation stitching features of the second time step and the simulation stitching features of at least one historical time step into the attribute prediction module to obtain the predicted simulation attribute features; the simulation attribute features are used to characterize the attribute data that affect the interaction between the robot and the simulation environment.

[0229] The second input module 1330 is used to input the simulation output command and target attribute features of the second time step into the reinforcement learning module to obtain the simulation output command of the second time step; the target attribute features are the predicted simulation attribute features or the simulation attribute features generated based on the attribute data of the sample objects.

[0230] The acquisition module 1340 is used to acquire motion rewards after controlling the robot's movement with simulation output instructions based on the second time step;

[0231] Training module 1350 is used to train the attribute prediction module based on the predicted simulation attribute features and the generated simulation attribute features, and to train the reinforcement learning module based on motion rewards through reinforcement learning.

[0232] When the control model includes an encoding module, the simulation generation module 1310 is also used to acquire the attribute data of the sample object and input the attribute data of the sample object into the encoding module to generate simulation attribute features.

[0233] In some embodiments, the training module 1350 is further configured to train the encoding module based on the predicted simulation attribute features and the generated simulation attribute features.

[0234] For example, motion rewards include at least one of the following: task-aware rewards and motion-aware rewards; task-aware rewards are used to encourage the robot to perform a desired motion sequence; motion-aware rewards are used to encourage the robot to perform smooth motion.

[0235] Task-aware rewards include at least one of the following: success rewards, used to encourage the robot to complete the interactive task; distance rewards, used to encourage the robot to reduce the distance between the robot's interactive device and the object to be interacted with; object state rewards, used to encourage the robot to increase the amount of joint position changes of the object to be interacted with; and interaction rewards, used to encourage the robot to perform the target action according to the expected motion sequence.

[0236] Motion perception rewards include at least one of the following: energy rewards to encourage the robot to reduce energy consumption; tracking position rewards to encourage the robot to reduce the difference between the actual position and the desired position; tracking rotation rewards to encourage the robot to reduce the difference between the actual angle and the desired angle; smoothness rewards to encourage the robot to perform smooth movements; and orientation rewards to encourage the robot to reduce movement deviations in the target direction.

[0237] In some embodiments, the acquisition module 1340 is further configured to scale the simulation output instructions to obtain simulation control instructions, and control the robot's movement based on the simulation control instructions.

[0238] In some embodiments, the acquisition module 1340 is further configured to generate the stiffness matrix of the robot based on the simulation output instructions; construct a dynamic model for impedance control based on the robot's mass inertia matrix, damping matrix, and stiffness matrix; and use the dynamic model to control the balance between the robot's motion and contact force.

[0239] The descriptions of the apparatus embodiments above are similar to those of the method embodiments above, and have similar beneficial effects. In some embodiments, the functions or modules included in the apparatus provided in this application can be used to perform the methods described in the method embodiments above. For technical details not disclosed in the apparatus embodiments of this application, please refer to the descriptions of the method embodiments of this application for understanding.

[0240] It should be noted that, in the embodiments of this application, if the above-described control method and control model training method are implemented in the form of software functional modules and sold or used as independent products, they can also be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the embodiments of this application, or the part that contributes to the related technology, can be embodied in the form of a software product. This software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, mobile hard drives, read-only memory (ROM), magnetic disks, or optical disks. Thus, the embodiments of this application are not limited to any specific hardware, software, or firmware, or any combination of hardware, software, and firmware.

[0241] This application provides a computer device including a memory and a processor. The memory stores a computer program that can run on the processor. When the processor executes the program, it implements some or all of the steps in the above-described method.

[0242] This application provides a computer-readable storage medium, which may be transient or non-transient, storing a computer program thereon. When executed by a processor, the computer program implements some or all of the steps in the above-described method.

[0243] This application provides a computer program including computer-readable code. When the computer-readable code is run in a computer device, the processor in the computer device performs some or all of the steps in the above-described method.

[0244] This application provides a computer program product, including a non-transitory computer-readable storage medium storing a computer program. When the computer program is read and executed by a computer, it implements some or all of the steps in the above-described method. This computer program product can be implemented specifically through hardware, software, or a combination thereof. For example, the computer program product can be embodied as a computer storage medium or a software product, such as a software development kit (SDK).

[0245] The descriptions of the various embodiments above tend to emphasize the differences between them, while their similarities or commonalities can be referred to each other. The descriptions of the above embodiments of devices, storage media, computer programs, and computer program products are similar to the descriptions of the above method embodiments and have similar beneficial effects. Technical details not disclosed in the corresponding embodiments should be understood with reference to the descriptions of the method embodiments of this application.

[0246] Figure 14 is a schematic diagram of the hardware entity of a computer device provided in an embodiment of this application. The hardware entity of the computer device 1400 includes a processor 1401 and a memory 1402. The memory 1402 stores a computer program that can run on the processor 1401. When the processor 1401 executes the program, it implements the steps in the method of any of the above embodiments.

[0247] The memory 1402 stores computer programs that can run on the processor. The memory 1402 is configured to store instructions and applications that can be executed by the processor 1401. It can also cache data to be processed or already processed (e.g., image data, audio data, voice communication data, and video communication data) in the processor 1401 and various modules in the computer device 1400. It can be implemented by flash memory or random access memory (RAM).

[0248] The steps of the control method or control model training method described above are implemented when the processor 1401 executes a program, which generally controls the overall operation of the computer device 1400.

[0249] This application provides a computer storage medium storing one or more programs that can be executed by one or more processors to implement the steps of the control method or control model training method as described in any of the above embodiments.

[0250] The descriptions of the storage medium and device embodiments above are similar to those of the method embodiments above, and have similar beneficial effects. For technical details not disclosed in the storage medium and device embodiments of this application, please refer to the descriptions of the method embodiments of this application for understanding.

[0251] The aforementioned processor can be at least one of the following: Application Specific Integrated Circuit (ASIC), DSP, Digital Signal Processing Device (DSPD), Programmable Logic Device (PLD), FPGA, CPU, controller, microcontroller, and microprocessor. It is understood that other electronic devices can also implement the functions of the aforementioned processor, and this application does not specifically limit the specific implementation.

[0252] The aforementioned computer storage media / memory can be ROM, Programmable Read-Only Memory (PROM), Erasable Programmable Read-Only Memory (EPROM), Electrically Erasable Programmable Read-Only Memory (EEPROM), Ferromagnetic Random Access Memory (FRAM), Flash Memory, Magnetic Surface Memory, Optical Disc, or Compact Disc Read-Only Memory (CD-ROM), etc.; or it can be various terminals that include one or any combination of the above-mentioned memories, such as mobile phones, computers, tablet devices, personal digital assistants, etc.

[0253] The above are merely preferred embodiments of this application and are not intended to limit this application. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the scope of protection of this application.

Claims

1. A control method, characterized in that, The method includes: Determine the first interaction pose for the object to be interacted with based on the image of the robot's environment. Based on the output command of the previous time step and the first interactive pose, generate the splicing features of the current time step; The concatenated features of the current time step and the concatenated features of at least one historical time step are input into the trained attribute prediction module to obtain the attribute features of the object to be interacted with; the attribute features are used to characterize the attribute data that affect the interaction with the robot. The attribute features and the concatenated features of the current time step are input into the trained reinforcement learning module to obtain the output instructions for the current time step.

2. The method according to claim 1, characterized in that, The trained attribute prediction module and the trained reinforcement learning module are obtained by training with the attribute data of the sample objects to be interacted with in the simulation environment. The attribute data is the attribute data that affects the interaction with the robot. In the training process, at any time step, the attribute prediction module adjusts its parameters based on the difference between the generated simulation attribute features and the predicted simulation attribute features at that time step; the generated simulation attribute features are obtained based on the attribute data of the sample object, and the predicted simulation attribute features are output by the attribute prediction module. The reinforcement learning module adjusts parameters based on motion rewards, which are obtained after the robot moves in response to the simulation output command. The simulation output command is output by the reinforcement learning module at any time step based on the generated simulation attribute features or the predicted simulation attribute features.

3. The method according to claim 1 or 2, characterized in that, The method further includes: Store the splicing features of the current time step in the history cache; Obtain the splicing features of at least one historical time step from the historical cache.

4. The method according to any one of claims 1 to 3, characterized in that, The method further includes: The output command is scaled to obtain the target control command; Based on the target control command, control the interactive device to execute the interactive action at the current time step.

5. The method according to any one of claims 1 to 4, characterized in that, The environmental image includes a depth image and a color image; determining the first interaction pose for the object to be interacted with based on the environmental image of the robot includes: Object recognition is performed on the color image to obtain a mask image of the object to be interacted with; The depth image is converted to obtain environmental point cloud data; Using the environmental point cloud data, multiple second interaction poses are determined; The first interaction pose is determined using the mask image among the plurality of second interaction poses.

6. The method according to any one of claims 1 to 5, characterized in that, The training method for the attribute prediction module includes: Based on the simulation output command of the first time step and the observation data of the second time step, the simulation stitching feature of the second time step is generated; the first time step is the previous time step of the second time step. The simulation stitching features of the second time step and the simulation stitching features of at least one historical time step are input into the attribute prediction module to obtain the predicted simulation attribute features; the simulation attribute features are used to characterize the attribute data that affect the interaction between the robot and the simulation environment. The simulation stitching features and target attribute features of the second time step are input into the reinforcement learning module to obtain the simulation output command of the second time step; the target attribute features are the predicted simulation attribute features or simulation attribute features generated based on the attribute data of the sample object; The attribute prediction module is trained based on the predicted simulation attribute features and the generated simulation attribute features; And / or, The training method for the reinforcement learning module includes: After controlling the robot's movement based on the simulation output command at the second time step, a motion reward is obtained; The reinforcement learning module is trained using the aforementioned exercise rewards through a reinforcement learning approach.

7. A method for training a control model, characterized in that, The control model includes an attribute prediction module and a reinforcement learning module; the training method includes: Based on the simulation output command of the first time step and the observation data of the second time step, the simulation stitching feature of the second time step is generated; the first time step is the previous time step of the second time step. The simulation stitching features of the second time step and the simulation stitching features of at least one historical time step are input into the attribute prediction module to obtain the predicted simulation attribute features; the simulation attribute features are used to characterize the attribute data that affect the interaction between the robot and the simulation environment. The simulation output command and target attribute features of the second time step are input into the reinforcement learning module to obtain the simulation output command of the second time step; the target attribute features are the predicted simulation attribute features or simulation attribute features generated based on the attribute data of the sample object. After controlling the robot's movement based on the simulation output command at the second time step, a motion reward is obtained; The attribute prediction module is trained based on the predicted simulation attribute features and the generated simulation attribute features, and the reinforcement learning module is trained based on the motion reward through reinforcement learning.

8. The method according to claim 7, characterized in that, The control model further includes an encoding module; the training method further includes: Obtain the attribute data of the sample object; The attribute data of the sample object is input into the encoding module to generate the simulation attribute features; The encoding module is trained based on the predicted simulation attribute features and the generated simulation attribute features.

9. The method according to claim 7 or 8, characterized in that, The motion reward includes at least one of the following: task-aware reward and motion-aware reward; the task-aware reward is used to encourage the robot to perform a desired motion sequence; the motion-aware reward is used to encourage the robot to perform smooth motion.

10. The method according to claim 9, characterized in that, The task-aware reward includes at least one of the following: Success rewards are used to encourage robots to complete interactive tasks; Distance rewards are used to encourage reducing the distance between the robot's interaction device and the object to be interacted with; Object state reward is used to encourage the robot to increase the amount of joint position change of the object to be interacted with; Interactive rewards are used to encourage the robot to perform the target action according to the desired motion sequence; And / or, The motion-sensing reward includes at least one of the following: Energy rewards are used to encourage robots to reduce energy consumption; Tracking location rewards are used to encourage the robot to reduce the difference between its actual location and its desired location; Tracking rotation rewards are used to encourage the robot to reduce the difference between the actual angle and the desired angle; Smoothness reward is used to encourage the robot to perform smooth actions, thereby enabling the robot to perform smooth motion; Directional rewards are used to encourage robots to reduce motion deviations in the target direction.

11. The method according to any one of claims 7 to 10, characterized in that, The simulation output command based on the second time step controls the robot's movement, including: The simulation output command is scaled to obtain the simulation control command; The robot's movement is controlled based on the simulation control commands.

12. The method according to any one of claims 7 to 11, characterized in that, The method further includes: The robot's stiffness matrix is ​​generated based on the simulation output instructions; An impedance control dynamic model is constructed based on the robot's mass inertia matrix, damping matrix, and stiffness matrix; the dynamic model is used to control the balance between the robot's motion and contact force.

13. A control component, characterized in that, The control component, applied to a robot, includes: The determining unit is used to determine a first interactive pose for the object to be interacted with based on the environmental image of the robot. The generation unit is used to generate the splicing features of the current time step based on the output instructions of the previous time step and the first interactive pose. The first input unit is used to input the spliced ​​features of the current time step and the spliced ​​features of at least one historical time step into the trained attribute prediction module to obtain the attribute features of the object to be interacted with; the attribute features are used to characterize the attribute data that affect the interaction with the robot. The second input unit is used to input the attribute features and the concatenated features of the current time step into the trained reinforcement learning module to obtain the output instruction of the current time step.

14. A robot, characterized in that, The robot includes the control components as described in claim 13.

15. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, implements the method described in claims 1 to 6, or any one of claims 7 to 12.

16. A computer program product containing instructions, characterized in that, When the computer program product is run on a computer, it causes the computer to perform the method described in any one of claims 1 to 6, or claims 7 to 12.

Images