Robot, and behavior planning method and control method therefor

Abstract

A behavior planning method for a robot. The behavior planning method for a robot (300) comprises: on the basis of an object pose of a target object, determining a target grasping pose of the robot grasping the target object, the target grasping pose comprising a target base pose and a target gripper pose; on the basis of the target base pose and a current base pose of the robot (300), determining a motion control strategy of mechanical legs (314) of the robot (300), the motion control strategy being used for controlling the mechanical legs (314) to drive a base (311) of the robot (300) to move, so that the base (311) reaches the target base pose; and on the basis of the target gripper pose and a current gripper pose of the robot (300), determining a grasping control strategy of a mechanical arm (313) of the robot (300), the grasping control strategy being used for controlling, when the base (311) reaches the target base pose, the mechanical arm (313) to drive a gripper (312) of the robot (300) to grasp the target object at the target gripper pose. Also provided are the robot (300) and a control method therefor.

Description

A robot and its behavior planning and control methods

[0001] This disclosure claims priority to Chinese Patent Application No. 2024115168278, filed on October 28, 2024, entitled "A Robot and Its Behavior Planning and Control Method", the entire contents of which are incorporated herein by reference. Technical Field

[0002] This disclosure relates to, but is not limited to, the field of robotics, and in particular to a robot and its behavior planning and control methods. Background Technology

[0003] The movement and manipulation of legged robots mainly include leg movements and robotic arm operations. Legged robots are widely used in fields such as home assistance, urban maintenance, disaster relief, and autonomous on-site operations. To enable legged robots to be better applied in these fields, a behavior planning and control scheme is needed to improve the stability of legged robot movements. Summary of the Invention

[0004] In view of the above, this disclosure provides a robot and its behavior planning and control methods, the technical solution of which is implemented as follows:

[0005] In a first aspect, a robot behavior planning method includes: determining a target grasping pose for the robot to grasp the target object based on the object pose of the target object; the target grasping pose includes a target base pose and a target gripper pose; determining a motion control strategy for the robot's mechanical leg based on the target base pose and the robot's current base pose; the motion control strategy is used to control the mechanical leg to move the robot's base so that the robot's base reaches the target base pose; and determining a grasping control strategy for the robot's robotic arm based on the target gripper pose and the robot's current gripper pose; the grasping control strategy is used to control the robotic arm to move the robot's gripper to grasp the target object in the target gripper pose when the base reaches the target base pose.

[0006] Secondly, a robot control method includes: acquiring a target grasping pose for the robot to grasp a target object; the target grasping pose includes a target base pose and a target gripper pose; acquiring a motion control strategy for the robot's mechanical legs; controlling the mechanical legs to move the robot's base so that the robot's base reaches the target base pose; acquiring a grasping control strategy for the robot's mechanical arm; and, when the base reaches the target base pose, controlling the mechanical arm to move the robot's gripper to grasp the target object in the target gripper pose.

[0007] Thirdly, a robot includes a planning module, a base, a gripper, a robotic arm, and robotic legs. The planning module is used to determine a target grasping pose for the robot to grasp the target object based on the object's pose; the target grasping pose includes a target base pose and a target gripper pose; based on the target base pose and the robot's current base pose, a motion control strategy for the robot's robotic legs is determined; the motion control strategy is used to control the robotic legs to move the robot's base so that the robot's base reaches the target base pose; based on the target gripper pose and the robot's current gripper pose, a grasping control strategy for the robot's robotic arm is determined; the grasping control strategy is used to control the robotic arm to move the robot's gripper to grasp the target object in the target gripper pose when the base reaches the target base pose.

[0008] Fourthly, a robot behavior planning device includes: a first determining module, configured to determine a target grasping pose for the robot to grasp the target object based on the object pose of the target object; the target grasping pose includes a target base pose and a target gripper pose; a second determining module, configured to determine a motion control strategy for the robot's mechanical leg based on the target base pose and the robot's current base pose; the motion control strategy is configured to control the mechanical leg to move the robot's base so that the robot's base reaches the target base pose; and a third determining module, configured to determine a grasping control strategy for the robot's robotic arm based on the target gripper pose and the robot's current gripper pose; the grasping control strategy is configured to control the robotic arm to move the robot's gripper to grasp the target object in the target gripper pose when the base reaches the target base pose.

[0009] Fifthly, a robot control device includes: a first acquisition module for acquiring a target grasping pose of the robot grasping a target object; the target grasping pose includes a target base pose and a target gripper pose; a second acquisition module for acquiring a motion control strategy for the robot's mechanical legs; a first control module for controlling the mechanical legs to move the robot's base so that the robot's base reaches the target base pose; a third acquisition module for acquiring a grasping control strategy for the robot's mechanical arm; and a second control module for controlling the mechanical arm to move the robot's gripper to grasp the target object in the target gripper pose when the base reaches the target base pose.

[0010] Sixthly, a computer device includes a memory and a processor, the memory storing a computer program executable on the processor, the processor executing the program to implement some or all of the steps in the above-described method.

[0011] In a seventh aspect, a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements some or all of the steps in the above-described method.

[0012] Eighthly, a computer program product comprising a computer program or instructions which, when executed by a processor, implement some or all of the steps in the above-described method.

[0013] In this embodiment, firstly, based on the object pose of the target object, a target grasping pose for the robot to grasp the target object is determined; the target grasping pose includes the target base pose and the target gripper pose. Secondly, based on the target base pose and the robot's current base pose, a motion control strategy for the robot's mechanical legs is determined; the motion control strategy can be used to control the mechanical legs to move the robot so that the robot's base reaches the target base pose. Finally, based on the target gripper pose and the robot's current gripper pose, a grasping control strategy for the robot's robotic arm is determined; the grasping control strategy can be used to control the robotic arm to move the robot's gripper to grasp the target object in the target gripper pose when the base reaches the target base pose. This enables the robot to move smoothly and grasp the target object accurately.

[0014] Based on motion control strategies, the base can be controlled to reach the target base pose, at which point the robot approaches the target grasping pose. Based on grasping control strategies, once the base reaches the target base pose, the robot's gripper can be controlled to reach the target gripper pose, at which point the robot ultimately achieves the target grasping pose. In the process of robot behavior planning, the integrity and correlation between the movement of the robotic legs and the operation of the robotic arm were considered. By combining the motion control of the robotic legs and the operation of the robotic arm using the target base pose and the target gripper pose, the coordination and stability of the legged robot's movement process are improved. Attached Figure Description

[0015] Figure 1 is a schematic diagram of the implementation process of a robot behavior planning method provided in an embodiment of this disclosure;

[0016] Figure 2 is a schematic diagram of the implementation flow of a robot control method provided in an embodiment of this disclosure;

[0017] Figure 3 is a schematic diagram of the composition structure of a robot provided in an embodiment of this disclosure;

[0018] Figure 4 is a schematic diagram of the composition structure of a robot provided in an embodiment of this disclosure;

[0019] Figure 5 is a schematic diagram of the composition structure of a robot provided in an embodiment of this disclosure;

[0020] Figure 6 is a schematic diagram of the composition structure of a robot provided in an embodiment of this disclosure;

[0021] Figure 7 is a schematic diagram of the composition structure of a robot provided in an embodiment of this disclosure;

[0022] Figure 8 is a schematic diagram of a robot grasping different objects in the real world according to an embodiment of this disclosure;

[0023] Figure 9 is a schematic diagram of the full-body grasping frame system provided in an embodiment of this disclosure;

[0024] Figure 10 is a schematic diagram of the implementation process of the full-body grasping framework provided in the embodiments of this disclosure;

[0025] Figure 11 is a schematic diagram of the implementation process of the motion control module provided in the embodiment of this disclosure;

[0026] Figure 12 is a schematic diagram of the implementation process of the sensing module provided in the embodiment of this disclosure;

[0027] Figure 13 is a schematic diagram of the implementation process of the operation module provided in the embodiment of this disclosure;

[0028] Figure 14 is a schematic diagram of the implementation process of the planning module provided in the embodiment of this disclosure;

[0029] Figure 15 is a schematic diagram of a robot grasping a target object according to an embodiment of this disclosure;

[0030] Figure 16 is a schematic diagram of the arm workspace and the full-body workspace provided in the embodiments of this disclosure;

[0031] Figure 17 is a schematic diagram of the dataset used in real-world testing provided in the embodiments of this disclosure;

[0032] Figure 18 is a schematic diagram of the composition structure of a robot behavior planning device provided in an embodiment of this disclosure;

[0033] Figure 19 is a schematic diagram of the composition structure of a robot control device provided in an embodiment of this disclosure;

[0034] Figure 20 is a schematic diagram of the hardware entity of a computer device provided in an embodiment of this disclosure. Detailed Implementation

[0035] To make the objectives, technical solutions, and advantages of this disclosure clearer, the technical solutions of this disclosure are further described below in conjunction with the accompanying drawings and embodiments. The described embodiments should not be regarded as limitations on this disclosure. All other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this disclosure.

[0036] In the following description, references to "some embodiments" describe a subset of all possible embodiments; however, it is understood that "some embodiments" may be the same subset or different subsets of all possible embodiments and may be combined with each other without conflict. The terms "first / second / third" are used only to distinguish similar objects and do not represent a specific ordering of objects. It is understood that "first / second / third" may be interchanged in a specific order or sequence where permitted, so that the embodiments of this disclosure described herein can be implemented in an order other than that illustrated or described.

[0037] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this disclosure pertains. The terminology used herein is for descriptive purposes only and is not intended to limit the scope of this disclosure.

[0038] This disclosure provides a robot behavior planning method, as shown in the flowchart of Figure 1. The method includes steps S101 to S103:

[0039] Step S101: Based on the object pose of the target object, determine the target grasping pose for the robot to grasp the target object; the target grasping pose includes the target base pose and the target gripper pose.

[0040] The above object pose includes the position and orientation of the target object.

[0041] In some implementations, the position of the target object includes its spatial location. The pose of the target object includes its orientation or angle relative to a coordinate system. The robot includes legged robots, such as quadruped robots. The robot includes mechanical legs and a mechanical arm. The mechanical legs are used to drive the robot's smooth movement and to move the robot's base, while the mechanical arm is used to move the robot's gripper. The target base pose includes the position and orientation of the robot's base, which can be achieved by controlling the movement of the mechanical legs; the target gripper pose includes the position and orientation of the robot's gripper, which can be achieved by controlling the movement of the mechanical arm.

[0042] In some implementations, a generalized orientation accessibility map is used to support six degrees of freedom (DOF) base positioning. These six DDFs include three translational DDFs (forward / backward, left / right displacement, and up / down movement) and three rotational DDFs (pitch, yaw, and roll). The generalized orientation accessibility map can represent candidate base poses for each of the six DDFs relative to the tool center point frame.

[0043] Step S102: Based on the target base pose and the robot's current base pose, determine the motion control strategy for the robot's mechanical legs; the motion control strategy is used to control the mechanical legs to drive the robot to move so that the robot's base reaches the target base pose.

[0044] The current base pose includes the current base position and the current base orientation. The target base pose includes the target base position and the target base orientation.

[0045] In this embodiment of the disclosure, the pose of the base can be changed by controlling the movement of the mechanical leg. Therefore, the movement of the mechanical leg can be planned so that the base moves from the current base pose to the target base pose.

[0046] In some implementations, a motion control strategy for the robotic leg is determined during the movement of the base from the current base posture to the target base posture. The motion control strategy can be used to: control the movement of the robotic leg so that the base moves from the current base posture to the target base position, and then control the movement of the robotic leg so that the base moves from the current base posture to the target base posture; or, control the movement of the robotic leg so that the base reaches the target base position with the target base posture.

[0047] Step S103: Based on the target gripper pose and the robot's current gripper pose, determine the robot's gripping control strategy; the gripping control strategy is used to control the robot arm to drive the robot's gripper to grasp the target object in the target gripper pose when the base reaches the target base pose.

[0048] The current gripper pose includes the current gripper position and the current gripper orientation, while the target gripper pose includes the target gripper position and the target gripper orientation.

[0049] In this embodiment of the disclosure, the gripper's pose can be changed by controlling the movement of the robotic arm. Therefore, the operation of the robotic arm can be planned to move the gripper from its current pose to a target gripper pose. In some embodiments, a gripping control strategy for the robotic arm is determined during the movement of the gripper from its current pose to the target gripper pose. The gripping control strategy can be used to: control the movement of the robotic arm to move the gripper from its current pose to the target gripper position, and then control the movement of the robotic arm to move the gripper from its current pose to the target gripper pose; or, control the movement of the robotic arm to move the gripper to the target gripper position with the target gripper pose.

[0050] In this embodiment, firstly, based on the object pose of the target object, a target grasping pose for the robot to grasp the target object is determined; the target grasping pose includes the target base pose and the target gripper pose. Secondly, based on the target base pose and the robot's current base pose, a motion control strategy for the robot's mechanical legs is determined; the motion control strategy can be used to control the mechanical legs to move the robot, so that the robot's base reaches the target base pose. Finally, based on the target gripper pose and the robot's current gripper pose, a grasping control strategy for the robot's robotic arm is determined; the grasping control strategy can be used to control the robotic arm to move the robot's gripper to grasp the target object in the target gripper pose when the base reaches the target base pose. In this way, by combining the motion control of the mechanical legs and the operation of the robotic arm using the target base pose and the target gripper pose, the robot can move smoothly and grasp the target object accurately.

[0051] In some embodiments, before performing step S101 to determine the target grasping pose of the robot to grasp the target object based on the object pose of the target object, the method may include steps S111 and S112:

[0052] Step S111: Obtain image information of the target object.

[0053] Image information can be image information acquired by image acquisition components mounted on the robotic arm, including but not limited to color image information, infrared image information, and depth image information. Color image information includes visual features such as the color, texture, and shape of the target object; infrared image information is used to characterize the temperature characteristics of the target object; and depth image information is used to characterize the spatial distance between the target object and the robot.

[0054] Step S112: Based on image information, identify and locate the target object to determine its pose.

[0055] The image information is segmented to obtain the features of the target object; based on the image information, the depth of the target object is predicted to obtain its spatial information. Based on the target object's features and spatial information, the object's pose is determined.

[0056] Specifically, the color image can first undergo preprocessing such as grayscale normalization, Gaussian filtering, and size standardization; then, an image segmentation method based on semantic segmentation networks (such as U-Net) can be used to segment the preprocessed color image into pixels to identify the target object and obtain the foreground region including the target object; then, the features of the target object can be extracted from the foreground region.

[0057] In one possible implementation, temperature features of different regions in an infrared image can be extracted, and contour information of the target object can be extracted from the infrared image based on the differences between the temperature features of different regions. Combining the contour information, a foreground region including the target object can be located from a color image. Then, features of the target object can be extracted from the foreground region.

[0058] In this embodiment of the disclosure, image information of the target object is acquired, and the target object is identified and located based on the image information to determine the object pose. This can yield an accurate object pose for subsequent prediction of the target grasping pose.

[0059] Taking image information including infrared and color images as an example; step S112, based on the image information, identifies and locates the target object and determines the object pose, which can be implemented as steps S121 to S123:

[0060] Step S121: Segment the target object in the color image to obtain the segmentation result.

[0061] For example, a Track Anything Model (TAM) is used to process the color image. The TAM may include a Segment Anything Model (SAM) and a memory module (XMem). The color image is segmented using the SAM to obtain an initial mask, and the XMem is used for tracking to obtain continuous real-time masks during robot movement, thus obtaining the segmentation result.

[0062] Step S122: Determine the depth of the target object based on the infrared and color images.

[0063] For example, based on infrared and color images, the depth of the target object is predicted using a learning-based infrared spot method with generalized grasping techniques (ASGrasp). After obtaining the segmentation results, the image positions and depths of the pixels in the region where the target object is located are substituted into the transformation relationship between the image coordinate system and the three-dimensional spatial coordinate system to obtain the spatial positions of the aforementioned pixels, i.e., the object pose of the target object.

[0064] Step S123: Determine the object pose of the target object based on the segmentation results and the depth of the target object.

[0065] That is, by combining the features of the target object in the segmentation results with the spatial information contained in the depth, the object pose of the target object is determined.

[0066] In this embodiment of the disclosure, the target object in the color image is segmented to obtain a segmentation result; the depth of the target object is determined based on the infrared image and the color image; and the object pose of the target object is determined based on the segmentation result and the depth of the target object. Thus, an accurate object pose is obtained using the segmentation result and the depth of the target object.

[0067] In some embodiments, step S101, which determines the target grasping pose of the robot to grasp the target object based on the object pose of the target object, can be implemented as steps S131 and S132:

[0068] Step S131: Determine the target gripper pose for the gripper to grasp the target object based on the object's pose. That is, determine the target gripper pose for the gripper to grasp the target object based on the position and orientation of the target object.

[0069] Step S132: Determine the target base pose based on the target gripper pose. That is, plan the target base pose based on the target gripper pose.

[0070] In this embodiment, the target base pose is a base pose that satisfies the target gripper pose, enabling coordinated movement between the robotic arm and the robot body. That is, the target base pose is a base pose that supports the gripper reaching the target gripper pose. Thus, when controlling the robot's movement based on the target base pose and the target gripper pose, coordinated movement between the robotic arm and the base can be achieved, ultimately enabling the robot's gripper to accurately reach the target gripper pose.

[0071] In some embodiments, step S132, which determines the target base pose based on the target gripper pose, can be implemented as steps S141 and S142:

[0072] Step S141: Based on the target gripper pose, determine multiple candidate base poses to enable the gripper to achieve the target gripper pose.

[0073] Specifically, the kinematic model between the base and the gripper can be calculated in reverse using homogeneous inverse transformation to obtain multiple candidate base poses that can support the gripper to reach the target gripper pose.

[0074] The above candidate base poses (i.e. potential base poses) reflect all possible base poses that can enable the gripper to reach the target gripper pose.

[0075] In some embodiments, base poses that exceed the range of motion, collide with the environment, or are below an accessibility threshold can be removed from a pool of candidate base poses. For example, based on a Generalized Oriented Reachability Map (GORM) technique, the accessibility when switching from the current base pose to each candidate base pose is predicted, a quality score for each candidate pose is obtained, and candidate poses with quality scores below a set threshold are removed, retaining the remaining candidate poses. Accessibility refers to the robot's ability to smoothly switch from the current base pose to a candidate base pose. This removes inferior candidate base poses that exceed the range of motion or collide with the environment, thereby optimizing the base poses in the workspace and obtaining high-quality candidate base poses.

[0076] Step S142: Based on the robot's current base pose, determine the target base pose that is the smallest distance from the current base pose from multiple candidate base poses.

[0077] In some implementations, a distance distribution is calculated based on the distance between the current base pose and the positions of each candidate base, and the minimum distance can be selected from the distance distribution as the target base pose.

[0078] In this embodiment of the disclosure, the robot can be encouraged to move closer to the target base pose that is closest to the current base pose, thereby improving the efficiency of robot movement.

[0079] In some embodiments, step S102, which determines the motion control strategy of the robot's mechanical leg based on the target base pose and the robot's current base pose, can be implemented as steps S151 and S152:

[0080] Step S151: Based on the target base pose and the current base pose, determine the moving speed of the mechanical leg and the angle of each leg joint during the process of the base moving from the current base pose to the target base pose.

[0081] The above-mentioned planning of the movement of the robotic leg may include planning the movement speed of the robotic leg and the angles of each leg joint, and may also include planning parameters such as the angular velocity of the leg joints and gait variables.

[0082] It should be noted that in this step, based on the target base pose and the current base pose, the expected movement speed of the robotic leg and the leg joint angle corresponding to each moment are obtained.

[0083] Specifically, the change from the current base pose to the target base pose can be solved using differential kinematics, which yields continuous movement speed and leg joint angles at each moment. The specific calculation steps will not be elaborated here.

[0084] In addition, in this step, the above-mentioned moving speed and leg joint angle can be determined directly based on the current base posture and the target base posture, see subsequent steps S161 to S162; or the first moving speed and the first leg joint angle can be determined first based on the current base position and the target base position, and then the second moving speed and the second leg joint angle can be determined based on the posture of the base when it is in the target base position and the posture of the target base, see subsequent step S163.

[0085] Step S152: Determine the motion control strategy based on the movement speed of the robotic leg and the angles of each leg joint.

[0086] In this embodiment, based on the target base pose and the current base pose, the moving speed of the robotic leg and the angles of each leg joint are determined during the base's movement from the current base pose to the target base pose. Based on the moving speed of the robotic leg and the angles of each leg joint, a motion control strategy is determined. Thus, the influence of the robotic leg's moving speed and leg joint angles on motion control is considered when determining the motion control strategy, allowing for the selection of a motion control strategy that facilitates smooth control of the robot to reach the target base pose. Consequently, the robot can move smoothly according to the motion control strategy.

[0087] In some embodiments, the influence of the mechanical leg's movement speed and leg joint angle on motion control can be considered, and a motion control strategy that facilitates smooth control of the robot to reach the target base posture can be selected.

[0088] Taking the target base pose, including the target base position and the target base orientation, as an example; the above step S151, based on the target base pose and the current base pose, determines the moving speed of the mechanical leg and the angle of each leg joint during the process of the base moving from the current base pose to the target base pose, can be implemented as steps S161 and S162:

[0089] Step S161: Based on the robot's current base position and target base position, determine the movement speed of the robotic legs and the angles of each leg joint during the process of the base moving from the current base position to the target base position. Step S162: Based on the robot's first posture when the base is at the target base position, and the target base posture, determine the angles of each leg joint during the process of the base moving from the first posture to the target base posture.

[0090] In this embodiment, the moving speed and joint angles of the robotic legs are first planned during the process of the base moving from the current base position to the target base position; then, the joint angles of the robotic legs are planned during the process of the base moving from the first posture to the target base posture. The first posture refers to the posture when the base is located at the target base position.

[0091] In this embodiment of the present disclosure, the base can be quickly moved to the target base position, and the pose of the base can be adjusted at the target base position using the shortest path.

[0092] Taking the target base pose, which includes the target base position and the target base orientation, as an example, step S151 above, based on the target base pose and the current base pose, determines the moving speed of the robotic leg and the angles of each leg joint during the process of the base moving from the current base pose to the target base pose, which can be implemented as step S163:

[0093] Step S163: Based on the current base pose and the target base pose, determine the movement speed of the robotic leg and the angle of each leg joint during the process of the base moving from the current base pose to the target base pose to the target base position.

[0094] As can be seen, the posture of the base is adjusted during the process of moving from the current base position to the target base position, so that the base reaches the target base posture. This allows the base to adjust its posture to the target base posture as it approaches the target object, improving the robot's posture switching efficiency.

[0095] As described above, the motion control strategy refers to controlling the base to move from its current position to the target position. Once the motion control strategy is determined, the robotic legs can be controlled to move the base according to the strategy's instructions, enabling the robot's base to reach the target position. The following steps S171 and S172 will be described in detail:

[0096] Step S171: Based on the motion control strategy, generate a set of leg motion commands for the robotic leg.

[0097] The leg movement command set includes five-dimensional commands, including x-axis, y-axis, z-axis, pitch, and yaw. To ensure smoother robot movement, roll is not included in the five-dimensional commands.

[0098] The following describes one possible way to generate a set of leg movement instructions.

[0099] First, calculate the total motion change of the robot base from the current base pose to the target base pose; then, according to the robot's preset step size, decompose the total motion change into multiple incremental changes, each of which corresponds to the parameter value of a leg movement command; finally, integrate the parameter values ​​of all leg movement commands to form a set of leg movement commands.

[0100] To ensure robot stability and prevent erroneous behavior, the values ​​of parameters in the leg movement command set are restricted based on joint height and pitch angle: θ ~ f, v x ,vy ,ω~g(h,θ), where h represents the joint height, the pitch angle θ satisfies the preset constraint f, and the x-axis movement speed is v. x The speed of movement along the y-axis is v y The angular velocity ω satisfies the preset constraint g(h,θ).

[0101] Step S172: Based on the set of leg motion commands, control the mechanical leg to drive the robot to move so that the base moves from the current base pose to the target base pose.

[0102] To enable the robot to walk naturally, a motion control model obtained through pre-training using a teacher-student method can be used. The set of leg movement commands is input into the motion control model to obtain the output control parameters. Based on the control parameters, the mechanical leg is controlled to move the robot's base from the current base posture to the target base posture. The teacher observation in the teacher-student model includes proprioception and privileged information. Proprietary perception includes at least one of the following: the robot's current leg movement command, gravity vector, angular velocity, joint angle, previous leg movement command, and gait variables. Privileged information includes at least one of the following: the robot's base velocity relative to the ground, the friction coefficient of each contact surface, mass parameters, and motor strength.

[0103] In some implementations, relevant parameters of the robot's leg motion are sampled in spherical coordinates with constant height and pitch angle, and randomization techniques and observation noise are used to determine relevant parameters of the robot's motion, in order to reduce the gap between simulated and actual robot motion.

[0104] In this embodiment of the disclosure, a reward function is defined to track the set of leg movement commands, and penalties are imposed on the robot's vertical (up and down) velocity, pitch angle, and angular velocity to achieve efficient and natural walking of the robot.

[0105] The following section introduces how to set the loss function during training.

[0106] After training a robust teacher model, it is refined into a student model. The teacher model uses a privileged information encoder to generate latent vectors. And generate actions through a multilayer perceptron (MLP). The student model uses a gated recurrent unit (GRU) instead of an encoder, and the GRU receives proprioceptive observation data. And generate the latent vector l t , l t and After connection, action a is generated via MLP. t .

[0107] For l t and at For supervision, the loss function is defined as follows:

[0108] In this embodiment of the present disclosure, a set of leg motion commands for the robotic leg is generated according to a motion control strategy. Based on the set of leg motion commands, the robotic leg is controlled to move the robot's base, causing the base to move from its current base posture to a target base posture. In this way, accurate control of the robotic leg's motion is achieved based on the set of leg motion commands.

[0109] In some embodiments, step S103 above, based on the target gripper pose and the robot's current gripper pose, determines the robot's robotic arm's grasping control strategy, which can be implemented as steps S181 to S183:

[0110] Step S181: Determine the motion trajectory of the robotic arm during the process of the gripper moving from the current gripper pose to the target gripper pose.

[0111] Motion planning is used to plan the motion trajectory of the robotic arm as the gripper moves from its current gripper pose to its target gripper pose. A whole-body reinforcement learning (RL) strategy is used to dynamically adjust the joint control parameters of the robotic arm, thereby solving the control error problem in the end effector control and enabling the gripper to reach the target pose more accurately.

[0112] In some embodiments, the movement of the image acquisition component mounted on the robotic arm can be restricted within a tracking sphere to reduce motion blur and prevent tracking loss. Furthermore, an accessibility map (RM) is used to define the tracking sphere, ensuring that the image acquisition component operates only within a dexterous workspace. Within the dexterous workspace, a motion planner generates motion trajectories.

[0113] During the movement of the robotic arm, the movement range of the image acquisition component on the robotic arm is restricted to a preset tracking ball. By constraining the movement of the image acquisition component through the tracking ball, the probability of motion blur is reduced, thereby preventing the image acquisition component from losing track of the target. The spatial range limited by the tracking ball can be called the dexterous working area, which can be defined by a reachability map (RM).

[0114] Step S182: Determine the angles of each joint of the robotic arm based on the motion trajectory. That is, the angles of each joint of the robotic arm can be determined according to the motion trajectory of the robotic arm.

[0115] Similarly, in this step, the above motion trajectory can be solved into the arm joint angles at each moment using differential kinematics. The specific solution steps will not be described here.

[0116] Step S183: Determine the grasping control strategy based on the angles of each joint of the robotic arm.

[0117] The implementation method of this step is similar to the method of determining the motion control strategy based on the leg joint angle introduced earlier, and will not be repeated here.

[0118] In this embodiment, the angles of each joint of the robotic arm are determined based on the motion trajectory of the gripper as it moves from its current gripper position to the target gripper position. Based on these joint angles, a gripping control strategy is determined. Thus, the gripper can be controlled to move to the target gripper position to accurately grasp the target object, according to the gripping control strategy.

[0119] As described above, the grasping control strategy refers to controlling the gripper to move from its current gripper position to the target gripper position. After determining the grasping control strategy, the robotic arm can be controlled to move the robot's gripper to grasp the target object according to the instructions of the grasping control strategy. The following steps S191 and S192 will be described in detail:

[0120] Step S191: Based on the grasping control strategy, generate a set of arm grasping instructions for the robotic arm.

[0121] Each arm grasping command in the arm grasping command set can be a six-dimensional command, including x-axis, y-axis, z-axis, pitch, yaw, and roll.

[0122] Step S192: Based on the set of arm grasping instructions, control the robotic arm to move the gripper from the current gripper position to the target gripper position to grasp the target object.

[0123] In this embodiment of the present disclosure, a set of gripping instructions for the robotic arm is generated according to a gripping control strategy. Based on the set of gripping instructions, the robotic arm is controlled to move the gripper from its current gripper position to the target gripper position to grasp the target object. In this way, accurate control of the robotic arm operation is achieved based on the set of gripping instructions.

[0124] In some embodiments, step S103, where the base reaches the target base posture, involves controlling the robotic arm to drive the robot's gripper to grasp the target object in the target gripper posture. This can be implemented as step S193:

[0125] Step S193: If the distance between the current base pose and the target base pose is less than or equal to a preset distance threshold, control the gripper to grasp the target object in the target gripper pose.

[0126] For example, during the robot's movement, the robot's motion can be tracked, and the reachability from the current grasping pose to the target grasping pose can be detected in real time based on the GORM technology mentioned above. If the reachability is less than or equal to a preset distance threshold, the gripper's state can be switched to the grasping stage and the target object can be grasped in the target gripper pose.

[0127] In this embodiment of the present disclosure, when the distance between the current base pose and the target base pose is less than or equal to a preset distance threshold, the gripper is controlled to grasp the target object in the target gripper pose. Thus, when the gripper is capable of grasping the target object, it automatically switches to the grasping phase and controls the gripper to grasp the target object, improving the efficiency of the gripper in grasping the target object.

[0128] The robot in the embodiments of this disclosure can be a legged robot, such as a quadruped robot. The robot includes a base, mechanical legs, and a mechanical arm. The base is connected to the mechanical legs and the mechanical arm and is a structural component that provides support for the robot as a whole. The end of the mechanical arm is equipped with a gripper, which can be a gripper, a suction cup, or other end effector, and is not limited thereto.

[0129] Next, let's describe the driving relationships between the aforementioned parts. The motor drives the robot's mechanical legs to move. Since the mechanical legs are connected to the base, they drive the robot's base to move. Since the base is connected to the robotic arm, the movement of the base also drives the movement of the robotic arm. As for the robotic arm, on the one hand, the robotic arm as a whole moves with the movement of the base; on the other hand, the individual joints included in the robotic arm can also rotate independently. Since the gripper is installed at the end of the robotic arm, the gripper moves with the movement of the robotic arm.

[0130] The coordinate system mentioned earlier refers to a three-dimensional coordinate system established with the robot as the reference. This three-dimensional coordinate system can be a coordinate system established with the Tool Center Point (TCP) as the origin, and this three-dimensional coordinate system can also be called the TCP frame. TCP refers to the functional point of action of the end effector, such as the center point of symmetry of the gripper.

[0131] For the target gripper pose, the target object can be identified and its pose determined based on information collected by sensors such as cameras. Then, the target gripper pose can be predicted using geometric model matching.

[0132] In some embodiments, based on the target gripper pose described above, the target base pose, which supports the gripper to reach the target gripper pose, can be predicted. That is, based on the determined target gripper pose, the target base pose that allows the gripper to be positioned in the target gripper pose is determined according to the kinematic relationship between the base and the gripper. Specifically, a kinematic model between the base and the gripper can be pre-established, which describes the spatial relationship between the gripper and the base during robot movement; then, the target gripper pose is substituted into the kinematic model to solve for the target base pose.

[0133] In some embodiments, a motion control strategy can be selected from a pre-defined motion control strategy based on the attitude difference between the target base attitude and the current base attitude and / or the distance between the target base position and the current base position. For example, the final motion control strategy can be determined based on the pre-defined correspondence between attitude differences and / or distances and motion control strategies.

[0134] In some embodiments, a grasping control strategy can be selected from a pre-defined grasping control strategy based on the attitude difference between the target grasper attitude and the current grasper attitude and / or the distance between the target grasper position and the current grasper position. For example, the final grasping control strategy can be determined based on a pre-defined correspondence between attitude differences and / or distances and grasping control strategies.

[0135] This disclosure provides a robot control method, as shown in the flowchart of FIG2, the method including steps S201 to S205:

[0136] Step S201: Obtain the target grasping pose of the robot grasping the target object; the target grasping pose includes the target base pose and the target gripper pose.

[0137] Step S202: Obtain the motion control strategy of the robot's mechanical leg based on the target base pose.

[0138] Step S203: Based on the motion control strategy, control the mechanical leg to drive the robot to move so that the robot's base reaches the target base posture.

[0139] Step S204: Obtain the robot's grasping control strategy based on the target gripper pose.

[0140] Step S205: Based on the grasping control strategy, when the base reaches the target base posture, control the robotic arm to drive the robot's gripper to grasp the target object in the target gripper posture.

[0141] The above robot control method can be understood by referring to the explanation of the above robot behavior planning method embodiment, and has similar beneficial effects as the above robot behavior planning method embodiment.

[0142] This disclosure provides a robot, as shown in FIG3. The robot 300 includes a planning module 301, a base 311, a gripper 312, a robotic arm 313, and robotic legs 314. The planning module 301 is used to determine a target grasping pose for the robot 300 to grasp the target object based on the object pose of the target object; the target grasping pose includes a target base pose and a target gripper pose; based on the target base pose and the current base pose of the robot 300, the planning module 301 determines a motion control strategy for the robotic legs 314 of the robot 300; the motion control strategy is used to control the robotic legs 314 to move the robot 300 so that the base 311 of the robot 300 reaches the target base pose; based on the target gripper pose and the current gripper pose of the robot, the planning module 301 determines a grasping control strategy for the robotic arm 313 of the robot 300; the grasping control strategy is used to control the robotic arm 313 to move the gripper 312 of the robot 300 to grasp the target object in the target gripper pose when the base 311 reaches the target base pose.

[0143] As shown in Figure 4, the robot 300 may also include a perception module 302. The perception module 302 is used to acquire image information of the target object; based on the image information, it identifies and locates the target object and determines the object pose.

[0144] As shown in Figure 5, the robot 300 may also include a motion control module 303. The motion control module 303 is used to generate a set of leg motion commands for the mechanical leg 314 based on a motion control strategy; and based on the set of leg motion commands, control the mechanical leg 314 to drive the robot 300 to move so that the base 311 moves from the current base pose to the target base pose.

[0145] As shown in Figure 6, the robot 300 may also include an operation module 304. The operation module 304 is used to generate a set of arm grasping instructions for the robotic arm 313 based on the grasping control strategy; based on the set of arm grasping instructions, it controls the robotic arm 313 to move the gripper 312 from the current gripper pose to the target gripper pose in order to grasp the target object.

[0146] As shown in Figure 7, the robot 300 may also include an image acquisition component 321, which is mounted on the robotic arm 313 and is used to acquire image information of the target object.

[0147] The following describes the application of the embodiments of this disclosure in real-world scenarios.

[0148] Legged robots with advanced maneuvering capabilities have significant potential to enhance household chores and urban maintenance. While progress has been made in developing robust motion control and precise maneuvering methods, seamlessly integrating these methods into a unified whole-body control system for practical applications remains a challenge.

[0149] The movement and manipulation of quadrupedal robots, integrating leg movements and robotic arm manipulation, has become an important research area due to its wide range of potential applications, including home assistance, urban maintenance, disaster relief, and autonomous field operations. Recent advances in reinforcement learning (RL) have enabled the development of comprehensive policies for the robot's whole-body movement and manipulation, allowing the robot to perform tasks involving seamless coordination of movement and object interaction. While end-to-end reinforcement learning has significantly improved movement capabilities, motion manipulation remains challenging due to the increased dimension of motion and the complex physical interactions involved. These challenges often result in generally low accuracy and limited generalization of motion manipulation policies, especially when grasping objects of different shapes, sizes, and materials. Figure 8 illustrates real-world scenarios of robots grasping different objects: the robot grasping an object in a cluttered environment 801, at a first height 802, and at a second height 803.

[0150] To improve the performance and versatility of full-body grasping systems, inspiration was drawn from the success of various grasp detection technologies. These methods have demonstrated strong performance in detecting grasping postures of various unseen objects (including challenging materials such as transparent or mirrored surfaces) in cluttered environments. By combining grasping posture detection with motion planning, these methods achieve very high accuracy, typically exceeding 90% success rate in grasping arbitrary objects in desktop environments.

[0151] This inspired the integration of leg motion with grasp detection to achieve high-performance, highly versatile motion and manipulation, thus achieving the best of both worlds. However, such integration is very difficult; directly applying grasp detection results to the arm motion planning of a legged robot is insufficient, as it ignores the necessary coordination between the body and arm movements of the legged robot.

[0152] In related technologies, legged robot control methods rely on control-based approaches to complete basic motion tasks, such as Model Predictive Control (MPC). These methods perform well in constrained and controllable environments, but typically require precise modeling and manual tuning. With the rise of deep neural networks, learning-based approaches have become more robust and adaptive, enabling robots to perform a wide range of actions with less human intervention. These learning methods have made significant progress in robust motion control, agile motor skills, dynamic jumping, fall recovery, complex parkour maneuvers, and performance in complex terrain and confined spaces.

[0153] In the field of legged robot motion manipulation, modularization and unified approaches are the two main development directions. The modular approach divides motion and operating systems into independent components, typically managing the motion components through off-the-shelf controllers and optimizing operational techniques according to specific tasks. This effectively simplifies the design process and enhances the system's flexibility and stability. The unified approach, on the other hand, aims to achieve seamless motion manipulation through whole-body control, such as simultaneously controlling leg joints and robotic arms using a unified strategy, thereby improving overall performance.

[0154] Control-based methods are effective in specific environments, but require precise modeling and complex manual adjustments, limiting the robot's adaptability to changing environments. While learning-based methods have made progress in many aspects, the challenge lies in how to organically combine motion and manipulative abilities.

[0155] While modular approaches enhance system flexibility, treating the arm and body as independent components limits workspace expansion and the effectiveness of full-body control. Unified approaches, while excelling in seamless control, often rely on remote operation, leading to insufficient autonomy. Furthermore, early unified methods suffered from limited arm pose and low tracking accuracy, issues not fully resolved even with the introduction of two-stage strategies. In addition, full-body control methods based on Simultaneous Localization and Mapping (SLAM) require substantial task data and external processing, limiting motion manipulation capabilities. Although existing research integrates high-level task planning with low-level control, its grasping accuracy and task generalization ability remain insufficient, and it requires significant training time.

[0156] Based on the above description, this disclosure proposes a Generalizable Quadrupedal Whole-Body Grasping (QuadWBG) framework for legged robots, employing a modular and unified approach. This modular whole-body grasping framework is used for a robust and generalizable whole-body motion manipulation controller based on a single arm-mounted camera. Robust low-level policies for executing commands in five-dimensional (5D) space are implemented through reinforcement learning (RL), and a high-level policy for perception-based grasping based on a Generalized Oriented Accessibility Map (GORM) is introduced. This whole-body grasping framework achieves an advanced first-pass grasping accuracy of 89% in real-world environments, successfully completing challenging tasks including grasping transparent objects. Extensive simulation and real-world experiments demonstrate that the system can effectively manage a wide workspace from the ground to above the body and perform diverse whole-body motion manipulation tasks.

[0157] The full-body grasping framework is a modular system, as shown in Figure 9, consisting of four key components: a motion control module 303, a perception module 302, a manipulation module 304, and a planning module 301. First, a reinforcement learning strategy is used to track five-dimensional commands, enabling the motion control module 303 to achieve robust robot movement. Simultaneously, the perception module 302 generates real-time object masks and grasping postures to guide the manipulation module in transitioning the arm from tracking to grasping within the robot frame. The planning module 301 is trained based on GORM to optimize the robot's base position and improve grasping performance. All strategies are trained using the Proximal Policy Optimization (PPO) algorithm in a simulation environment.

[0158] The full-body grasping frame is built on a quadruped robot and equipped with a robotic arm. For example, the end of the robotic arm is equipped with an electric gripper, and a camera is installed at the wrist of the robotic arm.

[0159] The flowchart of the operation of each module in the modular system is shown in Figure 10, which may include steps S1001 to S1004:

[0160] Step S1001: The perception module 302 receives the image signal, obtains the real-time grasping pose based on the neural network, and sends the real-time grasping pose to the planning module 301 and the operation module 304.

[0161] Step S1002: The planning module 301 receives the real-time grasped pose, determines the motion control command of the mechanical leg (i.e., the body motion command) based on the neural network, and sends the motion control command to the motion control module 303;

[0162] Step S1003: The operation module 304 receives the real-time grasping pose and controls the arm operation based on the neural network;

[0163] Step S1004: The motion control module 303 receives a motion control command and controls the robot's movement based on a neural network.

[0164] The four modules are described below:

[0165] (1) Motion control module;

[0166] The specific implementation process of the motion control module is shown in Figure 11, and may include steps S1101 to S1103:

[0167] Step S1101: Generate teacher's motion control strategy;

[0168] Based on privileged information and ontological perception, a teacher's motion control strategy is generated.

[0169] Step S1102: Generate student motion control strategy;

[0170] Based on ontological perception, online learning is supervised using teacher motor control strategies to generate student motor control strategies.

[0171] Step S1103: Experiment in a simulation environment.

[0172] The target motor position, determined by the teacher's motion control strategy and the student's motion control strategy, is tested in a simulation environment.

[0173] Flexible motion strategies are crucial for achieving high precision in whole-body motion control systems. Therefore, embodiments of this disclosure employ a teacher-student architecture and extend the five-dimensional instruction set to include the pitch angle and height at the robot's center. To ensure stability and avoid undesirable behavior, the instruction range is constrained based on the pitch angle and height: θ ~ f, v x ,v y ,ω~g(h,θ), it can be seen that the pitch angle θ satisfies the f constraint, and the x-axis movement speed is v x The speed of movement along the y-axis is v y The angular velocity ω satisfies the g(h,θ) constraint.

[0174] The observation of teacher behavior consists of proprioception and privileged information. proprioception Includes instructions (corresponding to the leg movement instructions in the aforementioned embodiments). Gravity vector angular velocity Joint position and velocity (i.e., joint angle) q t q t∈R 18 Previous action a t-1 ∈R 12 and gait variables Privileged information includes parameters that cannot be accessed in real time, such as the robot's base speed relative to the ground, the coefficient of friction of each contact surface, mass parameters, and motor strength.

[0175] This disclosure defines a reward function for tracking instructions to achieve efficient and natural walking; it penalizes the body's vertical velocity, roll, and pitch angular velocities; it uses a Raibert heuristic to determine the robot's foot position and foot spacing to prevent tripping, and the smoothness of movement for natural motion. Gait adjustment and energy rewards are applied to the robot to achieve a more stable and smooth gait.

[0176] The motion strategy needs to accurately track 5D commands despite disturbances generated during arm tracking and object grasping by the manipulator module. To this end, embodiments of this disclosure employ a sampling method based on height- and pitch-invariant spherical coordinates. Furthermore, wide-domain randomization techniques and observation noise are applied to reduce the gap between simulation and reality in robot dynamics (see Table 1). Among these strategies, increasing proprioceptive latency has proven crucial in preventing unstable motion caused by onboard computational delays.

[0177] Table 1 Randomly set parameters

[0178] (2) Sensing module;

[0179] The specific implementation process of the sensing module is shown in Figure 12, and may include steps S1201 to S1203:

[0180] Step S1201: ASGrasp capture;

[0181] Infrared images are used as input to ASGrasp to predict the depth of objects.

[0182] Step S1202: Image tracking;

[0183] Image segmentation and tracking are performed on the color image to obtain the image segmentation result.

[0184] Step S1203: Grab and select.

[0185] Real-time capture poses are generated by capturing selection.

[0186] To achieve real-time tracking and accurate grasping pose prediction, embodiments of this disclosure employ a Track Anything Model (TAM) and a generalized grasping technique (ASGrasp). After generating an initial mask using the Segment Anything Model (SAM) within the TAM, the memory module (XMem) outputs a real-time object mask based on the initial mask. Using infrared (IR) and color (RGB) images as input to ASGrasp, accurate depth prediction of transparent and reflective surfaces is achieved. The generated depth point cloud is then fed into a Geometric and Scene-aware Network (GSNet) model to generate a more accurate six-DOF grasping pose.

[0187] (3) Operation module;

[0188] The specific implementation process of the operation module is shown in Figure 13, and may include steps S1301 to S1304:

[0189] Step S1301: The motion planner plans the trajectory of the robotic arm;

[0190] The motion planner generates the robotic arm trajectory based on the target grasping pose, and the robotic arm trajectory is used to determine the actual motor position.

[0191] Step S1302: Determine the motor position using a differentiable kinematics model;

[0192] Based on the robotic arm trajectory, the motor position is determined using a differentiable kinematics model.

[0193] Here, the motor position is the joint angle of the robotic arm.

[0194] Step S1303: The PD controller determines the motor torque;

[0195] The appropriate motor torque is determined by the PD controller based on the motor location.

[0196] Step S1304: Control the movement of the robotic arm.

[0197] The movement of the robotic arm is controlled by the motor torque.

[0198] The manipulation module is designed to actively track and grasp target objects while adjusting the robot's basic motion. A motion planning method is employed, utilizing a whole-body RL strategy to address control errors in the end effector control. The system operates in two distinct phases: tracking and grasping.

[0199] During tracking, the motion of the camera mounted on the robot is confined within a predefined tracking sphere to reduce motion blur and prevent tracking loss. The tracking sphere is parameterized by its position and radius within the robot's frame. In this embodiment, a reachability map (RM) is used to define the tracking sphere, ensuring that the camera operates only within its dexterity workspace, i.e., an effective inverse kinematics (IK) solution is available in any orientation. The switching mechanism is built based on the reachability map and a threshold criterion; in each planning step, the reachability of the selected grasping posture is calculated using the RM. Once the threshold is reached, the system switches to the grasping phase. The motion planner can generate trajectories online, enabling the system to adapt to small, unexpected movements as it approaches the target.

[0200] (4) Planning module.

[0201] The specific implementation process of the planning module is shown in Figure 14, which may include steps S1401 to S1403:

[0202] Step S1401: Determine the generalized directional reachability graph (GORM);

[0203] Based on the real-time captured pose, a generalized orientation reachability map (GORM) is determined; based on the GORM, the fuselage reachability pose distribution can be determined.

[0204] Step S1402: Determine the planning strategy;

[0205] Based on the real-time captured pose, a planning strategy is determined and motion commands are generated.

[0206] Step S1403: Determine the motion control strategy.

[0207] Based on the motion instructions, a motion control strategy is generated.

[0208] Reachability maps (RM) are a pose quality metric commonly used to provide priors and targets for mobile maneuvering tasks. Oriented Reachability Maps (ORMs) can effectively represent potential basic poses relative to a tool center point (TCP) frame (corresponding to candidate base poses in the aforementioned embodiments). However, ORMs are typically applied to platform-based mobile maneuvering, where the robot's base is limited to a plane. This disclosure proposes a Generalized Oriented Reachability Map (GORM) that supports six degrees of freedom for robot base localization. As shown in Figure 15, using the GORM 1502, candidate base poses 1503 are determined within a cuboid frame, and the gripper grasps the object at the target pose 1501.

[0209] For any target pose (corresponding to the target gripper pose in the aforementioned embodiment) p∈SE(3) within the world framework, the distribution of potential bases in the world coordinate system is calculated by inverse RM. The space is improved by removing the following basic poses: 1) out of range of motion; 2) collision with the environment; 3) below the reachability threshold. Once the target pose is defined, GORM provides a high-quality distribution of potential base locations. By training an advanced policy, the distance between the current base pose and the nearest feasible location is minimized: r GORM =exp(-(min(d)) GORM )) 2 (2);

[0210] In each high-level decision step, a distance distribution dGORM is calculated using the sum of the Euclidean distance and the geodetic distance between the GORM and the current base pose. rGORM represents the minimum distance selected from the distance distribution to encourage the robot to approach the nearest candidate base pose. Since the GORM is defined within the target pose framework, it only needs to be computed once, making it highly efficient and well-suited for parallel training.

[0211] In vision-based control, training obstacle avoidance and object grasping strategies based on depth information is crucial. However, these methods often introduce noise addition and padding techniques to bridge the gap between simulation and reality, but these techniques can degrade system performance in precision tasks such as grasping. In contrast, the method of this disclosure does not rely on simulated depth images, but directly uses grasping pose detection results from off-the-shelf perception modules.

[0212] The definition of the observations for advanced strategies is given in Equation (3):

[0213] The vector Contains a rotation matrix rot t ∈R 9 and vector space mapping trans t ∈R 3 The gravity vector within the body frame is... The joint position is q t ∈R 18 And the last action is a t-1 ∈R 5 This action includes five-dimensional leg movement control commands.

[0214] In this embodiment of the disclosure, a simulation experiment of the full-body grasping frame is first conducted, in which the system is deployed on an onboard microcomputer.

[0215] Motion control module evaluation: The performance and energy efficiency of several motion controllers were compared, including the AsymAC controller, ROA controller, StateEstimator controller, and teacher-student controller. All methods were trained to track velocity commands under consistent conditions and tested in a simulated environment with significant randomization and observation noise to replicate real-world scenarios. Performance metrics were evaluated using command tracking reward and average torque, respectively. The teacher-student controller offered the optimal trade-off between low energy consumption and high performance and was therefore selected as the core of the motion controller.

[0216] An evaluation of the robot's full-body workspace was conducted: As shown in Figure 16, by tracking commands, the motion control strategy, guided by the arm's proprioceptive feedback, expanded the full-body workspace without sacrificing robustness. The sizes of the arm workspace 1601 and the full-body workspace 1602 were compared. Table 2 shows an example of the two-dimensional area and three-dimensional volume of the arm workspace 1601 and the full-body workspace 1602. It can be seen that the controller increased the effective workspace volume by 54% and the workspace area by 33%, effectively executing all commands under various body postures (leaning forward, leaning back, squatting).

[0217] Table 2 Size of Arm Workspace 1601 and Full-Body Workspace 1602

[0218] A general whole-body grasping test was conducted: The same simulation baseline as Visual Whole-Body Control (VBC) was used, with 34 objects categorized into 7 classes: ball, long box, square box, bottle, cup, bowl, and drill. In the picking experiments, the robot and object positions and orientations were randomly reset at the start of each round. 300 trials were performed on each object, and the success rate was evaluated. Success was defined as picking up the target object before 150 advanced steps. The method of this disclosure (i.e., the method with GORM reward), the method without GORM reward, and the baseline method were compared. For the method without GORM reward and the baseline method, VBC was used to acquire assistant rewards. The purpose of this comparison was to verify whether the method of this disclosure could accurately complete the movement and picking tasks of different categories of objects. Table 3 shows the success rates of picking up balls, long boxes, square boxes, bottles, cups, bowls, and drills using the method of this disclosure, the method without GORM reward, and the baseline method, as well as the standard deviation of the object grasping success rates for these three methods. The method of this disclosure achieved a significantly higher success rate across all test objects. It is noteworthy that the standard deviation of end-to-end RL methods such as VBC is 14.53, reflecting considerable variability. VBC performs well on simple and small objects, such as square boxes (80%), but struggles with more complex or larger objects, such as drill bits (53.33%) and long boxes (28.57%). In contrast, the method of this disclosure has a much smaller standard deviation of only 3.46, indicating that the method of this disclosure performs more consistently across all object types. Regardless of the size or geometric complexity of the object, the method of this disclosure exhibits consistent performance. This robustness is largely attributed to the integration of a grasping posture detector, which provides less redundant input to the high-level strategy, enabling the system to adapt more effectively to different objects and ensuring consistent performance even under challenging conditions. The success rate drops significantly after the GORM reward is removed, likely due to the inability to guide the arm into the optimal workspace, especially in tasks requiring precise manipulation skills.

[0219] Table 3 shows the success rates of object grasping using the method with GORM reward, the method without GORM reward, and the baseline method.

[0220] GORM-guided grasping was evaluated using the same environment as described above, but with a cup as the target object. Four different heights were set for the target object: 0 meters (floor), 0.3 meters (box), 0.75 meters (table), and 1 meter (shelf), with 500 trials per test. After the 150th high-level step in each test, the average reachability of the target pose relative to the body frame was calculated. Three strategies were compared: a depth-based occlusion strategy, a strategy without GORM rewards, and a strategy with GORM rewards. The strategy with GORM rewards used grasping detection results obtained from the perception module, while the strategy based on occlusion depth used occlusion depth images acquired from the camera sensor. The depth-based occlusion strategy and the strategy without GORM rewards were trained based on the proximity reward and auxiliary reward designed in VBC. As shown in Table 5, the reachability of grasping objects on the floor, box, table, and shelf was tested using the depth-based occlusion strategy, the strategy without GORM rewards, and the strategy with GORM rewards, respectively, and the reachability results were obtained. At all height levels, the masking depth-based strategy consistently performs poorly, especially when the target position is below the base's default range of motion. Without GORM rewards, the accessibility of this strategy decreases at different heights because it cannot find the optimal base grasping posture. In contrast, with GORM rewards, the strategy guides the motion module to find the optimal base position for grasping, maintaining high accessibility at all heights and maximizing performance regardless of object height. As shown in Table 3, the method with GORM rewards significantly improves the overall grasping success rate, indicating that guiding the robot to the optimal basic posture enables the manipulation module to perform more precise motion planning, thus making the system more robust and reliable in grasping tasks.

[0221] In this embodiment of the disclosure, the full-body grabbing frame is then evaluated in a real-world event.

[0222] Regarding the comparison of VBC objects: The method of this disclosure embodiment is compared with reinforcement learning methods and modular methods. The former can grasp objects of different heights, while the latter can only effectively handle table height settings. Fourteen real-world objects were collected and placed on the floor and table in random poses. As shown in Table 4, the GAMMA method, VBC method, and GROM reward method were used to test the grasping of objects on the floor, objects on the table, transparent objects, and arbitrary objects, respectively, and the grasping success rate results were obtained. The method of this disclosure embodiment performed best in all tasks. The GAMMA grasping method using a model-based low-level controller (i.e., a one-time grasping method) could not grasp objects on the floor. VBC has a lower success rate due to its use of an RL-based control system.

[0223] Table 4 compares the success rates of the GORM reward method, GAMMA grasping method, and baseline method under different object types and heights.

[0224] Table 5 compares the accessibility of the occlusion depth-based strategy, the strategy without GORM reward, and the strategy with GORM reward at different altitudes.

[0225] Experiments were conducted on arbitrary objects: Figure 17 shows the dataset used in real-world testing, which includes objects from the YCB dataset and common recyclable items such as plastic bottles, cans, and crumpled paper or plastic packaging. Additionally, the dataset includes small toys frequently used in grasping tests, and a large portion of the dataset consists of transparent objects of various shapes. During each trial, the dataset was shuffled, and a set of objects was randomly dropped from a height of 1 meter to simulate random orientation and position. After simply labeling the mask of the target object in the first frame, the automatic grasping process began. Table 6 provides the results of 100 trials conducted in a real-world environment, showing the success rate, tracking error rate, grasping error rate, and drop rate (objects falling after being grasped) of the object grasping test using the method of this embodiment. It can be seen that the method of this embodiment achieved a success rate of 89%. This demonstrates the versatility of the full-body grasping framework on various objects, thanks to its modular design that combines off-the-shelf perception models with powerful, learned motor skills.

[0226] Table 6 shows the results of 100 tests conducted in a real-world environment using the methods of the embodiments of this disclosure.

[0227] Experiments on transparent objects: Ten experiments were conducted on grasping transparent objects according to embodiments of this disclosure, as shown in Table 6. The results show that the grasping accuracy of a single camera reached 80% in cluttered scenes. This challenging task was accomplished through adaptive coordination of the components, inheriting the capabilities of the perception module without performance degradation.

[0228] Based on the above evaluation, this full-body grasping framework combines the learned five-dimensional basic motion strategy with a generalized orientational accessibility graph (GORM) to achieve precise and robust positioning and manipulation. This method effectively integrates precise manipulation with coordinated full-body motion, resulting in significant improvements in handling various tasks in both simulated and real-world scenarios.

[0229] Based on the foregoing embodiments, this disclosure provides a robot behavior planning device and a robot control method. The device includes various modules and units included in each module, which can be implemented by a processor in a computer device; of course, it can also be implemented by specific logic circuits. In the implementation process, the processor can be a central processing unit (CPU), a microprocessor unit (MPU), a digital signal processor (DSP), or a field programmable gate array (FPGA), etc.

[0230] Figure 18 is a schematic diagram of the composition structure of a robot behavior planning device provided in an embodiment of the present disclosure. As shown in Figure 18, the robot behavior planning device 1800 includes a first determining module 1801, a second determining module 1802 and a third determining module 1803.

[0231] The first determining module 1801 is used to determine the target grasping pose of the robot to grasp the target object based on the object pose of the target object; the target grasping pose includes the target base pose and the target gripper pose.

[0232] The second determining module 1802 is used to determine the motion control strategy of the robot's mechanical leg based on the target base pose and the robot's current base pose; the motion control strategy is used to control the mechanical leg to drive the robot's base to move so that the robot's base reaches the target base pose.

[0233] The third determining module 1803 is used to determine the grasping control strategy of the robot's robotic arm based on the target gripper pose and the robot's current gripper pose. The grasping control strategy is used to control the robotic arm to drive the robot's gripper to grasp the target object in the target gripper pose when the base reaches the target base pose.

[0234] In some embodiments, before determining the target grasping pose of the robot to grasp the target object based on the object pose of the target object, the robot's behavior planning device includes: an acquisition unit for acquiring image information of the target object; and a first determination unit for identifying and locating the target object based on the image information, and determining the object pose of the target object.

[0235] In some embodiments, the first determining unit includes: a first segmentation subunit, configured to segment the target object in the color image to obtain a segmentation result; a first determining subunit, configured to determine the depth of the target object based on the infrared image and the color image; and a second determining subunit, configured to determine the object pose of the target object based on the segmentation result and the depth of the target object.

[0236] In some embodiments, the first determining module includes: a second determining unit, configured to determine the target gripper pose for gripping the target object based on the object pose; and a third determining unit, configured to determine the target base pose based on the target gripper pose.

[0237] In some embodiments, the third determining unit includes: a third determining subunit for determining a plurality of candidate base poses to enable the gripper to reach the target gripper pose based on the target gripper pose; and a fourth determining subunit for determining the target base pose with the smallest distance from the plurality of candidate base poses based on the robot's current base pose.

[0238] In some embodiments, the second determining module includes: a fourth determining unit, configured to determine the moving speed of the mechanical leg and the angle of each leg joint during the process of the base moving from the current base position to the target base position based on the target base position and the current base position; and a fifth determining unit, configured to determine a motion control strategy based on the moving speed of the mechanical leg and the angle of each leg joint.

[0239] In some embodiments, the target base pose includes the target base position and the target base orientation; the fourth determining unit includes: a fifth determining subunit, used to determine the moving speed of the mechanical leg and the angle of each leg joint during the process of the base moving from the current base position to the target base position based on the robot's current base position and the target base position; and a sixth determining subunit, used to determine the angle of each leg joint of the mechanical leg during the process of the base moving from the first orientation to the target base orientation based on the robot's first orientation when the base is at the target base position and the target base orientation.

[0240] In some embodiments, the target base pose includes the target base position and the target base orientation; the fourth determining unit further includes: a seventh determining subunit, used to determine the moving speed of the mechanical leg and the angle of each leg joint during the process of the base moving from the current base pose to the target base position in the target base orientation based on the current base pose and the target base pose.

[0241] In some embodiments, the second determining module includes: a first generating unit, configured to generate a set of leg motion instructions for the robotic leg based on a motion control strategy; and a first controlling unit, configured to control the robotic leg to move the robot's base so that the base moves from the current base pose to a target base pose based on the set of leg motion instructions.

[0242] In some embodiments, the third determining module includes: a sixth determining unit, configured to determine the motion trajectory of the robotic arm during the process of the gripper moving from the current gripper pose to the target gripper pose; a seventh determining unit, configured to determine the angles of each arm joint of the robotic arm based on the motion trajectory; and an eighth determining unit, configured to determine a grasping control strategy based on the angles of each arm joint of the robotic arm.

[0243] In some embodiments, the third determining module further includes: a second generating unit, configured to generate a set of arm grasping instructions for the robotic arm based on a grasping control strategy; and a second controlling unit, configured to control the robotic arm to move the gripper from the current gripper pose to the target gripper pose to grasp the target object based on the set of arm grasping instructions.

[0244] In some embodiments, the third determining module further includes: a third control unit, configured to control the gripper to grasp the target object in the target gripper pose when the distance between the robot's current base pose and the target base pose is less than or equal to a preset distance threshold.

[0245] Figure 19 is a schematic diagram of the composition structure of a robot control device provided in an embodiment of the present disclosure. The robot control device 1900 includes a first acquisition module 1901, a second acquisition module 1902, a first control module 1903, a third acquisition module 1904, and a second control module 1905.

[0246] The first acquisition module 1901 is used to acquire the target grasping pose of the robot grasping the target object; the target grasping pose includes the target base pose and the target gripper pose.

[0247] The second acquisition module 1902 is used to acquire the motion control strategy of the robot's mechanical leg based on the target base pose.

[0248] The first control module 1903 is used to control the mechanical leg to drive the robot's base to move based on a motion control strategy, so that the robot's base can reach the target base posture.

[0249] The third acquisition module 1904 is used to acquire the grasping control strategy of the robot's robotic arm based on the target gripper pose.

[0250] The second control module 1905 is used to control the robotic arm to drive the robot's gripper to grasp the target object in the target gripper pose when the base reaches the target base pose, based on the grasping control strategy.

[0251] The descriptions of the apparatus embodiments above are similar to those of the method embodiments above, and have similar beneficial effects. In some embodiments, the functions or modules included in the apparatus provided by this disclosure can be used to perform the methods described in the method embodiments above. For technical details not disclosed in the apparatus embodiments of this disclosure, please refer to the descriptions of the method embodiments of this disclosure for understanding.

[0252] It should be noted that, in the embodiments of this disclosure, if the above methods are implemented as software functional modules and sold or used as independent products, they can also be stored in a computer-readable storage medium. That is, the technical solutions of the embodiments of this disclosure, or the parts that contribute to related technologies, can be embodied in the form of a software product. This software product is stored in a storage medium and includes several instructions to cause a computer device (such as a personal computer, server, or network device) to execute all or part of the methods of the various embodiments of this disclosure. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), magnetic disks, or optical disks. In other words, the embodiments of this disclosure are not limited to any specific hardware, software, or firmware, or any combination of hardware, software, and firmware.

[0253] This disclosure provides a computer device including a memory and a processor. The memory stores a computer program that can run on the processor. When the processor executes the program, it implements some or all of the steps in the above-described method.

[0254] This disclosure provides a computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements some or all of the steps in the above-described method. The computer-readable storage medium can be transient or non-transient.

[0255] This disclosure provides a computer program including computer-readable code. When the computer-readable code is executed in a computer device, a processor in the computer device performs some or all of the steps in the above-described method.

[0256] This disclosure provides a computer program product, which includes a non-transitory computer-readable storage medium storing a computer program. When the computer program is read and executed by a computer, it implements some or all of the steps in the above-described method. This computer program product can be implemented specifically through hardware, software, or a combination thereof. In some embodiments, the computer program product is specifically embodied as a computer storage medium; in other embodiments, the computer program product is specifically embodied as a software product, such as a software development kit (SDK), etc.

[0257] The foregoing descriptions of the various embodiments tend to emphasize the differences between them, while their similarities or commonalities can be referenced interchangeably. The descriptions of the above embodiments of the device, storage medium, computer program, and computer program product are similar to the descriptions of the above method embodiments and have similar beneficial effects. For technical details not disclosed in the embodiments of the device, storage medium, computer program, and computer program product of this disclosure, please refer to the descriptions of the method embodiments of this disclosure for understanding.

[0258] Figure 20 is a schematic diagram of a hardware entity of a computer device in an embodiment of the present disclosure. The hardware entity of the computer device 2000 includes a processor 2001, a communication interface 2002, and a memory 2003.

[0259] Processor 2001 typically controls the overall operation of computer device 2000.

[0260] Communication Interface 2002 enables computer devices to communicate with other terminals or servers over a network.

[0261] The memory 2003 is configured to store instructions and applications executable by the processor 2001, and can also cache data to be processed or already processed (e.g., image data, audio data, voice communication data, and video communication data) in the processor 2001 and various modules in the computer device 2000. It can be implemented using flash memory or random access memory (RAM). Data transfer between the processor 2001, the communication interface 2002, and the memory 2003 can be performed via the bus 2004.

[0262] It should be understood that the phrase "one embodiment" or "an embodiment" mentioned in the specification means that a specific feature, structure, or characteristic related to the embodiment is included in at least one embodiment of this disclosure. Therefore, "in one embodiment" or "in an embodiment" appearing throughout the specification do not necessarily refer to the same embodiment. Furthermore, these specific features, structures, or characteristics can be combined in any suitable manner in one or more embodiments. That is, the sequence number of the above steps / processes does not imply the order of execution; the execution order of each step / process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of this disclosure. The sequence numbers of the above embodiments of this disclosure are for descriptive purposes only and do not represent the superiority or inferiority of the embodiments.

[0263] In this document, the terms “comprising,” “including,” or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only the listed elements but also other elements not expressly listed. Without further limitation, an element defined by the phrase “comprising one…” does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element.

[0264] In the several embodiments provided in this disclosure, the disclosed devices and methods can be implemented in other ways. The device embodiments described above are merely illustrative; for example, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined, integrated into another system, or some features may be ignored or not executed. In addition, the coupling, direct coupling, or communication connection between the various components shown or discussed may be through some interfaces, and the indirect coupling or communication connection between devices or units may be electrical, mechanical, or other forms.

[0265] The units described above as separate components may or may not be physically separate, and the components shown as units may or may not be physical units; they may be located in one place or distributed across multiple network units; some or all of the units may be selected to achieve the purpose of this embodiment according to actual needs. Furthermore, the functional units in the embodiments of this disclosure may all be integrated into one processing unit, or each unit may be a separate unit, or two or more units may be integrated into one unit; the integrated unit may be implemented in hardware or in a combination of hardware and software functional units.

[0266] Those skilled in the art will understand that all or part of the steps of the above method embodiments can be implemented by hardware related to program instructions. The aforementioned program can be stored in a computer-readable storage medium. When the program is executed, it performs the steps of the above method embodiments. The aforementioned storage medium includes various media that can store program code, such as mobile storage devices, ROMs, magnetic disks, or optical disks.

[0267] Alternatively, if the integrated units described above are implemented as software functional modules and sold or used as independent products, they can also be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this disclosure, or the part that contributes to related technologies, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the methods of the various embodiments of this disclosure. The aforementioned storage medium includes various media capable of storing program code, such as mobile storage devices, ROMs, magnetic disks, or optical disks.

[0268] The above are merely embodiments of this disclosure, but the scope of protection of this disclosure is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this disclosure should be included within the scope of protection of this disclosure.

Claims

1. A method for planning the behavior of a robot, characterized in that, The method includes: Based on the object pose of the target object, the target grasping pose for the robot to grasp the target object is determined; the target grasping pose includes the target base pose and the target gripper pose. Based on the target base pose and the robot's current base pose, a motion control strategy for the robot's robotic legs is determined; the motion control strategy is used to control the robotic legs to move the robot's base so that the base reaches the target base pose. Based on the target gripper pose and the robot's current gripper pose, a gripping control strategy for the robot's robotic arm is determined; the gripping control strategy is used to control the robotic arm to drive the robot's gripper to grasp the target object in the target gripper pose when the base reaches the target base pose.

2. The method according to claim 1, characterized in that, Before determining the target grasping pose of the robot to grasp the target object based on the object pose of the target object, the method further includes: Acquire infrared and color images of the target object; The target object in the color image is segmented to obtain the segmentation result; The depth of the target object is determined based on the infrared image and the color image; Based on the segmentation results and the depth of the target object, the object pose of the target object is determined.

3. The method according to claim 1 or 2, characterized in that, The determination of the target grasping pose for the robot to grasp the target object based on the object pose of the target object includes: Based on the object pose, the target gripper pose for the gripper to grasp the target object is determined; Based on the target gripper pose, the target base pose is determined.

4. The method according to claim 3, characterized in that, Determining the target base pose based on the target gripper pose includes: Based on the target gripper pose, multiple candidate base poses are determined to enable the gripper to achieve the target gripper pose; Based on the robot's current base pose, a target base pose with the smallest distance from the current base pose is determined from a plurality of candidate base poses.

5. The method according to any one of claims 1 to 4, characterized in that, The process of determining the motion control strategy for the robot's robotic leg based on the target base pose and the robot's current base pose includes: Based on the target base seat posture and the current base seat posture, determine the moving speed of the mechanical leg and the angle of each leg joint during the process of the base moving from the current base seat posture to the target base seat posture; The motion control strategy is determined based on the moving speed of the mechanical leg and the angles of each leg joint.

6. The method according to claim 5, characterized in that, The target base posture includes the target base position and the target base attitude; The determination of the movement speed of the mechanical leg and the angles of each leg joint during the movement of the base from the current base position to the target base position, based on the target base position and the current base position, includes at least one of the following: Based on the robot's current base position and the target base position, determine the movement speed of the mechanical leg and the angle of each leg joint during the process of the base moving from the current base position to the target base position; based on the robot's first posture of the base when it is at the target base position, and the target base posture, determine the angle of each leg joint of the mechanical leg during the process of the base moving from the first posture to the target base posture. Based on the current base pose and the target base pose, determine the moving speed of the mechanical leg and the angle of each leg joint during the process of the base moving from the current base pose to the target base pose to the target base position.

7. The method according to any one of claims 1 to 6, characterized in that, The step of determining the grasping control strategy of the robot's robotic arm based on the target gripper pose and the robot's current gripper pose includes: Determine the motion trajectory of the robotic arm during the process of the gripper moving from the current gripper position to the target gripper position; Based on the motion trajectory, the angles of each joint of the robotic arm are determined; The grasping control strategy is determined based on the angles of each joint of the robotic arm.

8. The method according to any one of claims 1 to 7, characterized in that, When the base reaches the target base position, controlling the robotic arm to drive the robot's gripper to grasp the target object in the target gripper position includes: If the distance between the current base pose and the target base pose is less than or equal to a preset distance threshold, the gripper is controlled to grasp the target object in the target gripper pose.

9. A method for controlling a robot, characterized in that, The method includes: The target grasping pose of the robot is obtained; the target grasping pose includes the target base pose and the target grasper pose. The motion control strategy for the robot's mechanical legs is obtained based on the target base pose. Based on the motion control strategy, the mechanical leg is controlled to drive the robot's base to move, so that the robot's base reaches the target base posture; The grasping control strategy of the robot's robotic arm is obtained based on the target gripper pose; Based on the grasping control strategy, when the base reaches the target base posture, the robotic arm is controlled to drive the robot's gripper to grasp the target object in the target gripper posture.

10. A behavior planning device for a robot, characterized in that, The device includes: The first determining module is used to determine the target grasping pose of the robot to grasp the target object based on the object pose of the target object; the target grasping pose includes the target base pose and the target gripper pose; The second determining module is used to determine the motion control strategy of the robot's mechanical leg based on the target base pose and the robot's current base pose; the motion control strategy is used to control the mechanical leg to drive the robot's base to move so that the base reaches the target base pose; The third determining module is used to determine the grasping control strategy of the robot's robotic arm based on the target gripper pose and the robot's current gripper pose; the grasping control strategy is used to control the robotic arm to drive the robot's gripper to grasp the target object in the target gripper pose when the base reaches the target base pose.

11. A control device for a robot, characterized in that, The device includes: The first acquisition module is used to acquire the target grasping pose of the robot grasping the target object; the target grasping pose includes the target base pose and the target grasper pose; The second acquisition module is used to acquire the motion control strategy of the robot's mechanical leg based on the target base pose; The first control module is used to control the mechanical leg to move the robot's base based on the motion control strategy, so that the robot's base reaches the target base posture; The third acquisition module is used to acquire the grasping control strategy of the robot's robotic arm based on the pose of the target gripper. The second control module is used to control the robotic arm to drive the robot's gripper to grasp the target object in the target gripper posture when the base reaches the target base posture, based on the grasping control strategy.

12. A robot comprising a planning module, a base, a gripper, a robotic arm, and robotic legs, wherein: The planning module is used to determine the target grasping pose of the robot to grasp the target object based on the object pose of the target object; the target grasping pose includes the target base pose and the target gripper pose; based on the target base pose and the robot's current base pose, a motion control strategy for the robot's robotic legs is determined; the motion control strategy is used to control the robotic legs to move the robot's base so that the robot's base reaches the target base pose; based on the target gripper pose and the robot's current gripper pose, a grasping control strategy for the robot's robotic arm is determined; the grasping control strategy is used to control the robotic arm to move the robot's gripper to grasp the target object in the target gripper pose when the base reaches the target base pose.

13. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, implements the method of any one of claims 1 to 8 or 9.

14. A computer program product, characterized in that, It includes computer-readable code, in which, when the computer-readable code is executed in a computer device, the processor in the computer device performs the method of any one of claims 1 to 8 or 9.

Images