Robotic arm control method and apparatus, electronic device, and computer program product
By acquiring the pixel coordinates of the target object of the robotic arm to generate movement information, the robotic arm is controlled to move on the plane to meet the assembly conditions. This solves the problem of low assembly accuracy of robotic arms in unstructured scenarios in the prior art and realizes high-precision assembly operations.
Patent Information
- Application Number
- PCT/CN2025/146374
- Authority / Receiving Office
- WO · WO
- Patent Type
- Applications
- Current Assignee / Owner
- Priority Date
- 2025-05-09
- Filing Date
- 2025-12-27
- Publication Date
- 2026-11-12
AI Technical Summary
Existing robotic arm control solutions struggle to handle changes in the positional relationships of target objects in unstructured workpiece assembly scenarios, resulting in low operational accuracy and an inability to accurately complete assembly tasks.
By acquiring the pixel coordinates of the target object held by the robotic arm and the pixel coordinates of the target's movement position, movement information is generated, the robotic arm is controlled to move on the plane, and assembly is completed with a preset force when the assembly conditions are met. This reduces the conversion between the pixel coordinate system and the world coordinate system, and improves the flexibility and accuracy of the movement trajectory.
It improves the assembly accuracy and success rate of robotic arms in unstructured scenarios, reduces the risk of damage when the target object is misaligned, and enhances the safety and stability of assembly tasks.
Smart Images

Figure CN2025146374_12112026_PF_FP_ABST
Abstract
Description
Robotic arm control methods, devices, electronic equipment and computer program products
[0001] This application claims priority to Chinese Patent Application No. 202510600222.5, filed on May 9, 2025, entitled "Method, Apparatus, Electronic Device and Computer Program Product for Controlling a Robotic Arm", the entire contents of which are incorporated herein by reference. Technical Field
[0002] This application belongs to the field of robotic arm control technology, and in particular relates to a robotic arm control method, device, electronic device and computer program product. Background Technology
[0003] In the industrial sector, numerous repetitive tasks exist, such as shaft-hole alignment. With the continuous advancement of robotics technology, a large number of robotic arms have been introduced into industrial settings to replace human labor in performing these repetitive tasks. Currently, most robotic arm control solutions are designed for structured application scenarios. In structured application scenarios, the initial shape of the target object remains fixed, and its positional relationship with the target position is also unchanging. Therefore, in existing robotic arm control solutions, the robotic arm often operates according to a fixed trajectory to complete a predetermined task. For example, the robotic arm might grasp a target object from point A according to a preset posture and then move it to point B to accomplish the task.
[0004] However, in real factory environments, workpieces often roll or shift, causing random changes to their initial shape and the positional relationship between them. Therefore, existing robotic arm control solutions may fail to accurately complete predetermined tasks in practical applications, resulting in low accuracy in robotic arm operations. Summary of the Invention
[0005] In view of this, embodiments of this application provide a robotic arm control method, device, electronic device, and computer program product to improve the accuracy of robotic arm operation.
[0006] The first aspect of this application provides a robotic arm control method, including:
[0007] Obtain the first pixel coordinates of the first target object held by the robotic arm;
[0008] Obtain the second pixel coordinates of the second target object at the target movement position of the robotic arm;
[0009] Based on the first pixel coordinates and the second pixel coordinates, the movement information of the robotic arm on the plane is determined;
[0010] The robotic arm is controlled to move on the plane based on the movement information;
[0011] If the positional relationship between the first target object and the second target object meets the assembly conditions, the robotic arm is controlled to move in a preset direction according to the first preset force to complete the assembly of the first target object and the second target object; the assembly conditions are that the projection of the first target object on the plane where the second target object is located overlaps with the second target object.
[0012] In one possible implementation of the first aspect, the movement information includes a first movement speed of the robotic arm in the X direction of the plane and a second movement speed of the robotic arm in the Y direction of the plane;
[0013] Determining the movement information of the robotic arm on the plane based on the first pixel coordinates and the second pixel coordinates includes:
[0014] The first moving speed is calculated based on the difference between the horizontal coordinate of the first pixel coordinate and the horizontal coordinate of the second pixel coordinate;
[0015] The second moving speed is calculated based on the difference between the ordinate of the first pixel coordinate and the ordinate of the second pixel coordinate.
[0016] In one possible implementation of the first aspect, controlling the movement of the robotic arm on the plane based on the movement information includes:
[0017] The robotic arm is controlled to move on the plane based on the first moving speed, the second moving speed, and a preset time.
[0018] In one possible implementation of the first aspect, controlling the robotic arm to move in a preset direction according to a first preset force includes:
[0019] The robotic arm is controlled to move in a preset direction according to a first preset force, and a first feedback force in the preset direction is obtained;
[0020] If the first feedback force is within a preset reference force range, the robotic arm is controlled to stop moving in the preset direction;
[0021] The robotic arm is controlled to move in the X and / or Y directions of the second plane according to the second preset force, and a second feedback force is obtained; the direction of action of the second feedback force is opposite to the direction of action of the second preset force.
[0022] If the second feedback force is greater than or equal to the first threshold, then the robotic arm is controlled to release the first target object.
[0023] In one possible implementation of the first aspect, if the positional relationship between the first target object and the second target object satisfies the assembly conditions, then controlling the robotic arm to move in a preset direction according to a first preset force to complete the assembly of the first target object and the second target object includes:
[0024] Acquire a second image containing the first target and the second target;
[0025] Target detection is performed on the second image to obtain the third coordinates of the first target object in the second image and the fourth coordinates of the second target object;
[0026] If the difference between the horizontal coordinate in the third coordinate system and the horizontal coordinate in the fourth coordinate system is less than the second threshold, and the difference between the vertical coordinate in the third coordinate system and the vertical coordinate in the fourth coordinate system is less than the third threshold, then the robotic arm is controlled to move in a preset direction according to the first preset force to complete the assembly of the first target object and the second target object.
[0027] In one possible implementation of the first aspect, prior to obtaining the first pixel coordinates of the first target object held by the robotic arm, the following is included:
[0028] Target detection is performed on a third image containing the first target object to obtain the initial pose parameters corresponding to the first target object;
[0029] Generate motion parameters based on the initial pose parameters and the target pose parameters corresponding to the first target object;
[0030] The robotic arm is controlled to grasp the first target object according to the motion parameters.
[0031] In one possible implementation of the first aspect, when the first target object is a workpiece with a hole, the second target object is a workpiece with a pin; when the first target object is a workpiece with a pin, the second target object is a workpiece with a hole.
[0032] A second aspect of this application provides a robotic arm control device, comprising:
[0033] The first coordinate acquisition module is used to acquire the first pixel coordinates of the first target object held by the robotic arm.
[0034] The second coordinate acquisition module is used to acquire the second pixel coordinates of the second target object at the target movement position of the robotic arm;
[0035] The movement information determination module is used to determine the movement information of the robotic arm on the plane based on the first pixel coordinates and the second pixel coordinates;
[0036] A movement module is used to control the movement of the robotic arm on the plane based on the movement information;
[0037] An assembly module is used to control the robotic arm to move in a preset direction according to a first preset force if the positional relationship between the first target object and the second target object meets the assembly conditions, so as to complete the assembly of the first target object and the second target object; the assembly conditions are that the projection of the first target object on the plane where the second target object is located overlaps with the second target object.
[0038] A third aspect of this application provides an electronic device including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the robotic arm control method described in the first aspect above.
[0039] A fourth aspect of this application provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the robotic arm control method described in the first aspect above.
[0040] A fifth aspect of this application provides a computer program product that, when run on a computer, causes the computer to execute the robotic arm control method described in the first aspect.
[0041] Compared with the prior art, the embodiments of this application have the following advantages:
[0042] In this embodiment, the electronic device can determine the movement information of the robotic arm on a plane based on the first pixel coordinates of the first target object held by the robotic arm and the second pixel coordinates of the second target object at the target movement position of the robotic arm. The electronic device then controls the robotic arm to move on the plane based on this movement information, ensuring that the positional relationship between the first and second target objects meets the assembly conditions. Through the method provided in this embodiment, since the electronic device can generate the movement information of the robotic arm on the plane based on the first and second pixel coordinates, the movement trajectory of the robotic arm on the plane is not fixed but generated by the electronic device based on the relationship between the first and second pixel coordinates. When the position of the first target object changes, the first pixel coordinates also change; similarly, when the position of the second target object changes, the second pixel coordinates also change. The electronic device can plan the movement trajectory based on the changed first and / or second pixel coordinates, allowing the robotic arm to control the movement of the first target object based on the changed position of the second target object, ensuring precise assembly of the first and second target objects. Attached Figure Description
[0043] To more clearly illustrate the technical solutions in the embodiments of this application, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.
[0044] Figure 1 is a schematic diagram of a robotic arm control method provided in an embodiment of this application;
[0045] Figure 2 is a schematic diagram of an assembly process provided in an embodiment of this application;
[0046] Figure 3 is a schematic diagram of another robotic arm control method provided in an embodiment of this application;
[0047] Figure 4 is a schematic diagram of a robotic arm control device provided in an embodiment of this application;
[0048] Figure 5 is a schematic diagram of an electronic device provided in an embodiment of this application. Detailed Implementation
[0049] In the following description, specific details such as particular system architectures and techniques are set forth for illustrative purposes and not for limitation, in order to provide a thorough understanding of the embodiments of this application. However, those skilled in the art will understand that this application may also be implemented in other embodiments without these specific details. In other instances, detailed descriptions of well-known systems, apparatuses, circuits, and methods have been omitted so as not to obscure the description of this application with unnecessary detail.
[0050] In industrial production, especially in equipment manufacturing and assembly, there are numerous scenarios involving the automated docking, assembly, and maintenance of equipment components, such as factory workshops, warehousing and logistics centers, or outdoor assembly sites. In these scenarios, workers often need to perform a large number of repetitive tasks, such as shaft and hole alignment. With the increasing demand for unmanned and automated processes in industrial production, a large number of robotic arms are being introduced into industrial settings to replace human labor in performing repetitive tasks.
[0051] The scenarios for performing shaft-hole alignment operations have two main characteristics. Characteristic 1: Unstructured. The workpieces requiring shaft-hole alignment are diverse, varying in size, shape, and material. Furthermore, the workspace is complex, leading to potentially complex movement paths for the robotic arm. Additionally, the placement of the two workpieces to be assembled is often random due to potential rolling or displacement. Therefore, the execution scenarios for shaft-hole alignment operations are not static structures but rather highly variable unstructured scenarios. Characteristic 2: High-precision requirements. Shaft-hole alignment demands extremely high precision, with errors typically measured in millimeters. Due to these two characteristics, existing control schemes that guide the robotic arm along a preset trajectory may fail to accurately complete the task when applied to shaft-hole alignment operations, resulting in low accuracy.
[0052] In view of this, this application provides a robotic arm control method. Using this method, an electronic device can perform trajectory planning based on the correlation between a first target object and a second target object in an image, generating movement information. Then, the electronic device can control the robotic arm holding the first target object to move above the second target object, aligning the first and second target objects. When the positional relationship between the first and second target objects meets the assembly conditions, the electronic device can control the robotic arm holding the first target object to move towards the second target object, completing the assembly of the first and second target objects. Therefore, in this application embodiment, the robotic arm does not assemble according to a preset trajectory, but rather plans its trajectory based on real-time captured images. Thus, the method provided by this embodiment can improve the accuracy of robotic arm operation, thereby increasing the success rate of completing assembly tasks using the robotic arm.
[0053] The technical solution of this application will be described below through specific embodiments.
[0054] Referring to Figure 1, a schematic diagram of a robotic arm control method provided in an embodiment of this application is shown. This method can be applied to electronic devices mounted inside a robot, such as microcontroller units (MCUs), microprocessor units (MPUs), digital signal processors (DSPs), field-programmable gate arrays (FPGAs), and system-on-chips (SoCs); it can also be applied to electronic devices located outside the robot but with a communication link to the robotic arm for controlling the robotic arm, such as computers, tablets, mobile terminals, and servers. The above-mentioned robotic arm control method may specifically include the following steps:
[0055] S101. Obtain the first pixel coordinates of the first target object held by the robotic arm.
[0056] In this embodiment, when a user needs the robotic arm to perform an assembly operation, the user can issue an assembly command to the electronic device. The electronic device can respond to the user's assembly command by controlling the robotic arm on the robot to grasp the first target object corresponding to the assembly command. After the robotic arm grasps the first target object, the electronic device can control the robotic arm to continuously hold the first target object with a second preset force.
[0057] In this system, the robotic arm can grasp a first target object using its end effector. "Grasping" can refer to the robotic arm moving near the first target object and performing a rapid and accurate gripping or suction action to achieve initial contact and control. Grasping can emphasize the instantaneous action of the robotic arm acquiring the first target object; for example, the gripper at the end of the robotic arm quickly closing to clamp a workpiece. "Holding" can refer to the robotic arm maintaining a gripping or suction state on the target object with a certain force after grasping it, ensuring that the target object will not fall or move during subsequent operations. Holding can emphasize the continuous control the robotic arm maintains over the target object; for example, after grasping the workpiece, the gripper clamps it tightly with an appropriate force, ensuring that the workpiece remains stably held in the gripper during robotic arm movement or other operations. When the user initiates an assembly command that aligns shafts and holes, the first target object can be a workpiece with holes, and the second target object can be a workpiece with pins; alternatively, the first target object can be a workpiece with holes, and the second target object can be a workpiece with pins.
[0058] With the robotic arm holding the first target object, the electronic device can capture a first image containing both the first and second target objects using a camera on the robot. Specifically, the electronic device can capture the first image using a camera mounted on the wrist of the robotic arm, and / or using a camera mounted on the robot's head. The electronic device can perform target detection on the first target object in the first image and obtain the first pixel coordinates of the first target object. The first pixel coordinates can be the coordinates of the target assembly position of the first target object in the pixel coordinate system of the first image. Specifically, when the first target object is a workpiece with a hole and the second target object is a workpiece with a pin, the first pixel coordinates can be the coordinates of the hole on the first target object in the pixel coordinate system of the first image. When the first target object is a workpiece with a pin and the second target object is a workpiece with a hole, the first pixel coordinates can be the coordinates of the pin on the first target object in the pixel coordinate system of the first image.
[0059] In one possible implementation, before controlling the robotic arm to grasp the first target object, the electronic device can first capture a third image containing the first target object using a camera on the robotic arm's wrist and / or a camera on the robot's head. After acquiring the third image, the electronic device can perform target detection on the first target object in the third image to obtain the initial pose parameters corresponding to the first target object. Specifically, the initial pose parameters can include the first target object's current initial position parameters and initial attitude parameters. The initial position parameters can be used to represent the first target object's current specific position in space. The initial attitude parameters can be used to represent the first target object's current orientation in space, which can be represented by Euler angles (yaw angle, pitch angle, roll angle). After obtaining the first target object's current initial pose parameters, the electronic device can generate motion parameters based on the target pose parameters of the first target object preset by the developers. Then, the electronic device can control the robotic arm to grasp the first target object according to the motion parameters, so that the robotic arm holds the first target object according to the target pose.
[0060] S102. Obtain the second pixel coordinates of the second target object at the target movement position of the robotic arm.
[0061] In this embodiment, after the electronic device acquires a first image containing a first target object and a second target object, it can perform target detection on the second target object in the first image to obtain the first pixel coordinates of the second target object at the target movement position of the robotic arm. The second pixel coordinates can be the coordinates of the target assembly position of the second target object in the pixel coordinate system of the first image. The target assembly position can be the position used to connect the two target objects during assembly.
[0062] Specifically, when the first target object is a workpiece with a hole and the second target object is a workpiece with a pin, the second pixel coordinate can be the coordinate of the pin on the second target object in the pixel coordinate system of the first image. When the first target object is a workpiece with a pin and the second target object is a workpiece with a hole, the second pixel coordinate can be the coordinate of the hole on the second target object in the pixel coordinate system of the first image.
[0063] S103. Determine the movement information of the robotic arm on the plane based on the first pixel coordinates and the second pixel coordinates.
[0064] In this embodiment, after acquiring the first pixel coordinates and the second pixel coordinates, the electronic device can perform trajectory planning based on the first pixel coordinates and the second pixel coordinates to generate movement information for controlling the robotic arm on the plane. Specifically, the movement information generated by the electronic device can be the movement information of the robotic arm on the first plane. For example, when the second target object is a workpiece placed on the operating table, the first plane can be a plane parallel to the operating table at a certain height above the operating table.
[0065] In one possible implementation, after acquiring the first pixel coordinates and the second pixel coordinates, the electronic device can generate movement information of the robotic arm on the plane based on the positional difference between the first pixel coordinates and the second pixel coordinates. Specifically, the movement information generated by the electronic device may include a first moving speed of the robotic arm in the X direction of the plane and a second moving speed of the robotic arm in the Y direction of the plane.
[0066] In one possible implementation, when the first target object is a workpiece with a hole and only one hole, and the second target object is a workpiece with a pin and only one pin, the electronic device can acquire a first pixel coordinate and a second pixel coordinate. At this time, the electronic device can calculate a first moving speed based on the difference between the horizontal coordinates of the first and second pixel coordinates. The electronic device can also calculate a second moving speed based on the difference between the vertical coordinates of the first and second pixel coordinates. After acquiring the first and second moving speeds, the electronic device can control the robotic arm to move on the plane based on the first and second moving speeds and a preset time duration, so that the first target object reaches above the second target object, and the hole on the first target object aligns with the pin on the second target object.
[0067] Specifically, the formulas for calculating the first and second moving speeds by the electronic device can be as follows. x =pd1·(x1-x2) v y =pd2·(y1-y2)
[0068] Where p can be a conversion coefficient, representing the proportional relationship between visual data and movement speed. d1 can be the first correction coefficient in the X direction. d2 can be the second correction coefficient in the Y direction. x This can represent the first moving speed in the X direction. y This can represent the second moving velocity in the Y direction. x1 can represent the x-coordinate of the first pixel coordinate. x2 can represent the x-coordinate of the second pixel coordinate. y1 can represent the y-coordinate of the first pixel coordinate. y2 can represent the y-coordinate of the second pixel coordinate.
[0069] In one possible implementation, after acquiring the first and second moving speeds, the electronic device can calculate the resultant speed of the first and second moving speeds according to the parallelogram rule or the triangle rule. Then, the electronic device can control the robotic arm to move on the plane based on a preset time and the resultant speed.
[0070] In one possible implementation, after acquiring the first and second moving speeds, the electronic device can first control the robotic arm to move in one of the X or Y directions according to a first preset duration and the corresponding moving speed. Then, the electronic device can control the robotic arm to move in the other direction according to a second preset duration and the corresponding moving speed. For example, the electronic device can first control the robotic arm to move in the X direction of the plane according to the first preset duration and the first moving speed, and then control the robotic arm to move in the Y direction of the plane according to the second preset duration and the second moving speed.
[0071] In one possible implementation, when the first target object is a workpiece with holes, and the second target object is a workpiece with pins, and the number of pins is the same as the number of holes, the electronic device can acquire multiple first pixel coordinates and multiple second pixel coordinates. After acquiring the multiple first pixel coordinates and multiple second pixel coordinates, the electronic device can first generate the posture parameters of the robotic arm based on the multiple first pixel coordinates and multiple second pixel coordinates, and then adjust the gripping posture of the robotic arm according to the posture parameters so that the projection of the line connecting the multiple holes on the plane where the second target object is located is parallel to the line connecting the multiple pins.
[0072] Specifically, the electronic device can first calculate the first slope of the lines connecting multiple holes based on multiple first pixel coordinates, and then calculate the second slope of the lines connecting multiple pins based on multiple second pixel coordinates. Afterward, the electronic device can generate the robotic arm's posture parameters based on the slope difference between the first and second slopes.
[0073] After adjusting the gripping posture of the robotic arm, the electronic device can determine the pin associated with a hole on the first target object based on a preset correspondence between holes and pins, and determine the second pixel coordinate associated with the first pixel coordinate of that hole. Then, the electronic device can generate movement information based on the coordinate difference between the associated first and second pixel coordinates. The specific method by which the electronic device generates movement information based on a first pixel coordinate and a second pixel coordinate is detailed in the relevant content of this embodiment and will not be repeated here.
[0074] In one possible implementation, the first target object can also be a workpiece with a pin, and the second target object can be a workpiece with a hole. The reader can refer to the implementation in the case where the first target object is a workpiece with a hole and the second target object is a workpiece with a pin, and replace "hole" in the first target object with "pin" and "pin" in the second target object with "hole" to understand the specific implementation of this embodiment.
[0075] The method provided in this embodiment allows the electronic device to directly generate the movement information of the robotic arm based on the coordinates of the first and second target objects in the pixel coordinate system of the first image. Therefore, the method provided in this embodiment eliminates the need for coordinate transformation between the pixel coordinate system and the world coordinate system when generating movement information, and also eliminates the need for further coordinate transformation by combining the internal and external parameters of the camera. Thus, the method provided in this embodiment can reduce the computational load of the electronic device, thereby improving the reaction speed of the robotic arm.
[0076] Furthermore, since the movement information in this embodiment is calculated by the electronic device based on the coordinates of the target object in the pixel coordinate system, the robotic arm can be controlled by the method provided in this embodiment to ensure that the error after the first target object and the second target object are aligned is within 1 pixel, that is, the error after the first target object and the second target object are aligned is within 1 mm. Therefore, the method provided in this embodiment can improve the accuracy of the robotic arm when aligning the first target object and the second target object.
[0077] In one possible implementation, the electronic device can also input the first pixel coordinates and the second pixel coordinates into a trajectory planning model to generate the movement information of the robotic arm on the plane. The trajectory planning model can be trained by the electronic device using a user-input training dataset before controlling the robotic arm to complete the assembly operation. Specifically, the training dataset can include multiple first training coordinates, second training coordinates, and a desired trajectory. After acquiring the training model and training dataset, the electronic device can input the first and second training coordinates from the training dataset into the training model to generate an initial trajectory corresponding to the first and second training coordinates. Then, the electronic device can calculate a loss value based on the initial trajectory and the desired trajectory, and determine whether the loss value is less than or equal to a loss threshold. If the electronic device determines that the loss value is greater than the loss threshold, it can update the training model based on the loss value and regenerate the initial trajectory based on the updated training model to update the training model. The electronic device can continue to perform the above update operation until the loss value is less than or equal to the loss threshold. If the electronic device determines that the loss value is greater than the loss threshold, it can stop performing the update operation and determine the current training model as the trajectory planning model to generate movement information based on the first and second pixel coordinates. It should be noted that the trajectory planning model can be any trajectory planning model known to those skilled in the art, such as BEVGPT, Drive-WM, or Caspformer. The embodiments in this application are not intended to specifically limit the trajectory planning model.
[0078] S104. Control the robotic arm to move on the plane based on the movement information.
[0079] In this embodiment, after generating motion information, the electronic device can control the robotic arm to move on the plane based on the motion information.
[0080] S105. If the positional relationship between the first target object and the second target object meets the assembly conditions, the robotic arm is controlled to move in a preset direction according to the first preset force to complete the assembly of the first target object and the second target object.
[0081] In this embodiment, after the electronic device controls the robotic arm to move on the plane based on the movement information, it can determine whether the positional relationship between the first target object and the second target object meets the assembly conditions. Specifically, the assembly conditions can be that the projection of the first target object onto the plane containing the second target object overlaps with the second target object. More specifically, when the first target object is a workpiece with a hole and the second target object is a workpiece with a pin, the assembly conditions can be that the projection of the hole on the first target object onto the plane containing the second target object overlaps with the pin on the second target object. Similarly, when the first target object is a workpiece with a pin and the second target object is a workpiece with a hole, the assembly conditions can be that the projection of the pin on the first target object onto the plane containing the second target object overlaps with the hole on the second target object.
[0082] If the electronic device determines that the positional relationship between the first target object and the second target object meets the assembly conditions, the electronic device can control the robotic arm to move in a preset direction according to a first preset force. That is, the electronic device can control the robotic arm to move closer to the second target object to complete the assembly of the first and second target objects. The preset direction can be determined based on the assembly relationship between the first and second target objects. Specifically, when the first and second target objects are directly in an oblique assembly relationship, the preset direction can be a direction with an angle of less than 90 degrees to the plane containing the second target object. When the first and second target objects are directly in a perpendicular assembly relationship, the preset direction can be a direction perpendicular to the plane containing the second target object, that is, the angle between the preset direction and the plane containing the second target object can be equal to 90 degrees. The first preset force can be the externally applied force required for the robotic arm to move in the preset direction. The X and Y directions can be directions in a three-dimensional coordinate system constructed with the center of the first target object as the origin.
[0083] If the electronic device determines that the positional relationship between the first target object and the second target object does not meet the assembly conditions, the electronic device can continue to acquire a first image containing the first target object and the second target object, and regenerate the movement information based on the latest acquired first image, and continue to control the robotic arm to move on the plane based on the new movement information until the positional relationship between the first target object and the second target object meets the assembly conditions. That is, the electronic device can return to execute S101 to S104 until the positional relationship between the first target object and the second target object meets the assembly conditions.
[0084] The method provided in this embodiment allows the electronic device to generate movement information of the robotic arm on a plane based on the real-time acquired first pixel coordinates of the first target object and the second pixel coordinates of the second target object. Therefore, the method enables the robotic arm to move the first target object according to the actual situation of the first and second target objects in the scene during assembly tasks. Thus, the method provided in this embodiment can improve the accuracy of the robotic arm in performing assembly operations, thereby increasing the success rate of the robotic arm in completing assembly tasks.
[0085] Furthermore, since the electronic device only controls the robotic arm to assemble the first and second target objects after determining that the positional relationship between the first and second target objects meets the assembly conditions, the method provided in this embodiment can reduce the situation where the target objects are damaged due to misalignment during assembly. Therefore, the method provided in this embodiment can improve the safety of the robotic arm when performing assembly tasks.
[0086] Referring to Figure 2, a schematic diagram of an assembly process provided by an embodiment of this application is shown. Figure 2 may include a first target object 201 and a second target object 204, wherein the first target object 201 includes a hole 202 and the second target object 204 includes a pin 203. As shown in Figure 2(a), the first target object 201 held by the robotic arm can be a workpiece with a hole. The electronic device can control the movement of the robotic arm according to the contents of S101-S102 of this embodiment, so that the robotic arm moves the first target object 201 above the second target object 204, and aligns the hole 202 in the first target object 201 with the pin 203 in the second target object 204. As shown in Figure 2(b), after aligning the hole 202 in the first target object 201 and the pin 203 in the second target object 204, since the second target object 204 is located below the first target object 201, the electronic device can control the robotic arm to move downwards in a preset direction, thereby causing the robotic arm to move the first target object 201 downwards. As the robotic arm moves the first target object 201 downward, the pin 203 on the second target object can pass through the hole 202 on the first target object 201, thereby allowing the pin 203 on the second target object 204 to be inserted into the hole 202 on the first target object 201, thus completing the assembly of the first target object 201 and the second target object 204.
[0087] In one possible implementation, after the electronic device controls the robotic arm to move on a plane based on motion information, it can acquire a second image containing a first target object and a second target object via cameras on the robotic arm's wrist and / or head. After acquiring the second image, the electronic device can perform target detection on the first and second target objects in the second image to obtain a third coordinate of the first target object and a fourth coordinate of the second target object in the second image. Then, the electronic device can calculate the difference between the horizontal coordinate of the third coordinate and the horizontal coordinate of the fourth coordinate, as well as the difference between the vertical coordinate of the third coordinate and the vertical coordinate of the fourth coordinate.
[0088] If the electronic device determines that the difference between the horizontal coordinate in the third coordinate system and the horizontal coordinate in the fourth coordinate system is less than a preset second threshold, and the difference between the vertical coordinate in the third coordinate system and the vertical coordinate in the fourth coordinate system is less than a preset third threshold, then the electronic device can determine that the positional relationship between the first target object and the second target object meets the assembly conditions. The electronic device can then control the robotic arm to move in a preset direction to complete the assembly of the first target object and the second target object.
[0089] If the electronic device determines that the difference between the horizontal coordinate in the third coordinate system and the horizontal coordinate in the fourth coordinate system is greater than or equal to a preset second threshold, and / or the difference between the vertical coordinate in the third coordinate system and the vertical coordinate in the fourth coordinate system is greater than or equal to a preset third threshold, then the electronic device can determine that the positional relationship between the first target object and the second target object does not meet the assembly conditions. The electronic device can then return to execute S101 to S102 until the positional relationship between the first target object and the second target object meets the assembly conditions.
[0090] In one possible implementation, after determining that the positional relationship between the first target object and the second target object meets the assembly conditions, the electronic device can input a first preset force into a preset torque calculation formula to calculate the torque value of the robotic arm joints when the robotic arm moves in a preset direction. After calculating the torque value corresponding to the first preset force, the electronic device can control the joints of the robotic arm according to the torque value, so that the robotic arm moves towards the second target object in the preset direction.
[0091] Specifically, the formula for calculating torque is as follows:
[0092]
[0093] Where τ can represent the torque value of the robotic arm joint. F Z It can represent the first preset force in a preset direction. The Jacobian matrix can be used to represent the velocity of the robotic arm. This describes the relationship between the linear velocity of the robotic arm's end effector and the angular velocity of the robotic arm's joints. Specifically, It can be a 6xN matrix, where N can be a positive integer greater than or equal to 1. The first three lines can represent the linear velocities of the robotic arm's end effector in the X, Y, and preset directions, respectively. The last three rows can represent the angular velocities of the robotic arm joints rotating around the X, Y, and Z axes, respectively. [:,:3] can represent the operation on all elements in the first three columns of the matrix. Specifically, the symbol ":" in the above formula can represent all rows, and ":3" can represent the first three columns.
[0094] Figure 3 shows a flowchart illustrating the specific implementation of a robotic arm control method S103 according to the second embodiment of this application. Referring to Figure 3, compared to the embodiment described in Figure 1, S105 in the robotic arm control method provided in this embodiment includes: S1051 to S1054, which are detailed below:
[0095] S1051. Control the robotic arm to move in a preset direction according to the first preset force, and obtain the first feedback force in the preset direction.
[0096] In this embodiment, while controlling the robotic arm to move in a preset direction, the electronic device can continuously collect a first feedback force in the preset direction through a force sensor on the robotic arm. The direction of the first feedback force can be opposite to the direction of movement of the robotic arm.
[0097] S1052. If the first feedback force is within the preset reference force range, control the robotic arm to stop moving in the preset direction.
[0098] In this embodiment, the electronic device can continuously determine whether the acquired first feedback force is within a preset reference force range. If the electronic device determines that the first feedback force is within the reference force range, it can control the robotic arm to stop moving in the preset direction. If the electronic device determines that the first feedback force is not within the reference force range, it can continue to control the robotic arm to move in the preset direction and continue to acquire the first feedback force in the preset direction. The reference force range can be determined by the researchers based on multiple reference forces acquired during the experimental phase. Specifically, during the experimental phase, researchers can use a remote control device to manipulate the robotic arm to perform assembly actions on the first and second target objects. Simultaneously, the robotic arm can continuously acquire the feedback force during this process as a reference force to fit the reference force range.
[0099] S1053. Control the robotic arm to move in the X and / or Y directions of the second plane according to the second preset force, and obtain the second feedback force.
[0100] In this embodiment, after the electronic device controls the robotic arm to stop moving in a preset direction, it can control the robotic arm to move in the X and / or Y directions of a second plane using a second preset force, and obtain a second feedback force. The second plane can be the plane where the second target object is currently located. The second feedback force can be a force whose direction of action is opposite to that of the second preset force. After obtaining the second feedback force, the electronic device can determine whether the second feedback force is greater than or equal to a first threshold. If the second feedback force is less than the first threshold, the electronic device can control the robotic arm to move in a preset direction according to a third preset force, so that the first target object leaves the second target object, and repeat operations S101 to S104. The direction of action of the third preset force is opposite to that of the first preset force.
[0101] S1054. If the second feedback force is greater than or equal to the first threshold, control the robotic arm to release the first target object.
[0102] In this embodiment, if the electronic device determines that the second feedback force is greater than or equal to the preset first threshold, the electronic device can determine that the first target object and the second target object are successfully assembled, and the electronic device can control the robotic arm to release the first target object.
[0103] Specifically, the electronic device can first control the robot's robotic arm to move in the X direction with a second preset force and obtain a second feedback force in the X direction. The electronic device can determine whether the second feedback force in the X direction is greater than or equal to a first threshold. If the electronic device determines that the second feedback force in the X direction is less than the first threshold, the electronic device can determine that the first target object and the second target object are not fully assembled. The electronic device can then control the robotic arm to move in a preset direction according to a third preset force to make the first target object leave the second target object, and repeat operations S101 to S104. If the electronic device determines that the second feedback force in the X direction is greater than or equal to the first threshold, the electronic device can also control the robot's robotic arm to move in the Y direction with a second preset force and obtain a second feedback force in the Y direction. The electronic device can determine whether the second feedback force in the X direction is greater than or equal to the first threshold. If the electronic device determines that the second feedback force in the Y direction is less than the first threshold, the electronic device can determine that the first target object and the second target object are not fully assembled. The electronic device can then control the robotic arm to move in a preset direction according to a third preset force to make the first target object leave the second target object, and repeat operations S101 to S104. If the electronic device determines that the second feedback force in the Y direction is greater than or equal to the first threshold, the electronic device can determine that the first target and the second target have been assembled, and the electronic device can control the robotic arm to release the first target.
[0104] The method provided in this embodiment allows the electronic device to further determine whether the first target object and the second target object are fully assembled based on the feedback force in the preset direction, as well as the feedback force in the X and / or Y directions, after controlling the robotic arm to move in a preset direction. The first target object is only released when the feedback force exceeds a threshold. Therefore, the method provided in this embodiment reduces the probability of parts falling or being damaged due to incomplete assembly, thereby improving the stability and safety of the robotic arm performing assembly operations.
[0105] It should be noted that the sequence number of each step in the above embodiments does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of this application.
[0106] Referring to Figure 4, a schematic diagram of a robotic arm control device provided in an embodiment of this application is shown. Specifically, it may include a first coordinate acquisition module 401, a second coordinate acquisition module 402, a movement information determination module 403, a movement module 404, and an assembly module 405, wherein:
[0107] The first coordinate acquisition module 401 is used to acquire the first pixel coordinates of the first target object held by the robotic arm.
[0108] The second coordinate acquisition module 402 is used to acquire the second pixel coordinates of the second target object at the target movement position of the robotic arm;
[0109] The movement information determination module 403 is used to determine the movement information of the robotic arm on the plane based on the first pixel coordinates and the second pixel coordinates;
[0110] The movement module 404 is used to control the movement of the robotic arm on the plane according to the movement information;
[0111] Assembly module 405 is used to control the robotic arm to move in a preset direction according to a first preset force if the positional relationship between the first target object and the second target object meets the assembly conditions, so as to complete the assembly of the first target object and the second target object; the assembly conditions are that the projection of the first target object on the plane where the second target object is located overlaps with the second target object.
[0112] The motion information determination module 403 can also be used to calculate the first moving speed based on the difference between the horizontal coordinate in the first pixel coordinate and the horizontal coordinate in the second pixel coordinate; and to calculate the second moving speed based on the difference between the vertical coordinate in the first pixel coordinate and the vertical coordinate in the second pixel coordinate.
[0113] The moving module 404 can also be used to control the robotic arm to move on the plane according to the first moving speed, the second moving speed and the preset time.
[0114] Assembly module 405 can also be used to control the robotic arm to move in a preset direction according to a first preset force, and obtain a first feedback force in the preset direction; if the first feedback force is within a preset reference force range, then control the robotic arm to stop moving in the preset direction; control the robotic arm to move in the X and / or Y directions of the second plane according to a second preset force, and obtain a second feedback force; the direction of action of the second feedback force is opposite to the direction of action of the second preset force; if the second feedback force is greater than or equal to a first threshold, then control the robotic arm to release the first target object.
[0115] The assembly module 405 can also be used to acquire a second image containing the first target object and the second target object; perform target detection on the second image to acquire the third coordinate of the first target object and the fourth coordinate of the second target object in the second image; if the difference between the horizontal coordinate of the third coordinate and the horizontal coordinate of the fourth coordinate is less than a second threshold, and the difference between the vertical coordinate of the third coordinate and the vertical coordinate of the fourth coordinate is less than a third threshold, then the robotic arm is controlled to move in a preset direction according to the first preset force to complete the assembly of the first target object and the second target object.
[0116] The first coordinate acquisition module 401 can also be used to perform target detection on a third image containing the first target object, acquire the initial pose parameters corresponding to the first target object, generate motion parameters based on the initial pose parameters and the target pose parameters corresponding to the first target object, and control the robotic arm to grasp the first target object based on the motion parameters.
[0117] In the first coordinate acquisition module 401, when the first target object is a workpiece with a hole, the second target object is a workpiece with a pin; when the first target object is a workpiece with a pin, the second target object is a workpiece with a hole.
[0118] As the apparatus embodiments are basically similar to the method embodiments, they are described in a relatively simple manner. For relevant details, please refer to the description in the method embodiment section.
[0119] Referring to FIG5, a schematic diagram of an electronic device provided in an embodiment of this application is shown. As shown in FIG5, the electronic device 500 in this embodiment includes: a processor 510, a memory 520, and a computer program 521 stored in the memory 520 and executable on the processor 510. When the processor 510 executes the computer program 521, it implements the steps in the various embodiments of the above-described robotic arm control method, such as steps S101 to S105 shown in FIG1. Alternatively, when the processor 510 executes the computer program 521, it implements the functions of each module / unit in the above-described device embodiments, such as the functions of modules 401 to 405 shown in FIG4.
[0120] For example, the computer program 521 can be divided into one or more modules / units, which are stored in the memory 520 and executed by the processor 510 to complete this application. The one or more modules / units can be a series of computer program instruction segments capable of performing specific functions, which can be used to describe the execution process of the computer program 521 in the electronic device 500. For example, the computer program 521 can be divided into a trajectory planning module, a movement module, and an assembly module, with the specific functions of each module as follows:
[0121] The first coordinate acquisition module is used to acquire the first pixel coordinates of the first target object held by the robotic arm.
[0122] The second coordinate acquisition module is used to acquire the second pixel coordinates of the second target object at the target movement position of the robotic arm;
[0123] The movement information determination module is used to determine the movement information of the robotic arm on the plane based on the first pixel coordinates and the second pixel coordinates;
[0124] A movement module is used to control the movement of the robotic arm on the plane based on the movement information;
[0125] An assembly module is used to control the robotic arm to move in a preset direction according to a first preset force if the positional relationship between the first target object and the second target object meets the assembly conditions, so as to complete the assembly of the first target object and the second target object; the assembly conditions are that the projection of the first target object on the plane where the second target object is located overlaps with the second target object.
[0126] The electronic device 500 can be a microcontroller unit (MCU), microprocessor unit (MPU), digital signal processor (DSP), field-programmable gate array (FPGA), system-on-chip (SoC), or other devices. The electronic device 500 can also be a computer, tablet computer, mobile terminal, server, or other devices. The electronic device 500 may include, but is not limited to, a processor 510 and a memory 520. Those skilled in the art will understand that Figure 5 is merely an example of the electronic device 500 and does not constitute a limitation on the electronic device 500. It may include more or fewer components than shown, or combine certain components, or use different components. For example, the electronic device 500 may also include input / output devices, network access devices, buses, etc.
[0127] The processor 510 can be a Central Processing Unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor can be a microprocessor or any conventional processor.
[0128] The memory 520 can be an internal storage unit of the electronic device 500, such as a hard disk or RAM of the electronic device 500. The memory 520 can also be an external storage device of the electronic device 500, such as a plug-in hard disk, Smart Media Card (SMC), Secure Digital (SD) card, Flash Card, etc., equipped on the electronic device 500. Furthermore, the memory 520 can include both internal and external storage units of the electronic device 500. The memory 520 is used to store the computer program 521 and other programs and data required by the electronic device 500. The memory 520 can also be used to temporarily store data that has been output or will be output.
[0129] This application also discloses an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, it implements the robotic arm control method as described in the foregoing embodiments.
[0130] This application also discloses a computer-readable storage medium storing a computer program that, when executed by a processor, implements the robotic arm control method as described in the foregoing embodiments.
[0131] This application also discloses a computer program product that, when run on a computer, causes the computer to execute the robotic arm control methods described in the foregoing embodiments. The above embodiments are merely illustrative of the technical solutions of this application and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application, and should all be included within the protection scope of this application.
Claims
1. A robotic arm control method, characterized in that, include: Obtain the first pixel coordinates of the first target object held by the robotic arm; Obtain the second pixel coordinates of the second target object at the target position of the robotic arm; Based on the first pixel coordinates and the second pixel coordinates, the movement information of the robotic arm on the plane is determined; The robotic arm is controlled to move on the plane based on the movement information; If the positional relationship between the first target object and the second target object meets the assembly conditions, the robotic arm is controlled to move in a preset direction according to the first preset force to complete the assembly of the first target object and the second target object; The assembly condition is that the projection of the first target object onto the plane where the second target object is located overlaps with the second target object.
2. The method according to claim 1, characterized in that, The movement information includes the first movement speed of the robotic arm in the X direction of the plane and the second movement speed of the robotic arm in the Y direction of the plane; Determining the movement information of the robotic arm on the plane based on the first pixel coordinates and the second pixel coordinates includes: The first moving speed is calculated based on the difference between the horizontal coordinate of the first pixel coordinate and the horizontal coordinate of the second pixel coordinate; The second moving speed is calculated based on the difference between the ordinate of the first pixel coordinate and the ordinate of the second pixel coordinate.
3. The method according to claim 2, characterized in that, The step of controlling the robotic arm to move on the plane based on the movement information includes: The robotic arm is controlled to move on the plane based on the first moving speed, the second moving speed, and a preset time.
4. The method according to claim 1, characterized in that, The step of controlling the robotic arm to move in a preset direction according to a first preset force includes: The robotic arm is controlled to move in a preset direction according to a first preset force, and a first feedback force in the preset direction is obtained; If the first feedback force is within a preset reference force range, the robotic arm is controlled to stop moving in the preset direction; The robotic arm is controlled to move in the X and / or Y directions of the second plane according to the second preset force, and a second feedback force is obtained; the direction of action of the second feedback force is opposite to the direction of action of the second preset force. If the second feedback force is greater than or equal to the first threshold, then the robotic arm is controlled to release the first target object.
5. The method according to claim 1, characterized in that, If the positional relationship between the first target object and the second target object satisfies the assembly conditions, then the robotic arm is controlled to move in a preset direction according to a first preset force to complete the assembly of the first target object and the second target object, including: Acquire a second image containing the first target and the second target; Target detection is performed on the second image to obtain the third coordinates of the first target object in the second image and the fourth coordinates of the second target object; If the difference between the horizontal coordinate in the third coordinate system and the horizontal coordinate in the fourth coordinate system is less than the second threshold, and the difference between the vertical coordinate in the third coordinate system and the vertical coordinate in the fourth coordinate system is less than the third threshold, then the robotic arm is controlled to move in a preset direction according to the first preset force to complete the assembly of the first target object and the second target object.
6. The method according to any one of claims 1-5, characterized in that, Before obtaining the first pixel coordinates of the first target object held by the robotic arm, the process includes: Target detection is performed on a third image containing the first target object to obtain the initial pose parameters corresponding to the first target object; Generate motion parameters based on the initial pose parameters and the target pose parameters corresponding to the first target object; The robotic arm is controlled to grasp the first target object according to the motion parameters.
7. The method according to any one of claims 1-5, characterized in that, When the first target object is a workpiece with a hole, the second target object is a workpiece with a pin; when the first target object is a workpiece with a pin, the second target object is a workpiece with a hole.
8. A robotic arm control device, characterized in that, include: The first coordinate acquisition module is used to acquire the first pixel coordinates of the first target object held by the robotic arm. The second coordinate acquisition module is used to acquire the second pixel coordinates of the second target object at the target movement position of the robotic arm; The movement information determination module is used to determine the movement information of the robotic arm on the plane based on the first pixel coordinates and the second pixel coordinates; A movement module is used to control the movement of the robotic arm on the plane based on the movement information; An assembly module is used to control the robotic arm to move in a preset direction according to a first preset force if the positional relationship between the first target object and the second target object meets the assembly conditions, so as to complete the assembly of the first target object and the second target object. The assembly condition is that the projection of the first target object onto the plane where the second target object is located overlaps with the second target object.
9. An electronic device, characterized in that, The device includes a processor, a memory, and a computer program stored in the memory and executable on the processor, characterized in that, when the processor executes the computer program, the electronic device causes the robotic arm control method as described in any one of claims 1-7.
10. A computer program product, characterized in that, It includes a computer program, which, when run, causes the robotic arm control method as described in any one of claims 1-7 to be executed.
