Sugarcane grasping and placing system and method based on vision and mechanical arm end effector

Abstract

The present invention provides a sugarcane grasping and placing system and method based on vision and a mechanical arm end effector. The system comprises a visual recognition module, a six-degree-of-freedom mechanical arm module, a control module, and a placement sensor. The visual recognition module collects image data of a sugarcane piling area by means of an RGB-D camera, and recognizes the position, pose, and grasping point of a sugarcane. The mechanical arm module is equipped with an end effector capable of flexibly performing grasping and placing actions. The control module coordinates the operation of the modules to ensure the smooth progress of the grasping and placing process, and the placement sensor monitors the position of the sugarcane after placement, so as to achieve monitoring of placement precision. The present invention improves the efficiency and precision of sugarcane grasping and placement, achieves intelligence and automation during operation, and has significant economic and social benefits.

Description

A sugarcane grabbing and throwing system and method based on vision and robotic arm end effector Technical Field

[0001] This invention relates to the field of machine vision technology, and in particular to a sugarcane grabbing and throwing system and method based on vision and a robotic arm end effector. Background Technology

[0002] Sugarcane, as an important economic crop, requires harvesting and transportation as crucial steps in agricultural production. However, traditional sugarcane transportation methods rely primarily on manual labor, which is labor-intensive, inefficient, and unsuitable for large-scale planting operations. In recent years, with the rapid development of robotics and artificial intelligence, automated transportation systems based on robotic arms and visual recognition have gradually become a research hotspot.

[0003] The existing sugarcane grabbing system has the following problems:

[0004] Insufficient recognition accuracy: Due to the long and thin shape and complex stacking of sugarcane, existing vision systems have difficulty accurately identifying the position and posture of sugarcane.

[0005] Poor gripping stability: The surface of sugarcane is smooth, and traditional mechanical grippers are prone to slipping when gripping, resulting in gripping failure.

[0006] Low delivery efficiency: The existing system lacks precise control when delivering sugarcane after grabbing it, which can easily lead to uneven stacking of sugarcane and affect subsequent processing.

[0007] Therefore, designing a sugarcane grabbing and delivery system and method based on vision and robotic arm end effector can achieve efficient and accurate sugarcane grabbing and delivery, which has important practical significance. Summary of the Invention

[0008] In order to overcome the shortcomings of the existing technology, the purpose of this invention is to provide a sugarcane grabbing and throwing system and method based on vision and robotic arm end effector, which improves the efficiency and accuracy of sugarcane grabbing and throwing, and realizes intelligence and automation in the operation process.

[0009] To achieve the above objectives, the present invention provides the following solution:

[0010] A sugarcane grabbing and throwing system based on vision and a robotic arm end effector includes:

[0011] The visual recognition module collects image data of the sugarcane stacking area and identifies the sugarcane's position, posture, and gripping point;

[0012] The robotic arm module is equipped with a six-degree-of-freedom robotic arm for performing actions such as grasping and placing sugarcane; an end effector is provided at the six-degree-of-freedom robotic arm.

[0013] The control module, connected to the vision recognition module, the robotic arm module and the end effector, is used to coordinate the collaborative work of the vision recognition module, the robotic arm module and the end effector to realize the sugarcane grabbing and throwing process;

[0014] A delivery sensor is installed in the sugarcane delivery area and connected to the control module to monitor the position of the sugarcane after delivery, thereby achieving the accuracy of sugarcane delivery monitoring.

[0015] A sugarcane grasping and throwing method based on vision and a robotic arm end effector includes:

[0016] Collect image data of the sugarcane stacking area and identify the sugarcane's position, orientation, and gripping point;

[0017] The robotic arm module and end effector are controlled to perform actions based on the position, posture, and gripping point of the sugarcane in order to realize the sugarcane gripping and throwing process;

[0018] The sugarcane placement accuracy was monitored at the sugarcane placement area.

[0019] According to specific embodiments provided by the present invention, the present invention discloses the following technical effects:

[0020] This invention not only improves the efficiency and accuracy of sugarcane loading and unloading, but also realizes intelligent and automated operation, resulting in significant economic and social benefits. Attached Figure Description

[0021] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0022] Figure 1 is a schematic diagram of the system structure provided in an embodiment of the present invention;

[0023] Figure 2 is a flowchart of the method provided in an embodiment of the present invention.

[0024] Explanation of reference numerals in the attached diagram: 1-Vision recognition module, 2-Robotic arm module, 3-Control module, 4-Deployment sensor. Detailed Implementation

[0025] To make the above-mentioned objects, features and advantages of the present invention more apparent and understandable, the present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments.

[0026] Figure 1 is a schematic diagram of the system structure provided in an embodiment of the present invention. As shown in Figure 1, the present invention provides a sugarcane grabbing and throwing system based on vision and a robotic arm end effector, characterized in that it includes:

[0027] The visual recognition module 1 collects image data of the sugarcane stacking area and identifies the position, posture, and gripping point of the sugarcane.

[0028] Robotic arm module 2 is equipped with a six-degree-of-freedom robotic arm for performing actions of grasping and placing sugarcane; an end effector is provided at the six-degree-of-freedom robotic arm;

[0029] The control module 3 is connected to the vision recognition module 1, the robotic arm module 2 and the end effector, and is used to coordinate the collaborative work of the vision recognition module 1, the robotic arm module 2 and the end effector to realize the sugarcane grabbing and throwing process;

[0030] The delivery sensor 4 is installed in the sugarcane delivery area and connected to the control module 3. It is used to monitor the position of the sugarcane after delivery, so as to achieve the accuracy of sugarcane delivery monitoring.

[0031] Preferably, the end effector is a sugarcane gripper, which has an anti-slip structure and an adaptive clamping function to ensure gripping stability.

[0032] Specifically, robotic arm module 2 adopts a six-degree-of-freedom robotic arm design, enabling flexible movement in three-dimensional space and performing complex grasping and dropping actions. Each joint is equipped with a high-precision servo motor, ensuring smooth and rapid movement of the robotic arm. The robotic arm's motion control system is based on inverse kinematics algorithms, which can calculate the angles of each joint based on the three-dimensional coordinates of the target grasping point, thereby achieving precise positioning and movement. Control module 3 receives grasping point information from vision recognition module 1 in real time and dynamically adjusts the robotic arm's movement path to ensure that collisions with other objects are avoided during grasping and dropping.

[0033] The end effector is a specially designed sugarcane gripper designed to improve gripping stability and efficiency. The gripper's structure is made of lightweight, high-strength materials, ensuring sufficient strength and durability when gripping sugarcane. The inner surface of the gripper is designed with anti-slip textures to increase friction with the sugarcane surface and prevent slippage during gripping. Furthermore, the gripper's opening and closing mechanism is electrically or pneumatically driven, enabling rapid response to control commands and quick opening and closing of the gripper to accommodate sugarcane of different diameters and shapes.

[0034] To ensure stable gripping, the sugarcane gripper also features an adaptive clamping function. An internal pressure sensor monitors the clamping force in real time. When the gripper closes, the sensor detects the diameter and shape of the sugarcane being gripped and automatically adjusts the clamping force based on the feedback, ensuring sufficient gripping force without damaging the sugarcane. This adaptive function not only improves the success rate of gripping but also reduces the risk of damage due to over- or under-gripping, thereby enhancing the reliability and efficiency of the entire gripping and delivery system.

[0035] Preferably, the visual recognition module 1 includes:

[0036] An RGB-D camera was used to acquire image data of the sugarcane stacking area.

[0037] An image filtering submodule is used to filter the image data to obtain a filtered image.

[0038] The image enhancement submodule is used to enhance the filtered image to obtain an enhanced image;

[0039] An image recognition submodule is used to perform deep learning based on the enhanced image to identify the position, posture, and gripping point of the sugarcane.

[0040] Specifically, the core of the visual recognition module 1 is an RGB-D camera, which can simultaneously acquire color images and depth information of the sugarcane stacking area. First, the RGB-D camera transmits the captured image data to the image filtering submodule. The first step of this module is wavelet decomposition, where the image data is processed at multiple scales by the wavelet decomposition unit to obtain corresponding wavelet coefficients. These coefficients effectively represent different frequency components of the image. Next, the thresholding unit calculates the filtering threshold based on the image data size and decomposition scale, using the noise standard deviation and signal length to determine a suitable threshold for noise removal in subsequent filtering processes. Finally, the model building unit uses these thresholds to construct an image filtering model, and the filtering unit filters the original image data to obtain a clear filtered image, laying the foundation for subsequent image enhancement and recognition.

[0041] Preferably, the image filtering submodule includes:

[0042] The wavelet decomposition unit is used to perform wavelet decomposition on the image data at multiple scales to obtain the corresponding wavelet coefficients.

[0043] A threshold construction unit is used to construct a filtering threshold based on the size and decomposition scale of the image data; the formula for calculating the filtering threshold is as follows: Where, λ j σ represents the filtering threshold at the j-th decomposition scale. jN represents the noise standard deviation at the j-th decomposition scale. j This represents the signal length at the j-th decomposition scale, where j represents the decomposition scale.

[0044] The model building unit is used to construct an image filtering model using the filtering threshold; the calculation formula for the image filtering model is: Where, sign is the sign function, a is the first preset coefficient, b is the second preset coefficient, and ω j,k This represents the k-th wavelet coefficient at the j-th decomposition scale. Represents the wavelet coefficients after filtering;

[0045] The filtering unit is used to filter the image data using the image filtering model to obtain the filtered image.

[0046] Preferably, the image recognition submodule includes:

[0047] The background segmentation unit is used to segment the background of the enhanced image, remove non-cane regions, and obtain a segmented image.

[0048] An edge detection unit is used to extract the outline of sugarcane in the segmented image using an edge detection algorithm, and to fit the center line of the sugarcane using a Hough transform.

[0049] The computing unit is used to calculate the three-dimensional coordinates and orientation of the sugarcane based on its centerline.

[0050] The deep learning unit is used to input the three-dimensional coordinates and pose of the sugarcane, as well as the segmented image, into the trained Mask R-CNN model to obtain the optimized position, pose, and grasping points of the sugarcane.

[0051] Optionally, after obtaining the filtered image, the image enhancement submodule further processes it to improve the image's contrast and clarity. The enhanced image is then passed to the image recognition submodule. This submodule first processes the enhanced image through a background segmentation unit to remove non-sugarcane regions, obtaining a segmented image. Background segmentation can employ a thresholding method or a depth-based segmentation method to ensure that only the sugarcane portion is retained. Next, the edge detection unit applies an edge detection algorithm (such as Canny edge detection) to extract the outline of the sugarcane in the segmented image and combines it with a Hough transform to fit the sugarcane's centerline. This process effectively identifies the sugarcane's shape and location, providing basic data for subsequent 3D coordinate and pose calculations.

[0052] After extracting the sugarcane's outline and centerline, the computing unit calculates its 3D coordinates and pose based on this information. By combining RGB images and depth information, the system can accurately determine the sugarcane's position in 3D space. Subsequently, the deep learning unit inputs these 3D coordinates, pose, and segmented images into a trained Mask R-CNN model. This model, trained on a large number of sugarcane images, can identify the sugarcane's specific location, pose, and optimal grasping point. Finally, the results optimized by deep learning are fed back to the control module, providing precise guidance for the robotic arm's grasping actions and ensuring the efficiency and accuracy of the grasping process.

[0053] Furthermore, the control module is the core of the entire sugarcane grabbing and throwing system, responsible for coordinating the collaborative work between the vision recognition module, the robotic arm module, and the end effector. This module uses an embedded controller or industrial computer, integrating a real-time operating system to ensure efficient data processing and task scheduling. The control module exchanges data with each module through communication interfaces (such as CAN, RS-485, or Ethernet), receiving grabbing point information from the vision recognition module in real time and converting it into motion commands for the robotic arm. Simultaneously, the control module is also responsible for monitoring the system status, ensuring that each module remains synchronized during the grabbing and throwing process, avoiding operational errors caused by delays or erroneous commands. During the grabbing and throwing process, the control module first obtains the sugarcane's position and posture data from the vision recognition module. Based on this information, the control module uses inverse kinematics algorithms to calculate the target angles of each joint of the robotic arm and generates the corresponding motion trajectory. The control module monitors the robotic arm's motion status in real time, ensuring it moves smoothly to the grabbing position along the predetermined path. After grabbing, the control module instructs the robotic arm to move the sugarcane to the throwing area and adjusts the throwing angle and position based on feedback information from the throwing sensors, ensuring the sugarcane is accurately placed in the designated location. The sugarcane placement sensor is installed in the placement area to monitor the position and stacking status of the sugarcane after placement. This sensor can employ technologies such as laser rangefinders, infrared sensors, or cameras to acquire real-time depth information and sugarcane distribution within the placement area. The sensor feeds back the monitored data to the control module, which assesses the placement accuracy based on this data and makes adjustments as necessary. If sugarcane is found to have failed to be placed accurately in the designated location, the control module can instruct the robotic arm to re-grab it or adjust the placement strategy to ensure that the sugarcane is neatly stacked and meets subsequent processing requirements. This feedback mechanism not only improves placement accuracy but also enhances the system's intelligence level.

[0054] As shown in Figure 2, a sugarcane grasping and throwing method based on vision and a robotic arm end effector in this embodiment includes:

[0055] Step 100: Collect image data of the sugarcane stacking area and identify the sugarcane's position, orientation, and gripping point;

[0056] Step 200: Control the robotic arm module and end effector to perform actions according to the position, posture and gripping point of the sugarcane to realize the sugarcane gripping and throwing process;

[0057] Step 300: Monitor the sugarcane placement accuracy at the sugarcane placement area.

[0058] The beneficial effects of this invention are as follows:

[0059] (1) The visual recognition module of the present invention can collect image data of the sugarcane stacking area in real time, accurately identify the position, posture and gripping point of the sugarcane, thereby improving the gripping accuracy and reducing the gripping failure.

[0060] (2) The present invention can quickly complete the grabbing and delivery of sugarcane, significantly improve the efficiency of operation, meet the needs of large-scale sugarcane planting, and reduce operation time and cost.

[0061] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on the differences from other embodiments. The same or similar parts between the various embodiments can be referred to each other.

[0062] This document uses specific examples to illustrate the principles and implementation methods of the present invention. The descriptions of the above embodiments are only for the purpose of helping to understand the method and core ideas of the present invention. Furthermore, those skilled in the art will recognize that, based on the ideas of the present invention, there will be changes in the specific implementation methods and application scope. Therefore, the content of this specification should not be construed as a limitation of the present invention.

Claims

1. A sugarcane grabbing and throwing system based on vision and a robotic arm end effector, characterized in that, include: The visual recognition module collects image data of the sugarcane stacking area and identifies the sugarcane's position, posture, and gripping point; The robotic arm module is equipped with a six-degree-of-freedom robotic arm for performing actions such as grasping and placing sugarcane; an end effector is provided at the six-degree-of-freedom robotic arm. The control module, connected to the vision recognition module, the robotic arm module and the end effector, is used to coordinate the collaborative work of the vision recognition module, the robotic arm module and the end effector to realize the sugarcane grabbing and throwing process; A delivery sensor is installed in the sugarcane delivery area and connected to the control module to monitor the position of the sugarcane after delivery, thereby achieving the accuracy of sugarcane delivery monitoring.

2. The sugarcane grabbing and throwing system based on vision and robotic arm end effector according to claim 1, characterized in that, The end effector is a sugarcane gripper, which has an anti-slip structure and an adaptive clamping function to ensure gripping stability.

3. The sugarcane grabbing and throwing system based on vision and robotic arm end effector according to claim 1, characterized in that, The visual recognition module includes: An RGB-D camera was used to acquire image data of the sugarcane stacking area. An image filtering submodule is used to filter the image data to obtain a filtered image. The image enhancement submodule is used to enhance the filtered image to obtain an enhanced image; An image recognition submodule is used to perform deep learning based on the enhanced image to identify the position, posture, and gripping point of the sugarcane.

4. The sugarcane grabbing and throwing system based on vision and robotic arm end effector according to claim 3, characterized in that, The image filtering submodule includes: The wavelet decomposition unit is used to perform wavelet decomposition on the image data at multiple scales to obtain the corresponding wavelet coefficients. A threshold construction unit is used to construct a filtering threshold based on the size and decomposition scale of the image data; the formula for calculating the filtering threshold is as follows: Where, λ j σ represents the filtering threshold at the j-th decomposition scale. j N represents the noise standard deviation at the j-th decomposition scale. j This represents the signal length at the j-th decomposition scale, where j represents the decomposition scale. The model building unit is used to construct an image filtering model using the filtering threshold; the calculation formula for the image filtering model is: Where, sign is the sign function, a is the first preset coefficient, b is the second preset coefficient, and ω j,k This represents the k-th wavelet coefficient at the j-th decomposition scale. Represents the wavelet coefficients after filtering; The filtering unit is used to filter the image data using the image filtering model to obtain the filtered image.

5. The sugarcane grabbing and throwing system based on vision and robotic arm end effector according to claim 3, characterized in that, The image recognition submodule includes: The background segmentation unit is used to segment the background of the enhanced image, remove non-cane regions, and obtain a segmented image. An edge detection unit is used to extract the outline of sugarcane in the segmented image using an edge detection algorithm, and to fit the center line of the sugarcane using a Hough transform. The computing unit is used to calculate the three-dimensional coordinates and orientation of the sugarcane based on its centerline. The deep learning unit is used to input the three-dimensional coordinates and pose of the sugarcane, as well as the segmented image, into the trained Mask R-CNN model to obtain the optimized position, pose, and grasping points of the sugarcane.

6. A sugarcane grasping and throwing method based on vision and a robotic arm end effector, characterized in that, include: Collect image data of the sugarcane stacking area and identify the sugarcane's position, orientation, and gripping point; The robotic arm module and end effector are controlled to perform actions based on the position, posture, and gripping point of the sugarcane in order to realize the sugarcane gripping and throwing process; The sugarcane placement accuracy was monitored at the sugarcane placement area.

Images