Machine vision-based dynamic fine positioning method and system for smart laser weeding

Abstract

Disclosed are a machine vision-based dynamic fine positioning method and system for smart laser weeding. The method comprises: performing target detection on the basis of an image acquired by a coarse positioning camera, and coarsely positioning a bounding box of a weed; performing optical flow field calculation on the basis of the image acquired by the coarse positioning camera, so as to obtain a displacement offset; on the basis of the displacement offset, correcting the position of the coarse positioning bounding box in real time; on the basis of the corrected position, controlling a fine positioning camera to perform image acquisition; and, on the basis of an acquired image, calculating the central position of a weed root stem. According to the present invention, an accurate weed center pixel position is acquired by means of combining coarse positioning and fine positioning, and a motion direction and distance of a machine are predicted by means of an optical flow method, so as to achieve dynamic and accurate positioning, thereby solving the problem of low precision in visual recognition of weeds during existing laser weeding.

Description

A machine vision-based intelligent laser weeding dynamic precision positioning method and system Technical Field

[0001] This invention belongs to the field of artificial intelligence and agricultural laser weeding applications, and particularly relates to a machine vision-based intelligent laser weeding dynamic precision positioning method and system. Background Technology

[0002] Currently, laser weeding is being widely tested and promoted due to its advantage of minimal environmental harm. The visual positioning function of weeds is an important part of laser weeding.

[0003] Most visual positioning methods use a single camera for location, but in practical applications, a single camera has a limited field of view or captures blurry images of crops, making it impossible to identify all weeds or limiting the accuracy of identification. Chinese patent application CN117274566A, published on April 26, 2024, discloses a real-time weeding method based on deep learning and inter-plant weed distribution. It uses a neural network to identify the smallest detection box and uses the center of the detection box as the target location for weeds, resulting in limited accuracy. Chinese patent CN117882697A, published on April 16, 2024, discloses a precise and rapid intelligent laser weeding method. It uses a vision system and an optical ranging sensor to precisely control the movement of a galvanometer to ensure the laser spot's focal point covers the weed roots and stems, achieving effective removal. However, this method does not consider the impact of the dynamic movement of the weeding robot on the weeding positioning. Summary of the Invention

[0004] To address the issue of low accuracy in visual weed identification during existing laser weeding methods, this invention provides a machine vision-based intelligent laser weeding dynamic precision positioning method. This method combines coarse and fine positioning to obtain the precise pixel position of the weed center, and uses optical flow to predict the machine's movement direction and distance, thereby achieving dynamic precision positioning.

[0005] According to one aspect of the present invention, a machine vision-based intelligent laser weeding dynamic precision positioning method is provided, comprising:

[0006] Target detection is performed based on images acquired by a coarse-positioning camera, and the detection bounding box for weeds is coarsely positioned.

[0007] Optical flow field calculations are performed on images acquired by a coarse positioning camera to obtain the displacement offset.

[0008] Based on the displacement offset, the position of the coarse positioning detection box is corrected in real time;

[0009] The image acquisition is performed by controlling the precise positioning camera based on the corrected position.

[0010] The center position of the weed roots and stems was calculated based on the acquired images.

[0011] As a further technical solution, target detection is performed based on images acquired by a coarse positioning camera, including:

[0012] Images acquired by a coarse-localization camera are input into a target detection model built on an RT-DETR network. The target detection model performs feature extraction and target detection on the input images. It combines image features with target queries through self-attention and cross-attention mechanisms to perform target localization and classification, and outputs the position information of the weed detection box. The position information of the detection box includes the pixel coordinates of the top left corner vertex of the detection box and the width and height of the detection box.

[0013] As a further technical solution, optical flow field calculation is performed based on images acquired by a coarse positioning camera, including:

[0014] Based on an optical flow prediction network, displacement field information at the pixel level is extracted from two consecutively input images.

[0015] As a further technical solution, displacement field information based on the pixel level is separated, including:

[0016] The acquired image is divided into several regions of a set size;

[0017] Calculate the displacement of the center point of each region as the displacement of the current region;

[0018] Calculate the average displacement of all areas as the real-time motion displacement of the laser weeding robot.

[0019] As a further technical solution, the method also includes:

[0020] Based on the motion displacement and the time interval between two consecutively input images, the motion speed of the laser weeding robot is obtained.

[0021] As a further technical solution, the method further includes: correcting the position (m+dx, n+dy, w, h) of the coarse positioning detection box according to the motion displacement, where (m, n) represents the pixel coordinates of the upper left corner vertex of the detection box, w represents the width of the detection box, h represents the height of the detection box, and (dx, dy) represents the motion displacement.

[0022] As a further technical solution, after obtaining the image captured by the precision positioning camera, the method further includes: calculating the center position of the weed roots and stems in the image based on the skeleton line extraction algorithm.

[0023] As a further technical solution, calculating the center position of the weed roots and stems in the image includes:

[0024] Based on the acquired image, the background is segmented by converting the RGB channels to HSV channels;

[0025] Weed images were obtained using binarization and erosion algorithms.

[0026] The central skeleton line was extracted using the Zhang-Suen skeleton extraction algorithm;

[0027] The center point is calculated using intersection detection.

[0028] According to one aspect of the present invention, a machine vision-based intelligent laser weeding dynamic precision positioning system is provided, comprising:

[0029] The coarse localization module is used for target detection based on images acquired by the coarse localization camera, and coarsely localizes the detection boxes for weeds.

[0030] The displacement correction module is used to perform optical flow field calculation based on the image acquired by the coarse positioning camera, obtain the displacement offset, and correct the position of the coarse positioning detection box in real time based on the displacement offset.

[0031] The precision positioning module is used to control the precision positioning camera to acquire images based on the corrected position, and to calculate the center position of the weed roots and stems based on the acquired images.

[0032] According to one aspect of the present invention, an intelligent laser weeding robot is provided, which is equipped with a coarse positioning camera, a fine positioning camera, and a processor. The coarse positioning camera is disposed at the front end of the laser weeding robot and is used to acquire weed images with a wide field of view. The fine positioning camera is disposed at the bottom end of the laser weeding robot and is used to acquire weed images after position correction. The processor is used to receive the images acquired by the coarse positioning camera and the fine positioning camera to execute the steps of the machine vision-based intelligent laser weeding dynamic fine positioning method.

[0033] Compared with the prior art, the beneficial effects of the present invention are as follows:

[0034] 1. This invention achieves wide-angle weed identification while maintaining high identification accuracy through a combined camera solution of coarse and fine positioning;

[0035] 2. This invention utilizes an optical flow prediction network to calculate the displacement field without relying on other sensors. It corrects the coarse positioning detection area in real time based on the motion parameters of the weeding robot to control the fine positioning camera to move in advance, ensuring that the fine positioning can capture weeds and accurately identify the location of weed roots and stems.

[0036] 3. The coarse positioning of this invention adopts an RT-DETR network structure, which optimizes accuracy and ensures real-time deployment;

[0037] 4. In the precise positioning process of this invention, the central skeleton line method is used to extract the outline of the weeds, and then the precise root and stem position is calculated according to the geometric characteristics of the weeds. Attached Figure Description

[0038] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the accompanying drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the accompanying drawings described below are some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0039] Figure 1 is a flowchart illustrating a machine vision-based intelligent laser weeding dynamic precision positioning method according to an embodiment of the present invention.

[0040] Figure 2 is a schematic diagram of the positioning method provided in an embodiment of the present invention.

[0041] Figure 3 is a schematic diagram of the installation of the coarse positioning camera and the fine positioning camera provided in an embodiment of the present invention.

[0042] Figure 4 is a schematic diagram of optical flow information predicted by an optical flow prediction network according to an embodiment of the present invention.

[0043] Figure 5 is a schematic diagram of the future location prediction of crops (blue dashed box) and weeds (red dashed box) based on optical flow prediction and RT-DETR provided in an embodiment of the present invention. Detailed Implementation

[0044] It should be noted that:

[0045] This invention, based on deep learning for target detection, uses color images acquired by a standard RGB optical camera to calculate both coarsely located weed positions and finely located feature points on the stems of weeds targeted by laser. By optimizing and accelerating the deep learning network and deploying it on an engineering board, hardware requirements are reduced while still meeting the need for real-time weed identification and removal. This method, through the collection of data on various crops such as soybeans and corn seedlings, as well as weed data, enables rapid location of both weeds and crops. Furthermore, it uses optical flow field calculations to continuously correct and estimate the positions of monitored crops and weeds during machine movement, ultimately achieving precise weed identification with a positioning error of less than 1 mm.

[0046] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention. In addition, the technical features of the various embodiments or individual embodiments provided by the present invention can be arbitrarily combined to form new technical solutions. Such combinations are not bound by the order of steps and / or structural composition patterns, but must be based on the ability of those skilled in the art to implement them. When the combination of technical solutions is contradictory or cannot be implemented, it should be considered that such a combination of technical solutions does not exist and is not within the scope of protection claimed by the present invention.

[0047] This invention provides a machine vision-based intelligent laser weeding dynamic precision positioning method, which uses a combination of coarse and fine positioning to achieve precise positioning of the center pixel position of weeds, solving the problem of limited visual recognition accuracy in existing laser weeding methods. As shown in Figure 1, the method provided by this invention first obtains the coarse positioning position of the weed detection box based on the image acquired by the coarse positioning camera; then, it calculates the real-time displacement offset through an optical flow prediction network and uses this offset to correct the real-time position of the laser weeding robot; finally, it controls the fine positioning camera to acquire the position-corrected weed image and calculates the center position of the weed rootstock.

[0048] It should be noted that, to improve work efficiency, the laser weeding robot described in this embodiment of the invention rapidly processes weeds by controlling the direction of the galvanometer during its forward movement. This means that laser weeding is performed even while the robot is moving. Therefore, it is necessary to correct and predict the position of the weeds based on the robot's real-time position and control the galvanometer to move towards the target point in advance. This invention uses optical flow calculation to predict the machine's movement direction and distance, and precisely positions the camera to ensure the weed target is within the camera's field of view, identifying the weed outline and calculating the position of the weed roots, thus achieving dynamic and precise positioning.

[0049] As a preferred embodiment, please refer to Figure 2. The dynamic fine positioning method provided by this embodiment of the invention specifically includes the following steps:

[0050] Step 1: Image acquisition using a coarse positioning camera. The primary function of the coarse positioning camera is its wide field of view, meeting the need for wide-angle weed identification. This invention employs a 4-megapixel wide-field-of-view camera, which is mounted on the front end of the unmanned robot to perform coarse positioning of the weeds in advance.

[0051] This invention also installs multiple rows of galvanometer lasers at the bottom of the robot. Each galvanometer laser is equipped with a coaxial 1-megapixel fine positioning camera. The installation diagrams of the coarse and fine positioning cameras can be seen in Figure 3. This embodiment of the invention achieves wide-angle weed identification while maintaining high accuracy through the combined use of coarse and fine positioning cameras.

[0052] Step 2: Acquire coarse localization images and identify detection boxes. The coarse localization camera acquires color RGB channel images, and the RT-DETR neural network is used for target detection to initially locate the detection boxes for weeds. Compared to the commonly used YOLO series networks, the RT-DETR network has higher localization accuracy and meets the requirements of real-time computing. Under the test environment of Intel i7-12700KF processor and NVIDIA RTX3060 graphics card, it takes 6ms to process one image.

[0053] The overall framework of RT-DETR consists of a feature extraction network and a Transformer-based detection network. It combines image features with object queries through self-attention and cross-attention mechanisms to achieve object localization and classification.

[0054] The output of the RT-DETR network is the position information of the detected weed detection boxes. In this embodiment of the invention, the coarse localization visual output data format is a detection box of (m,n,w,h), where (m,n) are the pixel coordinates of the top-left vertex of the detected weed detection box, and (w,h) are the width and height of the detection box.

[0055] Step 3: Calculate the optical flow field and obtain the average offset. As shown in Figure 4, while obtaining the coarse localization position, the optical flow prediction network separates the displacement field information based on the pixel level by inputting two consecutive frames of images.

[0056] The optical flow prediction network employs a dual-branch and feature pyramid multi-scale principle to achieve end-to-end optical flow field training, resulting in higher accuracy for detecting small-scale and long-distance moving objects. It should be noted that when actually estimating the motion direction and speed of the weeding robot, optical flow information for every single pixel is not required; only an average velocity representing the robot's speed needs to be calculated.

[0057] The present invention adopts a method of block-based calculation of uniform displacement field. The entire image is evenly divided into regions of 100x100. The displacement of the center point of the region is taken as the displacement of the region. The average displacement of all regions is calculated as the real-time motion information of the robot. This motion information is denoted as (dx, dy).

[0058] Furthermore, by inputting the time interval t between two frames, the real-time movement speed of the weeding robot can be roughly estimated. Based on this movement speed, the galvanometer control unit of the laser weeding system can adjust the direction to burn weeds according to the speed and time. Furthermore, during the movement of the weeding robot, considering that galvanometer control also requires time, the position of the laser device's emission point can be adjusted according to the calculated real-time movement speed.

[0059] Step 4: Based on the average offset obtained in Step 3, the position of the coarse localization detection box can be corrected in real time to (m+dx, n+dy, w, h). Figure 5 shows the predicted positions of future crops (blue dashed box) and weeds (red dashed box) based on optical flow prediction and RT-DETR network.

[0060] Step 5: Control the precision positioning camera to acquire images and perform precision positioning calculations based on the acquired images. The precision positioning camera, coaxial with the laser galvanometer, moves to the corrected position via galvanometer control to acquire high-resolution weed images. Then, using a skeleton line extraction algorithm, the pixel-level position of the weed roots and stems is calculated to achieve precise weed removal.

[0061] Furthermore, the pixel-level positions of weed roots and stems are calculated based on the skeleton line extraction algorithm, mainly by converting RGB channels to HSV and extracting green information to identify the root, stem, and leaf parts. The specific process is as follows: First, the background is segmented by converting RGB channels to HSV channels; then, the weed image is obtained using binarization and erosion algorithms; next, the central skeleton line is extracted using the Zhang-Suen skeleton extraction algorithm; finally, intersection points are detected and the center point is calculated.

[0062] Based on the same inventive concept as the above-described method embodiments, this embodiment of the invention provides a machine vision-based intelligent laser weeding dynamic precision positioning system, comprising:

[0063] The coarse localization module is used for target detection based on images acquired by the coarse localization camera, and coarsely localizes the detection boxes for weeds.

[0064] The displacement correction module is used to perform optical flow field calculation based on the image acquired by the coarse positioning camera, obtain the displacement offset, and correct the position of the coarse positioning detection box in real time based on the displacement offset.

[0065] The precision positioning module is used to control the precision positioning camera to acquire images based on the corrected position, and to calculate the center position of the weed roots and stems based on the acquired images.

[0066] When locating a weeding robot using the dynamic precision positioning system provided in this embodiment of the invention, the position of the weeds is coarsely located by using color images acquired by a common RGB optical camera. At the same time, optical flow field calculation is performed to obtain real-time motion information, and the position of the coarsely located weeds is corrected using the real-time motion information. Then, the precision positioning camera is controlled to acquire images of the weeds after position correction in order to calculate the position of the feature points of the weed stems that are finally targeted by the laser. Through the combination of coarse and precision positioning, high-precision recognition is achieved while satisfying a wide field of view.

[0067] This invention utilizes an RT-DETR network structure for coarse localization to optimize accuracy and ensure real-time deployment. Simultaneously, it employs an optical flow prediction network for real-time motion information calculation, achieving real-time correction of the detection area without relying on other sensors. This ensures that fine-positioning image acquisition based on the corrected location can capture weeds and accurately identify the location of weed roots and stems. This invention deploys a deep learning network on an engineering board using optimized acceleration technology, reducing hardware requirements and meeting the needs of real-time weed identification and removal.

[0068] Based on the same inventive concept as the above-described method embodiments, this invention also provides an intelligent laser weeding robot, equipped with a coarse positioning camera, a fine positioning camera, and a processor. The coarse positioning camera is located at the front end of the laser weeding robot and is used to acquire weed images with a wide field of view. The fine positioning camera is located at the bottom end of the laser weeding robot and is used to acquire weed images after position correction. The processor is used to receive the images acquired by the coarse positioning camera and the fine positioning camera to execute the steps of the machine vision-based intelligent laser weeding dynamic fine positioning method.

[0069] The installation diagram of the coarse positioning camera and the fine positioning camera in the intelligent laser weeding robot provided in this embodiment of the invention can be seen in Figure 3. The coarse positioning camera is installed at the front end of the unmanned robot to perform coarse positioning of the weeds in advance. The fine positioning camera is installed at the bottom of the laser weeding robot, and multiple rows of galvanometer lasers are installed at the bottom of the laser weeding robot, with each galvanometer laser configured with a coaxial 1-megapixel fine positioning camera.

[0070] In this embodiment of the invention, the processor may be a general-purpose processor, a digital signal processor, an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components, capable of implementing or executing the methods, steps, and logic block diagrams disclosed in this embodiment of the invention. The general-purpose processor may be a microprocessor or any conventional processor. The steps of the methods disclosed in this embodiment of the invention can be directly manifested as being executed by a hardware processor, or executed by a combination of hardware and software modules within the processor.

[0071] Compared to traditional mechanical and chemical weeding, laser weeding offers precise targeting, high speed, and less environmental damage. However, existing laser weeding robots do not consider the impact of the robot's dynamic movement on weed positioning, resulting in low accuracy. To address this, this invention provides a machine vision-based intelligent laser weeding dynamic precision positioning method that considers the positioning problem during the robot's dynamic movement and improves recognition accuracy by combining coarse and fine positioning.

[0072] In summary, the present invention provides a method for accurately locating weed roots and stems. It employs a combined camera mounting structure for coarse positioning (large field of view) and fine positioning (precision), and utilizes RT-DETR and optical flow prediction networks to dynamically correct the real-time position of the coarse positioning detection box during robot movement. Based on the corrected position, the fine positioning camera is controlled to obtain clearer images. The weed outline is extracted using a center skeleton line method, and the precise root and stem center position is calculated based on the weed's geometric features, improving the recognition accuracy to within 1 mm.

[0073] In terms of coarse-grained localization recognition neural network testing, this invention tested YOLO5, YOLO8, Mamba-YOLO8, and RT-DETR networks based on the same batch of collected weed and soybean seedling data. The test results are shown in Table 1. Among them, RT-DETR achieved the best performance with a recognition accuracy of 92%, while also meeting the requirements for real-time deployment speed. Optical flow prediction can correct the position of the following detection box according to the different working speeds of the weeding robot, facilitating dynamic planning and distribution of multi-point weeding tasks. In addition, this invention improves weeding efficiency through an independent multi-mirror control scheme.

[0074] Table 1. Accuracy Comparison of Various Model Networks

[0075] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some or all of the technical features therein; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the technical solutions of the embodiments of the present invention.

Claims

1. A machine vision-based intelligent laser weeding dynamic precision positioning method, characterized in that, include: Target detection is performed based on images acquired by a coarse-positioning camera, and the detection bounding box for weeds is coarsely positioned. Optical flow field calculations are performed on images acquired by a coarse positioning camera to obtain the displacement offset. Based on the displacement offset, the position of the coarse positioning detection box is corrected in real time; The image acquisition is performed by controlling the precise positioning camera based on the corrected position. The center position of the weed roots and stems was calculated based on the acquired images.

2. The intelligent laser weeding dynamic precision positioning method based on machine vision according to claim 1, characterized in that, Target detection is performed based on images acquired by a coarse-positioning camera, including: Images acquired by a coarse-localization camera are input into a target detection model built on an RT-DETR network. The target detection model performs feature extraction and target detection on the input images. It combines image features with target queries through self-attention and cross-attention mechanisms to perform target localization and classification, and outputs the position information of the weed detection box. The position information of the detection box includes the pixel coordinates of the top left corner vertex of the detection box and the width and height of the detection box.

3. The intelligent laser weeding dynamic precision positioning method based on machine vision according to claim 1, characterized in that, Optical flow field calculation based on images acquired by a coarse positioning camera includes: Based on an optical flow prediction network, displacement field information at the pixel level is extracted from two consecutively input images.

4. The intelligent laser weeding dynamic precision positioning method based on machine vision according to claim 3, characterized in that, The displacement field information based on the pixel level is separated, including: The acquired image is divided into several regions of a set size; Calculate the displacement of the center point of each region as the displacement of the current region; Calculate the average displacement of all areas as the real-time motion displacement of the laser weeding robot.

5. The intelligent laser weeding dynamic precision positioning method based on machine vision according to claim 3, characterized in that, The method further includes: Based on the motion displacement and the time interval between two consecutively input images, the motion speed of the laser weeding robot is obtained.

6. The intelligent laser weeding dynamic precision positioning method based on machine vision according to claim 3, characterized in that, The method further includes: correcting the position (m+dx, n+dy, w, h) of the coarse positioning detection box according to the motion displacement, where (m, n) represents the pixel coordinates of the top left corner vertex of the detection box, w represents the width of the detection box, h represents the height of the detection box, and (dx, dy) represents the motion displacement.

7. The intelligent laser weeding dynamic precision positioning method based on machine vision according to claim 1, characterized in that, After obtaining the image captured by the precision positioning camera, the method further includes: calculating the center position of the weed roots and stems in the image based on the skeleton line extraction algorithm.

8. The intelligent laser weeding dynamic precision positioning method based on machine vision according to claim 7, characterized in that, Calculate the center position of the weed roots and stems in the image, including: Based on the acquired image, the background is segmented by converting the RGB channels to HSV channels; Weed images were obtained using binarization and erosion algorithms. The central skeleton line was extracted using the Zhang-Suen skeleton extraction algorithm; The center point is calculated using intersection detection.

9. A machine vision-based intelligent laser weeding dynamic precision positioning system, characterized in that, include: The coarse localization module is used for target detection based on images acquired by the coarse localization camera, and coarsely localizes the detection boxes for weeds. The displacement correction module is used to perform optical flow field calculation based on the image acquired by the coarse positioning camera, obtain the displacement offset, and correct the position of the coarse positioning detection box in real time based on the displacement offset. The precision positioning module is used to control the precision positioning camera to acquire images based on the corrected position, and to calculate the center position of the weed roots and stems based on the acquired images.

10. An intelligent laser weeding robot, characterized in that, The system is equipped with a coarse positioning camera, a fine positioning camera, and a processor. The coarse positioning camera is located at the front end of the laser weeding robot and is used to acquire weed images with a wide field of view. The fine positioning camera is located at the bottom end of the laser weeding robot and is used to acquire weed images after position correction. The processor is used to receive the images acquired by the coarse positioning camera and the fine positioning camera to execute the steps of the intelligent laser weeding dynamic fine positioning method based on machine vision as described in any one of claims 1 to 8.

Images