Method for extracting navigation line in trellis orchard

Abstract

The present invention relates to a method for extracting a navigation line in a trellis orchard. The method comprises three parts: part I, detecting fruit trees and support frames on two sides of a trellis orchard on the basis of an object detection model Yolo v7, acquiring corresponding corner points, and then performing screening to select feature corner points on the basis of a threshold filtering method; part II, implementing a method for implementing tree row line fitting in the trellis orchard by means of acquiring the obtained feature corner points and integrating temporal information of image frames on the basis of an RANSAC algorithm; and part III, implementing the fitting of a middle navigation line on the basis of tree row lines on the two sides. The present invention can effectively reduce the influence of noise, thereby improving the accuracy of tree row line fitting. Furthermore, the present invention can effectively handle image frames having a large amount of noise and outliers, and solve the problem of navigation line fitting failure caused by the loss of object detection boxes at row ends. The present invention reduces the manual configuration and maintenance costs of a system, has good scalability, is applicable to a variety of trellis scenarios, and can be integrated with various types of navigation sensors and hardware devices to meet the requirements of specific tasks, thus having broad application prospects.

Description

A method for extracting navigation lines in a pergola orchard Technical Field

[0001] This invention relates to the field of image processing technology, and more specifically to a method for extracting navigation lines in a pergola orchard. Background Technology

[0002] Existing technologies generally employ several visual navigation strategies: semantic segmentation (road surface, fruit trees, etc.) and traditional image morphology to extract navigation lines to guide robot movement. However, in vineyard scenarios, the foreground and background are similar, the road conditions are unstructured, and there is noise such as weeds. Therefore, traditional machine vision solutions based on threshold segmentation, grayscale changes, and edge detection are difficult to extract feature points of crops on both sides, thus failing to fit the middle navigation line well and control the robot's navigation in the trellis orchard effectively. Summary of the Invention

[0003] The purpose of this invention is to overcome the problems existing in the prior art and provide a method for extracting navigation lines in a pergola orchard, so as to realize autonomous navigation inside the pergola orchard.

[0004] The environment of a pergola orchard is complex and contains numerous and varied targets, resulting in significant noise. Therefore, before fitting the navigation line, threshold filtering is necessary to process the noise and obtain feature corner points. The RANSAC algorithm performs better when handling noisy data, so it is chosen as the basic fitting algorithm for treeline fitting. To address the issue of treeline fitting failure caused by the decreasing number of target detection boxes as the transport robot travels to the rear of the pergola, the RANSAC algorithm is integrated with image frame temporal information to achieve treeline fitting. Then, the angle bisector theorem is used to fit the intermediate navigation line, thereby calculating the corresponding parameters required for navigation and achieving autonomous navigation within the pergola orchard.

[0005] To achieve the above-mentioned technical objectives and effects, the present invention is implemented through the following technical solution:

[0006] A method for extracting navigation lines in a pergola orchard, the method comprising the following steps:

[0007] Step S1: Corner point filtering of the detection box based on threshold filtering;

[0008] Step S2: Treeline fitting based on RANSAC algorithm to fuse temporal information of image frames;

[0009] Step S3: Extract and fit the navigation lines based on the tree lines on both sides.

[0010] Furthermore, in step S1, the screening process is as follows:

[0011] Step S1.1: The robot is positioned in the center of the trellis orchard row, and the RGB camera is installed in the middle of the robot to acquire a real-time one-frame image;

[0012] Step S1.2: Detect targets on both sides of the shed using the YOLOv7 target detection algorithm, and output the diagonal points (x1, y1, x2, y2) of the target detection box on a frame of image, where the lower left corner is defined as (x1, y1) and the lower right corner is defined as (x2, y2).

[0013] Step S1.3: Set the resolution of a single frame image to half its width to ensure that the robot travels in the center of the orchard. Take the lower right corner (x2, y2) of the right line of all rectangular detection boxes on the left side of the single frame image as feature points, and take the lower left corner (x1, y2) of the left line of all rectangular detection boxes on the right side as feature points.

[0014] Step S1.4: Create a mask image, map the target bounding boxes onto the mask image, take the right line of the left target bounding box and the left line of the right target bounding box, and then threshold the feature points on the left and right sides. Based on the characteristics of the camera where near objects appear larger and far objects smaller and linear fitting, the threshold is limited to two regions. That is, the feature points of distant targets are in one class, and the threshold range is represented as [k1(width), k2(width)], where k1 and k2 are proportional coefficients. The feature points of near targets are in another class, and the threshold range is limited by two dimensions: width and height, which is represented as simultaneously satisfying [k3(width), k4(width)] and [k5(height), height]. The corner points of a single frame image that have been thresholded are added to list A and list B respectively.

[0015] Furthermore, in step S2, the fitting process is as follows:

[0016] Step S2.1: Obtain the tree line information on both sides of the previous frame. Obtain the fitted tree line parameters on both sides from the previous frame image, i.e., the slope K on both sides.

[0017] Step S2.2: Detect the grapevine trunk and support pole in the current frame, process the current frame, detect the grapevine trunk and support pole based on the YOLOv7 target detection algorithm, and output the feature corner points A and B on both sides;

[0018] Step S2.3: Fit the tree line of the current frame based on the RANSAC algorithm. Use the feature corner points A and B of the grape trunk and support pole detected in the current frame to fit the tree line on both sides using the RANSAC algorithm, and obtain the slope of the tree line on both sides at the same time.

[0019] Step S2.4: Validity judgment of navigation lines. The top and bottom of each row of grapevines will form a trapezoidal area in the overall image. The slopes Q1 and Q2 on both sides are set according to the two sides of the trapezoid based on the image pixel points.

[0020] Step S2.5: Based on the slopes Q1 and Q2 on both sides, set a threshold and compare the slopes of the tree lines on both sides obtained in the current frame. Assuming the slope on the left is K1 and the slope on the right is K2, the slope limit is defined by the following formula:

[0021] Step S2.6: Tree line update and selection. If the above formula is satisfied, the tree line of the current frame is determined to be valid; otherwise, the tree line of the current frame is determined to be invalid, and the tree line of the previous frame is retained as the tree line of the current frame.

[0022] Step S2.7: Repeat the process from step S2.1.

[0023] Furthermore, in step S3, the fitting process is as follows:

[0024] Step S3.1: Select the angle bisectors of the fitted tree line segments on both sides as the navigation center line. The slopes of the tree lines on the left and right sides are k1 and k2, respectively. The formula for calculating the slope of the angle bisector is:

[0025] Step S3.2: The angle bisector will also pass through the corner point (x3, y3) where the two tree lines intersect. The angle bisector can be calculated using the following formulas, where y1 and y2 correspond to the equations of the two tree lines respectively:

[0026] get: y3 = k1*x3 + b1;

[0027] Step S3.3: Solve the equations in steps S3.1 and S3.2 simultaneously to obtain the equation of the central navigation line: Y-y3=k3(X-x3);

[0028] Step S3.4: The angle between the actual road centerline in the image and the generated navigation line is the heading angle deviation θ. Assuming the centerline is x = m and the navigation line is y = kx + b, the formula for calculating the heading deviation θ is:

[0029] The horizontal distance between the intersection of the navigation line and the x-axis in the image is the lateral pixel deviation e, and the formula is:

[0030] Then, by transforming the coordinate system from pixel coordinate system to image coordinate system to camera coordinate system to world coordinate system, the heading angle deviation and lateral angle deviation are mapped to obtain the true values ​​of the parameters in the world coordinate system.

[0031] The beneficial effects of this invention are:

[0032] 1. This invention uses the YOLOv7 target detection model to extract corner points of raisins and support frames on both sides. Based on the threshold filtering method, the influence of noise can be effectively reduced, thereby improving the accuracy of tree line fitting. RANSAC is used as the basic fitting algorithm, which can effectively handle image frames with a lot of noise and outliers. A tree line fitting method based on RANSAC algorithm and image frame temporal information is proposed, which can well solve the problem of missing target detection boxes at the end of the row and the inability to fit navigation lines.

[0033] 2. The navigation system based on RANSAC fusion image frames of this invention can reduce the reliance on manual marking or pre-modeled navigation lines, and can autonomously extract navigation line information from sensor data, reducing the manual configuration and maintenance costs of the system. The navigation line extraction algorithm has strong scalability and can be applied to various trellis scenarios. The algorithm can be adjusted and customized according to the needs of specific scenarios to adapt to the different shapes and sizes of plant rows, thereby realizing diversified agricultural applications. In addition, the algorithm can also be integrated with various types of navigation sensors and hardware devices to meet the needs of specific tasks, such as autonomous navigation, automatic inspection, and spraying of intelligent agricultural machinery, thus having broad application prospects. Attached Figure Description

[0034] Figure 1 is a diagram showing the definition of target detection corner points according to the present invention;

[0035] Figure 2 is a diagram showing the definition of the feature corner points of the present invention;

[0036] Figure 3 is a feature corner point threshold filtering diagram of the present invention;

[0037] Figure 4 is a flowchart of the generation of tree row lines of temporal information of image frames based on the RANSAC algorithm of the present invention.

[0038] Figure 5 is a tree row fitting diagram based on the RANSAC algorithm of this invention;

[0039] Figure 6 is a diagram showing the slope definition of the trapezoidal regions formed on the top and bottom of the grapevine in the image according to the present invention;

[0040] Figure 7 is a diagram showing the fitting and generation of the navigation line according to the present invention;

[0041] Figure 8 is a diagram showing the navigation parameter acquisition method of the present invention;

[0042] Figure 9 is a flowchart of the overall navigation line extraction process of the present invention;

[0043] Figure 10 is a flowchart of the navigation line extraction algorithm of the present invention. Detailed Implementation

[0044] The present invention will now be described in detail with reference to the accompanying drawings and embodiments.

[0045] A method for extracting navigation lines in a pergola orchard, as shown in Figure 9, includes the following steps:

[0046] Step S1: Corner point filtering of the detection box based on threshold filtering;

[0047] Step S2: Treeline fitting based on RANSAC algorithm to fuse temporal information of image frames;

[0048] Step S3: Extract and fit the navigation lines based on the tree lines on both sides.

[0049] In step S1, the screening process is as follows:

[0050] Step S1.1: The robot is positioned in the center of the trellis orchard row, and the RGB camera is installed in the middle of the robot to acquire a real-time one-frame image;

[0051] Step S1.2: Detect targets on both sides of the trellis using the YOLOv7 target detection algorithm (both sides can be fruit trees and support poles, or one side can be fruit trees and one side can be support poles). As shown in Figure 1, output the diagonal points of the target detection box on a frame of the image as (x1, y1, x2, y2), where the lower left corner is defined as (x1, y1) and the lower right corner is defined as (x2, y2).

[0052] Step S1.3: Set the resolution of the single frame image to half the width to ensure that the robot travels in the center of the orchard. At the same time, in order to avoid obstacles, as shown in Figure 2, take the lower right corner (x2, y2) of the right line of all rectangular detection boxes on the left side of the single frame image as feature points, and take the lower left corner (x1, y2) of the left line of all rectangular detection boxes on the right side as feature points.

[0053] Step S1.4: Due to the presence of redundant noise in the single-frame image, a mask image is created, and the target bounding box is mapped to the mask image, as shown in Figure 3. The right line of the left target bounding box and the left line of the right target bounding box are taken. Then, the feature points on the left and right sides are thresholded. Based on the characteristics of the camera where near objects appear larger and far objects smaller and the linear fitting, the threshold is limited to two regions. That is, the feature points of distant targets are in one category, and the threshold range is represented as [k1(width), k2(width)], where k1 and k2 are proportional coefficients. The feature points of near targets are in another category. According to the observation of the real-time detection image, the threshold range is limited by two dimensions: width and height, which means that [k3(width), k4(width)] and [k5(height), height] are satisfied at the same time. The corner points on the single-frame image that have been thresholded are added to list A and list B respectively.

[0054] In step S2, as shown in Figure 4, the overall fitting process is as follows:

[0055] Step S2.1: Obtain the tree line information on both sides of the previous frame. Obtain the fitted tree line parameters on both sides from the previous frame image, i.e., the slope K on both sides.

[0056] Step S2.2: Detect the grapevine trunk and support pole in the current frame, process the current frame, detect the grapevine trunk and support pole based on the YOLOv7 target detection algorithm, and output the feature corner points A and B on both sides;

[0057] Step S2.3: Fit the tree line of the current frame based on the RANSAC algorithm. Use the feature corner points A and B of the grape trunk and support pole detected in the current frame to fit the tree line on both sides using the RANSAC algorithm, and obtain the slope of the tree line on both sides at the same time, as shown in Figure 5.

[0058] Step S2.4: Validity judgment of navigation line. Due to the "near-large and far-small" effect of the camera, when the camera is installed in front of the transport robot, the top and bottom of each row of grapevines will appear smaller and larger in the image, respectively. Therefore, a trapezoidal area will be formed as shown in Figure 6. Therefore, the slopes Q1 and Q2 on both sides are set according to the two sides of the trapezoid based on the image pixel points.

[0059] Step S2.5: Based on the slopes Q1 and Q2 on both sides, set a threshold and compare the slopes of the tree lines on both sides obtained in the current frame. Assuming the slope on the left is K1 and the slope on the right is K2, the slope limit is defined by the following formula:

[0060] Step S2.6: Tree line update and selection. If the above formula is satisfied, the tree line of the current frame is determined to be valid; otherwise, the tree line of the current frame is determined to be invalid, and the tree line of the previous frame is retained as the tree line of the current frame.

[0061] Step S2.7: Repeat the process from step S2.1.

[0062] In step S3, the fitting process is as follows:

[0063] Step S3.1: Based on the "nearer objects appear larger, farther objects appear smaller" effect, the tree lines fitted on both sides will eventually intersect at a point. Therefore, the angle bisector of these two line segments can be selected as the navigation center line. The slopes of the tree lines on the left and right sides are k1 and k2, respectively. The formula for calculating the slope of the angle bisector is:

[0064] Step S3.2: The angle bisector will also pass through the corner point (x3, y3) where the two tree lines intersect. The angle bisector can be calculated using the following formulas, where y1 and y2 correspond to the equations of the two tree lines respectively:

[0065] get: y3 = k1*x3 + b1;

[0066] Step S3.3: Solve the equations in steps S3.1 and S3.2 simultaneously to obtain the equation of the central navigation line: Y-y3=k3(X-x3). The intermediate navigation line obtained by the above method is shown in Figure 7.

[0067] Step S3.4: The angle between the actual road centerline in the image and the generated navigation line is the heading angle deviation θ. Assuming the centerline is x = m and the navigation line is y = kx + b, the formula for calculating the heading deviation θ is:

[0068] The horizontal distance between the intersection of the navigation line and the x-axis in the image is the lateral pixel deviation e, and the formula is:

[0069] Then, the mapping of heading angle deviation and lateral angle deviation is achieved through coordinate system transformation from pixel coordinate system to image coordinate system to camera coordinate system to world coordinate system, as shown in Figure 8, to obtain the true values ​​of parameters in the world coordinate system.

[0070] The above description is merely a preferred embodiment of the present invention and is not intended to limit the invention. Various modifications and variations can be made to the present invention by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A method of extracting a navigation line in a vineyard, characterized by, The method comprises the following steps: Step S1: threshold filtering-based detection frame corner point screening; Step S2: tree line fitting based on RANSAC algorithm fusion image frame time sequence information; Step S3: navigation line extraction fitting based on two-side tree lines.

2. The method of extracting a navigation line in a vineyard according to claim 1, characterized in that, In the step S1, the screening process is as follows: Step S1.1: the robot is arranged at the center position of the trellis vineyard row, and the RGB camera is installed at the middle of the robot, and a real-time image frame is acquired; Step S1.2: the target detection algorithm Yolov7 is used to detect the targets on both sides of the trellis, and the diagonal points of the target detection frame on the image frame are output (x1, y1, x2, y2), wherein the left lower corner point is defined as (x1, y1), and the right lower corner point is defined as (x2, y2); Step S1.3: the resolution size of the single image frame is taken as half of the width, so as to ensure that the robot travels in the center of the vineyard, and the right lower corner points (x2, y2) of the right lines of all rectangular detection frames on the left side of the single image frame are taken as feature points, and the left lower corner points (x1, y2) of the left lines of all rectangular detection frames on the right side are taken as feature points; Step S1.4: a mask mask image is made, the target frame is mapped to the mask image, the right line of the left target frame and the left line of the right target frame are taken, and the feature points on the left and right sides are limited by a threshold value, based on the characteristics of the camera that the near is large and the far is small and linear fitting, the threshold value is limited to two regions, i.e. the far target feature points are a class, and the threshold value range is represented as [k1(width), k2(width)], wherein k1 and k2 are proportional coefficients, the near target feature points are a class, and the threshold value range is limited in width width and height height, represented as [k3(width), k4(width)] and [k5(height), height] are satisfied, and the corner points on the single image frame that pass the threshold screening are added to list A and list B respectively.

3. The method of extracting a navigation line in a vineyard according to claim 2, characterized in that, In the step S2, the fitting process is as follows: Step S2.1: the tree line information on both sides of the last frame is acquired, and the parameters of the two-side tree lines that have been fitted from the last image frame are obtained, i.e. the corresponding slopes K on both sides; Step S2.2: the current frame of grape tree trunk and support rod is detected, the current frame is processed, the YOLOv7 target detection algorithm is used to detect the grape tree trunk and the support rod, and the feature corner points A and B on both sides are output; Step S2.3: the current frame of tree line is fitted based on the RANSAC algorithm, the feature corner points A and B of the grape tree trunk and the support rod detected in the current frame are used, and the RANSAC algorithm is used to fit the tree lines on both sides, and the slopes of the tree lines on both sides are obtained; Step S2.4: navigation line validity judgment, the top and bottom of each row of grape trees will form a trapezoidal region on the whole image, and the slopes Q1 and Q2 on both sides are set according to the two-side edge lines of the trapezoidal region; Step S2.5: Set a threshold value threshold based on the two side slopes Q1 and Q2, compare the slopes of the two side tree lines of the current frame, assume that the slope of the left side is K1 and the slope of the right side is K2, and the slope limit is defined as the following formula: Step S2.6: tree line update and selection, if the above formula is satisfied, it is judged that the tree line of the current frame is valid, otherwise, it is judged that the tree line of the current frame is invalid, and the tree line of the last frame is retained as the tree line of the current frame; Step S2.7: the process is repeated from step S2.

1.

4. The method of extracting a navigation line in a vineyard according to claim 3, characterized in that, In the step S3, the fitting process is as follows: Step S3.1: Select the angle bisector of the tree line segment fitted on both sides as the navigation center line, and the slopes of the tree lines on the left and right sides are k1 and k2, respectively. The slope formula of the angle bisector is calculated as: Step S3.2: The angle bisector will pass through the intersection point (x3, y3) of the two side tree lines. The intersection point (x3, y3) can be calculated by solving the following equations simultaneously, where y1 and y2 are the equations of the two side tree lines, respectively: obtained: y3=k1*x3+b1; Step S3.3: the equation of the central navigation line is obtained by simultaneously solving the equations in the step S3.1 and the step S3.2: Y-y3=k3(X-x3); Step S3.4: the included angle between the actual road centerline in the image and the generated navigation line is the heading angle deviation θ. Assuming that the centerline is x = m and the navigation line is y = kx + b, the heading angle deviation θ is calculated according to the following formula: The horizontal distance of the intersection point of the navigation line in the image and the x axis is the lateral pixel deviation e, and the formula is: The mapping of the heading angle deviation and the lateral angle deviation is realized through the coordinate system conversion of the pixel coordinate system, the image coordinate system, the camera coordinate system and the world coordinate system, and the real value of the parameter in the world coordinate system is obtained.

Images