Determination of a plant stem emerging points dependent wayline

Abstract

An agricultural machine (102) comprises a control unit (110) configured for executing a method for determining a plant stem emerging points (118) dependent wayline by receiving a captured image (600) of at least two plants (120) arranged in a crop row (116a - 116e) from an imaging unit (112), processing the captured image (600) by a trained model trained by annotated images (500) and determining a wayline connecting at least two plant stem emerging points (118) of the at least two plants (120) based on the processing by the trained model.

Description

Docket No. 25012WODETERMINATION OF A PLANT STEM EMERGING POINTS DEPENDENT WAYLINEFIELD

[0001] The present disclosure relates generally to a detection of plant stem emerging points of plants growing in an agricultural field and a determination of a wayline dependent on the detected plant stem emerging points.BACKGROUND

[0002] For different field operations during a crop cycle, it is advantageous to know the exact positions of objects in the agricultural field such as plants to determine a wayline along which an agricultural machine can be guided. The precision of the field operation depends on the precision of the wayline determination. I. e. in case of a precise wayline, plants can be treated very precisely, e. g. such as spraying, to avoid treatment of areas free of plants, or a position of an agricultural machine operating in the field can be controlled very precisely to avoid damage of the plants when moving through crop rows or to enable a precise treatment of the plants. The position of the plants may be detected by optical devices such as a camera.

[0003] The paper of Haug S. et al., entitled “Plant Stem Detection and Position Estimation using Machine Vision”, published in Workshop Proceedings of IAS-13, ser.13th Inti. Conf, on Intelligent Autonomous Systems, 2014, p. 483-490, discloses a method to determine a plant stem emerging point of a plant based on an image of the plant taken from a bird's view perspective. A detection rate of 80.4 % of ground truth stem positions could be achieved.BRIEF SUMMARY

[0004] A precise operation of an agricultural machine may depend on the preciseness of a wayline along which an agricultural machine may be guided. If the wayline is determined based on plant positions, a very reliable determination of the plant positions is required for the determination of a precise wayline. But a reliable detection of plant positions may be very difficult under some circumstances, for example if greatDocket No. 25012WOleaves of the plants hide the plant stems or if winds blowing over the agricultural field bend the plant stems. Thus, it would be beneficialto provide a method with a high detection rate of plant positions for an improved determination of a precise wayline and an improved wayline controlled operation of the agricultural machine in the agricultural field.

[0005] According to an aspect of the invention there is provided a method for determining a plant stem emerging points dependent wayline comprising receiving a captured image of at least two plants arranged in a crop row from an imaging unit, processing the captured image by a trained modeltrained by annotated images, and determining a wayline connecting at least two plant stem emerging points of the at least two plants based on the processing by the trained model.

[0006] The method may be executed by a control unit configured to carry out the method. The trained model including the annotated images may be stored in a memory of the control unit and may be applied by the control unitto process the captured image. The control unit may be integrated in an agricultural machine. The agricultural machine may be an agricultural vehicle such as a tractor, a harvester, a combine, a sprayer, etc., optionally connected with an implement. The agricultural machine may comprise an imaging unit. The imaging unit may be any optical device such as a camera, a LIDAR or a radar. The imaging unit may be used to capture images of the at least two plants, e. g. when the agricultural machine moves through crop rows of the agricultural field. The control unit may be connected with the imaging unit to receive the images.

[0007] The captured image may be a georeferenced or mapped image so that the geographical coordinates of objects in the image can be determined by the control unit. For example, the captured image may contain geographical coordinates of the position from which the image has been captured. The coordinates may be provided by a receiver for receiving position signals of a global navigation satellite system (GNSS) such as GPS, Galileo, etc. The receiver may be integrated in the imaging unit. A plant stem emerging point defines a point at which a plant grows out of the agricultural field. I. e., the plant gets visible at its plant stem emerging point when it grows out of the soil of the agricultural field. Hence, the plant and its position can be detected at ground level of the agricultural field. Advantageously, the plant stem emerging points may notDocket No. 25012WObe affected by winds, crooked plant stems or growing state of the leaves of the plants. Hence, the position of the plants may be reliably detected based on the plant stem emerging points captured by the imaging unit and contained in the captured image. Accordingly, a precise wayline can be determined based on the determination of the precise positions of the at least two plant stem emerging points. Since the position of the at least two plant stem emerging points can be determined at ground level, the wayline can be determined regardless of the plant heights, the position of the imaging unit, the growth state of the plants (e. g. leaves or crooked stem), stems bent by wind or weed surroundingthe plants.

[0008] The method may comprise determining a geographical position of each of the at least two plant stem emerging points.

[0009] Due to the georeferences contained in the captured image, the geographical positions of the at least two plant stem emerging points can be precisely determined (calculated) by the control unit. Based on the positions of the at least two plant stem emerging points, also the wayline along which an agricultural machine may be guided for a treatment of the plants can be determined very precisely by the control unit.

[0010] An annotation of an annotated image may contain at least two plant stem emerging points of plants arranged in a crop row.

[0011] The annotation may be of a first type of annotation. The annotation may be generated automatically by the control unit or manually by an user. The annotation may be used to improve the trained model for a better detection rate of the plant stem emerging points. Based on the trained model, the control unit may detect the at least two plant stem emerging points in the captured image with high accuracy for a precise determination of the positions of the plant stem emerging points. Based on the precisely determined positions of the plant stem emerging points, the wayline can be determined very precisely. The at least two plant stem emerging points of plants may be arranged in a common crop row, i. e. the same crop row.

[0012] The annotation of the annotated image may connect the at least two plant stem emerging points of plants arranged in a crop row.

[0013] The annotation may be of a second type of annotation in addition to or alternative to the first type of annotation. This annotation may also be generated automatically by the control unit or manually by the user. The annotation may be usedDocket No. 25012WOto improve the trained model for a better detection rate of the plant stem emerging points. Based on the trained model, the control unit may determine the wayline connecting the at least two plant stem emerging points directly from the image. The separate determination of the at least two plant stem emerging points as an additional method step may be omitted to improve the performance for determination of waylines. But the detection of the wayline may be more difficult compared to the detection based on the first type of annotations. Optionally, the control unit may combine both types of annotations to determine the wayline. Moreover, the at least two plant stem emerging points of plants may be arranged in a common crop row, too.

[0014] The annotation may be shaped as spline curves along a crop row containing the at least two plant stem emerging points of the at least two plants, wherein the thickness of the spline curve covers a diameter of a plant stem of the at least two plants.

[0015] If the thickness of the spline curve is too thin, the quality of the trained model may be poorer and may cause the control unit to not detect correctly a plant stem emerging point covered by the annotation. Instead, the trained model may be trained by more accurate representations of the plant stem emerging points providing an improved detection rate of the plant stem emerging points if the spline curves cover the diameter of the plant stem.

[0016] The annotation may be matched with an expected pattern of an agricultural field.

[0017] Hence, the spline curves may be validated by the expected pattern to improve the quality of the trained model. The expected pattern of the agricultural field may be related to a seed pattern of the plants, e. g. a distance from crop row to crop row, a distance from plant to plant within a crop row, parallelism of two or more crop rows, etc. which may be extracted by the control unit by means of image processing from the captured image. The control unit may additionally consider parameters of the imaging unit such as distance of the imaging unit to the ground of the agricultural field, orientation of the camera, or georeferences received by the GNSS receiver.

[0018] A parameter of the pattern of the agricultural field and / or the imaging unit may be manually parametrizable.

[0019] To improve the accuracy of the parameters, a user may manually adjust the values of the parameters. For example, the user may provide the exact values of theDocket No. 25012WOdistance of the imaging unit to the ground of the agricultural field and the orientation of the camera forthe control unit.

[0020] The method may comprise providing annotated images comprising an annotation with reference to at least two plant stem emerging points of a plants arranged in a crop rowand trainingthe trained model by the annotated images.

[0021] The detection rate of the plant stem emerging points depends on the quality of the trained model which can be influenced by the annotated images. An annotated image may be provided by adding automatically or manually an annotation containing at least two plant stem emerging points of plants of a crop row shown in an image. If the image shows multiple crop rows, a separate annotation may be added in respect of each crop row. The annotated image may be stored in a memory to create a stock of multiple annotated images to train the trained model. By means of a comparison of the captured image with the annotated images, the control unit may identify all plant stem emerging points shown in the captured image. Hence, the annotated images may be used, to improve the trained model for a better detection rate of the plant stem emerging points and for an improved performance to determine the plant stem emerging points dependent wayline.

[0022] The method may comprise generating a segmented image comprising at least two segmented plant stem emerging points based on the processing by the trained model.

[0023] A segmentation of the segmented image may comprise the at least two plant stem emerging points of the crop row. Hence, the wayline may be determined by the control unit by connecting the one segmented plant stem emerging point with the other segmented plant stem emerging point. If the segmentation comprises more than two segmented plant stem emerging points of a crop row, the control unit may determine the wayline based on a connection from segmented plant stem emerging point to segmented plant stem emerging point of the crop row.

[0024] The segmented image may comprise a segmentation containing the at least two segmented plant stem emerging points.

[0025] A segmentation may have the shape of a spot when a single plant stem emerging point has been segmented. Alternatively, the segmentation may be pathshaped. Moreover, the segmentation may represent a path connecting the at least twoDocket No. 25012WOsegmented plant stem emerging points. Accordingly, the control unit may derive the wayline directly from the path-shaped segmentation.

[0026] The wayline may extend along and within the segmentation connecting the at least two segmented plant stem emerging points.

[0027] I. e., the (path-shaped) segmentation may define boundaries for the wayline within those the wayline may extend. Thus, the control unit may determine a wayline of a more flexible shape compared to a wayline defined as an inflexible A-B line connecting a first and a second plant stem emerging point.

[0028] The method may comprise determining a distance between two plant stem emerging points based on the segmented image.

[0029] The distance may be used as a control parameter for the agricultural machine. For example, the agricultural machine may operate faster in case of longer distances or operate slower in case of smaller distances.

[0030] The method may comprise determining an expected distance between two plant stem emerging points, and detecting a missing plant if the distance between two plant stem emerging points may be greater than the expected distance between two plant stem emerging points.

[0031] When an agricultural field is planted, seeds may be equidistantly placed in the soil. Hence, it can be expected that the distance between all plant stem emerging points of a single crop row may be generally the same. So, the expected distance may be determined based on the captured image by the control unit as a distance that can be most often identified in the captured image. Also distances between plant stem emerging points of different crop rows may be the same. When a missing plant has been detected in a crop row, the agricultural machine may be triggered for a specific action.

[0032] The method may comprise mapping at least one of the plant stem emerging points.

[0033] For example, the control unit may generate a map with all determined plant stem emerging points. This map may be used for planning or executing subsequent field operations and may be transferred to other agricultural machines.

[0034] The method may comprise detecting a plant having a misaligned plant stem emerging point in respect to the wayline.Docket No. 25012WO

[0035] In such a case, the control unit may consider that the plant has not been regularly planted and may treat the plant as weed growing next to the wayline instead of crop growing in alignment with the wayline.

[0036] The method may comprise georeferencing the wayline.

[0037] The wayline may be georeferenced based on the georeferences of the captured image or the plant stem emerging points determined before. I. e., the control unit may determine the geographic coordinates of the wayline. The control unit may include the georeferenced wayline in a map. The georefernced wayline may also be used as input for a positional control to guide the agricultural machine or the implement through the agricultural field.

[0038] The orientation of a field of view of the imaging unit may be configured for capturing of plant stem emerging points of a first crop row and plant stem emerging points of a second crop row.

[0039] Hence, the captured image may comprise the first crop row and the second crop row for determining a first wayline aligned with the plant stem emerging points of the first crop row and a second wayline aligend with the plant stem emerging points of the second crop row simultaneously.

[0040] The orientation of the field of view of the imaging unit may be configured for capturing of plant stem emerging points of the first crop row in a first perspective and plant stem emerging points of the second crop row in a second perspective.

[0041] For example, the first perspective may show the first crop row from a left hand side and the second perspective may show the second crop row from a right hand side. This may result in an angled view of the first and second crop rows with a free (unhidden) view at the ground level of the agricultural field enabling a capturing of the plant stem emerging points by the imaging unit even if the plants have formed great leaves or plants have formed crooked stems or winds bend the plant stems.

[0042] As disclosed above, the control unit is configured to execute different actions. Each action may be implemented as one or more method steps of the method executable by the control unit. Hence, each action for which the control unit is configured to execute may be defined as a method step.

[0043] Within the scope of this application, it should be understood that the various aspects, embodiments, examples and alternatives set out herein, and individualDocket No. 25012WOfeatures thereof may be taken independently or in any possible and compatible combination. Where features are described with reference to a single aspect or embodiment, it should be understood that such features are applicable to all aspects and embodiments unless otherwise stated or where such features are incompatible.BRIEF DESCRIPTION OF THE DRAWINGS

[0044] Several aspects of the invention will now be described, byway of example only, with reference to the accompanying drawings, in which:

[0045] FIG. 1 illustrates an agricultural machine in an agricultural field.

[0046] FIG. 2 illustrates a simplified view of a control unit.

[0047] FIG. 3 illustrates an imaging unit.

[0048] FIG. 4 illustrates a flow chart of a method executable by the control unit of FIG.2.

[0049] FIG. 5 illustrates an annotated image of an agricultural field.

[0050] FIG. 6 illustrates a captured image of an agricultural field.

[0051] FIG. 7 illustrates a segmented image based on the captured image of FIG. 6.DETAILED DESCRIPTION

[0052] FIG. 1 shows an agricultural machine 102 operating in an agricultural field 104. Several plants 120 are growing on the agricultural field 104 in crop rows 116a to 116f. The crop rows 116a to 116f are parallel to each other. The plants 120 may be of a same type of crop, e. g. corn. Each plant 120 has a plant stem emerging point 118 at which the visible part of the plant 120 grows out of the soil of the agricultural field 104. The plant stem emerging points 118 may be spaced apart by a regular distance 122, e. g. a distance by which the seeds had been placed in the agricultural field 104 during a preceding planting process. The distance 122 from a plant stem emerging point 118 to another plant stem emerging point 118 may be approximately equidistant unless a plant is missing. In case of a missing plant 124, the distance from a plant stem emerging point 118 to another plant stem emerging point 118 may be greater as indicated by distance 126 than the distance 122 that may be considered as expected distance.Docket No. 25012WO

[0053] The agricultural machine 102 may be a vehicle 106 or a vehicle-implement combination (106, 108). The vehicle 106 may be an agricultural vehicle such as a tractor, a harvester, a combine, a sprayer or of any other type such as a truck. The vehicle 106 may generate a tractive force to tow an implement 108 through an agricultural field. The implement 108 may be fixed to the vehicle 106 or detachably connected with the vehicle 106. The implement 108 may be used for an operation in the agricultural field and may be of the type of a plough, a rake, a planter, a sprayer, a mower, a trailer, etc. Depending on the type of the implement 108, the implement 108 may comprise one or more tools such as a rake rotor, a mower knife, a seeding unit, a spray nozzle, a shovel, a dumper, etc. In FIG. 1 , the vehicle 106 is exemplarily shown as a harvester with a header attached to the front of the vehicle 106 as implement 108 for harvesting the plants 120.

[0054] The agricultural machine 102 also comprises a control unit 110 as shown in FIG. 2, an imaging unit 112 as shown in FIG. 3 and an inertial navigation system (INS) 114.

[0055] The inertial navigation system 114 provides position and time signals for determining an absolute position of the agricultural machine 102 at a specific point of time. The inertial navigation system 114 may comprise an inertial measurement unit (IMU) with a gyroscope for determining a vehicle speed, a vehicle acceleration and / or an inclination a of the vehicle 106. The inertial navigation system 114 may also comprise a global navigation satellite system (GNSS) receiver receiving position and time signals from a GNSS such as GPS or Galileo. The IMU may provide additional orientation information about the orientation and movement of the agricultural machine 102 for improvingthe accuracy of the position estimation and the reference points of the GNSS receiver. Based on received position and time signals, the agricultural machine 102 can move autonomously along a wayline.

[0056] FIG. 2 shows the control unit 110 comprising an I / O interface 202, a controller 204 and a memory 206. The I / O interface 202, the controller 204 and the memory 206 may be attached to a printed circuit board (PCB). The control unit 110 may receive and send signals or data via the I / O interface 202. For example, the control unit 110 may receive signals from the imaging unit 112 and the inertial navigation system 114. The I / O interface 202 may be a wireless interface or a connector. The controller 204 mayDocket No. 25012WOoptionally represent a combination of two or more networked controllers. The controller 204 may store the data or signals received by the control unit 110 in the memory 206. The memory 206 may contain additional data or executable computer program products, for example in terms of a computer-implemented method, that may be retrieved, processed or executed by the controller 204. Data or signals resulting from the processing of data or signals or from the execution of a computer program product may be stored to the memory 206 or sent to the I / O interface 202 by the controller 204. Regardless of FIG. 2 exemplarily illustrating the control unit 110 as a standalone control unit, the control unit 110 may represent a network of multiple control units distributed within the system.

[0057] FIG. 3 shows exemplarily an imaging unit 112. The imaging unit 112 may be of the type of a 2D-camera, a stereo camera or a time-of-flight (ToF) camera, for example. AToF camera could provide depth information and improve accuracy of detection and pose estimation. Depending on the type of the imaging unit 112, the imaging unit 112 may capture 2D or 3D images, gray-scale images, color images in any color space as for example in red-green-blue (RGB) color space, or multispectral images. Alternatively, the imaging unit 112 may be of a type other than a camera such as a LIDAR or a radar.

[0058] The imaging unit 112 may comprise several components such as at least one optical lens 304, an optional filter 306, a detector 308 and a processing circuitry 310. The optical lens 304 may collect and direct light from a field of view 302 of the imaging unit 112 through the filter 306 to the detector 308 and serve to focus and / or magnify images. The at least one optical lens 304 may be of the type of a fisheye lens, a rectilinear lens or any other standard and moderate wide-angle lens. A fish-eye lens may be of the type of a F-theta lens, a F-tan lens, a tailored distortion lens or a fovea lens, for example. A standard lens is typically defined as a lens with a focal length being approximately equal to the diagonal of the detector 308. This results in a field of view 302 that is rather similar to what human eyes see. Moderate wide-angle lenses have shorter focal lengths than standard lenses, typically ranging from 24 mm to 35 mm for full-frame cameras. These lenses offer a wider field of view 302 than standard lenses and can capture more of the scene in the frame. The optional filter 306 passes selected spectral bands such as ultraviolet, infrared or other bands. The detector 308 may be a digital image sensorthat converts electromagnetic energy to an electric signal andDocket No. 25012WOemploys image sensing technology such as charge-coupled device (CCD) technology and / or complementary metal oxide semiconductor (CMOS) technology. The processing circuitry310 may include a circuitry for amplifying and processingthe electric signal generated by the detector 308 to generate image data, which is passed to the one or more computing devices such as the control unit 110.

[0059] The imaging unit 112 may receive position and time signals from the inertial navigation system 114 for geo-referencing and time stamping of each captured image. The data captured by the imaging unit 112 is logged along with the position and time data gathered by inertial navigation system 114 allowing an accurate determination of the global position of objects contained in the captured images.

[0060] The imaging unit 112 may be moveable so that the pose (i. e. position and / or orientation) of the imaging unit 112 may change. The movement may be determined by a corresponding sensor as for example a position sensor.

[0061] FIG. 4 shows a flow chart of a method for determining a plant stem emerging points dependent wayline. The method may be at least partly a computer-implemented method stored as a computer program product in the memory 206 of the control unit 110. The control unit 110 is configured to carry out the method. Computer-implemented parts of the method may be executed by the controller 204 of the control unit 110. Non-computer-implemented parts of the method may be executed manually or by other components of the system. The method is described by way of example of several steps without any restriction in respect of the steps. That is, the number or the order of steps may be adapted, for example single steps may be excluded and / or added and executed earlier or later than described. When the method proceeds from one step to a next step, the previous step may still be active so that both the one and the next step may be executed in parallel. Accordingly, the control unit may execute two or more method steps in parallel.

[0062] The method starts at step S100 and proceeds to step S101. At step S101 , multiple annotated images are provided by adding at least one annotation to a stock of images each showing crop rows of plants growing in an agricultural field. The images may be taken from different agricultural fields, at different times and may show plants 120 at different growth states. So, each of the annotated images may be different from another. The at least one annotation may be added automatically by the control unitDocket No. 25012WO110 or manually by a user. For example, an annotated image may look like as the annotated image 500 as illustrated in FIG. 5 showing a part of an agricultural field 502 with several plants 506 arranged in crop rows 504 and six annotations 512a to 512f. Each of the six annotations 512a to 512f refer to at least two plant stem emerging points 118 of plants 506 arranged in a common crop row 504. 1, e., an annotation 512a to 512f contains at least two plant stem emerging points 118 of plants 120 arranged in the same crop row 504 and / or an annotation 512a to 512f connects at least two plant stem emerging points 118 of plants 120 arranged in the same crop row 504. An annotation may also connect at least two plant stem emerging points 508 in case of a missing plant 510 as for example annotation 512b or 512e. So, the number of plant stem emerging points 118 covered by an annotation may be different from annotation to annotation.

[0063] An annotation 512a to 512f may be shaped as a straight A-B line or as a spline curve along the crop row 504. Thus, the annotation can be adapted to the pathway of the crop row accordingly. As shown in FIG. 5, the thickness of each annotation 512a to 512f covers the diameters of each plant stems of the plants 506 covered by the corresponding annotation. So, a minimum width of an annotation may be defined by the largest diameter of all plant stems covered by the corresponding annotation.Additionally, the annotations 512a to 512f shown in FIG. 5 match with an expected pattern of the agricultural field 502. For example, the crop rows 504 are parallel to each other and extend from a bottom side to a top side. The distances 122 between the plant stem emerging points 118 of a crop row are expected to be equidistant. Moreover, the pattern may be defined by a row to row distance. Then, the control unit 110 may check whether the annotations 512a to 512f comply with the expected pattern and may detect an invalid annotation if the annotation does not match with the expected pattern, for example if an annotation would not be parallel or would extend from a left hand side to a right hand side. Hence, the control unit 110 may assure a high quality of the annotations 512a to 512f added to each annotated image 500.

[0064] The expected pattern may be determined by the control unit 110 based on the image to be annotated. For example, the control unit 110 may determine the pattern in terms of parameters such as the distance between two crop rows 504, the direction of the crop rows 504, the parallelism of the crop rows 504, etc. The control unit 110 may show the parameters to an user, e. g. on a display. The parameters may be manuallyDocket No. 25012WOparametrizable so that the user may refine or adjust the values of the parameters. For example, the user may correct or refine the value of a row to row distance that has been determined by the control unit 110. Moreover, parameters of the imaging unit 112 such as the positional height, angle or the orientation may also be manually parametrizable.

[0065] The method proceeds to step S102 for training the trained model by means of the annotated images. For this purpose, the annotated images are stored in the memory 206 of the control unit 110 to build or update a stock of annotated images. The trained model is also stored in the memory 206 of the control unit 110 and is implemented as a computer program product configured to detect plant stem emerging points 118 and / or a path connecting two (or more) plant stem emerging points 118 in an image captured by the imaging unit 112. The control unit 110 may detect plant stem emerging points 118 and / or a path connecting two (or more) plant stem emerging points 118 based on a comparison of the image captured by the imaging unit 112 with the stock of annotated images. The more annotated images are provided for the trained model, the better the quality and performance of the trained model may be improved.

[0066] The method proceeds to step S103 and the control unit 110 triggers the imaging unit 112 to capture an image of the agricultural field 104, for example when the agricultural machine 102 operates in the agricultural field 104 (see FIG. 1). The orientation of the field of view 302 of the imaging unit 112 is adjusted for capturing plant stem emerging points 118 of plants 120 of a first crop row in a first perspective and plant stem emerging points 118 of plants 120 of a second crop row in a second perspective.

[0067] FIG. 6 shows exemplarily a captured image 600 that may be captured by the imaging unit 112. The captured image 600 shows a part of the agricultural field 104 including the plants 120 of the agricultural field 104 arranged in crop rows 116a to 116f. The crop rows 116a to 116f are parallel to each other and extend from a bottom side to a top side of the captured image 600. As can be seen in FIG. 6, the plant stem emerging points 118 of plants 120 of crop rows 116a and 116b are captured from a left hand side perspective whereas the plant stem emerging points 118 of plants 120 of crop rows 116e and 116f are captured from a right hand side perspective (according to the specific orientation of the field of view 302 of the imaging unit 112). Compared to the crop rows 116c and 116d captured more from a centralized perspective, the plant stemDocket No. 25012WOemerging points 118 of plants 120 of crop rows 116a, 116b, 116e and 116f are not hidden by the leaves of the plants 120 and can be freely seen even if the plant stems of the plants 120 are crooked or bend by a wind blowing over the agricultural field 104.

[0068] When the captured image 600 has been captured by the imaging unit 112, position and time signals received from the inertial navigation system 114 are added to the captured image 600 for geo-referencing and time stamping of the captured image 600. The position signal represents the geographic coordinates and the time signal represents the point of time at which the captured image 600 has been captured by the imaging unit 112.

[0069] The method proceeds to step S104 and the control unit 110 receives the captured image 600 from the imaging unit 112. The control unit 110 may store the captured image 600 including the georeference and the time-stamp in the memory 206.

[0070] The method proceeds to step S105 and the control unit 110 starts processing the captured image 600 by means of the trained model trained by the annotated images. The trained model tries to apply the annotations 512a to 512f of the annotated images 500 containing plant stem emerging points 508 (as shown in FIG. 5) to the captured image 600 to detect the plant stem emerging points 118 and / or a path connecting two (or more) plant stem emerging points 118 along a crop row 116a to 116f in the captured image 600. As an intermediate result, the control unit 110 generates a segmented image 700 based on the captured image 600 as illustrated in FIG. 7.Compared to the captured image 600, the segmented image 700 is reduced to pathshaped segmentations 702a to 702f. Each of the segmentations 702a to 702f represents a segmented area of the captured image 600. Other areas of the captured image 600 have been removed by the control unit 110 to extract the segmentations 702a to 702f from the captured image 600.

[0071] The trained model of the control unit 110 may apply a combination of different annotations 512a to 512f of the annotated images on the captured image 600 to determine the segmentations 702a to 702f. Hence, each segmentation 702a to 702f extends along a corresponding crop row 116a to 116f analogously to the annotations 512a to 512f. Each segmentation 702a to 702f may contain at least two plant stem emerging points 118 of plants of the same crop row 116a to 116f. As can be seen in FIG.7, all plant stem emerging points 118 of the captured image 600 could be detected byDocket No. 25012WOthe control unit 110 and are covered by the segmentations 702a to 702f whereas large regions of the captured image 600 without any plant stem emerging point are removed. Thus, at least two plant stem emerging points 118 or a path connecting two plant stem emerging points 118 could be segmented by the control unit 110.

[0072] Then, the method proceeds to step S106 and the control unit 110 determines for each segmentation 702a to 702f a wayline 704a to 704f connecting at least two plant stem emerging points 118 of at least two plants 120 in the same crop row 116a to 116f. The path-shaped segmentations 702a to 702f may be used each as a boundary so that each wayline 704a to 704f extends along and within its corresponding segmentation 702a to 702f. In case of segmentations shaped as spots covering separate plant stem emerging points, a wayline may be determined as a spot to spot connection by the control unit 110. Hence, the waylines 704a to 704f are derived from the corresponding (path-shaped) segmentation 702a to 702f. Since each segmentation 702a to 704f is in alignment with a crop row 116a to 116f, each wayline 704a to 704f is aligned with a corresponding crop row 116a to 116f. Since the plant stem emerging points 118 of a single crop row 116a to 116f are covered by the corresponding segmentation 702a to 702f, the control unit 110 determines a plant stem emerging points dependent wayline.

[0073] After step S106, the control unit 110 may execute one or more optional steps S107. For example, the control unit 110 may optionally generate for each of the waylines 704a to 704f a georeference. For example, the control unit 110 may transfer the georeference included in the captured image 600 to the segmented image 700. Based on optical parameters of the imaging unit 112 stored in the memory 206 of the control unit 110, the control unit 110 can calculate the geographical relationship between the georeference and each object in the captured or segmented image 600, 700. For example, the control unit 110 may determine the geographic coordinates of the separate waylines 704a to 704f based on the georeference. Consequently, the control unit 110 may include the waylines 704a to 704f in a map of the agricultural field 104 and show the mapped waylines 704a to 704f on a display of the agricultural machine 102 together with the map. The georeferenced waylines 704a to 704f may also be used for guiding the agricultural machine 102 automatically through the agricultural field 104. The inertial navigation system 114 of the agricultural machine 102 may determine the current position of the agricultural machine 102 and check whether the positionDocket No. 25012WOmatches with a georeferenced wayline 704a to 704f. In case of a deviation, the control unit 110 may take control of the agricultural machine 102 to bring the position of the agricultural machine in alignment with the corresponding wayline 704a to 704f. For example, the control unit 110 may automatically control a steering system of the agricultural machine 102 to adjust the position of the agricultural machine 102.

[0074] At another optional step, the control unit 110 determines the geographical coordinates of the plant stem emerging points 118 based on the georeference of the captured image 600. Based on optical parameters of the imaging unit 112 stored in the memory 206 of the control unit 110, the control unit 110 can calculate the geographical relationship between the georeference and each plant stem emerging point 118 shown in the captured image 600. 1, e., the control unit 110 calculates the geographic coordinates of each plant stem emerging point 118. Consequently, the control unit 110 may include the plant stem emerging points 118 in a map of the agricultural field 104 and show the mapped plant stem emerging points 118 on a display of the agricultural machine 102 together with the map.

[0075] At another optional step, the control unit 110 checks whether each plant stem emerging point 118 is in alignment with one of the waylines 704a to 704f. If a plant stem emerging point 118 is not in alignment, the control unit 110 detects a plant having a misaligned plant stem emerging point in respect to the corresponding wayline. This plant stem emerging point 118 may be displayed in a map to indicate the misalignment. The control unit 110 may also display the captured image 600 where the misaligned plant stem emerging point 118 can be seen. The plant 120 with the misaligned plant stem emerging point 118 may be of a different type than the plants 120 having a plant stem emerging point 118 in alignment with a wayline 704a to 704f. The plant 120 with the misaligned plant stem emerging point 118 may be weed for example, growing around the other the plants 120. Accordingly, the control unit 110 may control the agricultural machine 102 for a special treatment of this misaligned plant.

[0076] At another optional step, the control unit 110 determines an expected distance between two plant stem emerging points 118 of plants 120 growing in the same crop row 116a to 116f. The expected distance may correspond to a regular distance 122, e. g. a distance by which the seeds had been placed in the agricultural field 104 during a planting process (see FIG. 1). So, the expected distance may be expected as anDocket No. 25012WO(approximately) equidistant distance (except in case of a missing plant 124). The expected distance 122 may be stored in the memory 206 of the control unit 110 as a parameter of the pattern of the agricultural field 104 so that the expected distance can be directly retrieved by the control unit 110. Moreover, the control unit 110 may determine the expected distance based on the captured or segmented image 600, 700, e. g. based on the georeferences of the plant stem emerging points 118 by calculating the distance between two geographical positions of the plant stem emerging points 118.

[0077] At another step, the control unit 110 determines at least one distance 122, 126 between two plant stem emerging points 118 of plants 120 growing in a common crop row 116a to 116f (see FIG. 1 ). For example, the control unit 110 may determine the at least one distance 122, 126 based on the captured or segmented image 600, 700, e. g. based on the georeferences of the plant stem emerging points 118 by calculating the distance between two geographical positions of the plant stem emerging points 118. Then, the control unit 110 compares the distance 122, 126 with the expected distance 122 and may detect a missing plant 124 if the distance 126 between the two plant stem emerging points 118 is greater than the expected distance 122. In case of a missing plant 124, the control unit 110 may indicate the missing plant 124 in the map on a display.

[0078] Then, the method proceeds to step S108 and ends. The method may be restarted again by the control unit 110.

[0079] All references cited herein are incorporated herein in their entireties. If there is a conflict between definitions herein and in an incorporated reference, the definition herein shall control.LISTING OF DRAWING ELEMENTS102 agricultural machine 110 control unit104 agricultural field 112 imaging unit106 vehicle 114 inertial navigation system 108 implement 116a crop rowDocket No. 25012WOb crop row 510 missing plant c crop row 512a annotation d crop row 512b annotation e crop row 512c annotation f crop row 512d annotation plant stem emerging point 512e annotation plant 512f annotation distance 600 captured image missing plant 700 segmented image distance 702a segmentation I / O interface 702b segmentation controller 702c segmentation memory 702d segmentation field of view 702e segmentation lens 702f segmentation filter 704a wayline detector 704b wayline processing circuitry 704c wayline annotated image 704d wayline agricultural field 704e wayline crop row 704f wayline plantplant stem emerging point

Claims

1. Docket No. 25012 / GBPCLAIMSWhat is claimed is:

1. A method for determining a plant stem emerging points (118) dependent wayline, comprising:Receiving a captured image (600) of at least two plants (120) arranged in a crop row (116a - 116e) from an imaging unit (112);Processing the captured image (600) by a trained model trained by annotated images (500);Determining a wayline connecting at least two plant stem emerging points (118) of the at least two plants (120) based on the processing by the trained model.

2. The method of claim 1 , comprising:determining a geographical position of each of the at least two plant stem emerging points.

3. The method of claim 1 or 2, whereinan annotation (512a - 512f) of an annotated image (500) contains at least two plant stem emerging points (508) of plants (506) arranged in a crop row (504).

4. The method of claim 3, whereinthe annotation (512a - 512f) of the annotated image (500) connects the at least two plant stem emerging points (508) of the plants (506) arranged in the crop row (504).

5. The method of claim 3 or 4, whereinthe annotation (512a - 512f) is shaped as a spline curve along the crop row (504) containingthe plant stem emerging points (508), whereinthe thickness of the spline curve covers a diameter of a plant stem of the plants (506).

6. The method of any one of claims 3 to 5, whereinthe annotation (512a - 512f) is matched with an expected pattern of an agricultural field (502).Docket No. 25012 / GBP7. The method of claim 6, whereina parameter of the pattern of the agricultural field (104) and / or the imaging unit (112) is manually parametrizable.

8. The method of any one of the preceding claims, comprising:Providing annotated images (500) comprising an annotation (512a - 512f) with reference to at least two plant stem emerging points (508) of plants (506) arranged in a crop row (504);Trainingthe trained model by the annotated images (500).

9. The method of any one of the preceding claims, comprising:Generating a segmented image (700) comprising at least two segmented plant stem emerging points (118) based on the processing by the trained model.

10. The method of claim 9, whereinthe segmented image (700) comprises a segmentation (702a - 702f) containing the at least two segmented plant stem emerging points (118).

11. The method of claim 10, whereinthe wayline extends along and within the segmentation (702a to 702f) connecting the at least two segmented plant stem emerging points (118).

12. The method of any one of claims 9 to 11, comprising:Determining a distance (122, 126) between two plant stem emerging points (118) based on the segmented image (700).

13. The method of claim 12, comprising:Determining an expected distance (122) between two plant stem emerging points (118);anddetecting a missing plant (124) if the distance (126) between two plant stem emerging points (118) is greater than the expected distance (122) between two plant stem emerging points (118).

14. The method of any one of claims 9 to 13, comprising:Mapping at least one of the plant stem emerging points (118).Docket No. 25012 / GBP15. The method of any one of the preceding claims, comprising:Detecting a plant having a misaligned plant stem emerging point in respect to the wayline.

16. The method of any one of the preceding claims, comprising:Georeferencing the wayline.

17. An agricultural machine (102), comprising:An imaging unit (112); anda control unit (110) configured to carry out the method of any one of the preceding claims.

18. The agricultural machine of claim 17, whereinan orientation of a field of view (302) of the imaging unit (112) is configured for capturing of plant stem emerging points (118) of a first crop row (116a, 116b) and plant stem emerging points (118) of a second crop row (116e, 116f).

19. The agricultural machine of claim 18, whereinthe orientation of the field of view (302) of the imaging unit (112) is configured for capturing of plant stem emerging points (118) of the first crop row (116a, 116b) in a first perspective and plant stem emerging points (118) of the second crop row (116e, 116f) in a second perspective.

Images