Method for monitoring the operation of a root crop harvester and monitoring system

Abstract

The invention relates to a method for monitoring the operation of a root crop harvester (1), comprising: a. obtaining images (15) from an image capturing unit (11), of harvested material (16, 17, 18), conveyed in the harvester (1); b. receiving the images (15) in one or more neural network algorithms (28) provided for applying segmentation and executing the one or more neural network algorithms (28) for analysing the received images (15) in order to detect at least part of the harvested material (16, 17, 18) in the images (15); c. determining one or more harvest parameters (29, 30, 31, 32) associated with the detected at least part of the harvested material (16, 17, 18). The invention further relates to a monitoring system (34), provided for carrying out this method and a harvester (1) provided with such monitoring system (34).

Description

[0001] METHOD FOR MONITORING THE OPERATION OF A ROOT CROP HARVESTER AND MONITORING SYSTEM

[0002] The invention relates to a method for monitoring the operation of a root crop harvester, comprising: a. obtaining images from an image capturing unit, of harvested material, conveyed in the harvester; b. detecting at least part of the harvested material in the images; and c. determining one or more harvest parameters associated with the detected at least part of the harvested material. The invention further relates to a monitoring system for monitoring the operation of a root crop harvester, to a computer program, to a computer-readable data carrier and to a harvester.

[0003] This application concerns harvesters for root crops such as e.g. potatoes, onions, red beets, carrots, ... Such harvesters may vary from machines which are pulled by a tractor or which are carried on a tractor to self-propelled vehicles. Such a harvester comprises a crop-digger section, which e.g. comprises one or more harvesting shares for the harvesting of root crops. Conveying means, comprising e.g. sieving units, a hedgehog, elevators, further conveyors... are provided in order to transport the root crops from the crop-digger section towards a bunker, which may be part of the harvester, or may be separate from the harvester, such as e.g. part of a truck or an external trailer, e.g. towed by a tractor or alternatively towards a big bag or a crate or a bag. Still more alternatively, the root crops may be deposited on the ground.

[0004] At harvesting of root crops, besides the root crops typically also by-products, such as haulm, soil, stones, ... are comprised in the harvested material. During transport from the crop-digger section towards the bunker, as much by-products as possible are removed, to minimize the by-products in the bunker. It is important to clean the root crops as well as possible, limiting damage to the root crops as much as possible. Given changing harvesting conditions, such as a change in soil type or soil humidity, or changes in harvest requirements for specific markets, or different crop types and crop varieties, amount and size of stones in the ground, inclination of the ground, . . . this is no sinecure. To this end, in a harvester, one or more operating parameters are typically adjustable by a harvester operator by taking operation actions in order to ensure that the harvester remains efficient in various operating circumstances. For example, the speed of a conveyor belt can be set and / or agitators and / or an interval between cleaning units and / or a slope of a hedgehog, . . .

[0005] Known harvesters typically include image capturing units for imaging one or more of the conveying means in the harvester, with the harvested material fed thereon. Images from these capturing units are visualised on a screen in the cabin so that the operator is able to see how efficiently the different conveying means are operating and how the harvested material moves through the harvester. Based on what the operator observes on these images, the operator can adjust one or more operating parameters, such as e.g. a conveyor angle, a conveying speed, ... For example, when the operator observes that there is a lot of soil on a sieving web of a potato harvester, the operator can increase the sieving web speed so that more soil falls through the sieving web. By allowing the operator to adjust the operating parameters based on what he sees on the images, the harvester can be set more efficient in various operating circumstances.

[0006] In order to aid the operator with this problem, it is e.g. in research projects and e.g. in W02021160607A1 proposed to use evaluation devices for automatically analysing images obtained by such image capturing units for determining related harvest parameters and generating operating parameters of the root crop harvester based on this analysis. Also in e.g. EP3557972B1 it is described to automatically derive one or more harvest parameters with a processing unit analysing such images.

[0007] In theory, the operator will be able to make more informed decisions using such harvest parameters and possibly operating parameters generated in function thereof, to improve the efficiency even further. Such harvest parameters can e.g. be the number of root crops, the size of root crops, the amount of haulm and / or clods within the harvested material, quality of the root crops, ... In practice, however, the computer vision systems which are proposed in this respect have a myriad of issues which make them not practically usable in real-life conditions. Most importantly, it is very difficult to ensure that the computer program for analysing the images works robustly in all kinds of operating circumstances. Furthermore, the harvested material which is conveyed on the conveying means is very disorganised and chaotic, making it more difficult to detect during operation.

[0008] In other applications, object detection by artificial intelligence (deep learning) using bounding boxes has become very widely used in image detection techniques. Deep learning or neural network algorithms consist of various layers of artificial neurons or artificial nodes that work together to learn and process information, similar to the neural network of a human mind. The grand advantage of said deep learning algorithms compared to more classic computer vision programs is that these deep learning algorithms are self-learning, so that the algorithms are able to adapt to various conditions and can be more robust in situations where the operating circumstances can change very quickly. With classic computer vision programs, per harvest parameter to detect, it must be decided which features are to be detected in an image. This quickly becomes cumbersome when the number of harvest parameter to be detected grows. Dependent on the number of features used, lots of parameters also have to be manually fine-tuned by the engineer.

[0009] The commonly used neural network algorithms using bounding boxes, are trained by feeding the neural network algorithm with images in which each object is inside a known labelled boundary box, which is often rectangular or square. The labelled boundary boxes are usually not actually drawn on the images, but are instead provided in a separate data file, comprising data which refers to specific pixels of the image, so that the neural network algorithm can correctly understand where each boundary box would be placed if it was drawn on the image. The algorithm will learn to automatically detect objects in the images that are similar to the object boxes which the algorithm is trained to recognize. However, applicant noticed that applying this method for detecting harvested material in root crop harvesters is practically not that feasible. Although it is possible to detect the root crops in ideal circumstances, this method is not robust enough to be able to track the root crops in all of the various operating conditions. Especially during wet conditions, when the root crops are mostly covered by wet soil that sticks to the root crops, these algorithms are not able to robustly and accurately make the difference between a clump of soil and a root crop covered by soil.

[0010] An object of the present invention is to provide a method for monitoring the operation of a root crop harvester by automatically detecting harvested material wherein this method is both more practically feasible and more efficient in detecting harvested material than current known methods.

[0011] This object is firstly obtained by providing a method for monitoring the operation of a root crop harvester, comprising: a. obtaining images from an image capturing unit, of harvested material, conveyed on conveying means in the harvester; b. detecting at least part of the harvested material in the images; c. determining one or more harvest parameters associated with the detected at least part of the harvested material; wherein step b comprises receiving the images in one or more neural network algorithms provided for applying segmentation, preferably semantic segmentation, for detecting the at least part of the harvested material in the images and comprises executing the one or more neural network algorithms for analysing the received images in order to detect the at least part of the harvested material.

[0012] By using one or more neural network algorithms provided for applying segmentation, it is possible to detect at least part of the harvested material in a way that is both more practically feasible and more efficient than with the currently known methods.

[0013] This method utilises one or more neural networks algorithms, with abovementioned advantages over classic computer vision programs.

[0014] The one or more neural network algorithms are now moreover provided for applying segmentation, which differs a lot from the earlier mentioned bounding boxes model. Instead of detecting each object individually, this neural network algorithm instead detects image regions by locating the boundaries of said objects, by selecting the pixels that ‘belong’ together in a region. For the at least part of the harvested material a respective segmentation mask is thus applied to the image, this segmentation mask specifying, for this at least part of the harvested material, a likelihood that the scene at the pixel shows an object belonging to the at least part of the harvested material. These pixels who belong together in a region share similar characteristics such as colour or brightness and are separated from other pixels belonging to other regions by boundaries that are detected by the neural network algorithm. This method is especially robust compared to other types of neural network algorithms and compared to classical computer vision programs, and e.g. makes it possible to detect root crops and / or one or more by-products such as e.g. haulm, soil, stones, ... in various operating circumstances. Depending on the harvest parameter to be determined, it is e.g. possible to detect root crops and / or one or more by-products such as e.g. haulm, soil, stones, . . . as will be explained further on.

[0015] Instead of other detection methods such as e.g. described in EP3557972B1 wherein detection is based on specifically chosen aspects of the harvested material to be detected, such as e.g. shape, colour and / or size detection, using segmentation, now a neural network algorithm is used, which is trained with labelled images (wherein the harvested material to be detected is labelled), where this algorithm itself decides which aspects to use for detection. This ensures that a much more general approach can be taken and the data can be annotated more accurately, as long as sufficient data are provided.

[0016] More preferentially, the one or more neural network algorithms will be provided for applying semantic segmentation. Semantic segmentation is a specific type of segmentation that does not distinguish between the individual objects within a region. This not only allows the one or more neural network algorithms to work much more efficiently since it no longer needs to recognize each individual object, but it also allows to detect by-products such as haulm and soil using this model since there is no need to recognize each individual ‘soil’ or ‘haulm’ -particle but it is instead only necessary to recognize regions of ‘soil’ and regions of ‘haulm’. Since each individual object is no longer recognized, it is no longer possible to count the exact number of objects, but this is quite irrelevant for the haulm and the soil.

[0017] For the one or more segmentation neural network algorithms, preferentially a convolutional neural network algorithm is chosen and more preferably a neural network algorithm with a contracting path and an expansive path, such as for example the U-Net algorithm. In regular neural networks, neurons are split into various layers, such as the input / output layer and multiple ‘hidden’ layers which apply filters on the input data, which is in this case an input image. The neurons of one layer are all interconnected to the neurons of the next layer. In contrast, a convolutional neural network algorithm takes advantage of the local spatial correlations within an image and instead only links the neurons of one layer to the ‘locally connected’ neurons of the next layer, so that the neural network algorithm can work much more efficiently and requires less memory. These convolutional neural networks have characteristic convolutional layers, which use multi-dimensional filters (e.g. 3x3, 4x4) that are convolved over the input image, so that features in the image such as edges are detected while maintaining the local correlations within an image. Some filters detect edges while others might detect colours, textures,. . .By linking several of these convolutional layers, very complicated features can be detected efficiently by the convolutional neural network algorithm, thus making such a neural network algorithm an excellent choice for (semantic) segmentation, as long as the features are relatively local within an image, such as is the case for the harvested material within the harvester.

[0018] The U-Net algorithm is a specific type of convolutional neural network algorithm that is especially fast and efficient for semantic segmentation. The U-Net algorithm has two paths: a contracting path and an expansive path. The contracting “convolutional” path is a normal convolutional neural network with multiple layers that recognizes features while reducing the spatial resolution of the input image. After this path, a final convolution operation is performed to generate a feature map, containing the semantic segmentation results, with very small spatial resolutions. The second expansive decoding path then again uses convoluting filters while increasing the spatial resolutions of the feature map until a full-sized feature map is returned at the end of the neural network algorithm. In addition, there are some additional connections between equally sized spatial resolutions within the first and second path. These so- called skip connections ensure that while the second path is up-scaling the feature map, the feature map remains as accurate as possible to the input image.

[0019] In order to adapt this neural network algorithm to the specific application within a root crop harvester, the obtained images will preferably first be pre-processed with data augmentation by classic machine operations to improve for example lightning, noise and saturation. Additionally, the images could be rescaled and / or rotated and / or cropped or similarly edited. This data augmentation can provide a significant improvement of the performance of the neural network algorithm.

[0020] The image capturing unit can be a regular camera or a video camera or any other type of image capturing device. Preferentially, this image capturing unit is adjustable by the operator and can pan and / or tilt and / or zoom in accordance with the preferences of the operator.

[0021] Since it can be very dark within the root crop harvester, it is necessary for image capturing units with shutter times to provide such shutter times sufficiently long so that the images are not too dark. However, the shutter times can of course not be too long, since otherwise the images will become very blurry, which would be detrimental for the performance of the one or more neural network algorithms. In specific embodiments, the root crop harvester comprises supplemental internal lights so that the shutter times of the images can become smaller.

[0022] In specific embodiments, the image capturing unit comprises a depth camera or a LiDAR camera system, so that the captured images are 3D images, which provide a lot more information than 2D images. These 3D images additionally provide depth info which allows an accurate volume measurement to be taken as an additional harvest parameter so that the one or more neural network algorithms are able to more efficiently detect the harvested material. For most applications such depth information will however not be required. In other specific embodiments, the image capturing unit comprises a hyperspectral camera, so that the product quality of the root crops can be determined as a harvest parameter.

[0023] Once the one or more harvest parameters are determined, these harvest parameters can be communicated to the operator of the harvester via a user interface, they can be used by other systems within the harvester and / or they can be stored within a data bank.

[0024] The one or more neural network algorithms are preferentially executed by one or more processing units that are specialised in high performance for neural network applications, such as for example a computer circuit that includes good CPU performance, good GPU performance and high-speed memory connections. Such a processing unit preferentially also includes one or more hardware and / or software GPU accelerators. In specific embodiments, such a processing unit can be integrated within the image capturing unit. Preferentially such a processing unit is a local control unit which is provided in the root crop harvester.

[0025] Preferentially, the root crop harvester is adjustable with at least one adjustable operating parameter. For example, the speed of a conveyor belt can be set and / or agitators and / or a distance between cleaning units and / or a slope of a hedgehog, . . .

[0026] The method then preferably comprises generating at least one operating parameter signal based on the one or more harvest parameters, for adjusting the at least one operating parameter of the root crop harvester.

[0027] This at least one operating parameter signal can for example be a number indicating how much the at least one operating parameter should be changed. Even more preferably, the method comprises automatically generating this at least one operating parameter signal based on the one or more harvest parameters, for adjusting at least one operating parameter of the root crop harvester.

[0028] The operating parameter signal can in certain embodiments be provided to the operator of the harvester, such as for example via a graphical user interface or via an audible or visual signal, . . . allowing the operator to make more informed decisions as to how to adjust the at least one operating parameter. Alternatively, at least one operating parameter can be adjusted by a control unit that adjusts the at least one operating parameter based on the at least one operating parameter signal. In addition, this control unit preferably includes means for the operator to override the control unit so that the operator is always able to adjust the at least one operating parameter. In addition, the at least one adjustable operating parameter preferably remains within a safety interval. This safety interval could be immutable in certain embodiments, and could be changeable by the operator in other embodiments. Preferably, the operator is notified via the aforementioned graphical user interface or via audible or visual signal when an operating parameter approaches one of the end points of the safety interval.

[0029] The obtained images are preferably taken with a specific field of view of the image capturing unit, i.e. a specific shot. Such field of view could e.g. be altered by turning such image capturing unit or by moving it otherwise with respect to a target, or when such target moves with respect to the image capturing unit or by zooming, etc. Preferably, the method comprises determining for the obtained images a detection subarea within the specific field of view, as a region of interest for further analysis, and the one or more neural network algorithms are provided for detecting the at least part of the harvested material within the detection sub-area as a region of interest.

[0030] Determining such detection sub-area in this way (and all further uses thereof as described below) is also useful in alternative methods for monitoring the operation of a root crop harvester, where the images are not similarly received in one or more neural network algorithms for applying segmentation, but are instead e.g. analysed with classic machine vision systems or with other types of neural network algorithms.

[0031] The possibility of choosing a detection sub-area, for example, in cases where the field of view is adaptable, such as by moving the image capturing unit and / or the subject with respect to the image capturing unit and / or by zooming, allows the operator to choose a field of view for the image capturing unit that gives him visual information in the cabin, beyond the information which is required for the analysis of the images, e.g. by the one or more neural network algorithms. Choosing the detection sub-area ensures that this does not affect the quality of the analysis of the one or more neural network algorithms. This can for example be desired when the operator wants to have a field of view wherein a conveyor is observed from an angle even though this angle is less suitable for the one or more neural network algorithms, while the chosen detection sub-area is easier to analyse for the one or more neural network algorithms. By choosing a detection sub-area it is also possible to purposefully neglect a zone of the field of view in which harvested material would normally not be present and would thus with high certainty not be detected. It is furthermore e.g. possible to purposefully neglect the detection of harvested material at places where such harvested material can get stuck, such as for example besides the elevators, so that the harvest parameters are not adversely affected by this ‘stuck’ harvested material.

[0032] In addition, the method further preferably comprises one or more transformations of the obtained images, such as a perspective transformation based on the detection subarea, in order to increase the robustness.

[0033] This detection sub-area can in some embodiments be chosen by the operator or can in some embodiments be automatically tracked by a tracking algorithm. The detection sub-area can be any shape, but is preferably either defined by a line or defined by a polygon and more preferentially a quadrilateral such as a rectangle or a trapezium. By choosing a trapezoidal shape, for example a perspective transformation can e.g. be applied to this trapezoidal shape.

[0034] Additionally, the method more specifically comprises determining for the obtained images multiple detection sub-areas within the specific field of view. Then each of the multiple detection sub-areas can be used for a specific purpose, such as for example a first detection sub-area being provided for a specific neural network algorithm, e.g. for detecting stuck haulm, while a second sub-area is provided for another neural network algorithm, e.g. for detecting amount of soil, or for example to give visual information to the operator. The method preferably further comprises selecting a specific detection sub-area of the multiple sub-areas as a region of interest based on the one or more operating parameters.

[0035] In specific embodiments, the image capturing unit is provided for obtaining images of harvested material conveyed on a hedgehog of the harvester. Said detection sub-area is then preferably situated at the top of the hedgehog. This detection sub-area is then preferentially determined by a baseline, which can more specifically be provided settable by the operator or can e.g. be automatically tracked by a tracking algorithm. Such baseline can define one detection sub-area at a specified side thereof, or can define two detection sub-areas at both sides thereof. Multiple such baselines can be set for determining multiple corresponding sub-areas. Instead of providing a baseline, the detection zone can e.g. be determined in the image as a rectangle or as a top edge of the hedgehog.

[0036] The method further preferably comprises determining said detection sub-area more than once. Since the image capturing unit is preferably adjustable by the operator, the field of view of the obtained images can vary depending on how the image capturing unit is adjusted. In addition, various adjustments to the operating parameters, for example the angle of a conveyor of the conveying means, also change the field of view of the obtained images widely. By allowing to determine the detection sub-area more than once, these various adjustments minimally affect the robustness and the performance of the one or more neural network algorithms.

[0037] To this end, in specific embodiments, a first said sub-area is determined at a first field of view of the image capturing unit, a second said sub-area is determined at a second field of view of the image capturing unit, the method further comprises determining the specific field of view at which the obtained images are taken and determining the detection sub-area based on:

[0038] - the determined specific field of view; said first sub-area; and said second sub-area.

[0039] More specifically, where the image capturing unit is provided for obtaining images of harvested material conveyed on a conveyor of the conveying means, for which the image capturing unit and this conveyor are arranged in an adjustable mutual position, the first sub-area is then preferably determined at a first mutual position and the second sub-area at a second mutual position. In order to determine the specific field of view the mutual position of the image capturing unit and the conveyor is then determined at taking of the obtained images.

[0040] Preferably, the one or more neural network algorithms are provided for detecting root crops as a first subpart of the at least part of the harvested material in the images such that when executing the one or more neural network algorithms, the received images are analysed for detecting the first subpart of the at least part of the harvested material and furthermore a root crop parameter is determined as one of the harvest parameters associated with the detected first subpart. In embodiments wherein at least one operating parameter signal is generated, the at least one operating parameter signal can then be generated based on the root crop parameter. Possible root crop parameter can e.g. be an amount of detected root crops, a size, such as e.g. an average size of the detected root crops, an estimated volume of the root crop parameters, a colour of root crops, a speed or acceleration of the root crops, the presence of root crops on a specific area of the conveying means, . . .

[0041] Alternatively and / or additionally, the one or more neural network algorithms are preferably provided for detecting haulm as a second subpart of the at least part of the harvested material in the images such that when executing the one or more neural network algorithms, the received images are analysed for detecting the second subpart of the at least part of the harvested material and furthermore a haulm parameter is determined as one of the harvest parameters associated with the detected second subpart. In embodiments wherein at least one operating parameter signal is generated, the at least one operating parameter signal can then be generated based on the haulm parameter. Possible haulm parameters can e.g. be an amount of haulm, an estimated volume, the presence of haulm on a specific area of the conveying means, . . .

[0042] Further alternative and / or complementary, the one or more neural network algorithms are preferably provided for detecting soil (which may also include clods) as a third subpart of the at least part of the harvested material in the images such that when executing the one or more neural network algorithms, the received images are analysed for detecting the third subpart of the at least part of the harvested material and furthermore a soil parameter is determined as one of the harvest parameters associated with the detected third subpart. In embodiments wherein at least one operating parameter signal is generated, the at least one operating parameter signal can then be generated based on the soil parameter. Possible soil parameters can e.g. be an amount of soil, an estimated volume, the presence of soil on a specific area of the conveying means, ...

[0043] Even further alternative and / or additionally, the one or more neural network algorithms are preferably provided for detecting one or more other harvested material than root crop, haulm and soil as a fourth subpart of the at least part of the harvested material, such as e.g. stones, or unexpected objects such as e.g. a football, etc. such that when executing the one or more neural network algorithms, the received images are analysed for detecting the fourth subpart of the at least part of the harvested material and furthermore a further by-product parameter is determined as one of the harvest parameters associated with the detected fourth subpart. In embodiments wherein at least one operating parameter signal is generated, the at least one operating parameter signal can then be generated based on the further by-product parameter. Possible further by-product parameters can e.g. be an amount of by-products, an estimated volume, the presence of by-products on a specific area of the conveyor, . . .

[0044] Since the features for each type of harvested material e.g. root crops, haulm, soil, ... are quite different from each other, each type of harvested material will preferably be detected by a respective neural network algorithm of the one or more neural network algorithms. Alternatively however, a single neural network algorithm can be executed that is provided for detecting several types of harvested material.

[0045] Preferably, the one or more neural network algorithms are provided for detecting all types of harvested material. The one or more neural network algorithms can thus be provided for detecting the first subpart and for detecting the second subpart and possibly for detecting the third subpart, when such third subpart is present and possibly for detecting the fourth subpart if one or more other types of harvested material are present. At least one neural network algorithm is then preferably provided for detecting the first subpart, at least one neural network algorithm for detecting the second subpart and possibly at least one neural network algorithm for detecting the third subpart and possibly at least one neural network algorithm for detecting the fourth subpart.

[0046] The one or more neural network algorithms are preferably additionally provided for applying segmentation, preferably semantic segmentation, for detecting at least part of the conveying means in the images and the method preferably further comprises checking whether for a received image the detected first subpart, the detected second subpart, possibly the detected third subpart, possibly the detected fourth subpart and the detected at least part of the conveying means amount to a predetermined percentage of this received image.

[0047] Preferably at least one neural network algorithm is specially provided for the detection of the at least part of the conveying means in the images. Alternatively however a single neural network algorithm can be provided for detecting the at least part of the conveying means and one or more types of the harvested material.

[0048] By detecting the conveying means and checking whether the sum of the detected harvested material and the conveying means amount to a predetermined percentage, it is possible to test the robustness of the detections with the one or more neural network algorithms. Especially in embodiments wherein a detection sub-area can be determined so that the detection sub-area comprises solely conveying means and harvested material, the sum of the percentages of the detected at least part of the conveying means and the detected at least part of the harvested material within this detection sub-area should be close to or approximately equal to 100% in case that the one or more neural network algorithms are sufficiently robust. It is possible that the sum of these detected parts is greater than 100%, e.g. somewhere something is seen both as roots crop and haulm, so that there is overlap in the detections. It is also possible that the sum is less than 100%. If the deviation is small, the detections can be considered as correct, whereas they can be considered as incorrect with larger deviations. If the combined percentage deviates from 100% it is also e.g. possible to correct the percentages in proportion until they add up to 100%. Where the detection sub-area comprises parts which cannot be considered as conveying means or harvested materials, such as e.g. a shield plate, compensation can be taken into account for such parts, e.g. taking a lower reference percentage for the sum of the detections of conveying means and harvested material as predetermined percentage. Also where conveying means are not detected and / or when not all harvested material is detected, this can be taken into account when determining the predetermined percentage.

[0049] Detecting all of the harvested material and at least part of the conveying means in an obtained image and checking whether the detected harvested material and at least part of the conveying means amount to a predetermined percentage of the image, is also useful in alternative methods for monitoring the operation of a root crop harvester, where the images are not similarly received in one or more neural network algorithms for applying segmentation, but are instead e.g. analysed with classic machine vision systems or with other types of neural network algorithms.

[0050] Furthermore, the one or more neural network algorithms are preferably additionally provided for applying bounding boxes for detecting a second at least part of the harvested material in the images, such that when executing the one or more neural network algorithms, the received images are analysed by applying bounding boxes for detecting the second at least part of the harvested material. This second at least part of the harvested material can be the same at least part of the harvested material as detected applying segmentation, or can differ from the at least part of the harvested material as detected applying segmentation.

[0051] Preferably at least one neural network algorithm is specially provided for applying bounding boxes for the detection of the second at least part of the harvested material in the images.

[0052] The combination of (semantic) segmentation and bounding boxes is especially useful when (semantic) segmentation is used to determine a said harvest parameter associated with one or more by-products and where bounding boxes are used to determine a said harvest parameter associated with the root crops. Even though using bounding boxes is less robust and less performant than using (semantic) segmentation, using bounding boxes the exact count of root crops can be determined, while with (semantic) segmentation the exact count of root crops can only be approximated as it is difficult to differentiate between overlapping root crops. Since by-products can have highly irregular shapes, it is less preferred to use bounding boxes in order to detect such byproducts.

[0053] Alternatively it is e.g. possible to use an alternative way of segmentation, such as e.g. instance segmentation as an alternative for applying bounding boxes in order to detect the second at least part of the harvested material if some count of root crops is required where semantic segmentation is used in order to detect the first said at least part of the harvested material in order to save computing capacity where an exact count is of lesser importance. Instance segmentation is a more specific type of segmentation for which also the separate detected instances are to be labelled. As a result the output does not only show the area where for example the root crops are situated, but also the number of root crops. In order to perform instance segmentation in contrast with semantic segmentation a new neural network algorithm needs to be trained and the data requires further labelling of the instances.

[0054] The one or more neural network algorithms are preferentially trained with labelled images of harvested material, conveyed on conveying means in a root crop harvester. In order for a neural network algorithm to be trained for a specific type of harvested material or conveying means, at least this specific type of harvested material or conveying means is to be labelled. Since the other types of harvested material can be ignored while labelling a specific type of harvested material, the labelling is easier and faster. Other elements in the images are preferably labelled as not being this specific type of harvested material or conveying means. Where one or more neural network algorithms are to be trained for different types of harvested material and / or conveying means, in order for the one or more neural network algorithms to be trained successfully, examples are preferably used which have been labelled to indicate all of the different elements on the images such as where each type of the harvested material is and where the conveying means are. This labelling can be performed manually, or can additionally be performed semi-automatically where a previous version of a neural network algorithm to be trained is used to label the elements to be labelled, which labelling by the previous version of the one or more neural network algorithms are then corrected manually. The one or more neural network algorithms itself can then be trained in the cloud or on a local device.

[0055] In specific embodiments, the one or more neural network algorithms are trained in several trainings with different sets of labelled images for different harvesting conditions and the one or more neural network algorithms are preferably provided for detecting the at least part of the harvested material according to such a harvesting condition. The method preferably additionally comprises obtaining a said harvesting condition, step b then further comprising receiving the obtained harvesting condition in the one or more neural network algorithms, such that when executing the one or more neural network algorithms, the received images are analysed for detecting the at least part of the harvested material in dependence of the obtained harvesting condition.

[0056] The obtained harvesting condition can e.g. be the sales market of the root crops to be harvested, such as whether the root crops are to be sold fresh, are to be provided for preservation or are to be provided for further processing in a factory, ....

[0057] The obtained harvesting condition can furthermore e.g. reflect the difference between ‘wet’ and ‘dry’ soil, and / or can reflect different soil types and / or different climatological circumstances, changes in harvest requirements for specific markets, different crop types and crop varieties, amount and size of stones in ground, inclination of ground and / or lighting conditions in which harvesting can take place, such as a difference between day or night, or cloudy or foggy, etc. which will cause images taken with image capturing units (possibly switching to nightview, infrared, black and white, ...) to vary,... although it is even more preferential to train the one or more neural network algorithms in order to have sufficient robustness in such said circumstances. It can be provided that the harvesting condition is manually submitted by an operator in order to obtain the harvesting condition. Alternatively or additionally, the harvesting condition can be automatically detected by a detection algorithm using images from the image capturing unit and / or one or more sensors, such as e.g. a humidity sensor, a pressure sensor, .... Alternatively or additionally the one or more neural network algorithms can detect the harvesting condition.

[0058] The one or more neural network algorithms can more specifically comprise multiple sub-algorithms for the different harvesting conditions. Alternatively, respective neural network algorithms for each of the different harvesting conditions may be used. In embodiments where the method comprises the generation of at least one operating parameter signal based on the one or more harvest parameters for adjusting the at least one operating parameter of the root crop harvester, the at least one operating parameter signal is preferably further based on the harvesting condition, so that the operating parameters are also dependent on the obtained harvesting condition, such as e.g. a quality parameter.

[0059] Further in specific embodiments, the image capturing unit is provided for obtaining images of harvested material conveyed on a conveyor of the conveying means, and step a further comprises obtaining images from a second image capturing unit which is provided for obtaining images of harvested material conveyed on the same conveyor. Step b then preferably additionally comprises similarly detecting at least part of the harvested material in the images obtained from this second image capturing unit and synchronising the detection of at least part of the harvested material in the images from the first said image capturing unit and the detection of at least part of the harvested material in the images from the second image capturing unit.

[0060] It is more specifically possible that step a further comprises obtaining images from a third, fourth, . . . image capturing unit, although two image capturing units are most preferential.

[0061] For analysis, analysing images of one image capturing unit can be much simpler, but an operator can e.g. prefer having different views of a conveyor for which different image capturing units are to be provided. In such case, the provided image capturing units can also be used for the method according to the invention.

[0062] Moreover, by using both a first and a second image capturing unit, it is e.g. possible to have multiple view angles at the conveying means, so that the harvested material can be detected more accurately. If only a first image capturing unit is used then it is often not possible to detect all of the harvested material since there can be significant overlap of the various harvested products within one view angle.

[0063] In order to synchronise the detection of the first said image capturing unit and second image capturing unit, there are multiple possibilities. The most preferential possibility is to detect the harvested material in the image of the first image capturing unit separately from the harvested material in the image of the second image capturing unit and then to combine the two determined harvest parameters by for example taking a maximum function of these harvest parameters. A less preferential possibility is to first merge the two images to a combined image and then to execute the one or more neural network algorithms to analyse this combined image. The main drawback with this method is that the two images do often not have the same spatial resolutions, especially when detection sub-areas are used or that images are taken from a different angle due to different settings of the different image capturing units, so that it is difficult to merge them into one image.

[0064] Still further in specific embodiments, the first said image capturing unit is provided for obtaining images of harvested material conveyed on a first conveyor of the conveying means and step a further comprises obtaining images from an additional image capturing unit which is provided for obtaining images of harvested material conveyed on an additional conveyor, different from the first conveyor. Step b then preferably additionally comprises similarly detecting at least part of the harvested material in the images obtained from this additional image capturing unit and step c additionally comprises similarly determining one or more additional harvest parameters associated with the detected at least part of the harvested material in the images from the additional image capturing unit. More preferably, at least one operating parameter signal is generated based on the one or more harvest parameters associated with the detected at least part of the harvested material in the images from the first said image capturing unit and based on the one or more additional harvest parameters associated with the detected at least part of the harvested material in the images from the additional image capturing unit. Preferably, the method further comprises obtaining sensor output data from at least one sensor in the root crop harvester and the at least one operating parameter signal is generated based on the obtained sensor output data. The at least one sensor can comprise a pressure sensor, a moisture sensor, an angle sensor, a wheel speed sensor, a tyre pressure sensor, a vehicle speed sensor, a brake pedal position sensor, a suspension articulation sensor, . . .

[0065] The method further preferably comprises calibrating the obtained sensor output data, either dependent or independent of the operating parameters.

[0066] The object of the invention is further obtained by a monitoring system for monitoring the operation of a root crop harvester, wherein the monitoring system comprises means for carrying out the method as discussed above.

[0067] The monitoring system preferably comprises one or more local control units on the root crop harvester for executing the one or more neural network algorithms. In addition, the monitoring system preferably comprise one or more network connections which allow access to services, data storage and / or applications via the internet or any other type of network such as for example a local area network. By utilising these one or more local control units, it is possible to use this monitoring system in circumstances where there is no good network connection e.g. in areas with bad internet connection, and thus ensures that the monitoring system is more robust. Additionally, the one or more local control units are preferably adapted for use within a root crop harvesters, for example that they are temperature resistant and / or vibration resistant and / or water resistant and / or dust resistant. A network connection allows for an operator to download and / or upload one or more of the one or more neural network algorithms to and / or from the one or more local control units.

[0068] The control system preferably further comprises one or more data banks, which can store the data within the monitoring system such as the images and / or sensor output data and / or the one or more harvest parameters and / or settings of the graphical user interface, ... The network connection allows for an operator to download and / or upload this data to and / or from the one or more data banks. The data which is stored in such a data bank can be retrieved later. These data banks can comprise any type of means for storing data, e.g. flash drives, Sd cards, hard drives, . . . The data banks can be directly integrated within the user interface and / or one or more local control units.

[0069] In addition, the object of the invention is obtained by a computer program, comprising instructions, which, when the computer program is executed on a monitoring system as discussed above causes the monitoring system to carry out the method as discussed above.

[0070] Furthermore, the object of the invention is obtained by a computer-readable data carrier having stored there on the computer program as discussed above.

[0071] Lastly, the object of the invention is obtained by a harvester comprising a monitoring system as discussed above.

[0072] This harvester preferably comprises a first image capturing unit and in addition preferably additional image capturing units. The harvester furthermore preferably comprises at least one sensor which generates sensor output data. The harvester preferably comprises CAN-bus connections and / or ethernet connections between its different components. The monitoring system and the one or more local control units, if present, are preferably adapted to be connectable to the CAN-bus connections and / or the ethernet connections to transmit / obtain the images and / or the sensor output data and / or the operating parameters and / or the operating parameter signals.

[0073] The present invention will now be explained in more detail by means of the following detailed description of methods for monitoring the operation of a root crop harvester, corresponding monitoring systems, computer programs, computer-readable data and harvesters according to the present invention. The sole aim of this description is to give explanatory examples and to indicate further advantages and particulars of the present invention, and can thus by no means be interpreted as a limitation of the area of application of the invention or of the patent rights defined in the claims. In this detailed description, reference numerals are used to refer to the attached drawings, in which in:

[0074] - Fig. 1 schematically illustrates a possible embodiment of a monitoring system according to the invention, implemented in a root crop harvester (1);

[0075] - Fig. 2 illustrates the possible field of view of a camera positioned above a sieving conveyor, with possible set detection sub-areas and with possible detected harvested material and possible detected conveying means using neural network algorithms of a monitoring system according to the invention;

[0076] - Fig. 3 illustrates the possible field of view of a camera positioned above a hedgehog, with possible set detection sub-areas and with possible detected harvested material and possible detected conveying means using neural network algorithms of a monitoring system according to the invention;

[0077] - Fig. 4 schematically illustrates a possible monitoring system according to the invention.

[0078] Figure 1 schematically illustrates a possible embodiment of a monitoring system according to the invention, implemented in a root crop harvester (1). Such a root crop harvester (1) comprises a crop-digger section with e.g. one or more harvesting shares, which are not illustrated. A sieving unit comprises sieving conveyors (3), haulm hooks (10) a feeding conveyor (4), a hedgehog (5) and axial rollers (6). This sieving unit is positioned downstream of the harvesting shares fortransporting root crops towards the ring elevator (7) and in the meantime, to sieve dirt from the harvested root crops. With the elevator (7), the root crops are then lifted towards a discharge conveyor (8) onto a transfer elevator (9). The feeding conveyor (4) preferably does not have a sieve function when passing through the ring elevator (7).

[0079] Other configurations are of course conceivable wherein e.g. the root crop harvester (1) is provided with a bunker and / or without axial rollers (6) and / or with haulm rollers and / or without hedgehog (5) and / or with the ring elevator (7) being positioned fully at the back of the root crop harvester (1) and / or without such ring elevator (7), and / or with additional or less conveyors (3, 4, 8, 9) and / or other transfer means etc.

[0080] The parts of the root crop harvester (1) can be of any known design.

[0081] In figure 1, the harvester (1) is provided with different cameras (11) as image capturing units:

[0082] 2 cameras (11) are positioned above the first sieving conveyor (3);

[0083] 2 cameras (11) are positioned above the third sieving conveyor (3);

[0084] 2 cameras (11) are positioned above the feeding conveyor (4);

[0085] 1 camera (11) is positioned above the hedgehog (5).

[0086] A camera (11) can also be positioned above the reading table (not illustrated).

[0087] Preferably at least 1 camera (11) is provided per said module (3, 4, 5). 2 cameras (11) can be placed above a said module (3, 4, 5) e.g. to provide an operator with a more pleasant view. If desired, even more cameras (11) could be provided per module (3, 4, 5). Alternatively, it is also possible to provide 1 camera (11) for several conveyors (3, 4, 5) next to each other. Such conveyors (3, 4, 5) are then preferably driven at a same speed. It is however also possible to split analysis of the obtained images (15) for conveyors (3, 4, 5) which are set up next to each other but which are running at different speeds. E.g. conveyors which fill an elevator evenly can be controlled separately.

[0088] With fewer cameras (11) related costs will be reduced. With multiple cameras (11) complexity of analysis of corresponding images (15) increases.

[0089] In fact, the harvester (1) can be provided with as many cameras (11) as desired. In practice, preferably e.g. 8 to 18 cameras (11) can be implemented standardly and a customer could request more cameras (11) as an option. Not all cameras (11) will then necessarily be used for monitoring, some might only be present to provide an operator with a desired view, e.g. for driving, such as a reverse camera or a haulm topper camera, etc.

[0090] Where a camera (11) is provided, the location thereof is preferably as central as possible on the respective module (3, 4, 5) above which it is placed. The root crop harvester (1) is now according to the invention provided with a monitoring system (34), which applies artificial intelligence for monitoring the operation of the root crop harvester (1), this with one or more neural network algorithms (28) provided for applying segmentation.

[0091] The monitoring system (34) is preferably provided so that it can be set how many cameras (11) are present so that the one or more neural network algorithm (28) will process images (15) taking this setting into account. The monitoring system (34) can be provided automatically to detect which cameras (11) are present and / or an operator could set which cameras (11) are to be taken into account using e.g. a user interface (13). It can then preferably be set which camera (11) is present at which position in the harvester (1), this preferably per module (3, 4, 5) and possibly also within said module (3, 4, 5).

[0092] The one or more cameras (11) could be supplemented with other image capturing units (11). In addition to one or more image capturing units (11) the harvester (1) is preferably provided with additional sensors (26), such as one or more pressure sensors and / or one or more angle sensors and / or one or more speed sensors and / or one or more valve setting sensors, . . . Thus, preferably a pressure sensor is provided on the first sieving conveyor (3). The filling of the ring elevator (7) could also be monitored using an image capturing unit or using a pressure sensor. Speed sensors for monitoring the speed of the conveyors could further be provided. It is also e.g. possible to provide a moisture sensor.

[0093] Furthermore as shown in figure 1, the monitoring system (34) within the harvester (1) is further preferably provided with a user interfaces (13) within the cabin (2) which allows the operator to make settings and / or on which obtained images (15) and possibly other sensor output data (27) can be shown to the operator and / or on which a generated operating parameter signal (33) can be presented to the operator, etc. Preferably, the control system (30) within the harvester (1) comprises one or more such user interfaces (13). Such user interfaces (13) can e.g. comprise a monitor and / or a touch-screen and / or a keyboard and / or a dial and / or voice activation, etc. Such user interfaces (13) can be provided in the cabin (2) of the harvester (1), but could also comprise mobile devices and / or remote devices.

[0094] The monitoring system (34) can further be provided with a network connection (14) which allows access to services, data storage and / or applications e.g. for obtaining settings via the internet. In certain embodiments of control systems (30) according to the invention, there could also be no network connection (14) or multiple network connections (14) present within the control system (30).

[0095] The control system (30) is further provided with one or more data banks which can store the data within the control system (30) such as the sensor output data (15,17) and / or the current operation actions (at) and / or settings with a user interface (13). These data banks can comprise any type of means for storing data, e.g. flash drives, Sd cards, hard drives, . . .

[0096] The one or more user interfaces (13) and / or the one or more network connections (14) and / or the one or more data banks are preferably adapted to be connectable to a data connection within the control system (30) such as a CAN-bus connection and / or an Ethernet connection.

[0097] The monitoring system (34) shown in figure 1 further comprises a local control unit (12) for executing the one or more neural network algorithms (22). Preferably, the monitoring system (34) comprises one or more such local control units (12), also known as a so-called edge device. The one or more local control units (12) are preferably adapted to be connectable to a data connection within the monitoring system (34) such as a CAN-bus connection and / or an Ethernet connection. Each local control unit (12) is preferably specialised in high performance for neural network applications, and preferably comprises a processing unit which contains a computer circuit that includes good CPU performance, good GPU performance and high-speed memory connections. Such a processing unit preferentially also includes one or more hardware and / or software GPU accelerators. In specific embodiments, such a processing unit can be integrated within an image capturing unit (11). Preferentially such a processing unit is a local processing unit which is provided within the root crop harvester (1).

[0098] For monitoring of the operation of the harvester (1), images (15) are obtained with one or more of said possible cameras (11) in the harvester (1). More specifically e.g. 2 images (15) per second (this is preferably settable) can be taken by each camera (11) so as not to overload the local control unit (12) which is provided for processing said images (15).

[0099] As illustrated in figure 4, the images (15) are loaded in one or more neural network algorithms (28) on one or more local control units (12). Each neural network algorithm (28) is executed for detecting at least part of the harvested material (16, 17, 18) in the images (15) and possibly for detecting at least part of the conveying means (19).

[0100] Such a control unit (12) can be integrated within the respective camera (11) but is preferably provided in the cabin (2). More specifically a Nvidia jetson can be chosen as such control unit( 12).

[0101] Images (15) taken by such cameras (11) are taken with a specific field of view, which - as illustrated in figures 2 and 3 - typically comprises parts of the harvester (1) which are not relevant for the desired analysis, e.g. when the operator wishes to have a view on images (15) taken by such cameras (11) which is not optimal for the required analysis.

[0102] In order to simplify the analysis, the monitoring system (34) is provided to allow the operator to set via the user interface (13) one or more detection sub-area (20, 21, 22, 24) of the specific field of view, such as e.g. illustrated in figures 2 and 3. Alternatively and / or additionally the monitoring system (34) could be provided to have such detection sub-area (20, 21, 22, 24) automatically determined e.g. by a neural network algorithm.

[0103] In figure 2, a first detection sub-area (20) is e.g. defined, selecting an area of the sieving conveyor (3) on which the harvested material (16, 17, 18) is conveyed, to be analysed by one or more corresponding algorithms (28), by selecting four corners (23) defining a quadrangle. Peripheral areas of the conveyor (3) which are not relevant for the further analysis are thereby excluded. A second detection sub-area (21) is analogously defined in figure 2, by selecting four corners (23) on the haulm hooks (10) above the sieving conveyor (3).

[0104] In figure 3, a first detection sub-area (22) is analogously defined, by selecting four comers (23) on the hedgehog (5). Furthermore in figure 3, a second detection sub-area

[0105] (24) is defined, by setting a baseline (25) on the hedgehog (5) above which the image (15) is more specifically further to be analysed. This second detection sub-area (24) is thereby situated at the top of the hedgehog (5).

[0106] Moreover, such cameras (11) will typically be adjustable by the operator and can pan and / or tilt and / or zoom in accordance with the preferences of the operator. Additionally, or alternatively, it is possible that modules (3, 4, 5) above which such cameras (11) are placed, can be set in different positions. In such case, it is not desirable to have said detection sub-areas (20, 21, 22, 24) for each possible setting of the camera (11) and / or the respective module (3, 4, 5). In such cases, preferably a limited number of possible settings is chosen, having different fields of view. For a first said field of view, a first said sub-area is then set (e.g. a first polygon (22) as in figure 3 or a first baseline (25) as in figure 3) and for a second said field of view, a second said sub-area (e.g. a second polygon (22) as in figure 3 or a second baseline

[0107] (25) as in figure 3) is set. Thereafter at each actual setting of the camera (11) and / or each actual setting of the respective module (3, 4, 5) the specific field of view is determined in relation to said first field of view and said second field of view. For this specific field of view, the respective detection sub-area (20, 21, 22, 24) is then not set by an operator or determined by a tracking algorithm, but obtained in function of the set first sub-area and the set second sub-area. More specifically e.g. for the hedgehog (5), a first polygonal sub-area can be set with the hedgehog (5) set at a first angle, a second polygonal sub-area can be set with the hedgehog (5) set at a second angle and at each third angle at which said hedgehog (5) is set, determining the specific field of view at which the images (15) are taken for analysis, the detection sub-area is further determined as a polygon by interpolation of the first polygonal sub-area and the second polygonal sub-area, in accordance with the ratio of the third angle to the first angle and the second angle. The images (15) taken by the cameras (11) can be taken with different shutter times. Such a shutter time, also known as the exposure time, is the length of time that the camera (11) is exposed to light while capturing an image (15). This shutter time can preferably be configured by the operator and / or could be automatically determined based on the brightness conditions within the module, e.g. via a light sensor. Having a slower shutter time makes the images (15) more vague while displaying them, but increases the robustness of the images (15) in various brightness conditions. In specific embodiments, it is also possible that the cameras (11) cycle between different shutter times. For example, a camera (11) could capture 2 images (15) per second with low exposure times and 2 images (15) per second with high exposure times, so that the images (15) with low exposure times can be used for the neural network algorithm (22) and the other images (15) can be used for the user interface (13).

[0108] The images (15) (or only one or more of said detection sub-area (20, 21, 22, 24) thereof, which for the analysis thereof are furthermore also referred to as images (15)) are pre-processed with data augmentation to improve e.g. lightning, noise, saturation, . . . It is e.g. possible to switch to nightview, infrared, black and white, . . .

[0109] Additionally, the images (15) can be rescaled (e.g. images (15) obtain with 704x400 pixels can be resize to 208x208 pixels in order not to overload the control unit (12)) and / or rotated and / or cropped and / or similarly edited.

[0110] Using one or more neural network algorithms (28), the images of a camera (11) are analysed for determining one or more harvest parameters (29, 30, 31, 32) associated with the detected at least part of the harvested material (16, 17, 18) and possibly taking the detected conveying means (19) into account.

[0111] A first neural network algorithm (28) can e.g. be provided for detecting root crops (16) as a first subpart of the harvested material (16, 17, 18) within a specific detection subarea (20, 21, 22, 24) and can possibly also be provided for determining one or more root crop parameters (29) based thereon. A second neural network algorithm (28) can e.g. be provided for detecting haulm (17) as a second subpart and possibly for determining one or more haulm parameters (30) based thereon and / or a third neural network algorithm (28) can e.g. be provided for detecting soil (18) as a third subpart and possibly for determining one or more soil parameters (31) based thereon and / or a further neural network algorithm (28) can e.g. be provided for detecting other harvested material than said root crops (16), haulm (17) or soil (18) as a further subpart of the harvested material (16, 17, 18) and for determining one or more further byproduct parameter (32) based thereon. Another neural network algorithm (28) can e.g. be provided for detecting at least part of the conveying means (19) within the specific detection sub-area (20, 21, 22, 24). It is also possible to have the one or more neural network algorithms (28) to send the detected subparts (16, 17, 18, 19) to a further algorithm (35) for determining one or more respective harvest parameters (29, 30, 31, 32) based thereon, as illustrated in figure 4.

[0112] Alternatively, it is also possible to have one single neural network algorithm (28) which is provided for detecting several types of harvested material (16, 17, 18) and possibly also for detecting conveying means (19). Preferably however separate neural network algorithms (28) are provided for each of the different types of harvested material (16, 17, 18) to be detected and for detecting the conveying means (19) as this simplifies the training of such neural network algorithm (28). Should 1 neural network algorithm (28) be used for detecting all types of harvested material (16, 17, 18) and the conveying means (19), then these should all be labelled for all images (15) used for training purposes, also when retraining is required. Using different neural network algorithms (16, 17, 18) for the different types, allows to use images (15) with specific conditions and, for example, only to retrain the neural network algorithm (28) for detecting soil on those images (15). Otherwise, it would be obliged to retrain for all types and, for the specific images for soil, also to label the other detected types.

[0113] In order to train the one or more neural network algorithms (28) images (15) are first collected of harvested material (16, 17, 18) conveyed in such a root crop harvester (1) and conveying means (19) with which said harvested material (16, 17, 18) is conveyed in this harvester (1). These images (15) are then labelled for training a neural network algorithm (28) for applying segmentation, this preferably on the full field of view thereof. Only after training, parts of the images (15) will be excluded by setting of said detection sub-area (20, 21, 22, 24).

[0114] Lightning, noise and saturation are used as data augmentation methods during training of the neural network algorithms (28) to make the models perform better. Data used for training can be artificially enlarged by e.g. change images (15) applying some classical operations thereon, so as e.g. mirroring, rotating, cropping ...

[0115] Labelling can be done manually or semi-automatically. It is also possible to use self- adversarial training.

[0116] Part of the neural network algorithms (28) is provided for applying segmentation, preferably semantic segmentation. For segmentation, the model can e.g. be based on U-Net as described above.

[0117] Some neural network algorithms (28) can be provided for applying bounding boxes. For bounding boxes, the model can e.g. more specifically be based on yolo (you only look once), but is then preferably specifically adapted for small items. The amount of filters and the amount of convolutional layers can be chosen less than typical for such models and a relatively small grid can be set to detect smaller objects.

[0118] It is also possible to provide some neural network algorithms (28) for applying other types of analysis.

[0119] In embodiments of a monitoring system (34) of the invention wherein all of the harvested material (16, 17, 18) are detected and wherein the conveying means are detected in the set detection sub-area (20, 21, 22, 24), it is further preferably provided to check whether for a received image (15) the detected first subpart, the detected second subpart, the possibly detected third subpart, the possibly detected fourth subpart and the detected at least part of the conveying means (19) amount to a predetermined percentage of this received image (15). If deviations are small, the detections can be considered correct, whereas they can be considered incorrect with larger deviations and possibly rescaled in order to obtain the predetermined percentage. It is furthermore possible to train the neural network algorithms (28) in several trainings with different sets of labelled images (15) for different harvesting conditions (37). It is then possible as illustrated in figure 4, to use e.g. sensor values (37) of sensors (26) provided in this respect and / or parameters (37) read over the internet (14), which are a measure for such harvesting conditions (37), as further input for the neural network algorithms (28) so that the received images (15) can be analysed by the respective neural network algorithm (28) in dependence of the obtained harvesting condition (37), for detecting the at least part of the harvested material (16, 17, 18), e.g. with ‘wet’ / ’dry’ soil as harvesting condition or climatological circumstances as harvesting condition, ... Instead of using such sensor values (37) or parameters (37) read over the internet (14), it is also possible to provide the monitoring system (34) so that an operator can set such harvesting condition as user input (38) using a user interface (13), e.g. with a sales market or quality parameter as harvesting condition, ... The harvesting condition can depend, for example, on the different soil types, climatic conditions, dryness of the soil, quality required, ... where it is now explicitly passed to the neural network algorithms (28) as a parameter to be taken into account in order to analyse the images (15) accordingly. E.g. where at “harvesting condition” A potatoes are nicely washed and at “harvesting condition” B they are very dirty, different evaluation criteria for segmentation are applied.

[0120] It is also possible to have the further determination of harvest parameters (29, 30, 31, 32) e.g. by the algorithm (35) provided in this respect, depend on sensor values (27) and / or parameters (27) read via the internet (14) and / or parameters entered by the operator as user input (38), ... as illustrated in figure 4.

[0121] It is furthermore possible to have the further determination of harvest parameters (29, 30, 31, 32) depend on the analysis of images (15) of different cameras (11).

[0122] In this respect it is e.g. possible to provide several image capturing units (11), each for obtaining images (15) of harvested material (16, 17, 18) conveyed on a same conveyor (3, 4, 5, 6, 7, 8, 9) of the harvester (1). The same or additional neural network algorithms (28) can then be provided for similarly detecting at least part of the harvested material (16, 17, 18) in the images (15) obtained from the different image capturing units (11), as illustrated in figure 4. The detections of at least part of the harvested material (16, 17, 18) in the images (15) from the different image capturing units (11) are then preferably synchronised. It is in this respect e.g. possible to combine respectively determined harvest parameters (29, 30, 31, 32) by for example taking a maximum function of these harvest parameters (29, 30, 31, 32). Alternatively it is e.g. possible first to merge the images (15) of the different cameras (11) to a combined image and then to execute the one or more neural network algorithms (28) to analyse this combined image.

[0123] For determining harvest parameter (29, 30, 31, 32) dependent on the analysis of images (15) of different cameras (11), it is furthermore possible to provide a first image capturing unit (11) for obtaining images (15) of harvested material (16, 17, 18) conveyed on a first conveyor (3, 4, 5, 6, 7, 8, 9) of the harvester (1) and one or more additional image capturing units (11) for obtaining images (15) of harvested material (16, 17, 18) conveyed on one or more additional conveyors (3, 4, 5, 6, 7, 8, 9) of the harvester (1), different from the first conveyor (3, 4, 5, 6, 7, 8, 9). The same or additional neural network algorithms (28) can then be provided for similarly detecting at least part of the harvested material (16, 17, 18) in the images (15) obtained from the one or more additional image capturing units (11), as illustrated in figure 4. The same or additional neural network algorithms (28) can then be provided for similarly detecting at least part of the harvested material (16, 17, 18) in the images (15) obtained from the different image capturing units (11), as illustrated in figure 4, before similarly determining one or more harvest parameters (29, 30, 31, 32).

[0124] Possible root crop parameter (29) can e.g. be an amount of detected root crops, a size, such as e.g. an average size of the detected root crops, an estimated volume of the root crop parameters, a colour of root crops, a speed or acceleration of the root crops, the presence of root crops on a specific area of the conveying means, . . .

[0125] Possible haulm parameters (30) can e.g. be an amount of haulm, an estimated volume, the presence of haulm on a specific area of the conveying means, . . . Possible soil parameters (31) can e.g. be an amount of soil, an estimated volume, the presence of soil on a specific area of the conveying means, ...

[0126] Possible further by-product parameters (32) can e.g. be an amount of by-products, an estimated volume, the presence of by-products on a specific area of the conveyor, . . . Obtained harvest parameters (29, 30, 31, 32) can be communicated to the operator via a said user interface (13) and / or can be used by other systems within the harvester (1) and / or can be stored within a data bank and / or can be shared remotely and / or can be used to determine operating parameter signals (33), ...

[0127] When the harvester (1) is adjustable with at least one adjustable operating parameter, the monitoring system (34) can further be provided with an additional algorithm (36) for generating at least one operator parameter signal (33), based on the one or more harvest parameters (29, 30, 31, 32) (associated with images (15) from one or more image capturing units (11)) for adjusting the at least one operating parameter of the root crop harvester (1), as illustrated in figure 4.

[0128] Such operator parameter signal (33) can e.g. be used to set:

[0129] - the speed of conveying means (3, 4, 6, 7, 8, 9); agitators; an interval between sieving conveyors (3); a slope of a hedgehog (5); adjusting of the spindles; drop height; adjust differential bypasses of e.g. a so-called flexyclean; intensity, angle, position, pressure and / or speed of a haulm roller and in case of a blockage change direction of rotation thereof; adjustment of axial rollers; opening adjustment, angle, rotational direction and / or position of a cleaning unit; fan blower control and intensity; haulm topper intensity, speed and positioning; height of crop digging section. The operator parameter signal (33) can also be based on sensor values (27) and / or parameters entered by the operator as user input (38), ... as illustrated in figure 4.

[0130] It is thus e.g. possible to take into account changes in soil type or soil humidity or changes in harvest requirements for specific markets, or different crop types and crop varieties, amount and size of stones in the ground, inclination of the ground, . . .

[0131] It is e.g. possible to provide the monitoring system (34) so that the operator can set a quality parameter, this e.g. both for the whole harvester (1) as e.g. per possible module (3, 4, 5), so that the operator can determine the quality with which he wishes to harvest. The quality parameter can be set manually by the user in the user interface (13). This can be a general overall quality parameter. Further alternative and / or complementary a quality parameter can be set for each module separately. The operating parameter signal (33) can then e.g. vary depending on the quality and e.g. the amount of haulm detected.

[0132] As an extra safety, the monitoring system (3) can furthermore be provided so that the operator can set a minimum value and a maximum value between which the operating parameter can vary.

[0133] The operator parameter signal (33) can be provided to the operator e.g. via the user interface (13) for setting the operating parameter and / or can be used to automatically set the operating parameter by the monitoring system (34). In the latter case, the monitoring system (34) is preferably provided with a possibility for the operator to override the setting.

[0134] The monitoring system (34) of the invention can e.g. be used to monitor the hedgehog (5) in a potato harvester (1).

[0135] In this respect, it is e.g. possible to detect the amount of potatoes (16) above the baseline (25) (see figure 3) e.g. using bounding boxes and / or to detect other harvested material (17, 18) and / or the conveying means (19) using semantic segmentation. The different harvested material (16, 17, 18) can be detected in series or parallel.

[0136] When potatoes (16) rise above the baseline (25), they can collide with the counter roll, which causes damage. Alternatively or in addition, if there is a lot of soil (18) above the baseline (25), then not enough soil can get rid of. The hedgehog (5) can then be put more vertical and to this end, a hedgehog angle setting parameter can be generated as operating parameter signal (36). This hedgehog angle setting parameter (36) can e.g. further be determined in dependence of the moisture conditions. E.g. where in dry conditions the angle of the hedgehog (5) is to be increased based on the detection of potatoes (16) above the baseline (25), in wet conditions, when there is too much dirt above the baseline, this detection of potatoes (16) can be disregarded and the filling of the hedgehog above the baseline (25) in general can be used to generate the hedgehog angle setting parameter (36) instead.

[0137] It is also possible to use a combination of the amount of haulm (17) and soil (18) detected above the baseline (25) to set the speed of the hedgehog (5) and possibly also setting the speed of other conveying means (3, 4, 6, 7, 8, 9) in the harvester (1). In this respect one or more speed setting signals can correspondingly be generated as operating parameter signal (36). E.g. when there is a lot of soil (18), one or more speeds can be increased, so that more soil falls through the sieving conveyors (3) and when here is less soil (18), such speeds can be lowered.

[0138] Such speed setting signal (36) can furthermore e.g. be determined dependent on e.g. said quality parameter, set by the operator. E.g. when quality is less important, cleaning can be maximised with higher speeds, whereas when quality is more important, cleaning is minimised with lower speeds.

[0139] Preferably also the pressure of the first sieving conveyor (3), e.g. detected using a pressure sensor (26), is taken into account for generating such speed setting signal (36) for setting the speed of the hedgehog (5) or other conveying means (3, 4, 6, 7, 8, 9). Furthermore, it is possible to track whether detected potatoes (16) remain hanging above the baseline (25), e.g. because haulm (17) got stuck to such potato (16). In such case, it is e.g. possible to show a popup on a screen (13) or to generate some other alert, to alert the operator accordingly.

[0140] The monitoring system (34) of the invention can furthermore e.g. be used to monitor the third sieving conveyor (3) (or a second or a further sieving conveyor (3)). In the illustrated embodiment two cameras (11) are provided above this third sieving conveyor (3). Agitators for this conveyor (3) are not separately controllable. In order to control beating of the conveyor (3) with the agitators, the soil (18) on this third sieving conveyor (3) can e.g. be monitored for generating an agitator activation signal as operating parameter signal (36). Since the side with the most soil is the most likely to clog, in order to monitor the soil (18) on this third sieving conveyor (3) the maximum of soil (18) detected with the left camera (11) and the maximum of soil (18) detected with the camera (11) to the right can e.g. be taken to determine the final soil percentage for generating the agitator activation signal (36). In addition to the soil (18) also the amount of detected conveying means (19) can be used to generate the agitator activation signal (36). Furthermore such agitator activation signal (36) can also be determined in dependence of said quality parameter, set by the operator. E.g. when quality is less important, cleaning can be maximised with higher agitation, whereas when quality is more important, cleaning is minimised with lower agitation. In addition, it is also possible to use the detection of haulm (17) stuck to potatoes (16) on the hedgehog (5) as mentioned above to check whether agitators are to be activated in generating such agitator activation signal (36). Potatoes (16) with haulm (17) stuck thereon can more easily be seen on the hedgehog (5).

[0141] A combination of detected soil (18), amount of detected conveying means (19) and haulm (17) detected can e.g. be used to generate a speed setting signal (36) for setting the speed of this third (or second or further) sieving conveyor (3) and possibly other conveying means (3, 4, 6, 7, 8, 9). As mentioned above, preferably also the pressure of the first sieving conveyor (3), e.g. detected using a pressure sensor (26) and / or e.g. a set quality parameter, ... is taken into account for generating such speed setting signal (36). For generating a speed setting signal (36) for setting the speed of the hedgehog (5) preferably also the amount of haulm (17) on the feeding conveyor (4) is taken into account. When there is a lot of haulm (17) the hedgehog (5) can thus be moved faster so it can be processed.

[0142] It is also possible to use the monitoring system (34) of the invention e.g. to monitor:

[0143] - the haulm hooks (10), similar to the third sieving conveyor (3); - the haulm hooks (10) to check whether haulm (17) gets stuck thereon, in order e.g. to generate a popup on a screen (13) or to generate some other alert, to alert the operator accordingly;

[0144] - the reading table, e.g. to determine the size of potatoes (16) entering the bunker, e.g. using bounding boxes, e.g. for classification into different classes and / or to determine yield, . . .

[0145] It is furthermore e.g. possible to detect a blockage in the harvester (1) using the monitoring system. E.g. if haulm (17) is detected which remains constantly detected in the same place over a certain time, this can be detected as a blockage and a corresponding operating parameter signal (33) can be generated and / or a notification can be presented to the operator.

[0146] The monitoring system (34) can in some embodiments be modularly provided. It is e.g. possible to provide a first module for monitoring and / or setting of the hedgehog, a second module for monitoring and / or setting of a said sieving conveyor, a third module for monitoring and / or setting of the haulm roller, a fourth module for monitoring the reading table, etc. Each of these modules can then be separately purchasable. Each of these modules can then be supplemental. It is then e.g. also possible for each of the modules to save determined harvest parameters in a data bank, consultable by the other modules, so that these harvest parameters (29, 30, 31, 32) can also be used by the other modules.

[0147] The in figure 1 illustrated harvester (1) is a potato harvester. The invention is however equally applicable to other type of root crop harvesters, such as e.g. harvesters (1) for harvesting onions or red beets or carrots, etc. In a carrot harvester e.g. similar monitoring with respect to the hedgehog can be implemented. Additionally and / or alternatively it is e.g. possible to implement a monitoring system for setting the harvesting depth depending on e.g. the height of haulm to be cut off and / or a monitoring system for adjusting the cleaning intensity depending on the number of roots that are excessively cut off by the cleaning brushes.

Claims

CLAIMS1. A method for monitoring the operation of a root crop harvester (1), comprising a. obtaining images (15) from an image capturing unit (11), of harvested material (16, 17, 18), conveyed in the harvester (1); b. detecting at least part of the harvested material (16, 17, 18) in the images (15); c. determining one or more harvest parameters (29, 30, 31, 32) associated with the detected at least part of the harvested material (16, 17, 18); characterised in that the step b comprises receiving the images (15) in one or more neural network algorithms (28) provided for applying segmentation, preferably semantic segmentation, for detecting the at least part of the harvested material (16, 17, 18) in the images (15) and comprises executing the one or more neural network algorithms (28) for analysing the received images (15) in order to detect the at least part of the harvested material (16, 17, 18).

2. A method according to claim 1, characterised in that the root crop harvester (1) is adjustable with at least one adjustable operating parameter and that the method comprises generating at least one operating parameter signal (33) based on the one or more harvest parameters (29, 30, 31, 32), for adjusting the at least one operating parameter of the root crop harvester (1).

3. A method according to any of the preceding claims, characterised in that the obtained images (15) are taken with a specific field of view of the image capturing unit (11) and that the method comprises determining for the obtained images (15) a detection sub-area (20, 21, 22, 24) within the specific field of view, and that the one or more neural network algorithms(28) are provided for detecting the at least part of the harvested material (16, 17, 18) within the detection sub-area (20, 21, 22, 24).

4. A method according to claim 3, characterised in that the image capturing unit (11) is provided for obtaining images (15) of harvested material (16, 17, 18) conveyed on a hedgehog (5) of the harvester, and that the detection sub-area (22, 24) is situated at the top of the hedgehog (5).

5. A method according to claim 3 or 4, characterised in that the image capturing unit (11) has an adjustable field of view, that a first sub-area is determined at a first field of view of the image capturing unit (11), that a second sub-area is determined at a second field of view of the image capturing unit (11), that the method further comprises determining the specific field of view at which the obtained images (15) are taken and that the detection sub-area (20, 21, 22, 24) is determined based on:- the determined specific field of view;- said first sub-area; and- said second sub-area.

6. A method according to claim 5, characterised in that the image capturing unit (11) is provided for obtaining images (15) of harvested material (16, 17, 18) conveyed on a conveyor (3, 4, 5, 6, 7, 8, 9) of the harvester (1), that the image capturing unit (11) and the conveyor (3, 4, 5, 6, 7, 8, 9) are arranged in an adjustable mutual position, that the first sub-area is determined at a first mutual position and the second sub-area is determined at a second mutual position and that the mutual position of the image capturing unit (1) and the conveyor (3, 4, 5, 6, 7, 8, 9) is determined in order to determine the specific field of view.

7. A method according to any of the preceding claims, characterised in that the one or more neural network algorithms (28) are provided for detectingroot crops (16) as a first subpart of the at least part of the harvested material (16, 17, 18) in the images (15) and / or for detecting haulm (17) as a second subpart of the at least part of the harvested material (16, 17, 18) in the images (15) and / or for detecting soil (18) as a third subpart of the at least part of the harvested material (16, 17, 18) in the images (15) and / or for detecting one or more other harvested material than root crops (16), haulm (17) and soil (18) as a fourth subpart of the at least part of the harvested material (16, 17, 18), such that when executing the one or more neural network algorithms (28), the received images (15) are analysed for detecting the first subpart and / or the second subpart and / or the third subpart and / or the fourth subpart of the at least part of the harvested material (16, 17, 18) and in that a root crop parameter (29) is determined as one of the harvest parameters (29, 30, 31, 32) associated with the detected first subpart and / or a haulm parameter (30) is determined as one of the harvest parameters (29, 30, 31, 32) associated with the detected second subpart and / or a soil parameter (31) is determined as one of the harvest parameters (29, 30, 31, 32) associated with the detected third subpart and / or a further by-product parameter (32) is determined as one of the harvest parameters (29, 30, 31, 32) associated with the detected fourth subpart.

8. A method according to claim 2 and 7, characterised in that the at least one operating parameter signal (33) is generated based on the root crop parameter (29) and / or based on the haulm parameter (30) and / or based on the soil parameter (31) and / or based on the further by-product parameter (32).

9. A method according to claim 7 or 8, characterised in that the one or more neural network algorithms (28) are provided for detecting the first subpart and for detecting the second subpart and possibly for detecting the third subpart and possibly for detecting the fourth subpart, that the one or more neural network algorithms (28) are additionally provided for applyingsegmentation, preferably semantic segmentation, for detecting at least part of conveying means (19) in the images (15), with which the harvested material (16, 17, 18) is conveyed in the harvester (1), and that the method further comprises checking whether for a received image the detected first subpart, the detected second subpart, the possibly detected (15) third subpart, the possibly detected fourth subpart and the detected at least part of the conveying means (19) amount to a predetermined percentage of this received image (15).

10. A method according to any of the preceding claims, characterised in that the one or more neural network algorithms (28) are additionally provided for applying bounding boxes for detecting a second at least part of the harvested material (16, 17, 18) in the images (15), such that when executing the one or more neural network algorithms (28), the received images (15) are analysed by applying bounding boxes for detecting the second at least part of the harvested material (16, 17, 18).

11. A method according to any of the preceding claims, characterised in that the one or more neural network algorithms (28) are trained with labelled images (15) of harvested material (16, 17, 18), conveyed in a root crop harvester (1).

12. A method according to claim 11, characterised in that the one or more neural network algorithms (28) are trained in several trainings with different sets of labelled images (15) for different harvesting conditions (37), that the one or more neural network algorithms (28) are provided for detecting the at least part of the harvested material (16, 17, 18) according to such a harvesting condition (37), that the method comprises obtaining a said harvesting condition (37) and that step b further comprises receiving the obtained harvesting condition (37) in the one or more neural network algorithms (28), such that when executing the one or more neural networkalgorithms (28), the received images (15) are analysed in dependence of the obtained harvesting condition (37), for detecting the at least part of the harvested material (16, 17, 18).

13. A method according to any of the preceding claims, characterised in that the image capturing unit (11) is provided for obtaining images (15) of harvested material (16, 17, 18) conveyed on a conveyor (3, 4, 5, 6, 7, 8, 9) of the harvester (1), that step a further comprises obtaining images (15) from a second image capturing unit (11) which is provided for obtaining images (15) of harvested material (16, 17, 18) conveyed on the same conveyor (3, 4, 5, 6, 7, 8, 9), and that step b additionally comprises similarly detecting at least part of the harvested material (16, 17, 18) in the images (15) obtained from this second image capturing unit (11) and synchronising the detection of at least part of the harvested material (16, 17, 18) in the images (15) from the first said image capturing unit (11) and the detection of at least part of the harvested material (16, 17, 18) in the images (15) from the second image capturing unit (11).

14. A method according to any of the preceding claims, characterised in that the first said image capturing unit (11) is provided for obtaining images (15) of harvested material (16, 17, 18) conveyed on a first conveyor (3, 4, 5, 6, 7, 8, 9) of the harvester (1), that step a further comprises obtaining images (15) from an additional image capturing unit (11) which is provided for obtaining images (15) of harvested material (16, 17, 18) conveyed on an additional conveyor (3, 4, 5, 6, 7, 8, 9) of the harvester (1), different from the first conveyor (3, 4, 5, 6, 7, 8, 9), that step b additionally comprises similarly detecting at least part of the harvested material (16, 17, 18) in the images (15) obtained from this additional image capturing unit (11) and that step c additionally comprises similarly determining one or more additional harvest parameters (29, 30, 31, 32) associated with the detectedat least part of the harvested material (16, 17, 18) in the images (15) from the additional image capturing unit (11).

15. A method according to claim 14 and claim 2 or 8, characterised in that the at least one operating parameter signal (33) is generated based on the one or more harvest parameters (29, 30, 31, 32) associated with the detected at least part of the harvested material (16, 17, 18) in the images (15) from the first said image capturing unit (11) and based on the one or more additional harvest parameters (29, 30, 31, 32) associated with the detected at least part of the harvested material (16, 17, 18) in the images (15) from the additional image capturing unit (11).

16. A method according to any of claims 2, 8 or 15, characterised in that the method further comprises obtaining sensor output data (27) from at least one sensor (26) in the root crop harvester (1) and in that the at least one operating parameter signal (33) is generated based on the obtained sensor output data (27).

17. A monitoring system for monitoring the operation of a root crop harvester (1), characterised in that the monitoring system comprises means for carrying out the method of any of claims 1 to 16.

18. A monitoring system according to claim 17 characterised in that the monitoring system comprises one or more local control units (12) on the root crop harvester (1) for executing the one or more neural network algorithms (28).

19. Computer program, comprising instructions, which, when the computer program is executed on a monitoring system according to claim 17 or claim 18 causes the monitoring system to carry out the method of any of claims 1 to 15.

20. Computer-readable data carrier having stored there on the computer program of claim 19.

21. Harvester (1), comprising a monitoring system according to any of claims16 to 17.

Images