Method for recognizing poor cut quality

Abstract

A method is provided for assessing the cutting performance of a crop harvesting machine. The method comprises the steps of obtaining a plurality of consecutive images of a harvested area, determining whether each of the plurality of consecutive images includes a symptom of poor cutting performance, determining a trend in the symptom of poor cutting performance in the plurality of consecutive images, determining whether the trend indicates a presence of the symptom of poor cutting performance, and if it is determined that the trend indicates the presence of the symptom of poor cutting performance, sending a signal regarding the presence of the symptom of poor cutting performance.

Description

METHOD FOR RECOGNIZING POOR CUT QUALITYTECHNICAL FIELD

[0001] The present invention relates to systems and methods for detecting symptoms of poor cutting performance on a crop harvesting machine.BACKGROUND OF THE INVENTION

[0002] Combine headers typically cut and gather a width of standing crop which is then transported inwardly to the center of the header where a feed draper and feed drum feed the crop rearward into the combine feederhouse. The crop is then processed by the combine. On a draper header, the inward transport of crop material is typically achieved using a left-hand side-draper and a right-hand sidedraper.

[0003] A primary function of any harvest header is cutting the crop, which consists of severing the upper portion of the crop from the lower portion that contains the roots located below the soil surface. The lower portion of crop material that remains is referred to as stubble. The height above the ground at which the crop is cut varies depending on crop type, condition, and agronomic practices.

[0004] While operating a harvester the operator must continuously monitor the stubble behind the header for any symptoms that indicate a potential problem with the machine. Problems with the machine can include damage, and incorrect, or suboptimal machine settings. Some common symptoms an operator can encounter during operation of a harvester include crop streaking, flagging, or pushing.SUMMARY OF THE INVENTION

[0005] According to one embodiment, there is provided a method for assessing the cutting performance of a crop harvesting machine. The method comprises the steps of obtaining a plurality of consecutive images of a harvested area, determining whether each of the plurality of consecutive images includes asymptom of poor cutting performance, determining a trend in the symptom of poor cutting performance in the plurality of consecutive images, determining whether the trend indicates a presence of the symptom of poor cutting performance, and if it is determined that the trend indicates the presence of the symptom of poor cutting performance, sending a signal regarding the presence of the symptom of poor cutting performance.

[0006] According to another embodiment, there is provided a system for assessing the cutting performance of a crop harvesting machine. The system comprises a sensor and an image processor. The sensor is configured to obtaining a plurality of consecutive images of a harvested area. The image processor is configured to determine whether each of the plurality of consecutive images includes a symptom of poor cutting performance, determine a trend in the symptom of poor cutting performance in the plurality of consecutive images, determine whether the trend indicates a presence of the symptom of poor cutting performance, and if the image processor determines that the trend indicates the presence of the symptom of poor cutting performance, the image processor is configured to send a signal regarding the presence of the symptom of poor cutting performance.BRIEF DESCRIPTION OF THE DRAWINGS

[0007] Advantages of the present invention will be readily appreciated as the same becomes better understood by reference to the following detailed description when considered in connection with the accompanying drawings wherein:

[0008] Figure 1 is a perspective view of a crop harvesting machine harvesting crop in accordance with one embodiment of the present invention;

[0009] Figure 2 is a top view of the crop harvesting machine of Figure 1;

[0010] Figure 3 is a side view of the crop harvesting machine of Figure 1 with optical sensors on one portion of the header;

[0011] Figure 4 is a side view of the crop harvesting machine of Figure 1 with optical sensors on another portion of the header;

[0012] Figure 5 is a rear view of the crop harvesting machine of Figure 4.

[0013] Figure 6 is a side view of the crop harvesting machine of Figure 1 with optical sensors on the harvester cab;

[0014] Figure 7 is a rear view of the crop harvesting machine of Figure 6.

[0015] Figure 8 is a schematic view of a system for recognizing symptoms of poor cutting performance in accordance with one embodiment of the present invention;

[0016] Figure 9 is a top view of the crop harvesting machine of Figure 1 where the harvested crop includes symptoms of poor cutting performance;

[0017] Figure 10 is an enlarged view of a crop head from one of the symptoms of poor cutting performance of Figure 9;

[0018] Figure 11 is a flow diagram illustrating one embodiment for recognizing symptoms of poor cutting performance in accordance with the present invention;

[0019] Figure 12 is a perspective view of the results of the process for recognizing symptoms of poor cutting performance using image segmentation;

[0020] Figure 13 is a perspective view of the results of the process for recognizing symptoms of poor cutting performance using object detection;

[0021] Figure 14 is a portion of an image of harvested crop obtained using the process for recognizing symptoms of poor cutting performance in accordance with the present invention;

[0022] Figures 15-17 reflect exemplary matrices used to track the occurrences of symptoms of poor cutting performance in accordance with the present invention; and

[0023] Figure 18 is a flow diagram illustrating one embodiment for determining whether a specific symptom of poor cut quality is present in accordance with the present invention.DETAILED DESCRIPTION OF EXAMPLE EMBODIMENTS

[0024] Referring to the Figures, where like numerals indicate like or corresponding parts throughout the several views, a crop harvesting machine for harvesting agricultural crops is shown generally at 10. Referring to Figures 1 and 2, the crop harvesting machine 10 includes a header 12 mounted on a combine harvester 14 for movement in a forward working direction D. The header 10 includes a front portion 16, a rear portion 18 and a bottom portion 19 extending laterally between opposite ends 20. Crop pick-up reels 22 are positioned generally above the front portion 16 of the header frame 12 for engaging the crops to be harvested. A cutter bar assembly 24 operatively extends across the front portion 16 of the header frame 12 between the opposite ends 20 for cutting the crops to be harvested. Gauge wheels 25 are pivotably coupled to the bottom portion 19 of the header 12. The harvester 14 includes a cab 26 mounted on the front of the harvester. Figures 1 and 2 also illustrate the harvested area 28 after the crop harvesting machine 10 has harvested the crop.

[0025] The crop harvesting machine 10 includes a system 30 for recognizing symptoms of poor cutting performance. The system 30 includes one or more optical sensors 32. The optical sensors 32 continuously observe the ground behind the header 12. The optical sensors 32 may be mounted anywhere on the crop harvesting machine 10 where they can obtain unobstructed views of the harvested area 28 along the entire width of the header 12. Referring to Figure 3, the optical sensors 32 preferably are mounted on the bottom portion 19 of the header 12 so that the sensor field of view 34 covers the harvested area 28 behind the cutter bar assembly 24. Referring to Figures 4 and 5, the optical sensors 32 also may be mounted on the rear portion 18 of the header 12 so that the sensor field of view 34 covers the harvested area 28 behind the cutter bar assembly 24. Alternatively, as reflected in Figures 6 and 7, the optical sensors 32 may be mounted on the sides of the cab 26 so that the sensor field of view 34 covers theharvested area 28 behind the cutter bar assembly 24. Although each example is depicted with two sensors, any number of sensors may be mounted onto the crop harvesting machine 10 without departing from the scope of the invention. The optical sensors 32 preferably comprise monocular cameras. Alternatively, the optical sensors 32 comprise other types of sensors, such as stereo (3D) cameras, scanning 2D or 3D LIDAR or radar, and / or any other type sensor that is configured to provide a 2D or 3D image or profile of the cut crop 28 behind the header 12.

[0026] Referring to Figure 8, the system 30 also includes an image processor 36 and a machine controller 38 mounted on the cab 26 of the harvester 14. The optical sensors 32 and the image processor 36 are configured to detect characteristics representative of symptoms indicating poor cutting performance including but not limited to streaking 60, flagging 62, and / or pushing 64, as reflected in Figure 9.

[0027] Crop streaking 60 consists of a mostly linear strip of uncut or poorly cut crop parallel to the direction D of travel. Streaking 60 most commonly occurs when there is damage to a cutting device on the cutter bar 24, specifically a broken section of a sickle knife. Streaking 60 can also occur when debris such as dirt or mud accumulates on the cutting device which obstructs the cutting components from the crop.

[0028] Flagging 62 is a term that describes seemingly random patterns of uncut, or remaining crop "heads" or "pods" 66, as reflected in Figure 10. Flagging 62 is commonly observed in thin, or sparse crop conditions as well as when operating at ground speeds that exceed the cutting capabilities of the cutting device on the cutter bar 24. The symptom of flagging 62 can also appear in crops where plant damage has occurred prior to harvest from weather events, such as heavy precipitation or hail, or insect damage to the plants.

[0029] Referring to Figure 9, pushing 64 occurs when the cutter bar 24 pushes crop debris or other foreign material forward which results in a patch of exposed soil. Pushing 64 is most commonly observed while harvesting crops that require the cutter bar 24 to be placed directly on the ground, e.g. for many pulsecrops including soybeans, but can also be observed in other crops when large objects, such as large rocks or other foreign objects, are pushed by the cutter bar 24.

[0030] Referring to Figure 8, the machine controller 38 is configured to make machine adjustments to the header 12 or the combine harvester 14 in order to resolve the symptom that has been detected. Machine parameters that can be adjusted include, but are not limited to: harvester ground speed 40, feederhouse position 42, gauge wheel position 43, reel height 44, reel fore-aft position 46, reel speed 48, header or float ground pressure 50, left hand wing trim 52, right hand wing trim 54, cutting device or knife speed 56, and / or header height 58.

[0031] Figure 11 illustrates an exemplary process 68 for recognizing symptoms of poor cutting performance. At the start of the process (step 70), the operator selects the crop type being harvested via a user interface device, typically a screen in the harvester cab 26, which is sometimes referred to as a virtual, or universal terminal (VT) (step 72). Alternatively, the crop type may be selected automatically by the monitoring system 30 based on the crop characteristics it recognizes using the optical sensors 32 and image processor 36. The system 30 then loads the trained deep-learning model corresponding to the type of crop being harvested (step 74). Although described using a deep-learning type model, other methods may be utilized without departing from the scope of the invention. The monitoring software pipeline then begins.

[0032] The system 30 obtains images of the harvested area 28 from the optical sensors 32 (step 76) and transmits them to the image processor 36 for analysis. The analysis involves an image segmentation function (step 78) where each image frame is analyzed for features representing different classes of poor cutting performance including streaking 60, flagging 62, and pushing 64. Although segmentation is the preferred method for detecting symptoms of poor cutting performance, other techniques may also be used, including object detection, without departing from the scope of the invention. The image processor 36 then determines whether a given symptom is detected. In every case of a detected symptom, an alert may be displayed to the operator via a user interface,which is typically located in the cab, but may be located remotely. Each different symptom will require adjusting a different machine parameter, or set of machine parameters, in order to resolve the symptom. Sometimes the resolution of the symptom will require manual intervention, such as in the case of a broken cutting component.

[0033] If the image processor 36 determines that streaking 60 is detected (step 80), an alert is displayed on the user interface device to notify the machine operator that a streak was detected (step 82), and a signal is transmitted to the machine controller 38 to halt the forward motion of the crop harvesting machine 10 (step 84). Alternatively, the signal may be transmitted to the user interface device to instruct the machine operator to halt the forward motion of the crop harvesting machine 10. The machine controller 38 or the machine operator may subsequently raise the header while crop is cleared from the cutter bar 24. At this point harvesting may resume, or if required, the cutting device may be stopped and inspected for damage, either manually or automatically.

[0034] If the image processor 36 determines that flagging 62 is detected (step 86), an alert is displayed to notify the machine operator that flagging was detected (step 88), and a signal is transmitted to the machine controller 38 to make any combination of adjustments including to lower the reel, increase the reel speed, reduce the height of the cutter bar assembly 24 by lowering the combine feederhouse or raising the gauge wheels 25, reduce the harvester ground speed and / or increase the speed of the cutting device (step 90). Alternatively, the signal may be transmitted to the user interface device to instruct the machine operator to make any combination of these adjustments.

[0035] If the image processor 36 determines that pushing 62 is detected (step 92), an alert is displayed to notify the machine operator that pushing was detected (step 94), and a signal is transmitted to the machine controller 38 to make any combination of adjustments including to halt the forward motion of the crop harvesting machine 10, raise the header 12 for a period of time before lowering it back to the ground, reduce the float ground force, and / or adjust the trim setting for either left-hand or right-hand wing, if the adjustment is available(step 96). Alternatively, the signal may be transmitted to the user interface device to instruct the machine operator to make any combination of these adjustments. After making adjustments to the header 12 and / or combine harvester 14 at steps 84, 90 and 96, or if no symptom of poor cutting performance is detected, the system 30 then returns to step 76 and obtains the next image from the optical sensor 32. Monitoring for any symptom continues after an adjustment is made to determine whether the adjustment was effective at reducing the instances of the symptom or eliminating the symptom entirely. Further adjustments may be made to machine parameters if symptoms persist. If a symptom is not reduced or eliminated, such as when the cutting device is damaged, a warning message may be presented to the operator and the crop harvesting machine 10 may be stopped.

[0036] Because streaking 60 typically consists of a singular length of uncut, or poorly cut crop, one would expect only a single instance of streaking 60 detected in the image frame. However, there may be more than one instance detected in circumstances where the streak does not appear to be continuous, or another detection technique is used such as object detection. For example, as reflected in Figure 12, when image segmentation is used to detect the symptoms of poor cutting performance, a single instance of streaking 60 is detected. By contrast, as reflected in Figure 13, when object detection is used to detect the symptoms of poor cutting performance, multiple, discontinuous instances of streaking 60 may be detected.

[0037] Flagging 62, on the other hand, consists of random patterns of crop heads 66. Thus, as reflected in Figures 12 and 13, regardless of the method used to detect symptoms of poor cutting performance, it is common for several instances of flagging 62 to be detected in the image frame when there is a problem. Generally a single instance of flagging 62 is not considered a cut quality issue. Therefore tracking the number of instances and the frequency at which symptoms are detected over time is important for recognizing problematic machine settings or operation.

[0038] To assist in tracking instances and frequency of poor cutting performance symptoms, the images of the harvested area 28 obtained along thewidth of the header 12 may be split into sections. For example, the image may be divided into three sections corresponding to a left "L" portion of the header 12, a center "C" portion of the header 12, and a right "R" portion of the header 12. Each of these sections may further be divided into three subsections for a total of nine subsections across the width of the header. For example, referring to Figure 14, the image reflecting the left L portion of the header may be divided into subsections "0," "1," and "2." Similarly, the image reflecting the center C portion of the header may be divided into subsections "3," "4" and "5," while the image reflecting the right R portion of the header may be divided into subsections "6," "7" and "8." The image processor 36 may record the number of occurrences of a given symptom identified in each subsection into a 1-dimensional matrix, or vector. In order to track trends over time, the image processor 36 may store the previous results. In addition, the image processor 36 may store an integrated value representing the cumulative instance count over time and a derivative value representing the rate of change of the number of detected instances. These values may all be stored in a single 2-dimensional matrix, which may be used for determining whether a symptom of poor cutting performance has occurred. For example, Figure 15 shows an exemplary matrix used to track the occurrence of streaking 60, Figure 16 shows an exemplary matrix used to track the occurrence of flagging 62, and Figure 17 shows an exemplary matrix used to track the occurrence of pushing 64. The first two rows of each matrix identify the section 98 (left L, center C or right R) and subsection or zone 100 of the images across the width of the header 12. The number 110 of symptoms identified in the current image frame 102 and the previous image frame 104 may be stored for each zone 100. The integral value 106 and derivative value 108 are also stored for each zone 100. Although illustrated with nine zones, the matrix could include any number of columns corresponding to any number of zones across the width of the header, and the number of rows may also be different depending on the application.

[0039] When a symptom is no longer being detected, a routine may be implemented that resets the derivative values 108 for the given symptom so that the detection logic resets and no further adjustments are made to the operatingparameters. This routine may be initiated when a symptom has not been detected for a period of time or in a number of consecutive image frames.

[0040] Each symptom may have its own set of rules for determining if an instance is present. These rules may be based on the values stored in the matrices described above, but also may be assessed directly without the use of a matrix. Figure 18 illustrates an exemplary process 112 for determining whether a specific symptom of poor cut quality is present. At the start of the process (step 114), the image processor 36 determines whether the integral value for the symptom is greater than a symptom threshold (step 116). If the image processor 36 determines that the integral value is greater than the threshold, which indicates that instances of the symptom have been persistently recognized in the image frame, the image processor 36 determines whether the derivative value for the symptom is greater than zero (step 118). If the image processor 36 determines that the derivative value is greater than zero, which indicates that the instance count is continuing to increase, the image processor 36 determines that the symptom exists (step 120) and the process ends (step 122). For example, if the streaking integral value is larger than a determined threshold value, and the derivative value is greater than zero, the image processor will indicate streaking is detected and the machine controller will perform an automatic adjustment of the corresponding machine parameter. In the case of flagging, the rule set may be similar where if the flagging integral value is larger than a determined threshold value and the flagging derivative value is larger than zero, the image processor will indicate flagging is detected and the machine controller will perform an automatic adjustment of the corresponding machine parameter. Similarly in the case of pushing, if the pushing integral value exceeds a determined threshold and the pushing derivative value is greater than zero, the image processor will indicate that pushing is detected and the machine controller will perform an automatic adjustment of the corresponding machine parameter. Though the rules appear to be identical, the integral and derivative threshold values used may be unique values for each symptom.

[0041] Returning to Figure 18, if at step 116, the image processor 36 determines that the integral value is not greater than the threshold, the image ioprocessor 36 determines that the symptom does not exist (step 124) and the process ends (step 122). If at step 118, the image processor 36 determines that the derivative value is greater than the threshold but is not greater than zero, the image processor 36 determines that the symptom does not exist (step 124) and the process ends (step 122).

[0042] The invention has been described in an illustrative manner, and it is to be understood that the terminology, which has been used, is intended to be in the nature of words of description rather than of limitation. Many modifications and variations of the present invention are possible in light of the above teachings. It is, therefore, to be understood that within the scope of the appended claims, the invention may be practiced other than as specifically described.

Claims

CLAIMS1. A method for assessing the cutting performance of a crop harvesting machine, the method comprising the steps of: obtaining a plurality of consecutive images of a harvested area; determining whether each of the plurality of consecutive images includes a symptom of poor cutting performance; determining a trend in the symptom of poor cutting performance in the plurality of consecutive images; determining whether the trend indicates a presence of the symptom of poor cutting performance; and if it is determined that the trend indicates the presence of the symptom of poor cutting performance, sending a signal regarding the presence of the symptom of poor cutting performance.

2. The method of claim 1, wherein the trend comprises a cumulative instance count of the symptom of poor cutting performance over time.

3. The method of claim 2, wherein the trend indicates the presence of the symptom of poor cutting performance when the cumulative instance count is greater than a threshold over time.

4. The method of claim 1 or claim 3, wherein the trend comprises a number of occurrences of the symptom of poor cutting performance over time.

5. The method of claim 4, wherein the trend indicates the presence of the symptom of poor cutting performance when the number of occurrences of the symptom of poor cutting performance increases over time.

6. The method of any one of claims 1-5, wherein the signal comprises a notification regarding presence of the symptom of poor cutting performance.

7. The method of any one of claims 1-6, wherein the signal comprises an instruction to modify a parameter on the crop harvesting machine.

8. The method of any one of claims 1-7, further comprising the step of categorizing the symptom of poor cutting performance as one of streaking, flagging and pushing.

9. A system for assessing the cutting performance of a crop harvesting machine comprising: a sensor configured to obtaining a plurality of consecutive images of a harvested area; and an image processor configured to: determine whether each of the plurality of consecutive images includes a symptom of poor cutting performance; determine a trend in the symptom of poor cutting performance in the plurality of consecutive images; determine whether the trend indicates a presence of the symptom of poor cutting performance; and if the image processor determines that the trend indicates the presence of the symptom of poor cutting performance, the image processor is configured to send a signal regarding the presence of the symptom of poor cutting performance.

10. The system of claim 9, wherein the trend comprises a cumulative instance count of the symptom of poor cutting performance over time.

11. The system of claim 10, wherein the trend indicates the presence of the symptom of poor cutting performance when the cumulative instance count is greater than a threshold over time.

12. The system of claim 9 or claim 11, wherein the trend comprises a number of occurrences of the symptom of poor cutting performance over time.

13. The system of claim 12, wherein the trend indicates the presence of the symptom of poor cutting performance when the number of occurrences of the symptom of poor cutting performance increases over time.

14. The system of any one of claims 9-13, wherein the signal comprises a notification regarding presence of the symptom of poor cutting performance.

15. The system of any one of claims 9-14, wherein the signal comprises an instruction to modify a parameter on the crop harvesting machine.

16. The system of any one of claims 9-15, wherein the image processor is configured to categorize the symptom of poor cutting performance as one of streaking, flagging and pushing.

Images