Machine and method for optical sorting product material in a product material flow

Abstract

An optical-based method and machine for sorting granular product material (1) in a product material flow (1.3) comprising an optical detection process measuring optical image data (8.1) of the product material flow (1.3) by one or more optical sensing devices (7) and processing the measured optical image data (8.1) for detecting abnormal objects (1.1.2) in the product material flow (1.3) using an optical detection and classification unit (8), and a sorting process for ejecting the abnormal objects (1.1.2) in the product material flow (1) using an ejection system (20) based on and / or triggered by the optical detection process.

Description

[0001] P1436PC00

[0002] Machine and Method for Optical Sorting Product Material in a Product Material Flow

[0003] Field of the Invention

[0004] The present invention relates to an optical-based machine and method for sorting granular product material objects in a product material flow comprising an optical detection process based on optical image data of the product material flow measured by one or more optical sensing devices, the optical-based sorting method comprising the steps of detecting product material objects in the product flow, classifying the detected product material objects as abnormal or normal based on the measured optical image data by an optical detection and classification unit, and ejecting an material object, if a material object is classified as abnormal, by triggering an ejection system by the optical detection and classification unit, wherein each measured pixel of the optical image data comprise at least pixel intensity and / or pixel color.

[0005] Particularly, the invention relates to a machine and a method for sorting and monitoring granular material and product material, like products consisting of small particles, particularly granular food or agriculture products such as grains, seeds, beans, pulses, nuts, fruits, oat, barley, maize, rice, peas, beans, nuts, etc. and vegetables including long green beans, wherein abnormal or undesired material objects represent a contamination of the material as desired. The term "object" is used herein in a wide sense as including all kinds of materials of granular and bulk products in the form of finer and coarser granular particles, that can be processed in large quantities.

[0006] Background of the Invention

[0007] In methods and machines of these types, a flow of material to be sorted and monitored is delivered through an imaging zone and a sorting or an ejection zone. In the imaging zone, an imaging system generates digital image data of the passingmaterial objects using optical imaging devices. A data processing unit identifies abnormalities, defects, irregularities, non-standard or simply undesired features of the material by analyzing the image data. In the sorting or the ejection zone, any material objects comprising an abnormality have been identified, and are removed or separated from the material flow. The removal is usually by way of one or more blasts of gas, such as air, from one or more ejectors disposed adjacent the product flow.

[0008] Examples for sorting and monitoring machines are disclosed in WO 2013 / 001303 Al or WO 2019 / 201786 Al.

[0009] Removing abnormal or undesired material objects allows the sorted material flow to meet a quality or health standard, while still optimizing the overall yield from the unsorted material flow to the best extent possible, in a given timeframe. The quality or health standards usually define individual maximum levels of contamination for different types of defects, abnormalities, or impurities. For example, the U. S. Food and Drug Administration considers oats a gluten-free grain under its gluten-free labeling regulations and only requires that packaged products with oats as an ingredient contain less than 20 parts per million of gluten objects overall. However, harvested oats from the field often contain much higher contamination of gluten-containing grains, such as barley, which is for example why sorting and analyzing the material becomes necessary.

[0010] As used herein, the term "abnormality" or "abnormal" should be understood to include both defects on material objects being sorted, whereas defects particularly mean unacceptable material objects in the product flow, such as blemished, misshaped, damaged, rotten pieces, foreign items, and any other contaminants in the product stream, and whole material objects which are unsatisfactory or undesired. While material that is referred to as abnormal matches to an undesirable material type with undesirable optical object characteristics, material with optical object characteristics that deviate from the predefined undesirable optical material characteristic is referred to as normal. The term "optical object characteristic" as used herein includes at least one optical material characteristic and / or optical object characteristic of a material object that can be measured with electromagnetic waves, including visible light, but also non-visible light such as infrared light and ultraviolet light. The measured electromagnetic waves include waves reflected from material objects and / or transmitted through material objects.Visible light is suitable, for example, for detecting broken material objects such as broken rice or nuts, or for detecting overripe or underripe fruit based on its color, or for detecting material objects with mold on the surface. Infrared (IR) is suitable, for example, for detecting undesirable / abnormal moisture content in material object, which can lead to grain spoilage, and also for detecting rancid material objects due to prolonged storage or exposure to heat, for example. Ultraviolet (UV) waves are used to detect the presence of certain pesticides that can fluoresce under UV light or for detecting unwanted organic matter such as bacterial contamination that can become visible under UV light. Fluorescence waves are used, for example, to detect aflatoxins, which are produced by fungi and fluoresce, in materials such as corn and peanuts.

[0011] Optical optical-based machines or optical-based sorting machines identify defects in the material flow being sorted by using known techniques, such as by continuously monitoring and analyzing optical images of material objects in the material flow taken at the imaging zone. Output signals from a data processing unit configured as an image analyzer can then be used to allow a control system to instruct a material ejection system to eject the identified material objects from the material flow.

[0012] Constant monitoring of materials may be required at various stages of industrial processes, which may involve large quantities of materials flowing through the process. Monitoring of the material flow can be performed by optical imaging systems configured to generate digital image data of the material objects that can then be processed by a data processing apparatus. The data signals can be measured by an imaging device such as digital cameras or scanners, optical readers, and the like. Signals from the imaging system input into a data processing apparatus can result appropriate process control actions and reports. Monitoring based on data from imaging systems can be provided for various reasons, can be applied to various types of materials, and can result a variety of actions, particularly in sorting the material or in pre-processing or post-processing steps.

[0013] For example, for raw material Intake for ensuring the quality and purity of a raw materials flow like grains, nuts, or fruit by removing foreign materials, damaged goods, or different varieties, quality control for detecting and removing of defectivefood items from a production line based on size, shape, color, or structural abnormalities that do not meet the defined quality standards. Furthermore, in the field of food safety using for example x-rays or infrared light to identify and remove contaminated material objects that could pose hazards, monitoring the packaging process is important to product freshness and to prevent contamination of the packed product.

[0014] Yet another example, sorting can comprise tasks such as grading or classifying material and removal of unwanted material from the material flow. For the sorting process, various analyses of the material flow are provided based on the optical image data. Information may be generated abouts material parameters such as quality, volume, quantity, particle size, mass, shape, color, type, moisture and so on. Information of other property related parameters may also be generated, e.g., amount of abnormal or defective objects, types of abnormalities or defects, product variations and so on.

[0015] There are numerous ways to sort and analyze a material flow and the above examples are not presented as an exhaustive list of sorting and quality monitoring processes. However, sorting and monitoring operations have conflicting requirements. A sorter operates in real-time as the material flows through the process and the processing of image data needs to be fast for the sorter be able to react. On the other hand, the material monitoring operations need to provide accurate analysis to comply with quality and health standards and requirements. The monitoring does not necessarily need to be in real-time. Because of the different requirements the current systems operate based on separated sorting and analysis functions, and often separate machines or systems are used for sorting and monitoring. However, the use of two separate systems incurs higher costs and longer processing times, increases data processing needs and often requires extra mechanical-handling of the material or even manual handling. Furthermore, to comply with quality or health standards on one hand and maintain high material processing throughput on the other hand, often results in false judgment of the quality of material and therefore in ejecting normal and nondefective material from the material flow. Such false positive material causes material loss and increases material costs.Summary of the Invention

[0016] It is one object of the present invention to provide a method and a machine for sorting material of a material flow, that overcomes the disadvantageous as explained above. In particular, it is an object of the invention to simultaneously identify abnormal material, analyze a quality and / or characteristic of the material, improve the efficiency of a sorting process, and increase the material yield and the material throughput through an optical-based machine. Furthermore, it is an object of the invention to provide a prediction about abnormal material in a material flow, prevent degrading of material quality over time and increase the process reliability.

[0017] According to the present invention, these objects are achieved, particularly, with the features of an optical-based method and an optical-based machine for sorting material of a material flow according to the independent claims. In addition, further advantageous embodiments can be derived from the dependent claims and the related descriptions.

[0018] An optical-based sorting method for sorting granular product material objects in a product material flow comprising an optical detection process based on optical image data of the product material flow measured by one or more optical sensing devices, the optical-based sorting method comprising the steps of detecting product material objects in the product flow, classifying the detected product material objects as abnormal or normal based on the measured optical image data by an optical detection and classification unit, and ejecting an material object, if a material object is classified as abnormal, by triggering an ejection system by the optical detection and classification unit, wherein each measured pixel of the optical image data comprise at least pixel intensity and / or pixel color. The optical-based sorting method comprising the steps of measuring for each detected material object an abnormal probability value for being abnormal based on the optical image data; marking a material object as abnormal, if the measured abnormal probability value of the material object exceeds a sorting threshold value, and triggering the ejection system to eject said material object marked as abnormal; marking a product material object for monitoring, if the measured abnormal probability value of the material object exceeds a monitoring threshold value and if the detected material object is marked as abnormal, and increasing a control indicator value gradually by countingover a predefined time interval or over a predefined number of product material objects the number of product material objects marked for monitoring; and triggering an alert signal if the control indicator value (given by the measured counts is detected to exceed a predefined alert threshold or deviates from previously measured counts by a predefined tolerance value, so that adjustments of the optical-based machine and / or sorting method can be made in real-time, thus improving the overall efficiency of the sorting of the material flow. Furthermore trend monitoring based on the control indicator value makes malfunctions of the optical-based machine and / or changes in the material flow measurable, enabling a preventive maintenance of the opticalbased machine and leading to a reducing in machine downtime. Furthermore a continuous improvement of the measuring of abnormal material objects is possible, based on control indicator values so that the sorting method is optimized continuously. Reporting and data analysis of the control indicator values ensure a highly automated and adaptable machine and sorting method resulting in an efficient sorting of the material flow and reducing the potential for errors. Furthermore the machine / sorting method simultaneously sorts and monitors the material flow as a consequence no extra mechanical-handling of the material or even manual handling is needed for the sorting and monitoring step.

[0019] Optional the sorting threshold is lower than the monitoring threshold to ensure a level of efficiency of identifying and removing abnormal material objects of the material flow, but as a consequence there are many normal material object marked as abnormal. Hence a statistic based on the material objects marked as abnormal is useless.

[0020] The monitoring threshold and sorting threshold are defined in order to optimize the quality of the sorted product material and the performance of the opticalbased sorting method. These thresholds define the criteria or limits that determine for each product material is considered acceptable or must be ejected. The estimation of quality of the detected material objects is done in real-time during the sorting of the material flow ensuring that sorted material object consistently meet quality standards. Furthermore, the monitoring threshold detects anomalies or problems at an early stage so that a timely response can be made, and material waste minimized.The optical detection and classification unit measures the abnormal probability values by analyzing the measured pixels of image data by ways of known image analysis methods as for example statistical pattern recognition, feature extraction, classification, or any other suitable method for measuring the abnormal probability value. The abnormal probability value can be understood as the likelihood that the optical detection and classification unit and the applied image analysis method, respectively, identifies measured product material object being abnormal.

[0021] The optical detection and classification unit measures the abnormal probability of material object for example by measuring pixels extracted from the digital image data according to known statistical methods for determining a probability. The relation between material labeled for quality control and material detected can for example be based on comparing the actually measured probability value of a labelled material object derived from the optical image data with the sorting threshold and / or the monitoring threshold, wherein the difference between the measured value and the threshold provides a trend indicator for the material object. Also, the relation can for example be based on comparing a quantity of material detected and a quantity of material labeled for quality control providing a trend indicator for an amount of material. Further, the relation can for example be expressed as a quotient of the material labeled for quality control and the material labeled for ejection. However, other mathematical relations may be used for determining the trend indicator.

[0022] Furthermore, a material object only labeled for ejection but not for monitoring / quality control is less likely correctly identified as abnormal, material also labeled for quality control monitoring is more likely a true abnormal material. As such the material object labeled for quality control monitoring may for example serve as a reference for truly abnormal material object. The optical detection and classification unit determines a control indicator as a product material quality control indicating a relation between material exceeding the sorting threshold and labeled for ejection, and material exceeding the monitoring threshold and labeled for monitoring / quality control. The control indicator is an estimate of the contamination of the product material with abnormal material.The control indicator serves as a material control for the sorting method. It is an indicator for the sorting efficiency and the sorting quality that is derived from optical measurements of the material or the material objects, respectively. It can be provided in real-time while analyzing and sorting material of the material flow. It is not necessary to subject material ejected from the flow to a quality analysis after separation from the flow. The method can easily be implemented in existing optical-based machine comprising an imaging system and an optical detection and classification unit for image and material analysis.

[0023] In an embodiment of the present invention the optical-based method comprises the steps of extracting an optical feature map representing optical features of one or more product classes for each detected material object based on the measured pixel values of the optical image data; generating class probability values for each material objects for the one or more product classes by matching the corresponding feature map of the material object to the corresponding product class, and assigning the class probability values to the material object; and generating the abnormal probability value for each of the material objects based on the class probability values and / or the product class so that the optical feature map of a detected material object represents optical object characteristics(s) characterizing optical measurable material characteristics for example volume, quantity, object size, shape, color, type, moisture, protein content, pesticides, unwanted organic matter such as bacterial contamination, or other material characteristics of a product material object. Whereas one or more feature maps are extracted for each detected material object. The one or more product classes are predefined wherein each product class representing at least one unwanted material characteristic. In a further embodiment each of the one or more product classes representing at least one unwanted material characteristic or one wanted material characteristic. The generated class probability of a material object quantifies the probability of such a detected material object comprising the product class material characteristics. Depending on the optical characteristics / properties of the material objects of the material flow to be sorted and the optical features to be detected for the detection of abnormal material objects, appropriate optical sensors / optical detection devices with different frequency ranges can be used and application-specific product classes can be predefined, whereas the frequency ranges may include non-visible wave lengths.In another embodiment of the invention the abnormal probability value of a material objects is generated based on the product class with the highest class probability value of the generated class probability values of the material objects so that the measured optical characteristic of the detected material object with the optically best matching product class determines the abnormal probability value of the detected material object. The product class with the highest class probability specifies the type of the abnormality of the material object thus product classes are predefined for the optical-based sorting process of the material flow.

[0024] In another embodiment of the invention the optical detection and classification unit comprises a convolutional neural network, the convolutional neural network generating the feature map by convolving pixels of the optical image data representing the material objects with several kernels representing optical characteristics of the product classes during an extraction process; generating a feature vector by flattening the feature maps by a flattening unit during the extraction process; generating for each detected material object the class probability for each product class by a fully connected neural network based on the feature vector, and generating for each detected material object at least one bounding box locating a region within the optical image data comprising the pixels of the material object in the optical image data by a bounding box regressor so that the convolutional neural network unit detects and classifies material objects based on optical image data by the pixels or a part of the pixels representing the material object in the optical image data and therefore can handle a high degree of complexity and variability of material objects optical characteristics of abnormal or normal objects defined by the product classes.

[0025] The generated feature map(s) for each material object represented by pixels in the optical image data represent the presence of specific optical characteristics / features of the product classes represented by one or more kernels for each product class so that product class optical features, hierarchical and non-hierarchical, are detectable by the convolutional neural network. The convolutional neural network comprises one or more convolutional layers and each of these convolutional layers generates one or more feature maps by convolving kernelsrepresenting optical characteristic(s) of a product class with pixels of the optical image data, for example from simple edges of a material object in early convolutional layer to complex shapes / characteristics in deeper convolutional layer.

[0026] In summary the feature maps are the output of one or more kernels applied to pixels of the optical image data or a preceding feature map(s). The feature maps represent the detected optical characteristics of the product classes in various spatial position of the optical image data. As a consequence the convolutional neural network automatically detects product classes characteristics / features without the need of manual feature engineering. The use of the feature maps of the convolutional neural network reduces the dimension of the optical image data while preserving the essential data for the detection and classification of normal or abnormal product material objects.

[0027] The flattening layer of the convolutional neural network is generating the one dimensional feature vector based on at least a part of the feature maps generated by the convolutional layer(s) so that all the optical characteristics of the product classes detected for the detected material object are transformed into the feature vector.

[0028] The convolutional neural network can learn from vast amounts of sample image data, enabling the optical detection and classification unit to distinguish between subtle optical differences in material objects characteristics and detect and classify material objects reproduced in the optical image data under various conditions such as different lighting, angles, or occlusions.

[0029] Optionally the inventive method comprises a training process using a large dataset of sample image records saved in a repository unit, training the neural network on this sample image records to learn the kernels and / or the fully connected neural network and / or bounding box regressor, validating the model to fine-tune its parameters, in particular the kernels, and / or fully connected neural network, and then deploying a resulting digital model for real-time detection and classification of material objects in the material flow via the optical image data. Over time, the convolutional neural network can also be retrained or updated with new sample image records to adapt to changes in material objects features / characteristics or to recognize newproduct classes of materials. This process reduces manual inspection requirements, increases accuracy and efficiency, and significantly outperforms traditional image processing techniques. The product classes defined by the trained kernels and / or fully connected neural network of the convolutional neural network are also named as "predefined" when the digital model is used for the sorting, as these are not changed during the sorting, but only during the training step for example.

[0030] In another embodiment of the invention the method comprises the steps of setting the abnormal probability value of the corresponding detected material object to the highest class probability value of the product classes assigned to the material object; and selecting for each detected material object the corresponding bounding box with the highest confidence by a non-maximum suppressor unit (NMS) so that the corresponding bounding box with the highest confidence by a non-maximum suppressor unit so that redundant detections of material objects are minimized, because object detection algorithms propose multiple bounding boxes around a detected material object. In the material object detection, the non-maximum suppressor unit (NMS) reduces the number of redundant bounding boxes corresponding to the detected material object. Each bounding box is assigned the corresponding confidence value, which reflects the NMS certainty that the bounding box contains the detected material object. The confidence value typically ranges from 0 to 1, where a higher value indicates a greater confidence in the detection. The NMS generates based on the confidence value of the bounding boxes for the detected material object the bounding box with the highest confidence while suppressing the bounding boxes with a lower confidence for the detected material object. As a consequence NMS reduces false positives and improves the overall accuracy of the material object detection and ensures that each detected material object within the optical image data is represented by a single and best fitting bounding box.

[0031] Furthermore, by eliminating numerous extraneous bounding boxes, NMS reduce the computational burden during the post-processing stage, which is beneficial especially for processing optical images data in real-time.

[0032] In another embodiment of the invention the optical detection and classification unit is connected to a repository for searching and loading a sorting threshold and / or a monitoring threshold and / or a class label of a product class and / or a control indicator value so that the monitoring threshold and sorting threshold aredefined in order to optimize the quality of the sorted product material and the performance of the optical-based sorting method. These thresholds define the criteria or limits that determine for each product class when a product material of such product class is considered acceptable or must be ejected. Each product class comprises a monitoring threshold resulting in a real-time check of the quality of the detected material objects during the sorting process of the material flow on a product class specific base ensuring that sorted material object consistently meet quality standards especially definable for each product class. Furthermore, the monitoring threshold detects anomalies or problems at an early stage so that a timely response can be made, and material waste minimized. Each product class comprising a sorting threshold ensure that only detected material object that match product class specific standards defined by the kernels are ejected if the corresponding product class is labeled as abnormal. This results in an increase of the productivity of the optical-based sorting method a reduction of waste of material. Furthermore, the product class specific sorting thresholds enables the optical-based method to comply with product class specific safety-related limits and meet product class specific regulatory standards by reliably removing for example contaminated material objects. Furthermore no extra mechanical-handling of the material or even manual handling is needed for the sorting and monitoring step.

[0033] In another embodiment of the invention the optical detection and classification unit the one or more product classes of the repository for the opticalbased sorting so that the optical-based machine and / or optical-based sorting method are configurable by the HMI and / or a configuration signal for example by setting the product classes defining the optical features / characteristics of abnormal and / or normal material objects of the product flow before starting the sorting process.

[0034] In another embodiment of the invention the product classes are labeled as abnormal or normal, wherein the abnormal probability value of such detected material object assigned to a product class labeled as normal is set to 0 marked as normal so that material objects assigned to a product class labeled as abnormal are taken in consideration for ejection by the optical-based machine only.Brief Description of the Drawings

[0035] The present invention will be explained in more detail below relying on examples and with reference to these drawings in which:

[0036] Figure 1 shows a schematical illustration of a material optical-based machine applying a method for sorting material of a material flow according to the present invention,

[0037] Figure 2a shows a schematical illustration of an optical detection and classification unit according to the present invention,

[0038] Figure 2b shows a schematical illustration of convolutional neural network (CNN) unit according to the present invention,

[0039] Figure 3 shows a flowchart of process steps of the method for sorting material of a material flow according to the present invention, and

[0040] Figure 4 shows a graph illustrating a relation between an abnormality trend indicator as a sorting control and a quantity of abnormal material objects in a defined amount of material.

[0041] Figure 5 shows a schematical illustration of imaging system according to the present invention,

[0042] Figure 6 shows another schematical illustration of convolutional neural network (CNN) unit according to the present invention,

[0043] Figure 7 shows another schematical illustration of convolutional neural network (CNN) unit according to the present invention,Figure 8 shows another schematical illustration of the optical detection and classification unit, the Neural Network the Repository according to the present invention,

[0044] Figure 9 shows schematically an exemplary CNN structure 12.1 for the inventive optical-based image classification and monitoring,

[0045] Figure 10 shows schematically an exemplary Alert signal generating structure.

[0046] Detailed Description of the Preferred Embodiments

[0047] Figure 1 schematically illustrates a material optical-based machine applying a method for sorting material of a granular material objects 1.1 in a product material flow 1.3 according to the present invention. The material optical-based machine is configured for sorting the product material objects 1.1 of the material flow 1.3 comprising normal product material objects 1.1.1 and abnormal product material objects 1.1.2. As explained earlier, abnormal objects 1.1.2 can be any kind of undesired objects that differ with respect to the majority of material objects 1.1 in the material flow 1. While normal objects 1.1.1 represent the majority of objects 1.1 in the material flow 1 , the abnormal objects 1.1.2 differ in some specific material characteristics from these normal objects 1.1.1. The material optical-based machine executes an opticalbased sorting method for sorting the granular product material objects 1.1 in the product material flow 1.3 comprising an optical detection process measuring optical image data 8.1 by an optical sensing device 7 of the product material flow 1.3, detecting product material objects 1.1 of the product flow 1.3 and classifying the abnormal product material objects 1.1.2 of the detected product material objects 1.1.1 / 1.1.2, and ejecting material objects 1.1.2 if a material objects 1.1.2 is classified as abnormal by triggering the ejection system 20 by an optical detection and classification unit 8, wherein each measured pixel of the optical image data 8.1 comprises at least pixel intensity and / or pixel color. The optical sensing device 7 comprising an imaging devices 7.1 and 7.2 wherein the imaging devices 7.1 and / or 7,2may be implemented as a RGB, near infrared or x-ray sensor, or a combination of these, for example.

[0048] The sorting method comprising the steps of measuring for each detected material object 1.1.1 / 1.1.2 a probability parameter value (also referred herein as abnormal probability value) 8.2.3 for a detected material object 1.1.1 / 1.1.2 of being abnormal, wherein the measurement is based on the optical image data 8.1; marking a material object 1.1.1 / 1.1.2 as abnormal if the measured abnormal probability value 8.2.3 of the material object 1.1.1 / 1.1.2 exceeds a sorting threshold 8.3.1.2, and triggering the ejection system 20 to eject said material object 1.1.1 / 1.1.2 marked as abnormal; marking a product material object 1.1.1 / 1 .1.2 for monitoring, if the measured abnormal probability value 8.2.3 of the material objects 1.1.1 . / 1.1.2 exceeds a monitoring threshold value 8.3.1.3 and if the detected material object 1.1.1 / 1.1.2 is marked as abnormal, and increasing a control indicator value 8.4 gradually by counting over a predefined time interval 8.4.1 or over a predefined number 8.4.2 of product material objects 1.1.1 / 1.1.2 the number of product material objects 1.1.1 / 1.1.2 marked for monitoring; and triggering an alert signal 8.6 if the control indicator value 8,4 given by the measured counts is detected to exceed a predefined alert threshold 8.5.1 or deviates from previously measured counts by a predefined tolerance value 8.5.2.

[0049] For measuring the probability parameter value 8.2.3, the inventive sorting method can e.g. comprise the steps of (i) extracting a feature map 8.2.1.1 representing optical characteristics / features of one or more product classes 8.3 for each detected material object 1.1.1 / 1.1.2 based on the measured pixel values of the optical image data 8.1; (ii) generating a class probability value 8.2.7 for each material objects 1.1.1 / 1.1.2 for the one or more product classes 8.3 by matching the corresponding feature map(s) 8.2.1.1 of the material object 1.1 .1 / 1.1.2 to the corresponding product class 8.3, and assigning the class probability values 8.2.7 to the material object

[0050] 1.1.1 / 1.1.2; and (iii) generating and / or measuring the probability parameter value 8.2.3 for each of the material objects 1.1.1 / 1.1.2 based on the class probability 8.2.7 values and / or the product class 8.3.

[0051] Each detected material object 1.1.1 / 1.1.2 includes a respective object parameter 8.2, wherein the object parameter 8.2 comprises an object class 8.2.6 and wherein the object class 8.2.6 of material object 1.1.1 / 1.1.2 is assigned to the productclass 8.3 with the highest class probability 8.2.7 value of the generated class probability values 8.2.7 of such detected material object 1.1.1 / 1.1.2. Furthermore the abnormal probability value 8.2.3 of such material objects 1.1.1 / 1.1.2 is generated based on the product class 8.3 with the highest class probability 8.2.7 value of the generated class probability values 8.2.7 of the material objects 1.1.1 / 1.1.2.

[0052] In the present embodiment of the invention each product class 8.3 comprises a class parameter 8.3.1 with one or more kernels 8.3.1.1 representing material characteristics / features of such a product class 8.3, a class label 8.3.1.4, the sorting threshold 8.3.1.2 and monitoring threshold 8.3.1.3 applicable for such a product class 8.3, wherein each product class 8.3 is markable as normal or abnormal by the class label 8.3.1.4 by the optical detection and classification unit 8 and wherein the abnormal probability value 8.2.3 generated for a detected material objects 1.1.1 assigned to a product class 8.3 marked as normal by the object class 8.2.6 is set to 0 and the abnormal probability value 8.2.3 of a detected material objects 1.1.1 / 1.1.2 assigned to a product class 8.3 marked as abnormal is set to the generated class probability 8.2.7 value of the product class 8.3 assigned as the object class 8.2.6 by the object class 8.2.6. The optical detection and classification unit 8 marks a detected material object 1.1.1 / 1.1.2 as abnormal / normal based on the abnormal probability 8.2.3 of such a material object 1.1.1 / 1 .1.2 and the sorting threshold 8.3.1 .2 of the product class 8.3 assigned to such a material object 1.1.1 / 1.1.2 by the object class 8.2.6. Furthermore the optical detection and classification unit 8 marks a detected material object 1.1.1 / 1.1.2 for monitoring based on the abnormal probability 8.2.3 of such a material object 1.1.1 / 1.1.2 and the monitoring threshold 8.3.1.3 of the product class 8.3 assigned to such a material object 1.1.1 / 1.1.2 by the object class 8.2.6.

[0053] For carrying out the inventive method of sorting a material flow system of this example of a material optical-based machine comprises a hopper 2 for storing the product material 1 , a feeder 3 for feeding material 1 into the material flow 1.3 and at least one chute 4 guiding the material 1.1 in a defined direction through the material optical-based machine. The material optical-based machine comprises the one or more optical sensing devices 7 for generating the optical image data 8.1 of the product material objects 1.1 of the product flow 1 .3, the optical detection and classification unit 8 for detecting the abnormal product material objects 1.1.2 of the product flow 1.3 in the optical image data 8.1 and classifying the detected materialobjects 1.1.1 / 1.1.2 as normal or abnormal based on the measured optical image data 8.1 and the ejection system 20 for ejecting the product material objects 1.1.2 classified as abnormal by triggering the ejection system 20 by the optical detection and classification unit 8, wherein the optical image data 8.1 comprise pixels with at least pixel intensity and / or pixel color. Furthermore each detected material object 1.1.1 / 1.1.2 comprises the object parameter 8.2 comprising a monitoring flag 8.2.4, an ejection flag 8.2.2 for marking the respective material object 1.1.2 for ejection by the ejection system 20 respectively marking the respective material object 1.1.2 as abnormal, the abnormal probability value 8.2.3, the object class 8.2.6. The monitoring flag 8.2.4 is settable by the optical detection and classification unit 8, if the detected material object 1.1.2 is marked for monitoring by the optical detection and classification unit 8.

[0054] Furthermore, the optical-based machine comprises an imaging zone 5 for gathering the digital image data 8.1 of the material 1.1 and an ejection zone 6 for ejecting the abnormal product material objects 1.1.2 by the ejection system 20 as illustrated in Fig. 1 and 5. The optical sensing device 7 of the present example comprises a first imaging device 7.1 and a second imaging device 7.2 for measuring digital the pixels of the image data 8.1 of the material objects 1.1 passing through the imaging zone 5. Depending on the embodiment variant of the material optical-based machine, the imaging devices 7.1 and 7.2 may be digital cameras or digital line scanners that are set up opposite to each other for monitoring observation planes 7.11 and 7.22, respectively, that are located in the imaging zone 5. The material objects 1.1 may be free falling through these observation planes 7.11 and 7.22 and the imaging devices 7.1 and 7.2 capture the optical image data 8.1 from opposite sides of the material flow 1.3. Further, the optical-based machine comprises a separation plate 21 for keeping the abnormal product material objects 1.1.2 separate from the normal product material objects 1.1.1.

[0055] Each of the imaging devices 7.1 and 7.2 is connected to the optical detection and classification unit 8 via a data link 7.3 and 7.4, respectively. The optical detection and classification unit 8 is configured for process the pixels of the optical image data 8.1 to determine an abnormal probability value 8.2.3 for each material object 1.1 being abnormal. Optional the optical image data 8.1 measured by a RGB image signal imaging devices 7.1 and 7.2 implemented as RGB, whereas the optical image data 8.1 is generated by three separate channels (r) corresponding to theprimary colors: red (R), green (G), and blue (B). Each channel (r) encodes the pixels intensity of its respective color as a matrix of pixel values, where each pixel intensity can typically range from 0 (no contribution of the color) to 255 (full contribution of the color) for the 8-bit optical image data 8.1. The combination of these three channel's (r) intensities at each pixel location in the optical image data 8.1 determines the final color of the pixel. The optical image data 8.1 measured by the RGB imaging devices 7.1 and 7.2, the optical image data 8.1 is a three-dimensional array with dimensions corresponding to the optical image data's 8.1 height (h), width (w), and the three-color channels (r). The combination of these three channels / layers (r) of the pixels per sensing device 7.1 / 7.2 represents the material objects 1.1.1 / 1.1.2 measured by the sensing devices 7.1 / 7.2 in the material flow 1.1. herein named as "detected material objects". Due to the two images devices 7.1 / 7.2 and the three channels RGB each of the detected material objects 1.1.1 / 1.1.2 is represented several times by a different layer of pixels in the optical image data 8,1.

[0056] The product class 8.3 are rice, broken rice, and ergot rice, wherein the ergot rice and the broken rice are labelled as abnormal, for example. Each of the product class 8.3 is defined by the predefined kernels 8.3.1.1 representing optical material characteristics of a product class 8.3 optical detectable by the imaging system 7 comprising at least one material characteristics such as quality property, volume, quantity, particle size, shape, color, type, moisture, protein content and so on detectable by at least one of the sensing devices 7. The optical-based machine is configurable of performing the sorting for different types of product material 1, wherein abnormal product material objects 1.1.2 are defined and / or configurable by the definition of the product classes 8.3 and the configuration of the at least one product class 8.3 as abnormal. This possibility of configuring and defining the product class 8.3 makes the sorting process performed by the optical-based machine very flexible regarding the definition of abnormal product material objects 1.1.2 and allows an optimal use of the technical possibilities of the connected image devices 7.1 and 7.2 based on the predefined kernels 8.3.1.1 of product classes 8.3 labeled as abnormal and feature maps 8.2.1.1 of detected material objects 1.1, wherein the kernels 8.3.1.1 represent material characteristics / features detectable by at least one of the image devices 7.1 and 7.2.The optical detection and classification unit 8 comprises an analysis algorithm 15 which is configured for labelling material 1.1 for ejection according to the pre-defined sorting threshold 8.3.1.2, if the measured abnormal probability value 8.2.3 of a detected product material object 1.1 exceeds the sorting threshold value 8.3.1.2 of the product class 8.3 assigned by the object class 8.2.6 of the respective material object 1.1.1 / 1.1 ,2then the of said detected product material object 1.1 is marked as abnormal by the analysis algorithm 15 by setting the ejection flag 8.2.2 of the respective material object 1.1.2. Further, the analysis algorithm 15 is configured for labelling detected material objects 1.1 for quality control according to the monitoring threshold 8.3.1.3, which is higher than the sorting threshold 8.3.1.2 of the product class 8.3 assigned by the object class 8.2.6 of the respective material object 1.1.1 / 1.1.2, if the measured abnormal probability value 8.2.3 of a detected product material object 1.1 exceeds the monitoring threshold value 8.3.1.3 of the product class 8.3 assigned by the object class 8.2.6 of the respective material object 1.1.1 / 1.1.2 then the monitoring flag 8.2.4 of said detected product material object 1 .1 is set to monitoring by the analysis algorithm 15. Also, the analysis algorithm 15 is configured for determining the control indicator 8.4 indicating a relation between material 1.1 labeled for ejection and material labeled for quality control.

[0057] Thus, the material optical-based machine is capable of simultaneously performing a sorting process and a quality monitoring process. The material 1.1 of the material flow 1.3 is simultaneously assessed for being normal or abnormal as well as with respect to the quality of the separation process / and therefore with respect to the quality of the material output from the material optical-based machine on the basis of an assumed or a preset sorting efficiency.

[0058] Depending on the embodiment variant of the material optical-based machine, the optical detection and classification unit 8 comprises an appropriate computing arrangement comprising at least one data processor 9, at least one memory 10, a comparator unit 11, optionally a convolutional neural network (CNN unit) 12.1, a counting unit 13 and a signal generator 14, as well as internal circuitries and components to perform the method described herein.

[0059] The applied CNN unit 12.1 structure consists of a number of layers or multibuilding blocks / units 12.1.i, where each layer in the CNN unit 12.1 has its function: (i) Theconvolutional layer 12.1.1 consists of a collection of convolutional kernels 8.3.1.1 or filters. The input image 8.1 is expressed as N-dimensional metrics and is convolved with these filters 8.3.1.1 to generate the output feature map(s) 8.2.1.1 ; (ii) a pooling layer 12.1.1.3, which has as main task the sub-sampling of the feature maps 8.2.1 .1. These feature maps 8.2.1.1 are generated by following the convolutional operations. Thus, this allows to shrink large-size feature maps 8.2.1.1 to create smaller feature maps 8.2.1.1. Concurrently, it maintains the majority of the optical characteristics (or features) in every step of the pooling stage. In a similar manner to the convolutional operation, both the stride and the kernel 8.3.1.1 are initially size-assigned before the pooling operation is executed; (iii) The rectifier or ReLU (rectified linear unit) 12.1.1.2, is an activation function, which can e.g. be implemented in the present inventive system. Activation function (non-linearity), which maps the input to the output, where the input value is generated by computing the weighted summation of the neuron input along with its bias (if present). This means that the activation function makes the decision as to whether or not to fire a neuron with reference to a particular input by creating the corresponding output. For the present invention, non-linear activation layers can be employed after all layers with weights (i.e. learnable layers, such as FC layers 12.1.3 and convolutional layers 12.1.1) in the used CNN unit 12.1. This non-linear performance of the activation layers means that the mapping of input to output will be non-linear; moreover, these layers give the CNN unit 12.1 the ability to learn the complex optical characteristics / features. The activation function must also have the ability to differentiate, which is a significant feature, as it allows error back-propagation to be used to train the network. In the present inventive system, for example, a ReLU rectifier 12.1.1.2 can be used, which converts the whole values of the input to positive numbers. In the present context, lower computational load is one of the important benefit of ReLU 12.1.1.2 over the application of other activation functions. However, other activation functions may be also suitable for specific applications; (iv) a flattening unit 12.1.2 can e.g. further be implemented generating a vector based on the output of the convolution layer 12.1.1. The fully connected neural network uses the output of the flattening unit 12.1.2 as input for generating the class probability 8.2.7 of the one or more product classes 8.3 with the class probability values 8.2.7 as output, (v) a bounding box regressor 12.4 further be implemented generating for each detected material object 1.1.1 / 1.1.2 at least one bounding box 8.2.5 with respective coordinates as illustrated in Figure 9. The fully connected layer 12.1.3 is located at the end of the CNN unit 12.1. Inside the FC layer 12.1.3, each neuron is connected to all neurons of theprevious layer, providing a fully connected (FC) approach. It is utilized as the CNN classifier. It follows the method of the conventional multiple-layer perceptron neural network, as it is a type of feed-forward ANN. The input of the FC layer 12.1.3 comes from the last pooling layer 12.1.1.3 or convolutional layer 12.1.1. This input is in the form of a vector, which is created from the feature maps 8.2.1.1 after flattening. The output of the FC layer 12..1.3 represents the final CNN unit 12.1 output.

[0060] In summary, the CNN unit 12.1 can be used for the present inventive system which consists of numerous convolution sub-layers 12.1.1.1 preceding sub-sampling (pooling) layers 12.1.1.2, while the ending layers are fully connected neural network unit 12.1.4. The input x of each layer in the CNN unit 12.1 can be organized in three dimensions: height, width, and depth, or m x n x r , where the height (m) is equal to the width (n) optionally. The depth (r) is also referred to as the channel number as illustrated in figure 6. For example, in an RGB image, the depth (r) is equal to three. The kernels (filters) 8.3.1.1 available in each convolutional layer 12.1.1 are denoted by k and also have three dimensions ( n x n x q ), similar to the input image; here, however, n must be smaller than m, while q is either equal to or smaller than r. In addition, the kernels 8.3.1.1 are the basis of the local connections, which share similar parameters (bias bk and weight Wk) for extracting k feature maps hk / 8.2.1.1 with a size of ( m - n - 1) each and are convolved with input, as mentioned above. The convolution layer 12.1.1 generates a dot product between its input and the weights as in the equation below, but the inputs are undersized areas of the initial image size. By applying the nonlinearity or an activation function to the convolution-layer output, the following relation can be given: hk= f (Wk-x+bk). In the next step, every feature map 8.2.1.1 in the sub-sampling layers can be down-sampled. This leads to a reduction in the network parameters, which accelerates the training process and in turn enables handling of the overfitting issue. For all feature maps 8.2.1.1, the pooling function (e.g. max or average) can be applied to an adjacent area of size pxp , where p is the size of the kernel 8.3.1.1. Finally, the fully connected neural network 12.1.3 receive the mid- and low-level material features and create the high-level abstraction, which represents the last-stage layers as in a typical neural network. The classification scores can be generated using the ending layer [e.g. support vector machines (SVMs) or the like. For a given instance, every score represents the abnormal probability 8.2.3 of a specific product class 8.3 of the two or more classes 8.3.The convolutional layer 12.1.1 comprises the bounding box regressor 12.4 , the flattening unit 12.1.2 and the fully connected neural network 12.1.3 using the output of the flattening unit 12.1.2 as input and generating for each detected material object 1.1 the class probability 8.2.7 value 8.2.3 for each product class 8.3 and a bound box value 8.2.5 as output. The classification layer can e.g. comprise a softmax unit that applies the function sm (yi) = evi / j (evi) .

[0061] The flatting unit 12.1.2 generates the material vector 8.2.1.2 based on the output of the convolution layer 12.1.1. The fully connected neural network 12.1.3, in particular each neuron of the fully connected neural network 12.1.3 applies a linear transformation to the material vector 8.2.1.2 of the flattening unit 12.1.2 through a weight's matrix. As a result, all possible connections layer-to-layer are present, meaning every input of the material vector 8.2.1.2 influences every output of fully connected neural network 12.1.3 output vector. The material vector 8.2.1.2 represents optical material characteristics of detected material objects 1.1 extracted by the kernels 8.3.1.1 representing material characteristics of the predefined product classes 8.3.

[0062] Preferably the CNN unit 12.1 is used to predict the product class 8.3 abnormal probability values 8.2.3 and bounding boxes 8.2.5 of each of detected material objects 1.1 simultaneously, wherein the location of a product material object 1.1 is represented by the bounding box 8.2.5 that encompasses the outline of the product material object 1.1. Each bounding box 8.2.5 is a bounding box data assigned, whereas the bounding box data comprises a center top left corner, a width and a high of the bounding box 8.2.5. The bounding box values 8.2.5 is saved in the memory 10. The advantage of using the bounding box 8.2.5 is that it is computationally efficient to compute and represent the detected material object 1.1 in the optical image data 8.1 is very high.

[0063] The detection and classification unit 8 comprises the non-maximum suppression unit (NMS) 8.7 optionally. The NMS 8.7 refining the output of the CNN unit 12.1. It is not a component of the CNN unit 12.1 itself, but a post-processing step that improves the quality of the object parameter 8.2 of detected material objects 1.1 detection by eliminating multiple, redundant bounding box(es) 8.2.5 detected for the same product material object 1.1 and measuring the product class 8.3 with a highest confidence score measured by the NMS 8.7 for each detected material object1.1.1 / 1.1.2. When the CNN unit 12.1 processes the optical image data 8.1 for detecting product material object 1.1 in the optical image data 8.1, it often measures multiple detected material objects 1.1.1 / 1.1.2 for the same product material object 1.1. This can occur due to the way CNN unit 12.1 scans the optical image data 8.1 with overlapping windows or anchor boxes, leading to several potential bounding box(es) 8.2.5 for a single product material object 1.1, each with its own confidence score, whereas the confidence score is measured by an Intersection over a Union (loU) metric. The nonmaximum suppression (NMS) 8.7 operates on the principle of retaining the best bounding box 8.2.5 for a particular detected product material object 1.1 and discarding the rest. The "best" box is typically the one with the highest confidence score. The final list of bounding boxes 8.2.5 after being processes by the NMS 8.7 is used as a part of the output of the classification step of the sorting process. In summary, the NMS 8.7 ensures that each detected product material object 1.1 is represented by only one bounding box 8.2.5 and assigned to one product class 8.3, thereby reducing duplicates, clarifying the output, and making the detection and classification 8 results more useful for subsequent processing or analysis.

[0064] The Convolutional Neural Network 12.1 detects product material objects 1.1 by applying the bounding box regressor 12.4 to the feature maps 8.2.1.1 in real-time and classifies each detected material object 1.1 in the bulk material flow 1.3 according to the product class 8.3 with the respective class probability 8.2.7 value for each product class 8.3 applying the FC layer 12.1.3 to the material vector(s) 8.2.1.2. The Convolutional Neural Network 12.1, as a deep learning structure, can be specifically designed for processing the pixels of the optical image data 8.1, in particular detection and classification of the material objects 1.1 represented by the pixels in the optical image data 8.1. In the present case, compared to alternative classification structures, the Convolutional Neural Network 12.1 requires less preprocessing as the Convolutional Neural Network 12.1 automatically learns the predefined kernels 8.3.1.1 defining the one or more product classes 8.3 either hierarchical or non-hierarchical representations of material objects 1.1.1 / 1.1.2 in the optical image data 8.1. Furthermore, the Convolutional Neural Network 12.1 processes the spatial information of the optical image data 8.1 and needs less computational resources for the processing of the optical image data 8.1 than other type of neural networks and therefore process even the high-resolution optical image data 8.1 in real-time.In an advantageous embodiment the material optical-based machine according to the invention CNNs adapted for object detection can be trained with very large image datasets of sample images records 8.1.1 of each product class 8.3 using deep learning. In general, training the Neural Network 12 with very large sample image records 8.1.1 improves the classification accuracy. Furthermore, training the Neural Network 12 with very high number of sample image records 8.1.1 improves the robustness and reliability of using the probability value 8.2.3 to infer whether classification is correct. The detection and classification unit 8 comprising the repository 10 with the sample image records 8.1.1 and the neural network 12 comprising (i) a machine learning interface 12.3 for receiving the sample image records 8.1.1 from the repository 10 and (ii) a machine-learning-based modelling engine 12.2 for generating at least one digital model structure 12.2.1 based on the sample image records 8.1.1.and the CNN unit 12.1 for detecting material objects 1.1 in the optical image data 8.1 and classifying each detected material objects 1.1.1 / 1.1.2 to the one or more product class 8.3 / 8.3.1.1.1.5 with a probability value 8.2.3 each of the product classes 8.3, so that the CNN unit 12.1 is trained to detect the optical characteristics of at least one or more sample product classes 8.1.1.1.5 and, in particular, learns the product characteristics of the product classes 8.3 mapped in the kernels 8.3.1.1 of the CNN unit 12.1 and makes them available for the sorting process for detecting and classifying the product material objects 1.1 .1 / 1.1 .2. Optionally to the CNN unit 2.1 a supervised training is applied to generate the at least one digital model structure 12.2.1 based on sample optical image data 8.1.1.1.1 and sample material objects 8.1.1.1.3 represented in the sample optical image data 8.1.1.1.1, whereas each of the sample material object 8.1.1.1.3 is labeled with a corresponding sample product class 8.1.1.1.5 and a sample bounding box 8.1.1.1.6 so that the kernels 8.3.1.1 (also known as filters) of the convolutional layers 12.1.1 automatically learn to detect material characteristics defined by the sample product classes' 8.1.1.1.5 from an optical sample image data 8.1.1.1.1 during the training process. Initially, the kernels 8.3.1.1 of the convolutional layer 12.1.1 are filled with random weights, and through the process of backpropagation and optimization (such as stochastic gradient descent), these weights are adjusted to minimize loss on the sample image records 8.1.1 values. As the CNN unit 12.1 learns, the kernels 8.3.1.1 in the early convolutional sub-layers 12.1.1.1 typically learn to recognize simple material characteristics of the sample product classes 8.1.1.1.5, while kernels 8.3.1.1 in the deeper convolutional sub-layers 12.1.1.1 learn to detect more complex and abstract material characteristics of the sample product classes 8.1.1.1 .5. Here arethree examples of the types of material characteristics that the CNN unit 12.1 kernels 8.3.1.1 may learn to detect: (i) edge detection: In the initial layers of the CNN unit 12.1 the often learn to detect edges and simple optical patterns. This might include vertical, horizontal, or diagonal edges, for example. This is analogous to edge detection operators used in image processing, such as the Sobel filter or Prewitt filter. The feature maps 8.2.1.1 from these filters indicate the locations and strengths of the detected edges of a detected sample material object 8.1.1.1.3 within the optical sample image data 8.1.1.1.1 and (ii) texture detection: As the optical sample image data 8.1.1.1.1 progresses through the CNN sub-layers 12.1 .1.1, some kernels 8.3.1.1 are trained to recognize textures or repetitive patterns. This could be the texture or the pattern of wheat kernel 1.1.1 / 1.1.2. These optical material characteristics enable the CNN unit 12.1 to distinguish material objects 1.1.1 / 1.1.2 between different product classes 8.3 based on the optical material characteristics measurable by the sensing unit 7. Some CNN sub-layers 12.1.1. layers, kernels 8.3.1.1 specialize in detecting parts of material objects 1.1.1 / 1.1.2. For example, in the CNN unit 12.1 trained to recognize abnormal rice, some kernels 8.3.1.1 might become tuned to detect brokens, while others might focus on detecting mold on the rice or ergot. Kernels 8.3.1.1 can extract hierarchical optical material characteristics that are increasingly global and specific to the sorting application of the optical-based sorter, which allows the CNN unit 12.1 to generate the accurate class probability values 8.2.7 for the classifications based on the complex combinations of optical material characteristics of the product classes 8.3, whereas these optical characteristics of the product classes 8.3 / the sample product classes 8.1.1.1.5 are represented by the kernels 8.3.1.1 of the CNN 12.1.

[0065] The CNN Unit 12.1 comprising the machine learning interface 12.3 for receiving the sample image records 8.1.1 from the repository 10 and the machine-learning-based modelling engine 12.2 for generating the at least one digital model structure 12.2.1 based on the sample image records 8.1.1, the sample image records 8.1.1 at least comprise the optical sample image data 8.1.1.1.1 with the sample material object 8.1.1.1.3 reproduced in the sample image data 8.1.1.1.1 and / or at least one sample product class 8.1.1.1.5 of the sample material object 8.1.1.1.4 represented in the optical sample image data 8.1.1.1 .1, and the digital model structure 12.2.1 being trained by applying the sample image records 8.1.1 as input values, wherein the at least one digital model structure 12.2.1 is applied to sample image data 8.1.1.1.1 using the one or more machine learning model structures 12.2.1 by generating the objectclass 8.2.6 with a corresponding probability value 8.2.3 for the sample material object 8.1.1.1.3 in a given optical sample image data 8.1.1.1.1 using the at least one digital model structure 12.2.1 for obtaining at least one sample product class 8.1.1.1.5 and the corresponding class probability 8.2.7 value being transmitted to the analysis algorithm 15.

[0066] The analysis algorithm 15 is applied to the at least one product classes 8.3 configured abnormal. In other words, for each material object 1.1 that is associated with a product class 8.3 configured abnormal, the probability value 8.2.3 is compared to the corresponding sorting threshold 8.3.1.2 for the product class 8.3 assigned to such a material object 1.1.1 / 1.1.2 by the respective object class 8.2.6, and if exceeded then the underlying material object 1.1 in the material 1.3 is ejected by the material ejection system 20.

[0067] Preferably, the memory 10 comprises a product class repository 10.1 comprising a plurality of product classes 8.3 each associated with the corresponding class parameter 8.3.1. The class parameter 8.3.1 comprises a product class identifier and / or the sorting threshold 8.3.1.2 and / or the monitoring threshold 8.3.1.3 and / or the class label 8.3.1.4 for labeling the associated product class 8.3 as normal or abnormal. The product class repository 10.1 is connectable with the analyze algorithm 15 and / or the CNN unit 12.1 and / or a user interface 22 and wherein the product class repository 10.1 provides a search functionality for the connectable units. Optional the memory 10 comprises a configuration repository 10.2 for grouping the one or more product classes 8.3 of the product class repository 10.1 to a predefined sorting configuration, wherein a product class 8.3 is associated with one or more sorting configuration. Furthermore, the configuration repository 10.2 comprises the at least one digital model structure 12.2.1 and / or several product classes 8.3.

[0068] Furthermore the optical detection and classification unit 8 comprises the data processor 9, the repository 10 for saving and searching data, the comparator unit 11 for comparing two or more data and generating a result signal, the counting unit 13 for counting events triggered by a trigger signal and adding up a value to a counter value, the signal generator 14 for generating a trigger signal based on a input signal, the analysis algorithm 15 for processing data and / or signals based on instruction saved in the repository 10, the control indicator 8.4 and a control signaling 8.5 as illustrated inFig.l and Fig. 10. Furthermore the control indicator 8.4 comprises the predefined time interval 8.4.1 and the predefined number 8.4.2 of product material objects 1.1.1 / 1.1.2. The control signaling 8.5 comprises the predefined alert threshold 8.5.1 and the predefined tolerance value 8.5.2. Furthermore, the configuration repository 10.2 comprises the corresponding predefined time interval 8.4.1 and / or the predefined number 8.4.2 of product material objects 1.1 .1 / 1.1 .2 and / or the predefined alert threshold 8.5.1 and / or the predefined tolerance value 8.5.2.

[0069] Optional the optical recognition and classification unit 8.1 comprises a data interface for connecting external data sources, wired or wireless, to the product class repository 10.1 and / or the configuration repository 10.2 with an external resource for sending and / or receiving predefined product classes 8.3 with associated class parameters 8.3.1 and / or predefined configurations between the external data sources and the optical recognition and classification unit 8.1. Furthermore, the data interface is used for importing the sample image records 8.1.1 for the training of the CNN Unit 12.1. In a further embodiment, the data interface is used for exporting the sample image records 8.1.1 for training the digital model structure off-line. Such sample image records 8.1.1 comprise historical sample image records 8.1.1 gathered by optical detection of the optical-based machine and / or sample image records 8.1.1 gathered by an optical detection mean of mobile device and / or sample optical image records 8.1.1 generated by the analysis algorithm 15.

[0070] Furthermore, for configuring one or more product classes 8.3 as abnormal, the material optical-based machine comprises the user interface 22 connectable to the optical recognition and classification unit 8 via a data link 7.5, wherein the configuration of the product classes 8.3 as abnormal / normal can be set for the optical recognition and classification unit 8 via the user interface 22, for example manually, or a predefined configuration of the configuration repository 10.2 can be selected and the optical recognition and classification unit 8 is set accordingly. For example, in a rice sorting process, one product class 8.3 could be configured as "broken rice" and another product class 8.3 could be configured as "ergot" as abnormal, resulting in the optical optical-based machine sorting a granular material flow 1.3 of rice and the detectionand classification unit 8 ejecting broken rice and / or ergot rice to obtain a high-quality rice by applying the sorting process.

[0071] Figure 3 illustrates a flowchart for process steps of the method for sorting material of a material flow 1.3 according to the present invention. In summary, the method comprises an analyzing process gathering the optical image data 8.1 of the product material 1 provided by the imaging system 7 and processing the optical image data 8.1 for identifying abnormal material objects 1.1.2 using the optical detection and classification unit 8, and a sorting process for ejecting abnormal material objects 1.1.2 using the material ejection system 20. For steering the material ejection system 20 the optical detection and classification unit 8 is connected with the material ejection system 20 and sends for each abnormal material object 1.1.2 labeled as eject an ejection instruction to the material ejection system 20.

[0072] In a data transmission step 50 the optical detection and classification unit 8 receives the digital optical image data 8.1 from the imaging system 7 via the data links 7.3 and 7.4. In an analyzing step 52 the optical detection and classification unit 8 determines an abnormal probability value 8.2.3 indicating the likelihood of a material object 1.1 matching a product class 8.3 labeled as abnormal based on the optical image data 8.1 of the material object 1.1 and assigns the abnormal probability value 8.2.3 to the material object. 1.1. For example, in a material flow 1.3 processing oats configured to classify oats and barley, an object looking like or classified as barley is identified by the optical detection and classification unit 8 based on the predefined kernels 8.3.1.1 representing the barley product class 8.3 and an abnormal probability value 8.2.3 is determined by assessing the optical image data 8.1 of the potential barley object 1.1. In case the optical image data 8.1 for assessing optical characteristics for example a size of the potential barley object 1.1 is impacted by blur the likelihood of an incorrect assessment is high. Therefore, a probability value 8.2.3 for the product class 8.3 barley is for example only 52% but it is still more likely a barley object 1.1. than an oat object 1.1. In contrast to that, in case the optical image data 8.1 for assessing the size of the potential barley object 1.1 are of very good quality with a high resolution the likelihood of an incorrect assessment is low. Therefore, an abnormal probability value 8.2.3 for matching the product class 8.3 barley of size is for example 100%, which means it is most likely a barley object 1.1 and not an oat object 1.1. According to a preferred embodiment of the present invention, in the analysis step19

[0073] 52, the optical recognition and classification unit 8 determines the class probability 8.2.7 for the one or more product classes 8.3 based on the optical image data 8.1 of the detected material object 1.1.1 / 1.1.2 and assigns the class probability 8.2.7 value 8.2.3 for each product class 8.3 to the material object 1 .1. Finally the NMS 8.7 generates and assigns the product class 8.3 with the highest class probability 8.2.7 to the detected material object 1.1.1 / 1.1.2. In the present example, this is the product class 8.3 "Barley" configured as abnormal.

[0074] In a comparing step 54 the optical detection and classification unit 8 compares the abnormal probability value 8.2.3 of detected product material object 1.1 with sorting threshold value 8.3.1.2 of a respective product class 8.3 wherein if the measured probability value 8.2.3 of detected product material object 1.1 exceeds the sorting threshold value 8.3.1.2 of a respective product class 8.3, said product material object 1.1.2 is assigned to the respective product class 8.3 and ejected by the ejection system 20 during the sorting process, if the respective product class 8.3 is labeled as abnormal. Furthermore, in the comparing step 54 the optical detection and classification unit 8 compares the probability value 8.2,3 of detected product material object 1.1 with monitoring threshold value 8.3.1.3 of a respective product class 8.3, wherein if the measured probability value 8.2.3 of a detected material object 1.1.2 exceeds the monitoring threshold value 8.3.1.3 for a respective product class 8.3 and if the respective product class 8.3 is a labeled as abnormal, said product material object 1.1.2 is marked for monitoring increasing the control indicator value 8.4 gradually by counting over the predefined time interval 8.4.1 or over the predefined number 8.4.2 of the product material objects 1.1.1 / 1.1.2 the number of product material objects 1.1.1 marked for monitoring. That means a material object 1.2 can be labelled for ejection but not for quality control or it can have an ejection label and a quality control label, depending on its assigned abnormal probability value 8.2.3. In the example, the potential barley object 1.1.2 with a 52% abnormal probability value 8.2.3 is labeled for ejection, but not for quality control purposes because it does not exceed the monitoring threshold 8.3.1.3. The potential barley object 1.1.2 with a 100% probability value 8.2.3 is labeled for ejection and for quality control because it exceeds both thresholds 8.3.1.2 / 8.3.1.3. The sorting threshold 8.3.1.2 and the monitoring threshold 8.3.1.3 for the product classes 8.3 assigned to the barley object 1.1.2 by the object class 8.2.6 and corresponding predefined optical class parameter 8.3.1.1 can be stored in the optical detection and classification unit 8, or they can be received from therepository 10, for example. For example, the sorting threshold 8.3.1.2 for barley objects 1.1.2 in oat material 1.1 is 50% and the monitoring threshold 8.3.1.3 is 99.98%.

[0075] The optical detection and classification unit 8 labels material objects 1.1 for ejection from the material flow 1.3, if the probability value 8.2.2 exceeds the sorting threshold 8.3.1 for the at least one material parameter 1.2 and / or product class 1.3 in step 56.1, and labels material object 1.1 for quality control if the probability value 8.2.3 exceeds the monitoring threshold 8.3.1.3 for the at least one material parameter 1.2 and / or product class 1.3 in step 56.2.

[0076] The Trend indicator indicates a relation between the objects labelled for quality control and the total number of objects over a defined time interval. A high quotient indicates a high contamination of the product material 1.1 / 1.2 with unwanted material.

[0077] In an embodiment of the present invention, the optical detection and classification unit 8 determines the control indicator 8.4 as a sorting quality control in an indication step 58. The control indicator 8.4 indicates a relation between material object 1.1 exceeding the sorting threshold 8.3.1.2 and labeled for ejection and material exceeding the monitoring threshold 8.3.1.3 and labeled for quality control 56.2. For example, a quotient of the number of all potential barley objects labeled for ejection 56.1 and the number of all potential barley objects labeled for quality control 56.2 over a defined time interval serves as the control indicator 8.4. In this case the control indicator 8.4 represents a trend indicator. A high quotient indicates a large number of objects that are very likely incorrectly identified as barley and therefore incorrectly ejected from the material flow 1.3. A low quotient indicates a low number of false assessments of a material object 1.1 being barley, which corresponds to a high quality of the sorting method and a high efficiency of the material optical-based machine.

[0078] The control indicator 8.4 is output as a result of the material sorting method in an output step 60. The control indicator 8.4 provides an assessment of the quality of the contamination of the product material 1.1 / 1.2 with unwanted objects and / or the sorting analysis and may provide guidance for adjusting the thresholds 8.3.1.2 / 8.3.1.3. For example, in case the sorting threshold 8.3.1 .2 is too low many product material objects 1.1 are ejected due to false assessment as being abnormal. These productmaterial objects 1.1 will not be labeled for quality control 56.2, which is reflected in the control indicator 8.4. The output can be used as feedback signal for training the CNN unit 12.1, which improves the thresholds 8.3.1.2 / 8.3.1.3 to optimize the sorting method.

[0079] In summary, the material sorting method uses a first threshold 8.3.1.2 that is for sorting and a second threshold 8.3.1.3 that is for monitoring. This second threshold 8.3.1.3 is higher than the sorting threshold 8.3.1.2, typical values are 50% and 99.98% for the sorting and monitoring thresholds 8.3.1.2 / 8.3.1.3, respectively, as indicated in the above example. The CNN unit 12.1 of the optical detection and classification unit 8 generates the class probability values 8.2.7 for the product class(es) 8.3 by the object class 8.2.6 for every material object 1.1 that it detects, e.g., this material is an "oat", or this material is a "barley". In addition, the NMS 8.7 assigns the "best fitting" product class 8.3 with the abnormal probability value 8.2.3 to each detected material object 1.1.1 / 1.1.2. For example, the CNN unit 12.1 may output a confidence of 52% for one material object 1.1 classified as barley and 100% for another material object 1.1 classified as barley. In this example, both barley objects are rejected by the material optical-based machine, but only the second barley object is counted for the purposes of monitoring by the counting unit 13. The method is generic for any sorting application. It is described here for the example in terms of sorting barley from oats as an example of its usefulness to the food industry, which needs a solution to estimate gluten contaminants, such as barley, to efficiently produce gluten-free oats.

[0080] The problem of simultaneous sorting and monitoring is solved by the introduction of a second qualifying threshold in form of the monitoring threshold 8.3.1.3. The key point is that a low threshold 8.3.1.2 is required for sorting to ensure a high level of efficiency of removing undesired material objects 1.1, but as a consequence there are many false detections of good product mistakenly identified as undesired. Hence, any sorting statistics are useless for analysis because the high number of mistakes of good product is a noise that dominates over the signal of the correctly identified undesired objects. Whereas a high threshold 8.3.1 .3, though useless as a sorting threshold 8.3.1.2 because it does not detect all the undesired objects 1.1, incurs little to no confusion of good product as defects. This minimal confusion is essential for the monitoring process because typically the contamination levels are extremely low, which means there is much more good product than undesired product). The result of the monitoring is not an absolute measurement but is a useful indicator. In case thecontrol indicator 8.4 is observed over time it can serve to predict a change of the contamination of material 1 ahead of time.

[0081] Figure 4 shows a graph illustrating a relation between the abnormality trend indicator, a barley trend indicator BTI as outlined above, and a quantity of barley objects representing abnormal product material objects 1.1.2 in a defined amount of 10kg oats representing the desired material 1.1.1.

[0082] The barley trend indicator BTI is used to infer when the accepted output product from the material optical-based machine exceeds a quality standard. The barley trend indicator BTI is provided a regression over several abnormality analysis indicated as dots. For example, provided above, for gluten-free oats, a common accepted quality requirement is to be less than 20 PPM. A conservative value of the efficiency of the material optical-based machine is applied to predict the percentage of defects 1.1.2 that are removed by the machine. For example, a typical material optical-based machine may be at least 95% efficient at removing a product class 8.3 of abnormal objects 1.1.2, like the barley objects 1.1.2 in the oats material 1. Then the corresponding maximum allowed barley contamination at input to the material optical-based machine is 1000 PPM. For a typical oat application, 1000 PPM corresponds to 225 barley objects 1.1.2 in 10 kg of oats, indicated by the vertical dashed line in the graph. Hence, on the graph, a warning level WL on the Barley Trend Indicator is set to about 70. If the Barley Trend Indicator exceeds this warning level, then an alert 8.6 is signaled to the mill quality control room.

[0083] This way the quality of the material sorting method can be assessed in realtime while a material sorting process is in action. The method for sorting material according to the invention can easily be integrated in existing optical-based machines by implementing the analysis algorithm 15 into the optical detection and classification unit 8 of the machine. It is not necessary to replace or add devices to the machine.List of references

[0084] 0 Optical Product Material Sorter

[0085] 1 Granular product material

[0086] 1.1 Product material object

[0087] 1.1.1 Normal product material object

[0088] 1.1.2 Abnormal product material object

[0089] 1.3 Product material flow of granular product material 2 Hopper

[0090] 3 Feeder

[0091] 4 Chute

[0092] 5 Imaging zone

[0093] 6 Ejection zone

[0094] 7 Optical sensing device

[0095] 7.1 First imaging device

[0096] 7.2 Second imaging device

[0097] 7.3 Data link

[0098] 7.4 Data link

[0099] 7.5 Data link

[0100] 8 Optical detection and classification unit

[0101] 8.1 Optical image data

[0102] 8.1.1 Sample image records

[0103] 8.1.1.1 Optical sample image data 8.1.1.3 Sample material object

[0104] 8.1.1.5 Sample product classes

[0105] 8.1.1.6 Sample bounding box

[0106] 8.2 Object parameter

[0107] 8.2.1 Optical object parameter

[0108] 8.2.1.1 Feature maps

[0109] 8.2.1.2 Material vector

[0110] 8.2.2 Ejection flag

[0111] 8.2.3 Measured probability parameter

[0112] 8.2.4 Monitoring flag

[0113] 8.2.5 Bounding box

[0114] 8.2.6 Object class8.2.7 Class probability(ies)

[0115] 8.3 Product class

[0116] 8.3.1 Class Parameter

[0117] 8.3.1.1 Kernel(s)

[0118] 8.3.1.2 Sorting threshold

[0119] 8.3.1.3 Monitoring threshold

[0120] 8.3.1.4 Class label

[0121] 8.4 Control indicator

[0122] 8.4.1 Predefined time interval

[0123] 8.4.2 Predefined number of product material objects (1.1.1 / 1.1.2) 8.5 Control signaling

[0124] 8.5.1 Predefined alert threshold

[0125] 8.52 Predefined tolerance value

[0126] 8.6 Alert signal

[0127] 8.7 Non-maximum suppression unit / NMS

[0128] Data processor

[0129] 10 Memory / Repository unit

[0130] 10.1 Product class repository

[0131] 10.2 Configuration repository

[0132] 11 Comparator unit

[0133] 12 Artificial intelligence structure / neural network unit / machine learning unit 12.1 Convolutional Neural Network / CNN unit

[0134] 12.1.1 Convolution layer / Feature Extractor

[0135] 12.1.1.1 Convolutional sub-layer

[0136] 12.1.1.2 Rectifier / Rectified Linear Unit (ReLU)

[0137] 12.1.1.3 Pooling layer

[0138] 12.1.2 Flattening Unit

[0139] 12.1.3 Fully Connected Neural Network / FC Layer / Classifier 12.2 Machine-learning-based modelling engine

[0140] 12.2.1 Digital model structure

[0141] 12.3 Machine learning interface

[0142] 12.4 Bounding box regressor

[0143] 13 Counting unit

[0144] 14 Signal generator

[0145] 15 Analysis structure20 Ejection system

[0146] 21 Separation plate

[0147] 22 User Interface

[0148] 50 Data transmission step

[0149] 52 Analysis step

[0150] 54 Comparison step

[0151] 56.1 Labelling for ejection

[0152] 56.2 Labelling for quality control 58 Indicator step

[0153] 60 Output step

[0154] WL Warning Level

Claims

36Claims1. An optical-based sorting method for sorting granular product material objects (1.1.1 / 1.1 .2) in a product material flow (1.3) comprising an optical detection process based on optical image data (8.1) of the product material flow (1.3) measured by one or more optical sensing devices (7), the optical-based sorting method comprising the steps of detecting product material objects (1.1.1 / 1.1.2) in the product flow (1.3), classifying the detected product material objects (1.1.1 / 1.1.2) as abnormal or normal based on the measured optical image data (8.1) by an optical detection and classification unit (8), and ejecting a material object (1.1.2), if a material object (1.1.2) is classified as abnormal, by triggering an ejection system (20) by the optical detection and classification unit (8), wherein each measured pixel of the optical image data (8.1) comprise at least pixel intensity and / or pixel color, characterized by the steps ofmeasuring for each detected material object (1.1 .1 / 1.1.2) a probability value (8.2.3) for being abnormal based on the optical image data (8.1);marking a material object (1.1.1 / 1.1.2) as abnormal, if the measured probability value (8.2.3) of the material object (1.1.1 / 1.1.2) exceeds a sorting threshold value (8.3.2), and triggering the ejection system (20) to eject said material object (1.1.1 / 1.1.2) marked as abnormal;marking a product material object (1.1.1 / 1.1 .2) for monitoring, if the measured probability value (8.2.3) of the material object (1.1.1 / 1.1.2) exceeds a monitoring threshold value (8.3.3) and if the detected material object (1.1.1 / 1.1.2) is marked as abnormal, and increasing a control indicator value (8.4) gradually by counting over a predefined time interval (8.4.1) or over a predefined number (8.4.2) of product material objects (1.1.1 / 1.1.2) the number of product material objects(1.1.1 / 1.1.2) marked for monitoring; andtriggering an alert signal (8.6) if the control indicator value (8,4) given by the measured counts is detected to exceed a predefined alert threshold (8.4.1) or deviates from previously measured counts by a predefined tolerance value (8.5.2).

372. An optical-based method for sorting granular product material objects (1.1.1 / 1.1.2) in a product material flow (1.3) according to claim 1, further characterized by the steps ofextracting an optical feature map (8.2.1.1) representing optical features of one or more product classes (8.3) for each detected material object (1.1.1 / 1.1.2) based on the measured pixel values of the optical image data (8.1);generating class probability values for each material objects (1.1.1 / 1.1.2) for the one or more product classes (8.3) by matching the corresponding feature map (8.2.1.1) of the material object (1.1.1 / 1.1.2) to the corresponding product class (8.3), and assigning the class probability values to the material object (1.1.1 / 1.1.2); andgenerating the abnormal probability value (8.2.3) for each of the material objects (1.1.1 / 1.1 .2) based on the class probability values and / or the product class.

3. An optical-based method for sorting granular product material objects (1.1.1 / 1.1.2) in a product material flow (1.3) according to claim 2, characterized in that the abnormal probability value (8.2.3) of a material objects (1.1.1 / 1.1.2) is generated based on the product class 8.3 with the highest class probability value of the generated class probability values of the material objects (1.1.1 / 1.1.2).

4. An optical-based method for sorting granular product material objects (1.1.1 / 1.1.2) in a product material flow (1.3) according to one of the claims 1 to 3, characterized in that the optical detection and classification unit (8) comprises a convolutional neural network (12.1), the convolutional neural network (12.1)generating the feature map (8.2.1.1) by convoluting pixels of the optical image data (8.1) representing the material objects (1.1.1 / 1.1.2) with several kernels representing optical characteristics of the product classes (8.3) during the extraction process;generating a feature vector (8.2.1.2) by flattening the feature maps (8.2.1 .1 ) by a flattening unit (12.1.2) during the extraction process;generating for each detected material object (1.1.1 / 1.1.2) the class probability for each product class (8.3) by a fully connected neural network (12.1.3) based on the feature vector (8.2.1.2), andgenerating for each detected material object (1.1.1 / 1.1.2) at least one bounding box (8.2.5) locating a region within the optical image data (8.1) comprising the pixels of the material object (1.1.1 / 1.1.2) in the optical image data (8.1) by a bounding box regressor ( 12.1.4) .

5. An optical-based method for sorting granular product material objects (1.1.1 / 1.1.2) in a product material flow (1.3) according to claim 4, further characterized by the steps ofsetting the abnormal probability value (8.2.3) of the corresponding detected material object (1.1.1 / 1.1.2) to the highest class probability value of the product classes (8.3) assigned to the material object (1.1.1 / 1 .1.2); andselecting for each detected material object (1.1.1 / 1.1.2) the corresponding bounding box (8.2.5) with the highest confidence by a non-maximum suppressor unit (8.7).

6. An optical-based method for sorting granular product material objects (1.1.1 / 1.1.2) in a product material flow (1.3) according to one of the claims 1 to 5, characterized in that the optical detection and classification unit (8) is connected to a repository (10) for searching and loading a sorting threshold (8.3.2) and / or a monitoring threshold (8.3.3) and / or a class label (8.3.4) of a product class (8.3) and / or a control indicator value (8.4).

7. An optical-based sorting system for sorting granular product material objects (1.1.1 / 1.1 .2) in a product material flow (1.3) comprising an optical detection process based on optical image data (8.1) of the product material flow (1.3) measured by one or more optical sensing devices (7), the optical-based sorting method comprising the steps of detecting product material objects (1.1.1 / 1.1.2) in the product flow (1.3), classifying the detected product material objects (1.1.1 / 1.1.2) as abnormal or normal based on the measured optical image data (8.1), and ejecting an materialobject (1.1.2), if a material object (1.1.2) is classified as abnormal, by triggering an ejection system (20) by the optical detection and classification unit (8), wherein each measured pixel of the optical image data (8.1) comprise at least pixel intensity and / or pixel color, characterized by the steps ofin that the optical-based sorting system comprises an optical detection and classification unit (8) for measuring for each detected material object (1.1.1 / 1.1.2) a probability value (8.2.3) for being abnormal based on the optical image data (8.1);in that the optical detection and classification unit (8) comprises means for marking a material object (1.1.1 / 1.1.2) as abnormal, if the measured probability value (8.2.3) of the material object (1.1.1 / 1 .1.2) exceeds a sorting threshold value (8.3.2), and triggering the ejection system (20) to eject said material object (1.1.1 / 1.1.2) marked as abnormal;in that the optical detection and classification unit (8) comprises means for marking a product material object (1.1.1 / 1.1 .2) for monitoring, if the measured probability value (8.2.3) of the material object (1.1.1 / 1.1.2) exceeds a monitoring threshold value (8.3.3) and if the detected material object (1.1.1 / 1.1.2) is marked as abnormal, and increasing a control indicator value (8.4) gradually by counting over a predefined time interval (8.4.1) or over a predefined number (8.4.2) of product material objects (1.1.1 / 1.1 .2) the number of product material objects (1.1.1 / 1.1.2) marked for monitoring; andin that the optical-based sorting system comprises a signal generator (14) for triggering an alert signal (8.6) if the control indicator value (8,4) given by the measured counts is detected to exceed a predefined alert threshold (8.4.1) or deviates from previously measured counts by a predefined tolerance value (8.5.2).

Images