System and method to scan for and classify microparticles

Abstract

A system and method are provided. A first scan component conducts a first scan of a zone for microparticles. One or more controllers detect a microparticle in the zone based on the first scan. A second scan component conducts a second scan of the microparticle. The one or more controllers determines one or more properties of the microparticle based on the second scan. The one or more controllers can automatically classify the microparticle based on the one or more properties.

Description

U.S. PATENT Attorney Docket No.: 129834-872886 (8375 CLA)Via EFS-web SYSTEM AND METHOD TO SCAN FOR AND CLASSIFY MICROPARTICLESCROSS-REFERENCE TO RELATED APPLICATION

[0001] This application claims the benefit of U.S. Provisional Patent Application No.63 / 754,893, filed in the U.S. Patent and Trademark Office on February 6, 2025, which is incorporated herein by reference in its entirety for all purposes.FIELD

[0002] The present disclosure relates generally to systems and methods to scan for microparticles and classify microparticles.BACKGROUND

[0003] Microparticles, such as microplastics, are present in nearly all environments. Microparticles, which are less than 5 mm in size, can be directly introduced into the environment or be the result of erosion and degradation of larger material. One feature of microparticles, afforded by their size, is that they can be carried and distributed throughout the environment by a variety of factors including waterways, tides, and wind.

[0004] Microparticles, such as microplastics, are an important source of pollution in the environment. Microplastics have gained significant attention given the potential effects of plastics on organisms and food chains, combined with their extremely long lifetime in the environment.BRIEF DESCRIPTION OF THE DRAWINGS

[0005] FIG. 1A illustrates a diagram of an example of a system to scan and classify microparticles.

[0006] FIG. IB illustrates a diagram of an example of the system.

[0007] FIG. 2 illustrates a diagram of a second scan component.

[0008] FIG. 3 illustrates a diagram of averaged absorbance spectra of different classifications of material.

[0009] FIG. 4 illustrates a schematic diagram of a controller which may be employed, for example, as shown in FIGS. 1A-3.

[0010] FIG. 5 illustrates an example neural network architecture.U.S. PATENT Attorney Docket No.: 129834-872886 (8375 CLA)Via EFS-web

[0011] FIG. 6 illustrates a flow chart of an example method for classifying microparticles.DETAILED DESCRIPTION

[0012] It will be appreciated that for simplicity and clarity of illustration, where appropriate, reference numerals have been repeated among the different figures to indicate corresponding or analogous elements. In addition, numerous specific details are set forth in order to provide a thorough understanding of the examples described herein. However, it will be understood by those of ordinary skill in the art that the examples described herein can be practiced without these specific details. In other instances, methods, procedures and components have not been described in detail so as not to obscure the related relevant feature being described. Also, the description is not to be considered as limiting the scope of the examples described herein. The drawings are not necessarily to scale and the proportions of certain parts may be exaggerated to better illustrate details and features of the present disclosure.

[0013] Detection and / or surveying of microparticles is increasingly important. Microparticles, such as microplastics, can cause harm to terrestrial, aerial, and / or aquatic organisms, as the microparticles can accumulate in tissues and / or interact with metabolic pathways. For example, microplastics have gained significant attention given the potential effects of plastics on organisms and food chains, combined with their extremely long lifetime in the environment. Policies regarding environmental mitigation, use / recycle of plastics are limited by our ability to monitor the presence of microplastics in the environment.

[0014] Real-time surveys of microparticles in the field require robust instruments, rapid acquisition, and minimal processing. In some examples, to detect and classify microparticles such as microplastics, near infrared (NIR) spectroscopy is a robust tool that can detect molecular composition regardless of spectral contamination by environment (e.g., water and / or organic matter). The systems and methods disclosed herein are designed for simple and efficient spectral acquisition of microparticles, such as consumer plastic microparticles of varying shapes and sizes. Data augmentation with measured contaminant spectra has been used to generate machine-learning based classification models that can identify molecular compositions in microparticles that are contaminated with environmental noise.

[0015] Conventional methods for tracking microparticles often involve physical surveying teams collecting samples from the field, sorting the samples by size, cleaning the samples,U.S. PATENT Attorney Docket No.: 129834-872886 (8375 CLA)Via EFS-web potentially sorting them visually, and then using spectroscopy to identify the composition of the collected plastics. These methods are time consuming and require the transport of material, which can very limiting in remote environments. Additionally, conventional methods have high cost and are very labor intensive, which can be a very slow process. As such, workflow optimization and sample pre-screening are needed.

[0016] In the present disclosure, the scans and classifications of the microparticles can be done without the need of removing environmental noise or contamination. In some examples, the scans and classifications can be performed in real time. In some examples, the scans and classifications can be performed in situ.

[0017] In some examples, the system and method can include conducting a first scan of a zone in the environment. The first scan can be performed by a first scan component. The first scan component can include an optical camera that is operable to perform a fast, wide-ranged scan (e.g., larger field of view) for potential microparticles. The first scan can be used, by a controller, to filter out non-particulates and detect the desired microparticle(s). In some examples, the controller can utilize a trained neural network to detect the microparticle, filter the zone for environmental noise, such as non-microparticle particulates. In some examples, the first scan component 104

[0018] Once the particle has been detected from the first scan, a second scan component can conduct a scan of the microparticle. The second scan component can include a near infrared (NIR) spectrometer. With the second scan, one or more properties of the microparticle can be determined, by a controller, so that the microparticle can be classified. In some examples, the controller can utilize a trained neural network to determine the properties of the microparticle, and classify the microparticle.

[0019] Spectroscopies, such as those used for plastic identification, can include NIR spectroscopy because of NIR spectroscopy’s sensitivity to molecular vibrations, which are unique to different molecular compositions. In the 6000 cm-1 to 11000 cm-1 region, NIR spectroscopy provides insights into the overtones and combination bands of vibrational modes related to CH, CO, NH, and OH bonds. For different plastics, these bands can be highly characteristic. In the present disclosure, NIR spectroscopy provides a desired size, portability, robustness, and rapid acquisition times. In some examples, the NIR scan component can include a room temperature Indium-Gallium-Arsenide detector, may not use a laser source, and can be relatively insensitive to mechanical vibrations.U.S. PATENT Attorney Docket No.: 129834-872886 (8375 CLA)Via EFS-web

[0020] To train a neural network architecture, environmental noise such as wet plastics or plant-contaminated plastics can be included in the training data set. The disclosed method of detection, spectral processing, and augmenting classifier algorithms can produce 98+% accuracy detecting 9 common plastics, water, and plant-matter on samples that have been spectrally contaminated with environmental noise, such as water and / or plant spectra. For example, samples as small as 100 micrometers can be successfully identified. In some examples, samples smaller than 100 micrometers can be identified.

[0021] As an example, if the desired microparticles to be identified and classified are microplastics, the optical camera can conduct the first scan, and the controller can detect the microplastic in the environment based on the first scan. The NIR component can then automatically conduct a second scan of the microplastic. Properties of the microplastic, such as molecular composition, particle size, location, source, and / or color, can be determined. The second trained neural network can then automatically classify the microplastic based on the properties. For example, the microplastic may be classified as Polyethylene terephthalate (PET), high-density polyethylene (HDPE), low-density polyethylene (LDPE), polyvinyl chloride (PVC), polypropylene (PP), polystyrene (PS), nylon, polylactic acid (PLA), and / or rubber bands. The location and properties (e.g., size, shape, color, composition, etc.) of the microplastic can then be logged, and such data can be utilized for surveys, clean up actions, regulations, etc.

[0022] The disclosure now turns to FIGS. 1A and IB, which illustrates an example of a system 100 that is operable to automatically scan, detect, and classify microparticles 20 within a zone 12 of an environment 10. As shown in FIG. 1A, the system 100 is capable of operating in a natural environment. For example, the system 100 can be operable to detect the microparticle(s) 20 within an environment 10 where the microparticle(s) 20 remains within non-micoparticle particulates 14, such as sand, rocks, dirt, liquid (e.g., water), and / or biological matter (e.g., wood particles, plants, and / or biological organisms, alive, not alive, contaminants, etc.). In some examples, the system 100 can be operable to scan and detect microparticles 20 in situ. A zone 12 can include one or more microparticles 20.

[0023] In some examples, the system 100 can be operable to scan and detect microparticles 20 within a static environment 10, such that the environment 10 is substantially not moving. For example, the system 100 can be operable to scan a static environment 10 such as the beach where the sand is substantially not moving. In another example, the system 100 can be operable to scanU.S. PATENT Attorney Docket No.: 129834-872886 (8375 CLA)Via EFS-web a static environment 10 where a sample of an area is collected (e.g., by a vacuum), and the contents are dumped out. For example, the sample can be collected from any suitable environment such as workplace, home, public spaces, etc., and the sample can then be scanned and the microparticles 20 classified, as discussed herein. This can be done to provide a survey of microparticles within that environment, for example determine the microparticles 20 in the sample, such as microplastics. The system 100 can then determine the most likely origin of microplastic microparticles 20 from materials known to be in the workplace, home, or public-space environment. The system 100 can then provide suggestions for further steps such as to remove or replace items in that environment based on the scan and classification. In some examples, the system 100 can automatically perform actions based on the scans and classification of the microparticles 20, such as automatically starting a robot vacuum cleaner, ordering cleaning materials, recommend actions to abate microplastics, etc. The system 100 can scan the contents which still include the non-microparticle particulates 14 and environmental background 10.

[0024] In at least one example, the system 100 can be operable to scan and detect microparticles 20 in an environment 10 with a dynamic background. For example, the system 100 can be operable to scan and detect microparticles 20 in moving liquid such as water (e.g., a stream, ocean, etc.). In some examples, the system 100 can be operable to scan and detect microparticles 20 in the air.

[0025] In at least one example, the system 100 can be operable to scan, detect, and classify the microparticles 20 in real time. In some examples, the system 100 can be operable to scan, detect, and / or classify the microparticles 20 automatically, without user input and / or assistance.

[0026] Conventionally, microparticles, such as microplastics, are surveyed by collecting samples from the environment such as water or soil samples, extracting the microparticles from the samples, and investigating their composition using specialized instruments (such as Raman microscope). The Raman microscopy approach is labor intensive, low-throughput, and requires high-precision laboratory -grade equipment. The microparticles are studied against a plain (usually white) background. Accordingly, conventional methods do not have an issue detecting microparticles because conventional detection is done by a person - rather than by the system 100, as disclosed herein.

[0027] In at least one example, the microparticles 20 can be less than about 5 millimeters in size. The size of the microparticle 20 can be determined by any of width, length, height, diameter,U.S. PATENT Attorney Docket No.: 129834-872886 (8375 CLA)Via EFS-web etc. The shape of the microparticle 20 may not be uniform, so the size may not be uniform depending on the measurement taken. However, the size can be generally determined, for example an average of measurements about the microparticle 20 and / or taking one measurement, without deviating from the scope of the disclosure. In some examples, the microparticles 20 can be less than about 1 millimeter in size. In some examples, the microparticles 20 can be less than about 0.75 millimeters in size. In some examples, the microparticles 20 can be less than about 0.5 millimeters in size. In some examples, the microparticles 20 can be less than about 0.4 millimeters in size. In some examples, the microparticles 20 can be less than about 0.3 millimeters in size. In some examples, the microparticles 20 can be less than about 0.2 millimeters in size. In some examples, the microparticles 20 can be less than about 0.1 millimeters.

[0028] In some examples, the microparticles 20 can include man-made materials, such as ceramics, microplastics, etc. Microplastics can include, without being limiting, polyethylene terephthalate (PET), high-density polyethylene (HDPE), low-density polyethylene (LDPE), polyvinyl chloride (PVC), polypropylene (PP), polystyrene (PS), nylon, polylactic acid (PLA), acrylonitrile butadiene styrene (ABS), and / or rubber. In some examples, the microparticles 20 can include metallic particles. In some examples, the microparticles 20 can include asbestos. In some examples, the microparticles 20 can include any man-made material that is not naturally found in the environment 10. For example, the microparticles 20 can be any material that can be considered pollution for the environment 10. In some examples, the microparticles 20 may include more than one compositional material, for example more than one plastic. In some examples, the microparticles 20 may have undergone chemical changes and / or have contaminants from weathering and / or being within an environment 10. In some examples, the microparticles 20 may have dye(s), coatings, etc.

[0029] As shown in FIGS. 1A and IB, the system 100 can include a body 102. The body 102 can include any suitable components such as arms 152 (as shown, for example, in FIG. IB), wheels (e.g., wheels 154 as shown, for example, in FIG. IB), legs, housing(s) 150 (as shown, for example, in FIG. IB), etc. to implement the functions necessary.

[0030] In some examples, the body 102 can be handheld. In some examples, the body 102 can be moved by the user. In some examples, the body 102 can power and / or move, for example, with a motor, a battery, solar panels, etc. In some examples, the body 102 can be operable to move autonomously. In some examples, the body 102 can move in a pre-programmed path. In someU.S. PATENT Attorney Docket No.: 129834-872886 (8375 CLA)Via EFS-web examples, the body 102 can be programmed to search and / or scan in a predetermined area. FIG. IB, for example, shows the system 100 in the form of a rover that can move across the environment 10 on wheels 154. In some examples, the body 102 can be movable and suitable for the environment 10. For example, the body 102 may be water resistant and / or waterproof such that the system 100 can operate in a water-based environment such as a beach, a stream, etc. In some examples, the body 102 may include legs and / or wheels that can transport the system 100 across any terrain in the environment 10, such as over carpet, sand, rocks, etc.

[0031] The body 102 can include a first scan component 104, a second scan component 106, and one or more controllers 400, 500 communicatively coupled with the first scan component 104 and the second scan component 106. The disclosure herein discusses one controller 400, 500. However, in some examples, more than one controller 400, 500 may be utilized and be in communication with one another to perform the functions discussed herein. For example, the first scan component 104 may include a corresponding controller 400, 500 while the second scan component 106 may include a separate controller 400, 500. In at least one example, as illustrated in FIG. IB, the first scan component 104 and the second scan component 106 can be included on one or more arms 152. For example, as illustrated in FIG. IB, the first scan component 104 and the second scan component 106 are included on one arm 152. In some examples, the first scan component 104 can be located on one arm 152 while the second scan component 106 can be included on a separate arm 152. In some examples, the first scan component 104 and the second scan component 106 can be moved independently such that the first scan component 104 can perform a scan on a location of the zone 12 different than the location that the second scan component 106 is performing a scan. In some examples, the zone 12 can be within an area and / or a container, and the first scan component 104 and / or the second scan component 106 can translate about a grid above the zone 12. For example, the first scan component 104 and / or the second scan component 106 can be positioned on an X-Y component that translates the first scan component 104 and / or the second scan component 106 along an X-Y grid.

[0032] In some examples, as shown in FIGS. 1A and IB, any of the one or more controller 400, 500 can be in communication with an external device 30, 400. The external device 30, 400 can include a server, a computer, a tablet, a phone, a watch, etc. In some examples, the controller 400, 500 can send data to the external device 30, 400. In some examples, the controller 400, 500 can send the data such as location, scanned microparticle properties (e g., measured spectra,U.S. PATENT Attorney Docket No.: 129834-872886 (8375 CLA)Via EFS-web molecular composition, particle size, location, source, and / or color) to the external device 30, 400. In some examples, the external device 30, 400 functions as a controller 400 for the system 100 and performs analysis. In some examples, the controller 400, 500 can be operable to receive signals and / or data from the external device 30, 400, for example analysis and / or location, instructions (e.g., for movement, adjustments to any components, etc.). The controller 400, 500 can then adjust the system 100, for example components of the body 102, the first scan component 104, and / or the second scan component 106 accordingly. In some examples, the external device 30, 400 can send messages and / or alerts to the system 100, and the system 100 may alert a user accordingly (e.g., via sound, image on a display, text on a display, etc.).

[0033] In at least one example, the first scan component 104 can be operable to conduct a first scan of the zone 12 for microparticles 20. The first scan can be to locate microparticles 20 within the environment 10. In at least one example, the first scan component 104 can include an optical camera 1040. In some examples, the first scan component 104 can include a light source 1042 that is operable to illuminate the zone 12 of the environment 10. The optical camera can be operable to take the first scan, which can include one or more image(s) and / or video(s) of the environment 10. The first scan component 104 can transmit the first scan to the controller 400, 500. The system 100 can be operable to automatically detect potential microparticle 20 candidates using the optical camera, with the controller 400, 500 running machine vision algorithms. In some examples, the controller 400, 500 can automatically detect potential microparticle 20 candidates with a first trained neural network. The vision algorithms can be designed to detect small microparticles against various natural backgrounds such as sand, rocks, dirt, and / or biological matter (e.g., wood particles, plants, or animals).

[0034] The controller 400, 500 can be operable to detect a microparticle 20 in the zone 12 based on the first scan by the first scan component 104. In at least one example, the controller 400, 500 can be operable to filter the zone 12 for the microparticle 20 based on the first scan. In the example where the first scan component 104 includes the optical camera, the filtering of the first scan including image(s) and / or video(s) can be performed quickly and efficiently. In at least one example, the filtering can include filtering out environmental noise in the zone 12, which can include non-microparticle particles 14, water, sand, etc. In at least one example, the microparticles 20 can be distinguished regardless of color, shape, size, composition, etc. In some examples, the controller 400, 500 can filter for the microparticle 20 using the first trained neural network.U.S. PATENT Attorney Docket No.: 129834-872886 (8375 CLA)Via EFS-web

[0035] Based on the first scan by the first scan component 104, the second scan component 106 can conduct a second scan of the detected microparticle 20. In at least one example, based on the first scan, a location of the detected microparticle 20 can be logged. The second scan can include moving the second scan component 106 to the location to be directed at the microparticle 20. In some examples, the second scan component 106 can be moved automatically, without user input and / or assistance. For example, once the microparticle 20 is detected by the controller 400, 500 based on the first scan, the controller 400, 500 can automatically move the second scan component 106 so that the second scan component 106 is pointed at the microparticle 20 to perform the second scan.

[0036] The first scan component 104 can detect a plurality of microparticles 20 at one time based on the first scan having a wider range. However, the second scan component 106 may only scan a small range (e g., less than 5 microparticles 20 at a time, in some examples less than 3 microparticles 20 at a time, in some examples 1 microparticle 10 at a time). Accordingly, for quick results, the first scan component 104 can scan a wide range while the second scan component 106 focuses in on the microparticles 20 once detected.

[0037] In at least one example, the second scan component 106 can include a spectrometer, for example as illustrated in FIGS. 1A-2. In at least one example, the second scan component 106 can include a near-infrared (NIR) instrument operable to measure an infrared spectrum.

[0038] The spectrometer can be operable to collect light scattered and / or reflected from the microparticle 20. For example, referring to FIG. 2, the second scan component 106 can include an illumination component 1060 operable to emit a light and a spectrometer 1062 (e.g., a NIR instrument) operable to collect the reflected light signal from the microparticle 20. The illumination component 1060 can include a halogen lamp. The illumination component 1060 can include a fiber coupling 200 operable to couple a fiber optic 202 to the illumination component 1060. An emission collimator 204 can be operable to focus the emitted light 250 from the fiber optic 202 to the microparticle 20. In at least one example, the emission collimator 204 can be coupled with a mount 206. The light is scattered and / or reflected from the microparticle 20, and the remitted light 252 can be collected by the spectrometer 1062. For example, the remitted light 252 can be collected by a collection collimator 204. The light is then passed to the spectrometer 1062 via the fiber optic 202 and a fiber coupling 200.U.S. PATENT Attorney Docket No.: 129834-872886 (8375 CLA)Via EFS-web

[0039] In at least one example, the remitted light 252 can be collected at an angle 250A in relation to the emitted light 250. The remitted light 252 may be scattered and / or reflected to be collected at an angle 250A due to the microparticle 20 likely having jagged edges and / or rounded edges. In some examples, the angle 250A can be between about 1 degree and about 89 degrees. In some examples, the angle 250A can be between about 25 degrees and about 65 degrees. In some examples, the angle 250A can be between about 35 degrees and about 55 degrees. In some examples, the angle 250A can be about 45 degrees.

[0040] Based on the second scan, one or more properties of the microparticle 20 can be determined. For example, the properties can include molecular composition, particle size, location, source, and / or color. In at least one example, second scan component 106 can be operable to transmit data from the second scan to a controller 400, 500. The controller 400, 500 can then assess the data and determine the properties of the microparticle 20. In at least one example, the controller 400, 500 can determine the properties of the microparticle 20 using a second trained neural network.

[0041] For example, the composition of the microparticle 20 can produce a characteristic reflectance spectrum of each microparticle 20 in the near-IR. Spectra can be analyzed in real time using a machine learning classifier of the second trained neural network trained on particle data for rapid identification of the microparticle composition. For example, FIG. 3 illustrates an exemplary chart 300 of averaged absorbance spectra of each category of material 302.

[0042] In at least one example, the microparticle 20 can be smaller than a focus of the second scan component 106. Accordingly, the signal is low, and there is more noise. Additionally, there may be background noise, for example from the environment 10, as the microparticles 20 are not pristine as would be in conventional methods. Accordingly, the second trained neural network can be trained to differentiate background noise with the microparticle 20 itself.

[0043] In at least one example, the controller 400, 500 can classify the microparticle 20 based on the determined properties based on the second scan. In some examples, the controller 400, 500 can automatically (e.g., without user input and / or assistance), the microparticle 20 based on the determined properties. For example, the molecular composition can be determined against a possible list of polymers. As another example, the sources of the microparticles 20 (e.g., from a tire, a plastic bottle, couch, etc.) can be categorized or classified, for example by color, fibers, etc. In at least one example, the controller 400, 500 can classify the microparticle 20 using the secondU.S. PATENT Attorney Docket No.: 129834-872886 (8375 CLA)Via EFS-web trained neural network. In some examples, the accuracy of the controller 400, 500 classifying the microparticle can be estimated based on the signal-to-noise of the measured spectrum from the second scan.

[0044] In at least one example, the properties and / or classification of the microparticles 20 can be logged. For example, a survey can be created. In some examples, statistics can be calculated. In some examples, distribution of the microparticles 20 and corresponding classifications can be determined. In some examples, a map of the microparticles 20 and / or classifications can be logged. This information can help further training of the system 100 with reinforcement learning algorithms to enhance detection capabilities, using negative detections from the system 100 to improve the classifier.

[0045] In some examples, based on the classifications and / or properties of the microparticles 20, the system 100 can move to find areas with maximal concentrations, smartly map an area, focus on areas with highest concentrations, and / or downsample in other areas.

[0046] In at least one example, the first scan component 104 and / or the second scan component 106 can be continuously scanning. In some examples, the first scan component 104 can continuously scan, and once a microparticle 20 is detected, the second scan component 106 can be activated. In some examples, the first scan component 104 can only be activated when a predetermined environment 10 is detected. In some examples, the first scan component 104 can only be activated when a user activates the first scan component 104 (e g., presses a button or sends a signal from an external device 30).

[0047] FIG. 4 is a block diagram of an exemplary controller 400. Controller 400 is configured to perform processing of data and communicate with the first scan component 104 and / or second scan component 106, for example as illustrated in FIGS. 1 A, IB, and 2. While FIGS.1 A-2 illustrate one controller 400, in some examples, more than one controller 400 may be utilized. For example, the first scan component 104 may be communicatively coupled with a first controller 400 while the second scan component 106 may be communicatively coupled with a second controller 400. In operation, controller 400 communicates with one or more of the components discussed herein and may also be configured to communication with remote devices / systems.

[0048] As shown, controller 400 includes hardware and software components such as network interfaces 410, at least one processor 420, sensors 460 and a memory 440 interconnected by a system bus 450. Network interface(s) 410 can include mechanical, electrical, and signalingU.S. PATENT Attorney Docket No.: 129834-872886 (8375 CLA)Via EFS-web circuitry for communicating data over communication links, which may include wired or wireless communication links. Network interfaces 410 are configured to transmit and / or receive data using a variety of different communication protocols, as will be understood by those skilled in the art.

[0049] Processor 420 represents a digital signal processor (e.g., a microprocessor, a microcontroller, or a fixed-logic processor, etc.) configured to execute instructions or logic to perform tasks in a wellbore environment. Processor 420 may include a general purpose processor, special-purpose processor (where software instructions are incorporated into the processor), a state machine, application specific integrated circuit (ASIC), a programmable gate array (PGA) including a field PGA, an individual component, a distributed group of processors, and the like. Processor 420 typically operates in conjunction with shared or dedicated hardware, including but not limited to, hardware capable of executing software and hardware. For example, processor 420 may include elements or logic adapted to execute software programs and manipulate data structures 445, which may reside in memory 440.

[0050] Sensors 460, which may include first scan component 104 and / or second scan component 106 as disclosed herein, typically operate in conjunction with processor 420 to perform measurements, and can include special-purpose processors, detectors, transmitters, receivers, and the like. In this fashion, sensors 460 may include hardware / software for generating, transmitting, receiving, detection, logging, and / or sampling, or other parameters.

[0051] Memory 440 comprises a plurality of storage locations that are addressable by processor 420 for storing software programs and data structures 445 associated with the embodiments described herein. An operating system 442, portions of which may be typically resident in memory 440 and executed by processor 420, functionally organizes the device by, inter alia, invoking operations in support of software processes and / or services 444 executing on controller 400. These software processes and / or services 444 may perform processing of data and communication with controller 400, as described herein. Note that while process / service 444 is shown in centralized memory 440, some examples provide for these processes / services to be operated in a distributed computing network.

[0052] It will be apparent to those skilled in the art that other processor and memory types, including various computer-readable media, may be used to store and execute program instructions pertaining to the fluidic channel evaluation techniques described herein. Also, while the description illustrates various processes, it is expressly contemplated that various processes mayU.S. PATENT Attorney Docket No.: 129834-872886 (8375 CLA)Via EFS-web be embodied as modules having portions of the process / service 444 encoded thereon. In this fashion, the program modules may be encoded in one or more tangible computer readable storage media for execution, such as with fixed logic or programmable logic (e.g., software / computer instructions executed by a processor, and any processor may be a programmable processor, programmable digital logic such as field programmable gate arrays or an ASIC that comprises fixed digital logic. In general, any process logic may be embodied in processor 420 or computer readable medium encoded with instructions for execution by processor 420 that, when executed by the processor, are operable to cause the processor to perform the functions described herein.

[0053] Additionally, the controller 400 can apply machine learning, such as a neural network or sequential logistic regression and the like, to determine relationships between the reflected signals from the first scan component 104 and / or the second scan component 106. For example, a deep neural network may be trained in advance to capture the complex relationship between an optical image with microparticles in a non-sterile or non-clean environment. Also, a deep neural network may be trained in advance to capture the complex relationship between near-infrared spectra and properties of microparticles. This neural net can then be deployed in the estimation of the classification of the microparticles. As such, the determination of microparticles and properties of the microparticles can be more accurate. More details regarding an example neural network is discussed in further detail below in FIG. 5.

[0054] FIG. 5 illustrates an example neural network architecture, according to one aspect of the present disclosure. Architecture 500 includes a neural network 510 defined by an example neural network description 501 in rendering engine model (neural controller) 530. The neural network 510 can represent a neural network implementation of a rendering engine for rendering media data. The neural network description 501 can include a full specification of the neural network 510, including the neural network architecture 500. For example, the neural network description 501 can include a description or specification of the architecture 500 of the neural network 510 (e.g., the layers, layer interconnections, number of nodes in each layer, etc.); an input and output description which indicates how the input and output are formed or processed; an indication of the activation functions in the neural network, the operations or filters in the neural network, etc.; neural network parameters such as weights, biases, etc.; and so forth.

[0055] The neural network 510 reflects the architecture 500 defined in the neural network description 501. In this example, the neural network 510 includes an input layer 502, whichU.S. PATENT Attorney Docket No.: 129834-872886 (8375 CLA)Via EFS-web includes input data, such images of microparticles 20 (e.g., microplastics) and / or environment 10 (e.g., non-particulate materials 14). In at least one illustrative example, the input layer 502 can include data representing a portion of the input media data such as a patch of data or pixels (e.g., a 128 x 128 patch of data) in an image corresponding to the input media data (e.g., that of microparticles 20 (e.g., microplastics) and / or environment 10 (e.g., non-particulate materials 14)).

[0056] The neural network 510 includes hidden layers 504A through 504N (collectively “504” hereinafter). The hidden layers 504 can include n number of hidden layers, where n is an integer greater than or equal to one. The number of hidden layers can include as many layers as needed for a desired processing outcome and / or rendering intent. The neural network 510 further includes an output layer 506 that provides an output (e.g., identification and / or classification of microparticle 20) resulting from the processing performed by the hidden layers 504. In one illustrative example, the output layer 506 can provide an identification and / or classification of the microparticles 20.

[0057] The neural network 510 in this example is a multi-layer neural network of interconnected nodes. Each node can represent a piece of information. Information associated with the nodes is shared among the different layers and each layer retains information as information is processed. In some cases, the neural network 510 can include a feed-forward neural network, in which case there are no feedback connections where outputs of the neural network are fed back into itself. In other cases, the neural network 510 can include a recurrent neural network, which can have loops that allow information to be carried across nodes while reading in input.

[0058] Information can be exchanged between nodes through node-to-node interconnections between the various layers. Nodes of the input layer 502 can activate a set of nodes in the first hidden layer 504A. For example, as shown, each of the input nodes of the input layer 502 is connected to each of the nodes of the first hidden layer 504A. The nodes of the hidden layer 504A can transform the information of each input node by applying activation functions to the information. The information derived from the transformation can then be passed to and can activate the nodes of the next hidden layer (e.g., 504B), which can perform their own designated functions. Example functions include convolutional, up-sampling, data transformation, pooling, and / or any other suitable functions. The output of the hidden layer (e.g., 504B) can then activate nodes of the next hidden layer (e.g., 504N), and so on. The output of the last hidden layer can activate one or more nodes of the output layer 506, at which point an output is provided. In someU.S. PATENT Attorney Docket No.: 129834-872886 (8375 CLA)Via EFS-web cases, while nodes (e.g., nodes 508A, 508B, 508C) in the neural network 510 are shown as having multiple output lines, a node has a single output and all lines shown as being output from a node represent the same output value.

[0059] In some cases, each node or interconnection between nodes can have a weight that is a set of parameters derived from training the neural network 510. For example, an interconnection between nodes can represent a piece of information learned about the interconnected nodes. The interconnection can have a numeric weight that can be tuned (e.g., based on a training dataset), allowing the neural network 510 to be adaptive to inputs and able to learn as more data is processed.

[0060] The neural network 510 can be pre-trained to process the features from the data in the input layer 502 using the different hidden layers 504 in order to provide the output through the output layer 506. In an example in which the neural network 510 is used to identify and / or classify microparticles 20, the neural network 510 can be trained using training data that includes example images, properties, and / or features of microparticles 20. For instance, training images and / or data can be input into the neural network 510, which can be processed by the neural network 510 to generate outputs which can be used to tune one or more aspects of the neural network 510, such as weights, biases, etc.

[0061] For example, in training the first trained neural network for filtering the first scan for microparticles, a plurality of classes of microparticles are scanned.

[0062] For example, 12 classes of plastics can be sampled and scanned. In such an example, polyethylene terephthalate (PET), high-density polyethylene (HDPE), low-density polyethylene (LDPE), polyvinyl chloride (PVC), polypropylene (PP), polystyrene (PS), nylon, polylactic acid (PLA), and rubber bands can be collected. Additionally, acrylonitrile butadiene styrene (ABS) sheets (1 / 16”), polyvinyl chloride (PVC) sheets (1 / 32”) and films (0.016”), polystyrene (PS) sheets (1 / 32”), and neoprene rubber sheets (1 / 64”) can be utilized. Furthermore, plant-based materials such as cardboard and white paper as well as plant materials like wood, bark, and dry grass can be collected. Both plant-based materials and plant materials can be categorized as ‘plant-based’ data, while all the collected rubber bands and purchased neoprene rubber can be categorized as ‘rubber’. All the samples can be provided and / or cut into small pieces in the size range of 5 millimeters and below.U.S. PATENT Attorney Docket No.: 129834-872886 (8375 CLA)Via EFS-web

[0063] The first trained neural network can be trained with images and / or videos of the samples with and without environmental background noise. The environmental background noise can include dirt, sand, rocks, liquid (e.g., water, oil, etc.), and / or biological matter (e.g., wood particles, plants, and / or animals).

[0064] The second trained neural network can be trained using NIR spectra of the samples. In some examples, the samples can be provided with and without environmental background noise. For example, the samples can be categorized as ABS, Nylon, PET, PE, PLA, PP, PS, PVC, Rubber, Plant-Based, and Water. Water spectra can be collected, for example, by measuring a wetted stone, similar to a wet sand environment found on a beach. For example, a total of 119 microparticle spectra, 8 plant-based spectra, 1 water spectrum, and 5 blank spectra can be collected. The blank spectra can be augmented with uncorrelated Gaussian noise only.

[0065] Using the basis set of plastic and background (water and plant) spectra, two larger training sets can be generated in parallel via data augmentation. In this way, there can be two training sets with identical basis sets and noise, but that differ in the presence of environmental backgrounds. In all, 48000*2 spectra can be generated for training the algorithms and 48000*2 spectra can be generated for validation. An additional 48000*2 spectra with Gaussian noise ratios of 0 to 2 can be generated for further characterization of the models’ performance at high noise levels.

[0066] A second order polynomial background can be subtracted from each spectrum. The polynomial can be fit to baseline points between 7550 cm’1to 7850 cm’1and 9550 cm’1to 10500 cm’1. Each NIR spectrum can be normalized to a maximum of 1 between the range of 6500 cm’1to 9200 cm’1, which is the region that contains the CH stretch first overtone + CH bend combination band (vCH® -I-and the CH stretch second overtone band (vCH®) , among other identifying features. The wavelength information was discarded as each spectrum can be identical in this regard. The 512 points of the spectra can be used as predictor variables for classification. Each observation (spectrum) was labeled with the material category.

[0067] The collected plastics can be identified using resin identification codes, crossvalidated against known samples with NIR spectroscopy, and compared with literature spectra.

[0068] In some examples, the first trained neural network and / or the second trained neural network can include historical data. For example, as the system 100 performs scans and analysisU.S. PATENT Attorney Docket No.: 129834-872886 (8375 CLA)Via EFS-web of the scans, the historical data from those scans can be utilized in continuing training the first trained neural network and / or the second trained neural network.

[0069] In some cases, the neural network 510 can adjust weights of nodes using a training process called backpropagation. Backpropagation can include a forward pass, a loss function, a backward pass, and a weight update. The forward pass, loss function, backward pass, and parameter update is performed for one training iteration. The process can be repeated for a certain number of iterations for each set of training media data until the weights of the layers are accurately tuned.

[0070] For a first training iteration for the neural network 510, the output can include values that do not give preference to any particular class due to the weights being randomly selected at initialization. For example, if the output is a vector with probabilities that the object includes different product(s) and / or different users, the probability value for each of the different product and / or user may be equal or at least very similar (e.g., for ten possible products or users, each class may have a probability value of 0.1). With the initial weights, the neural network 510 is unable to determine low level features and thus cannot make an accurate determination of what the classification of the object might be. A loss function can be used to analyze errors in the output. Any suitable loss function definition can be used.

[0071] The loss (or error) can be high for the first training dataset (e.g., images) since the actual values will be different than the predicted output. The goal of training is to minimize the amount of loss so that the predicted output comports with a target or ideal output. The neural network 510 can perform a backward pass by determining which inputs (weights) most contributed to the loss of the neural network 510, and can adjust the weights so that the loss decreases and is eventually minimized.

[0072] A derivative of the loss with respect to the weights can be computed to determine the weights that contributed most to the loss of the neural network 510. After the derivative is computed, a weight update can be performed by updating the weights of the filters. For example, the weights can be updated so that they change in the opposite direction of the gradient. A learning rate can be set to any suitable value, with a high learning rate including larger weight updates and a lower value indicating smaller weight updates.

[0073] The neural network 110 can include any suitable neural or deep learning network. One example includes a convolutional neural network (CNN), which includes an input layer andU.S. PATENT Attorney Docket No.: 129834-872886 (8375 CLA)Via EFS-web an output layer, with multiple hidden layers between the input and out layers. The hidden layers of a CNN include a series of convolutional, nonlinear, pooling (for downsampling), and fully connected layers. In other examples, the neural network 510 can represent any other neural or deep learning network, such as an autoencoder, a deep belief nets (DBNs), a recurrent neural networks (RNNs), etc.

[0074] Referring to FIG. 6, a flowchart is presented in accordance with an example embodiment. The method 600 is provided by way of example, as there are a variety of ways to carry out the method. The method 600 described below can be carried out using the configurations illustrated in FIGS. 1A-5, for example, and various elements of these figures are referenced in explaining example method 600. Each block shown in FIG. 6 represents one or more processes, methods, or subroutines, carried out in the example method 600. Furthermore, the illustrated order of blocks is illustrative only and the order of the blocks can change according to the present disclosure. Additional blocks may be added or fewer blocks may be utilized, without departing from this disclosure. The example method 600 can begin at block 602.

[0075] At block 602, a first scan component conducts a first scan of a zone for microparticles. In at least one example, the first scan component can include an optical camera.

[0076] At block 604, one or more controllers detect a microparticle in the zone based on the first scan. The zone for the microparticle can be filtered based on the first scan. The filtering can include filtering out environmental noise in the zone. The environmental noise can include nonmicroparticle particulates. In at least one example, the controller(s) can detect the microparticle using a first trained neural network.

[0077] In at least one example, based on the first scan, a location of the microparticle can be logged.

[0078] At block 606, a second scan component conducts a second scan of the microparticle. In at least one example, the second scan component can include a spectrometer. The second scan component can include a near-infrared instrument. The spectrometer can be operable to collect light scattered and / or reflected form the microparticle. In at least one example, the microparticle can be smaller than a focus of the second scan component. In some examples, two or more microparticles can be scanned within the same focal region.

[0079] In at least one example, the second scan can include moving, automatically, the second scan component to the location to be directed at the microparticle.U.S. PATENT Attorney Docket No.: 129834-872886 (8375 CLA)Via EFS-web

[0080] At block 608, the one or more controllers determines one or more properties of the microparticle based on the second scan. In some examples, the controller(s) can determine the properties of the microparticle using a second trained neural network. In at least one example, the properties can include, without being limiting, at least one of the following: molecular composition, particle size, location, source, and / or color.

[0081] At block 610, the one or more controllers can automatically classify the microparticle based on the one or more properties.

[0082] The examples shown and described above are only examples. Even though numerous characteristics and advantages of the present technology have been set forth in the foregoing description, together with details of the structure and function of the present disclosure, the disclosure is illustrative only, and changes may be made in the detail, especially in matters of shape, size and arrangement of the parts within the principles of the present disclosure to the full extent indicated by the broad general meaning of the terms used in the attached claims. It will therefore be appreciated that the examples described above may be modified within the scope of the appended claims.

Claims

U.S. PATENT Attorney Docket No.: 129834-872886 (8375 CLA)Via EFS-web CLAIMSWhat is claimed is:

1. A method comprising:conducting, with a first scan component, a first scan of a zone for microparticles; detecting, by one or more controllers, a microparticle in the zone based on the first scan; conducting, with a second scan component, a second scan of the microparticle; determining, by the one or more controllers, one or more properties of the microparticle based on the second scan; andautomatically classifying, by the one or more controllers, the microparticle based on the one or more properties.

2. The method of claim 1, wherein the first scan component includes an optical camera.

3. The method of claim 1, wherein the second scan component includes a spectrometer.

4. The method of claim 3, wherein the second scan component includes a near-infrared instrument.

5. The method of claim 3, wherein the spectrometer is operable to collect light scattered and / or reflected from the microparticle.

6. The method of claim 1, wherein the detecting of the microparticle in the zone further includes:filtering, using a first trained neural network, the zone for the microparticle based on the first scan.

7. The method of claim 6, wherein the filtering includes filtering out environmental noise in the zone, wherein the environmental noise includes non-microparticle particulates.U.S. PATENT Attorney Docket No.: 129834-872886 (8375 CLA)Via EFS-web 8. The method of claim 1, wherein, based on the first scan, a location of the microparticle is logged; wherein the second scan includes moving, automatically, the second scan component to the location to be directed at the microparticle.

9. The method of claim 1, wherein the one or more properties includes spectrum, molecular composition, particle size, location, source, and / or color.

10. The method of claim 1 , wherein the microparticle is smaller than a focus of the second scan component.

11. A system comprising:a first scan component operable to conduct a first scan of a zone for microparticles; a second scan component; andone or more controllers communicatively coupled with the first scan component and the second scan component, the one or more controllers operable to:detect a microparticle in the zone based on the first scan;cause the second scan component to conduct a second scan of the microparticle; determine one or more properties of the microparticle based on the second scan; andautomatically classify the microparticle based on the one or more properties.

12. The system of claim 11, wherein the first scan component includes an optical camera.

13. The system of claim 11, wherein the second scan component includes a spectrometer.

14. The system of claim 13, wherein the second scan component includes a near-infrared spectrometer.

15. The system of claim 13, wherein the spectrometer is operable to collect light scattered and / or reflected from the microparticle.U.S. PATENT Attorney Docket No.: 129834-872886 (8375 CLA)Via EFS-web 16. The system of claim 11, wherein the one or more controllers is further operable to:filter, using a first trained neural network, the zone for the microparticle based on the first scan.

17. The system of claim 11, wherein the filtering includes filtering out environmental noise in the zone, wherein the environmental noise includes non-microparticle particulates.

18. The system of claim 11 , wherein, based on the first scan, a location of the microparticle is logged; wherein the second scan includes moving, automatically, the second scan component to the location to be directed at the microparticle.

19. The system of claim 11, wherein the one or more properties includes spectrum, molecular composition, particle size, location, source, and / or color.

20. The system of claim 11, wherein the first scan component, with the first scan, is operable to detect two or more microparticles, wherein the second scan component is operable, with the second scan, to scan the two or more microparticles within a same focal region.

Images