Multimodal optical feedback system for characterization and control of crop growth

Abstract

The present disclosure provides a system and method for automated monitoring, quantification, and control of plant health from micro and macro scales using optical sensing and imaging modalities, environmental sensing, data analysis, image processing, and machine learning. Intelligent feedback control of plant health employed in the system reacts in real-time to changes in plant health to enable precision agriculture applications like intelligent irrigation or determination of optimal fertigation parameters. In turn, the system results in improved agricultural outcomes such as increased fruit production or unique flavor profiles. The system further provides a means for automated and non-invasive testing of novel treatments for disease or adverse conditions, or to improve crop yields.

Description

[0001] MULTIMODAL OPTICAL FEEDBACK SYSTEM FOR CHARACTERIZATION AND CONTROL OF CROP GROWTH

[0002] FIELD

[0003] The present disclosure relates to automated and intelligent methods for measuring the health and development status of plants and carrying out feedback control for precision agriculture such as optimizing crop yields or investigating novel technologies.

[0004] BACKGROUND

[0005] With mounting threats from factors such as climate change, precision agricultural technologies aim to optimally manage the growth and development of crops by leveraging advancements in sensors, cloud computing, nanotechnology, robotics, and machine learning. In tandem, these systems reduce resource waste, time cost, and human labour requirements and improve crop productivity (such as fruit yields). Current systems suffer from narrow usecases (e.g. greenhouse environment management), or the use of one modality (e.g. hyperspectral imaging) providing limited insights.

[0006] SUMMARY

[0007] The present disclosure comprises a system or systems capable of the following optical sensing and imaging modalities which combines facility analysis instrumentation in addition to portable or field capable instrumentation. Facility instrumentation includes in any combination: widefield fluorescence, high-resolution confocal fluorescence, fluorescence lifetime, high-resolution Raman spectroscopy and resonance energy transfer.Field capable instrumentation for use by an operator in the field include low resolution widefield fluorescence, low-resolution confocal fluorescence, RGB, and low-resolution Raman spectroscopy and hyperspectral / multispectral. For in the field analysis additional environmental sensors are integrated in or linked to the main system processing unit to provide data on aspects including but not limited to soil moisture content (using for example a capacitive soil moisture sensor, temperature and humidity, air and soil quality. Further, the system includes image / data acquisition and control software to coordinate imaging and sensor data collection from each modality, automatically trigger hardware like excitation and detection accessories, and store the captured data for analysis.

[0008] System components can also be connected through wireless sensor networks (WSN) or Internet of Things (loT) protocols with optional cloud computing and data storage (e.g. Google Firebase or AWS). The final aspect of the invention is an automated analysis pipeline which uses data processing and analysis techniques alongside machine learning models to output insights and facilitate control of environment variables. Insights can include, but are not limited to, locations of molecules or materials, efficiency of biochemical processes, water stress, nutrient deficiencies, optimal light requirements, and presence of pests or disease. These insights are then used to trigger actions like irrigation, temperature control, lighting, fertilization, or other treatments, through software and hardware contained within or linked to the main system processing unit. Environmental sensors and actuators are embedded directly into the environment (e.g. in the ground or in a greenhouse) or attached to a mobile robotic device (e.g. a rover or drone).Thus, there is disclosed a computer-controlled multimodal system for automated monitoring, quantification, and control of agricultural plant health and growth, comprising:

[0009] a computer controller including a processor and memory storing executable instructions, the processor configured to:

[0010] communicate with a first network of microscale facility-domiciled instruments that characterize microscopic properties of a sample;

[0011] communicate with a second network of macroscale and microscale portable field instruments that operate on macroscopic sections of the sample, the second network including environmental sensors;

[0012] collect data from each instrument in the first and second networks and combine the data into a unified data structure or database;

[0013] execute a data processing to clean, denoise, and prepare the unified data for analysis;

[0014] analyze the prepared data and determine key characteristics of the sample indicative of plant health;

[0015] perform data correlation by comparing health metrics at microscopic and macroscopic scales to obtain an overall quantification of plant health;

[0016] generate insights by comparing the health metrics to expected values for healthy plants and provide recommendations for remedial actions; and automatically initiate changes in the plant’s environment via connected devices based on the health impact assessment and feedback from the environmental sensors.The present disclosure further provides a computer controlled method for automated multi-modal monitoring, quantification, and control of agricultural plant health and growth, comprising:

[0017] storing executable instructions in a memory of a processor forming part of a computer controller;

[0018] the processor configured for:

[0019] communicating with a first network of microscale facility-domiciled instruments that characterize microscopic properties of a sample;

[0020] communicating with a second network of macroscale and microscale portable field instruments that operate on macroscopic sections of the sample, the second network including environmental sensors;

[0021] collecting data from each instrument in the first and second networks and combine the data into a unified data structure or database;

[0022] executing a data processing to clean, denoise, and prepare the unified data for analysis;

[0023] analyzing the prepared data and determine key characteristics of the sample indicative of plant health;

[0024] performing data correlation by comparing health metrics at microscopic and macroscopic scales to obtain an overall quantification of plant health;

[0025] generating insights by comparing the health metrics to expected values for healthy plants and provide recommendations for remedial actions; andautomatically initiating changes in the plant’s environment via connected devices based on the health impact assessment and feedback from the environmental sensors.

[0026] The present disclosure also provides a computer controlled multimodal system for automated monitoring, quantification and control of agricultural plant health and growth, comprising:

[0027] a) a computer controller including a processor and memory storing executable instructions connected to one or both of:

[0028] i) a first network of microscale facility domiciled instruments that characterize microscopic properties of a sample;

[0029] ii) a second network of macroscale and microscale portable field instruments which operate on large macroscopic sections of a sample, said second network including environmental sensors;

[0030] b) a central a data consolidation software module 64 in communication with each of the instruments in the first and second networks and programmed with instructions to collect data that is output from each instrument in the first and second networks and programmed to combine the output from each instrument into a unified data structure or database;

[0031] c) a data processing software module 68 in communication with software module 64 programmed with instructions with techniques to clean, denoise, and prepare the data for analysis which is applied to the unified data structure or database obtained from the data software module 64 to produce prepared data;d) a data analysis software module 70 in communication with data processing module 68 programmed with instructions to receive the prepared data and analyze the prepared data to determine key characteristics of the sample used to assess the health status of the agricultural plant;

[0032] e) a micro-macro scale correlations software module 88 programmed with instructions to compare health metrics at the microscopic and macroscopic scales which are indicative of similar phenomenon within the sample are compared to obtain a final, overall quantification of the phenomenon;

[0033] an insight generation software module 92 programmed with instructions to compare the health metrics obtained in the micro-macro scale correlations module 88 to each other and expected or nominal values, and a resulting health impact insight is conveyed to a user with metrics that are readily discernable, and if appropriate recommend remedial actions; and

[0034] an agricultural event triggers software module 96 connected to devices configured for changing the agricultural plant’s real-world environment and being programmed with instructions to automatically initiate changes in the agricultural plant’s real-world environment are automatically triggered upon receiving the health impact assessments, to optimize the health of the agricultural plant and based on feedback from the environmental sensors.

[0035] The data analysis software module 70 includes any one or combination of:

[0036] a) a computer vision and imaging processing software module 72 programmed with instructions to apply computer vision and image processing techniques to acquired images from optical imaging techniques in the first andsecond networks at any scale to determine physical characteristics including existence, number, location, and size of anatomical features, cellular structures and tracked materials;

[0037] b) a statistical and probabilistic techniques software module 76 programmed with instructions to apply mathematical operations and algorithms to the prepared data to summarize the dataset, measure relationships between variables, and provide predictions;

[0038] c) a machine learning software module 80 programmed with instructions to make predictions and provide insights from the prepared dataset without human intervention to develop models to be trained and optimized using the prepared data itself;

[0039] d) a descriptive / exploratory / diagnostic analysis software module 84 programmed with instructions to determine a layout or distribution of the data and extract key information can either be used on its own or passed into an analysis software module; and

[0040] wherein the data analysis steps carried out the software modules in a), b), c) and d) are carried out independently, concurrently, or in any subset or combination.

[0041] A further understanding of the functional and advantageous aspects of the disclosure can be realized by reference to the following detailed description and drawings.

[0042] BRIEF DESCRIPTION OF THE DRAWINGSThe disclosure will be more fully understood from the following detailed description thereof taken in connection with the accompanying drawings, which form part of this application, and in which:

[0043] Figures 1A and 1B one two (2) sheets shows an example system flowchart;

[0044] Figure 2 shows a schematic of intelligent irrigation system based on multispectral sensing;

[0045] Figure 3 is a flowchart of irrigation decision-making from multispectral sensing system;

[0046] Figure 4 shows a schematic of an intelligent illumination system based on multispectral imaging; and

[0047] Figure 5 shows a schematic of a system for cellular-level health quantification using a combination of optical techniques.

[0048] DETAILED DESCRIPTION

[0049] A detailed description is provided below to facilitate a thorough understanding of the disclosed embodiments and connections thereof. The description is not limited to any particular example included herein.

[0050] Various embodiments and aspects of the disclosure will be described with reference to the details discussed below. The following description and drawings are illustrative of the disclosure and are not to be construed as limiting the disclosure. Numerous specific details are described to provide a thorough understanding of various embodiments of the present disclosure. The Figures are not to scale. Further, in certain instances, well-known or conventionaldetails are not described in order to provide a concise discussion of embodiments of the present disclosure.

[0051] As used herein, the terms, “comprises” and “comprising” are to be construed as being inclusive and open ended, and not exclusive. Specifically, when used in the specification and claims, the terms, “comprises” and “comprising” and variations thereof mean the specified features, steps or components are included. These terms are not to be interpreted to exclude the presence of other features, steps or components.

[0052] As used herein, the term “exemplary” means “serving as an example, instance, or illustration,” and should not be construed as preferred or advantageous over other configurations disclosed herein.

[0053] As used herein, the terms “about” and “approximately”, when used in conjunction with ranges of dimensions of particles, compositions of mixtures or other physical properties or characteristics, are meant to cover slight variations that may exist in the upper and lower limits of the ranges of dimensions so as to not exclude embodiments where on average most of the dimensions are satisfied but where statistically dimensions may exist outside this region. It is not the intention to exclude embodiments such as these from the present disclosure. Unless otherwise specified, the terms “about” and “approximately” mean plus or minus 25 percent or less.

[0054] It is to be understood that unless otherwise specified, any specified range or group is as a shorthand way of referring to each and every member of a range or group individually, as well as each and every possible sub-range or sub-group encompassed therein and similarly with respect to any sub-ranges or sub-groups therein. Unless otherwise specified, the present disclosure relatesto and explicitly incorporates each and every specific member and combination of sub-ranges or sub-groups.

[0055] As used herein, the term "on the order of", when used in conjunction with a quantity or parameter, refers to a range spanning approximately one tenth to ten times the stated quantity or parameter.

[0056] Broadly, disclosed herein, the present disclosure provides a computer-controlled multimodal system for automated monitoring, quantification, and control of agricultural plant health and growth, comprising a computer controller including a processor and memory storing executable instructions. The processor is configured to communicate with a first network of microscale facility-domiciled instruments that characterize microscopic properties of an agricultural sample and communicate with a second network of macroscale and microscale portable field instruments that operate on macroscopic sections of the agricultural sample, The second network includes environmental sensors. The processor is programmed with executable instructions to collect data from each instrument in the first and second networks and combine the data into a unified data structure or database execute data processing to clean, denoise, and prepare the unified data for analysis.

[0057] The processor is programmed with executable instructions to analyze the prepared data and determine key characteristics of the sample indicative of plant health and perform data correlation by comparing health metrics at microscopic and macroscopic scales to obtain an overall quantification of plant health, and generate insights by comparing the health metrics to expected values for healthy plants and provide recommendations for remedial actions. The processor is programmed with executable instructions to automaticallyinitiate changes in the plant’s environment via connected devices based on the health impact assessment and feedback from the environmental sensors.

[0058] The present disclosure proves a computer controlled method for automated multi-modal monitoring, quantification, and control of agricultural plant health and growth, comprising storing executable instructions in a memory of a processor forming part of a computer controller. The processor is programmed for communicating with a first network of microscale facility-domiciled instruments that characterize microscopic properties of a sample, communicating with a second network of macroscale and microscale portable field instruments that operate on macroscopic sections of the sample, the second network including environmental sensors. The processor is programmed for collecting data from each instrument in the first and second networks and combine the data into a unified data structure or database and executing a data processing to clean, denoise, and prepare the unified data for analysis. The processor is programmed for analyzing the prepared data and determine key characteristics of the sample indicative of plant health and performing data correlation by comparing health metrics at microscopic and macroscopic scales to obtain an overall quantification of plant health. The processor is programmed for generating insights by comparing the health metrics to expected values for healthy plants and provide recommendations for remedial actions and automatically initiating changes in the plant’s environment via connected devices based on the health impact assessment and feedback from the environmental sensors.

[0059] Thus, there is disclosed herein is a system and method for automated monitoring, quantification, and control of plant health and growth throughcomprehensive sensing technologies, intelligent data analysis, and real-time environmental feedback. This system aims to enable precision agriculture applications like intelligent irrigation, optimized lighting (including wavelength and light flux), or novel crop treatment research by providing a single platform for monitoring, analysis, and control. The system is comprised of: (i) optical sensing and imaging devices capable of hyper / multispectral, RGB, widefield fluorescence, confocal fluorescence, fluorescence lifetime, Raman, and / or resonance energy transfer techniques; (ii) environmental sensor devices for measuring soil quality, soil moisture, ambient temperature, ambient humidity, atmospheric pressure, precipitation, and other factors; (iii) actuators for control of plant growth factors that can be embedded in the environment or portable (e.g. sprinklers for irrigation or a mobile robot for depositing fertilizer); (iv) a main processing unit (e.g., single-board computer); (v) data acquisition, control, and communication software to enable simultaneous capture of data from all modalities and control the flow of data through the system; and (vi) optionally cloud and wireless sensor technologies for data storage and transfer throughout the system and back-end computation.

[0060] Many optical imaging and sensing modalities such as fluorescence and hyperspectral imaging have shown promise in investigating and physiologically characterizing plants on micro and macro-scales. By correlating information from these modalities, a comprehensive assessment of the status and needs of a plant can be obtained.

[0061] The system and method disclosed here will be described in more detail with reference to the Figures. Figure 1, shown as Figures 1A and 1B on two (2) sheets, shows an example system flowchart. Along the top of the flowchartare two (2) sections: on the left-hand side are microscale facility domiciled techniques, and on the right-hand side are macroscale and microscale portable / field techniques. Microscale techniques operate on the microscopic aspects of a sample, such as cellular or molecular structures. In contrast, macroscale techniques operate on large sections of a sample. These macroscopic techniques may be operated at a scale that can be seen by the human eye, but not required. Facility techniques are those that must be carried out in a dedicated facility (e.g., laboratory) where the specialized equipment for the system is located, whereas portable / field techniques consist of equipment that can easily be moved and brought into the working environment. These techniques may refer to sensing and / or imaging uses as appropriate.

[0062] With respect to the microscale facility techniques, high-resolution widefield fluorescence microscopy 14, high-resolution confocal fluorescence microscopy 18, fluorescence lifetime 22, resonance energy transfer 26, and high-resolution Raman spectroscopy 30 techniques provide a wide range of information on cellular structures, biochemical processes, and the efficiency of agricultural treatments. High-resolution widefield fluorescence microscopy 14 can be used to tag, track, and identify cellular structures and plant anatomy in 2D. By leveraging autofluorescence, cellular structures and general anatomy can be seen. Engineered fluorescent molecules can be introduced as probes for specific identification of any microstructures or foreign nanomaterials of interest within a plant. High-resolution confocal fluorescence microscopy 18 enables internal 3D visualization of cellular structures and anatomy.

[0063] Measurement of the fluorescent lifetime 22 and resonance energy transfer 26 can be used to monitor specific molecules and their interactions, forbiochemical processes like photosynthesis. Finally, high-resolution Raman spectroscopy 30 can be used to investigate any changes in the structure or vibration of critical proteins and other molecules.

[0064] With respect to macroscale portable / field techniques, hyperspectral / multispectral 38 and RGB imaging 34 provide information on the health of whole tissues and plant-wide physiological functions. Hyperspectral and multispectral imaging 38 have demonstrated great potential by providing concrete indicators of plant health using light reflectance at different wavelengths. A relationship between a physiological property or biochemical process and the spectral response of the plant at certain wavelengths is called a vegetation index (VI). There are over 500 Vis developed to monitor various attributes such as water content, photosynthetic efficiency, and senescence. Traditional RGB imaging 34 can be used to automatically identify factors like disease, pests, or drought by evaluating the physical appearance of large regions of tissue. The field technique of low-resolution widefield fluorescence 46 provides a large-scale 2D view of plant tissues or whole crops using autofluorescence or fluorescence probes. The field technique of low-resolution confocal fluorescence 50 adds onto this technique to provide a 3D view.

[0065] Microscale techniques may also be employed in the field, such as low-resolution Raman spectroscopy 42 which provides information on molecular and chemical properties similar to Raman techniques performed in a facility with potentially lower resolution.

[0066] The above noted modalities 14, 18, 22, 26, 30, 46, 50, 38, 42 and 34 can be combined with various environment sensors 60 for attributes including but not limited to soil moisture, air temperature, humidity, air quality, and soilcomposition. Combining all these modalities results in a high-dimensional, informationally rich dataset that can be correlated and analyzed to provide actionable insights or enable automatic control. Machine learning, computer vision, and intelligent image processing techniques can be carried out in a pipeline to automate and expedite the analysis for real-time use.

[0067] In the data consolidation 64 step, data from each modality 14, 18, 22, 26, 30, 46, 50, 38, 42, 34, 60 or any combination of these modalities is combined into a unified data structure or database located in a central processing and computational unit or in a cloud storage solution.

[0068] In the data processing 68 step, techniques to clean, denoise, and prepare the data for analysis are applied to the unified data structure or database obtained from the data consolidation 64 step. Outliers, missing values, irrelevant information, and errors are removed from the data. Data values may be transformed from one representation to another to facilitate later analysis methods. Structure, organization, sorting, and formatting of data may be changed to facilitate desired analysis methods. Subsets of the data may be deliberately partitioned for use in different analysis modules as desired. Image data may be processed to isolate pixel regions of interest, apply masks or filters, or perform other image transformations. The output is a set of data free from erroneous values, organized in a manner that is conducive for the desired analysis.

[0069] The system includes a data analysis module 70 which includes several interconnected analysis modules including a computer vision and imaging processing software module 72, a statistical and probabilistic techniquesmodule 76, a machine learning module 80, and other descriptive / exploratory / diagnostic analysis module 84. In the computer vision and image processing step 72, statistical and probabilistic techniques step 76, machine learning step 80, and other descriptive / exploratory / diagnostic analysis step 84, the prepared data is analyzed to determine key characteristics of the sample that can be used to assess the health status of the plant.

[0070] General data analysis techniques are used to explore the data and uncover key information and patterns. Data visualization is carried out to assess the makeup of the data and relationships within the dataset. Further, summary metrics or descriptive statistics provide an overview of the entire dataset. Computer vision and image processing step 72 techniques are applied to acquired images from optical imaging techniques 14, 18, 22, 26, 46, 50, 38 and 34 at any scale to determine physical characteristics such as existence, number, location, and size of anatomical features, cellular structures, tracked materials, and other visual features. Statistical and probabilistic techniques in step 76 are mathematical operations and algorithms applied to the processed data to summarize the dataset, measure relationships between variables, and provide predictions.

[0071] Machine learning step 80 is used to make predictions and provide insights from the processed dataset without human intervention. The models developed are trained and optimized using the data itself. For example, machine learning can be used to predict water content and water stress in a leaf. Other types of data analysis, such as descriptive / exploratory / diagnostic analysis in step 84 can be done to better assess the layout or distribution of the data and extract key information can either be used on its own or passed into adifferent analysis module. These data analysis steps can be carried out independently, concurrently, and in any subset or combination. The output from these modules is a comprehensive and concise set of metrics extracted from the preprocessed data which illustrates the health status of the plant at the microscopic and macroscopic environment scales.

[0072] In the micro-macro scale correlations step 88, compatible metrics at the microscopic and macroscopic scales which are indicative of similar phenomenon within the sample are compared to obtain a final, overall quantification of the phenomenon. For example, chloroplast counts within a leaf obtained from widefield or confocal fluorescence imaging at the microscopic scale are compared to leaf chlorophyll Vis obtained from multispectral spectroscopy at the macroscopic scale to quantify chlorophyll production within the plant. After this step, a set of metrics that quantify physiological phenomenon and the overall health status of the plant is obtained.

[0073] In the insight generation step 92, the health metrics obtained in the previous step are compared to each other and expected or nominal values, and the resulting health impact is conveyed to a user in plain language with metrics that are understandable for a layperson. Remedial actions may also be recommended. For example, if metrics relating to photosynthetic efficiency indicate lower than expected rates, the user is informed of a low photosynthetic efficiency, the current rate, and changes that may promote greater photosynthetic efficiency. These insights are organized into categories based on the aspect of plant health they highlight.In the agricultural event triggers step 96, changes in the real-world environment are automatically triggered upon receiving an insight, to optimize the health of the plant given the current state. For example, if an insight is received indicating that the plant is suffering from drought stress, the irrigation system is automatically triggered. In this step, categories of insights are coupled with real-world actions.

[0074] Example Embodiments

[0075] The following example embodiments are meant to illustrate the present techniques and are not in any way to be construed as limiting.

[0076] Embodiment 1: Multispectral sensing system for inteiiigent irrigation Figure 2 shows an embodiment in which a system 100 comprises a multispectral sensor 38 (Figure 1A), LED light sources 104 a single board computer 108, and a water pump 112. The system 100 captures light reflectance from a plant leaf at wavelengths ranging from 410-940nm then calculates vegetation indices (Vis) related to water content and drought stress to quantify the water needs of a plant. The relevant Vis are fed into machine learning models to determine whether a plant requires irrigation and automatically triggers the pump when necessary. Temperature / humidity sensors 116 and soil moisture sensors 120 are integrated to aid in environment control and provide additional data for use in the irrigation decision-making pipeline. Figure 3 illustrates the logical flowchart of actions carried out by the system 100.

[0077] Referring to Figures 1A and 1B, 2, and 3, for data consolidation 64, data is collected from all sensors using the single-board computer 108 and AmazonWeb Services (AWS) and stored in a single location in the cloud. The data processing step 68 occurs on the single-board computer 108 to calculate Vis from the sensor data 168, and clean and structure the VI data for analysis.

[0078] Exploratory and descriptive data analysis 84, statistical and probabilistic techniques in step 76, and machine learning step 80 methods are carried out on the single-board computer 108 to generate metrics indicative of plant health and thereby the water requirements. Feature selection 172 techniques like analysis of variance (ANOVA) are used to find a set of the most statistically significant Vis for use in further analysis and understand the relationship between Vis and over / underwatering. Candidate supervised and unsupervised machine learning models are developed 176 using a pre-collected dataset 164 obtained from the sensors. Models use the ANOVA-selected features as inputs and output a prediction regarding the current water needs of the plant. An optimized machine learning model 188 is obtained from the candidate models developed 176.

[0079] Real-time data 180 is consolidated and processed on the single-board computer 108. For the real-time data 180, only pre-selected Vis are calculated in the processing step 184. The data is then passed into the optimized machine learning model 188 to obtain a water requirement prediction. The VI results from the macroscale multispectral sensing are compared to cellular features related to water content determined via microscopy techniques, as a macromicro correlation step 88. In particular, transmission and fluorescence microscopy images provide insight on cell vacuole size and count that enables determination of relative water volume within plant leaves. Additionally, measurements from the environmental sensors for temperature, humidity, andsoil moisture provide macroscopic information on the current context and previous agricultural changes. Together, the macroscopic and microscopic indicators provide a comprehensive assessment of water content and stress within the plant. Insight generation in step 92 from the initial model prediction and the macro-micro correlation analysis then provides users information regarding irrigation requirements and a final irrigation decision step 192. The water pump 112 is triggered in step 196 (Figure 3) to provide the plant with adequate irrigation, as the agricultural event trigger step 96 of Figure 1B. Embodiment 2: Multispectral imaging system for evaluation of plant photosynthesis and intelligent illumination

[0080] Figure 4 shows another embodiment in which the system 200 comprises a main processing unit 216, a periphery single-board computer 220, low-cost infrared 208 and RGB cameras 212, controllable LED light sources 224, and a mechanical filter wheel 204. The system 200 captures light reflectance by taking images of the leaf with each camera under each filter sequentially, wherein pixel intensity is correlated to reflected light intensity. An algorithm for region of interest detection is applied to isolate the plant tissue in the image, then relevant Vis are calculated by subtracting, adding, or dividing pixels as necessary between images. These Vis are then used as indicators for the health of a plant. In particular, the Photochemical Reflectance Index (PRI), which is sensitive to changes in carotenoid pigments in leaves, correlates to photosynthetic efficiency and thus the productivity of the plant. The quality (colour, strength, etc.) of the environmental lighting can be adjusted as desired to promote productivity (e.g. increased blue light to promote chlorophyll production).Referring to Figures 1 and 4, data consolidation step 64, data processing step 68, and data analysis techniques step all occur on the main processing unit 216. In data consolidation in step 64, all images captured for a single experiment are retrieved from the periphery single-board computer 220 and placed into a common location in memory for processing. For data processing in step 68, the consolidated images are cropped to the region of interest and pixel-thresholding is applied. Image processing and computer vision 76 techniques are then carried out on the new images on the main processing unit 216. Image segmentation is performed to isolate the plant tissue in the images. Pixel intensities are subtracted, added, multiplied, or divided between images captured with different filters according to VI equations.

[0081] Presence and quantity of carotenoids, chlorophyll, and other pigments within chloroplasts or other locations in a cell can be determined by confocal fluorescence microscopy, which can then be related to chlorophyll and light-use Vis during the micro-macro scale correlations step 88. Insights regarding light use and photosynthetic processes are generated in step 92 from the calculated Vis and the micro-macro correlations. Light quality (including wavelength, flux, timing) is automatically adjusted using a rules-based algorithm depending on the generated insights, as the agricultural event triggered in step 96 by the system 200.

[0082] Embodiment 3: Optical imaging and spectroscopies for cellular evaluation of plant healthFigure 5 shows another embodiment in which the system 300 carries out widefield, confocal, and lifetime fluorescence imaging, alongside Raman spectroscopy, to investigate plant health from cellular characteristics. The system 300 consists of laser or lamp light sources 304, a fluorescence microscope 316, a RGB camera 328, a laser scanning head 312, a time-resolved spectral detection system 324, a confocal detection system 320, control units 308, and a main processing unit 332. Excitation light from laser or lamp light sources 304 passes through the fluorescence microscope 316 to the biological sample which triggers fluorescence of natural plant structures and injected fluorescent probes, as well as light scattering. The emitted light from the sample is then captured by the RGB camera 328, confocal detection system 320, and time-resolved spectral detection system 324. Acquisition and timing for each optical technique are controlled using peripheral control units 308 and the captured data is passed to a main processing unit 332. The data obtained includes three-dimensional spatial image stacks, time-wavelength images, and spectral profiles.

[0083] Referring to Figures 1 and 5, data obtained from each optical technique is consolidated 64, processed 68, and analyzed 72 on the main processing unit 332. As the data has a variety of forms, many processing techniques can be employed to reduce the data size, remove irrelevant information, and correct for noise or errors. For example, image data may be cropped to regions of interest orthresholded to enable object detection. Spectral data may undergo smoothing, baseline subtraction, peak detection, orfeature extraction. Similarly, many data analysis techniques can be leveraged to extract key information regarding the cellular characteristics of the sample. For example, computervision and image processing 76 techniques like object detection can be used to identify, locate, count, and otherwise quantify cellular structures within plant tissue. Overall, the data analysis pipeline intakes the processed data and outputs information regarding locations and counts of structures or probes within plant tissue, rate and efficiency of biochemical processes, and other cellular information. RGB, multispectral imaging, environment sensors and other macro-scale techniques are correlated 88 with the cellular information obtained to provide context on the factors affecting the health of entire plants or crops, such as presence of pests, on-going drought, or progressing disease. Insights on the cellular and holistic health of the plant are generated from outputs of the data analysis pipeline compared to the macro-scale measurements 92. Many agricultural actions can be triggered 96 from this system, including changes in lighting, irrigation, fertilizer, temperature, and / or humidity.

[0084] Embodiments

[0085] The present system and method provide an intelligent, comprehensive, multimodal system for determination and control of plant health and growth, comprising: a main processing unit, secondary processing units, optical sensing and imaging devices, environmental sensors and actuators, wired and wireless communication technologies, acquisition and control software, cloud technologies, and a data analysis pipeline.

[0086] The determination of plant health and growth status is established through measurement, analysis, and comparison of biological and environmental characteristics at the macro and microscale.The measurement data is acquired using optical imaging and / or sensing methods and environmental sensors, all connected to a main processing unit.

[0087] The optical imaging and / or sensing modalities include, but are not limited to, widefield fluorescence, confocal fluorescence, fluorescence lifetime, Raman spectroscopy, resonance energy transfer, RGB, and hyperspectral, in any combination.

[0088] The sensors include, but are not limited to, sensors for water quality, soil moisture, soil composition, soil quality, soil temperature, air temperature, humidity, air quality, air composition, wind speed, wind direction, atmospheric pressure, precipitation, and location.

[0089] Components of measuring data are connected to the main processing unit and each other via wired connections with established communication protocols, and / or wireless technologies.

[0090] Components measuring data can take the form of handheld, portable, or mobile platforms, be directly embedded in the environment, or exist in dedicated facilities.

[0091] The data at one timestamp from all sources can be consolidated into a data packet (hypercube) and data acquisition, data storage, and communications are controlled using custom software.

[0092] A cloud approach can be adopted for data management and processing, including cloud database, backend cloud computing, a user interface, and internet of things (loT) or wireless sensor networks (WSN) style connectivity throughout the system.A custom, intelligent, and automated data analysis pipeline takes in realtime measured data and provides insights on plant health and growth.

[0093] The data analysis pipeline includes steps such as data processing and validation, feature extraction, statistical and probabilistic techniques, image processing, computer vision, and machine learning models.

[0094] The insights can be used to research and develop novel agricultural technologies.

[0095] The insights can trigger agricultural events, such as, but not limited to, lighting or temperature changes, irrigation, or application of fertilizer.

[0096] The agricultural events are carried out by actuators and mechatronic devices located within the plant’s environment, such as sprinklers or mobile robots.

[0097] The insights and / or triggered events lead to improved agricultural outcomes such as, but not limited to, flavor engineering based on flavonoid development insights, faster plant development by increasing photosynthetic efficiency using chlorophyll insights, increased fruit yield from optimal use of fertilizer and water, or fruit maturation control using temperature and humidity.

[0098] The system components for measurement and analysis of plant health and growth are used in a dedicated facility, out in the field, ora combination of the field and facility.

Claims

CLAIMS1. A computer controlled multimodal system for automated monitoring, quantification and control of agricultural plant health and growth, comprising: a) a computer controller including a processor and memory storing executable instructions connected to one or both of:i) a first network of microscale facility domiciled instruments that characterize microscopic properties of a sample;ii) a second network of macroscale and microscale portable field instruments which operate on large macroscopic sections of a sample, said second network including environmental sensors;b) a central a data consolidation software module 64 in communication with each of the instruments in the first and second networks and programmed with instructions to collect data that is output from each instrument in the first and second networks and programmed to combine the output from each instrument into a unified data structure or database;c) a data processing software module 68 in communication with software module 64 programmed with instructions with techniques to clean, denoise, and prepare the data for analysis which is applied to the unified data structure or database obtained from the data software module 64 to produce prepared data;d) a data analysis software module 70 in communication with data processing module 68 programmed with instructions to receive the prepareddata and analyze the prepared data to determine key characteristics of the sample used to assess the health status of the agricultural plant;e) a micro-macro scale correlations software module 88 programmed with instructions to compare health metrics at the microscopic and macroscopic scales which are indicative of similar phenomenon within the sample are compared to obtain a final, overall quantification of the phenomenon;an insight generation software module 92 programmed with instructions to compare the health metrics obtained in the micro-macro scale correlations module 88 to each other and expected or nominal values, and a resulting health impact insight is conveyed to a user with metrics that are readily discernable, and if appropriate recommend remedial actions; andan agricultural event triggers software module 96 connected to devices configured for changing the agricultural plant’s real-world environment and being programmed with instructions to automatically initiate changes in the agricultural plant’s real-world environment are automatically triggered upon receiving the health impact assessments, to optimize the health of the agricultural plant and based on feedback from the environmental sensors.

2. The system according to claim 1 , wherein the data analysis software module 70 includes any one or combination of:a) a computer vision and imaging processing software module 72 programmed with instructions to apply computer vision and image processing techniques to acquired images from optical imaging techniques in the first and second networks at any scale to determine physical characteristics includingexistence, number, location, and size of anatomical features, cellular structures and tracked materials;b) a statistical and probabilistic techniques software module 76 programmed with instructions to apply mathematical operations and algorithms to the prepared data to summarize the dataset, measure relationships between variables, and provide predictions;c) a machine learning software module 80 programmed with instructions to make predictions and provide insights from the prepared dataset without human intervention to develop models to be trained and optimized using the prepared data itself;d) a descriptive / exploratory / diagnostic analysis software module 84 programmed with instructions to determine a layout or distribution of the data and extract key information can either be used on its own or passed into an analysis software module; andwherein the data analysis steps carried out the software modules in a), b), c) and d) are carried out independently, concurrently, or in any subset or combination.

3. The system according to claims 1 or 2, wherein the environmental sensors are configured for measuring any one or combination of soil quality, soil moisture, ambient and soil temperature, ambient humidity, humidity, air quality, air composition, wind speed, wind atmospheric pressure, precipitation and location.

4. The system according to claims 1 , 2 or 3, wherein the instruments of the first network includeany one or combination of instrument for measuring fluorescence lifetime 22, and resonance energy transfer, high-resolution instrument configured for high resolution widefield fluorescence 14, instrument configured for high resolution confocal fluorescence 18, instrument for high resolution Raman spectroscopy 30, and an instrument for measuring resonance energy transfer 26, andthe second network includes said environmental sensors 60 andany one or combination of instruments configured for hyper / multispectral imaging 38, RGB 34, low-resolution widefield fluorescence 46, low resolution confocal fluorescence 50 and low resolution Raman spectroscopy 42.

5. The system according to any one of claims 1 to 4, wherein the microscopic properties of the sample include any one or combination of cellular and / OF molecular structures.

6. The system according to anyone of claims 1 to 5, wherein the agricultural plant is located in an open outdoor environment, and wherein the devices configured for changing the agricultural plant’s real-world environment include any one or combination of water / irrigation systems, fertilizer dispensing systems and pest control systems.

7. The system according to anyone of claims 1 to 5, wherein the agricultural plant is located in an indoor environment, and wherein the devices configured for changing the agricultural plant’s real-world environment include any one or combination of water / irrigation systems, fertilizer dispensing systems, temperature control systems, lighting control systems for controlling wavelength and / or light flux light intensity and pest control systems.

8. The system according to any one of claims 1 to 7, configured for multispectral sensing, including said multispectral sensor 38 configured to receive light emitted by LED light sources and being connected to a microprocessor 108, the microprocessor 108 being connected toa temperature and humidity sensor 116,a soil moisture sensor 120, anda waterpump 112 that is connected to a source of water and a liquid dispensing head that is aimed at any given plant(s);wherein in operation said system calculates temperature and humidity and the soil moisture and captures light reflectance from a plant leaf of any given plant at wavelengths ranging from about 41 Onm to about 940nm and the microprocessor 108 is programmed with instructions to calculate vegetation indices (Vis) related to water content and drought stress to quantify water or moisture needs of a plant, and in the event water is required by the plant, the microprocessor 108 is programmed with instructions to activate the water pump 112 to activate the pump 112 to dispense water onto the given plant(s).

9. The system according to claim 8, wherein the vegetation indices are input into machine learning models to determine whether a plant requires irrigation and automatically triggers the pump when necessary.

10. The system according to claim 9, wherein data is collected from all sensors using the microprocessor 108 and Amazon Web Services (AWS) and stored in a single location in the cloud, and wherein data processing occurs on the microprocessor 108 to calculate the Vis from the sensor data, and cleans and structures the VI data for analysis.

11. The system according to claim 10, wherein data analysis, statistical and probabilistic techniques and machine learning methods to calculate the vegetation indices are carried out on the microprocessor 108 to generate metrics indicative of plant health and thereby the water requirements.

12. The system according to claim 10, wherein once the vegetation indices undergo feature selection to find a set of the most statistically significant Vis for use in further analysis and understand the relationship between Vis and over / underwatering.

13. The system according to claim 12, wherein the feature selection is carried out using analysis of variance (ANOVA).

14. The system according to any one of claims 1 to 7, configured as an intelligent illumination system based on multispectral imaging, and comprisinga main processing unit 216, a periphery single-board computer 220 connected to the main processing unit 216, and an infrared camera 208 and an RGB camera 212, controllable LED light sources 224 connected to periphery single-board computer 220, and a mechanical filter wheel 204 connected to the main processing unit 216, and wherein in operation the system 200 captures light reflectance by taking images of a given leaf with each camera under each filter sequentially, wherein pixel intensity is correlated to reflected light intensity, the main processor being programmed with an algorithm for region of interest detection which is applied to isolate the plant tissue in the image, and thereafter relevant vegetation indices are calculated by subtracting, adding, or dividing pixels as necessary between images, and wherein these vegetation indices are used as indicators for the health of the given plant, andadjusting an output of the controllable LED light sources 224, including wavelength, flux intensity and timing based on the calculated vegetation indices15. The system according to claim 14, wherein the data consolidation step 64, data processing step 68, and data analysis techniques step all occur on the main processing unit 216, in the data consolidation step 64, all images captured for a single experiment are retrieved from the periphery single-board computer 220 and placed into a common location in memory for processing, and for data processing step 68, the consolidated images are cropped to the region of interest and pixel-thresholding is applied to produce new images, and image processing and computer vision step 72 techniques are then carried out on the new images on the main processing unit 216, and wherein image segmentationis performed to isolate the plant tissue in the images, and wherein pixel intensities are subtracted, added, multiplied, or divided between images captured with different filters according to vegetation indices equations.

16. The system according to claims 14 or 15, wherein the vegetation indices are Photochemical Reflectance Index (PRI), which is sensitive to changes in carotenoid pigments in leaves, and correlates to photosynthetic efficiency and thus the productivity of the plant, and wherein a light quality of environmental lighting output from the controllable LED light sources 224 can be adjusted as desired to promote productivity, wherein increased blue light promotes chlorophyll production.

17. The system according to any one of claims 1 to 7, configured for cellular-level health quantification using a combination of optical instruments, including laser or lamp light sources 304 connected to a laser scanning head 312 which is connected to a fluorescence microscope 316 and a time-resolved spectral detection system 324 and a confocal detection system 320, an RGB camera 328 is connected to the fluorescence microscope 316, and a main processing unit 332 connected to the RGB camera 328 in addition to the the time-resolved spectral detection system 324 and the confocal detection system 320, and peripheral control units 308 are connected to each instrument except the main processing unit 332, and wherein in operation,excitation light from laser or lamp light sources 304 passes through the fluorescence microscope 316 to the biological sample which triggersfluorescence of natural plant structures and injected fluorescent probes, as well as light scattering, and whereinthe emitted light from the sample is then captured by the RGB camera 328, confocal detection system 320, and time-resolved spectral detection system 324, and whereinacquisition and timing for each optical instrument are controlled using the peripheral control units 308 and the captured data is passed to a main processing unit 332, and wherein the data obtained includes three-dimensional spatial image stacks, time-wavelength images, and spectral profiles.

18. The system according to claim 17, wherein data obtained from each optical instrument is consolidated in step 64, processed in step 68, and analyzed in step 72 on the main processing unit 332, as the data has a variety of forms, processing techniques are used to reduce the data size, remove irrelevant information, and correct for noise or errors.

19. The system according to claims 17 or 18, wherein Insights on the cellular and holistic health of the plant are generated from outputs of the data analysis, and based the assessment of the plant’s health, agricultural actions can be triggered in step 96 from this system, including any one or combination of changes in lighting, irrigation, fertilizer, temperature, and / or humidity.

20. A computer-controlled multimodal system for automated monitoring, quantification, and control of agricultural plant health and growth, comprising:a computer controller including a processor and memory storing executable instructions, the processor configured to:communicate with a first network of microscale facility-domiciled instruments that characterize microscopic properties of a sample;communicate with a second network of macroscale and microscale portable field instruments that operate on macroscopic sections of the sample, the second network including environmental sensors;collect data from each instrument in the first and second networks and combine the data into a unified data structure or database;execute a data processing to clean, denoise, and prepare the unified data for analysis;analyze the prepared data and determine key characteristics of the sample indicative of plant health;perform data correlation by comparing health metrics at microscopic and macroscopic scales to obtain an overall quantification of plant health;generate insights by comparing the health metrics to expected values for healthy plants and provide recommendations for remedial actions; and automatically initiate changes in the plant’s environment via connected devices based on the health impact assessment and feedback from the environmental sensors.

21. A computer controlled method for automated multi-modal monitoring, quantification, and control of agricultural plant health and growth, comprising:storing executable instructions in a memory of a processor forming part of a computer controller;the processor configured for:communicating with a first network of microscale facility-domiciled instruments that characterize microscopic properties of a sample;communicating with a second network of macroscale and microscale portable field instruments that operate on macroscopic sections of the sample, the second network including environmental sensors;collecting data from each instrument in the first and second networks and combine the data into a unified data structure or database;executing a data processing to clean, denoise, and prepare the unified data for analysis;analyzing the prepared data and determine key characteristics of the sample indicative of plant health;performing data correlation by comparing health metrics at microscopic and macroscopic scales to obtain an overall quantification of plant health; generating insights by comparing the health metrics to expected values for healthy plants and provide recommendations for remedial actions; andautomatically initiating changes in the plant’s environment via connected devices based on the health impact assessment and feedback from the environmental sensors.

Images