System for detecting dirty or obstructed lenses of an imager

Abstract

An environmental monitoring system for a vehicle with lens obstruction detection. The environmental monitoring system includes at least one imager module having a lens defining a field of view and that is configured to capture a series of images of an environment defined by the field of view. A display module is configured to generate a graphic of the images defining an outer graphic boundary. A processor and a memory are provided. The memory includes instructions that, when executed by the processor, cause the processor to generate integral images from the captured images, the integral images including segmentations each with an input pixel data. The processor is further configured to review the integral images for the presence of an obstruction on the lens within the field of view.

Description

Atty. Docket No. AUTO 05264T GEN010 FP1450AWOSYSTEM FOR DETECTING DIRTY OR OBSTRUCTED LENSES OF AN IMAGER CROSS REFERENCE TO RELATED APPLICATION

[0001] This application claims priority under 35 U.S.C. §119(e) upon U.S. Provisional Patent Application No. 63 / 790,173, entitled "SYSTEM FOR DETECTING DIRTY OR OBSTRUCTED LENSES OF AN IMAGER," filed on April 17, 2025, by Andrew J. Oosterhouse et al., the entire disclosure of which is incorporated herein by reference.FIELD OF THE DISCLOSURE

[0002] The present disclosure generally relates to a monitoring system for a vehicle that detects a dirty or otherwise obstructed lens of an imager.SUMMARY OF THE DISCLOSURE

[0003] According to one aspect of the present disclosure, an environmental monitoring system for a vehicle is provided, the environmental monitoring system includes lens obstruction detection. The environmental monitoring system includes at least one imager module having a lens defining a field of view and that is configured to capture a series of images of an environment defined by the field of view. A display module is configured to generate a graphic of the images defining an outer graphic boundary. A processor and a memory are provided. The memory includes instructions that, when executed by the processor, cause the processor to generate integral images from the captured images, the integral images including segmentations each with an input pixel data. The processor is further configured to review the integral images for the presence of an obstruction on the lens within the field of view.

[0004] According to another aspect of the present disclosure, an environmental monitoring system for a vehicle is provided, the environmental monitoring system includes lens obstruction detection. The environmental monitoring system includes at least one imager module having a lens defining a field of view and that is configured to capture a series of images of an environment defined by the field of view. A display module is configured to generate a graphic of the images defining an outer graphic boundary. A processor and a memory are provided. The memory includes instructions that, when executed by the processor, cause the processor to generate integral images from the captured images, the integral images including segmentations each with an input pixeldata. The processor is further configured to detect edges of objects within the captured images, sharpen the detected edges, and quantify the number of sharpened pixels within each of the segmentations. The processor is further configured to detect one or more segmentations that include a low number of sharpened pixels for a period of time, and assign the one or more segmentations with the presence of an obstruction on the lens within the field of view.

[0005] According to one aspect of the present disclosure, an environmental monitoring system for a vehicle is provided, the environmental monitoring system includes lens obstruction detection. The environmental monitoring system includes at least one imager module having a lens defining a field of view and that is configured to capture a series of images of an environment defined by the field of view. A display module is configured to generate a graphic of the images defining an outer graphic boundary. A processor and a memory are provided. The memory includes instructions that, when executed by the processor, cause the processor to generate integral images from the captured images, the integral images including segmentations each with an input pixel data. The processor is further configured to review the segmentations in each of a plurality of sequentially captured integral images for input pixel data that remains static within a predetermined threshold.

[0006] The present disclosure generally provides an environmental monitoring system for a vehicle with lens obstruction detection. The environmental monitoring system utilizes the computational efficiency of integral images, monitors sharpened pixels of detected edges within the images, or combinations thereof to detect obstructions (e.g., dirt, ice, and / or other obstructions) to a lens associated with a camera of the environmental monitoring system.

[0007] These and other features, advantages, and objects of the present disclosure will be further understood and appreciated by those skilled in the art by reference to the following specification, claims, and appended drawings.BRIEF DESCRIPTION OF THE DRAWINGS

[0008] In the drawings:

[0009] FIG. 1 is a top plan view of a vehicle in an environment utilizing an environmental monitoring system, according to an aspect of the disclosure;

[0010] FIG. 2A is a front view of a full display mirror incorporating a display module of an environmental monitoring system, according to an aspect of the disclosure;

[0011] FIG. 2B is a schematic view of a full display mirror for an environmental monitoring system, according to an aspect of the disclosure;

[0012] FIG. 3A is a schematic view of an image segmented into a grid pattern with input pixel values, according to an aspect of the disclosure;

[0013] FIG. 3B is a schematic view of an integral image segmented into a grid pattern with input pixel value sums, according to an aspect of the disclosure;

[0014] FIG. 4A is a partially schematic front view of an integral image segmented into a grid pattern, according to an aspect of the disclosure;

[0015] FIG. 4B is a partially schematic front view of an integral image segmented into a grid pattern for obstruction detection under a first mode of operation, according to an aspect of the disclosure;

[0016] FIG. 4C is a partially schematic front view of an integral image segmented into a grid pattern for obstruction detection under a second mode of operation, according to an aspect of the disclosure;

[0017] FIG. 4D is a partially schematic front view of an integral image segmented into a grid pattern for obstruction detection under a third mode of operation, according to an aspect of the disclosure; and

[0018] FIG. 5 is a schematic view of a control system of an environmental monitoring system, according to an aspect of the present disclosure.DETAILED DESCRIPTION

[0019] The present illustrated embodiments reside primarily in combinations of method steps and apparatus components related to a monitoring system for a vehicle that detects a dirty or otherwise obstructed lens of an imager. Accordingly, the apparatus components and method steps have been represented, where appropriate, by conventional symbols in the drawings, showing only those specific details that are pertinent to understanding the embodiments of the present disclosure so as not to obscure the disclosure with details that will be readily apparent to those of ordinary skill in the art having the benefit of the description herein. Further, like numerals in the description and drawings represent like elements.

[0020] For purposes of description herein, the terms "upper," "lower," "right," "left," "rear," "front," "vertical," "horizontal," and derivatives thereof, shall relate to the disclosure as oriented in FIGS. 2A and 2B. Unless stated otherwise, the term "front" shall refer to the surface of the device closer to an intended viewer of the device, and the term "rear" shall refer to the surface of the device further from the intended viewer of the device. However, it is to be understood that the disclosure may assume various alternative orientations, except where expressly specified to the contrary. It is also to be understood that the specific devices and processes illustrated in the attached drawings, and described in the following specification, are simply exemplary embodiments of the inventive concepts defined in the appended claims. Hence, specific dimensions and other physical characteristics relating to the embodiments disclosed herein are not to be considered as limiting, unless the claims expressly state otherwise.

[0021] The terms "including," "comprises," "comprising," or any other variation thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements does not include only those elements but may include other elements not expressly listed or inherent to such process, method, article, or apparatus. An element preceded by "comprises a . . . " does not, without more constraints, preclude the existence of additional identical elements in the process, method, article, or apparatus that comprises the element.

[0022] Referring initially to FIGS. 1-5, reference numeral 10 generally designates an environmental monitoring system of a vehicle 12. The environmental monitoring system 10 includes at least one imager module 14 having a lens 15 that defines a field of view 19 and configured to capture a series of images 16A of an environment 18 defined by the field of view 19 relative to (e.g., outside of and / or behind, etc.) an associated vehicle 12. A display module 20 is configured to generate a graphic 22 of the images 16A and defines an outer graphic boundary 24. A control system 100 under some implementations, may include a processor 104 and a memory 106 or, alternatively, one or more integrated controllers. In some implementations, the memory 106 includes instructions that, when executed by the processor 104, cause the processor 104 to perform various steps. In some implementations, an integrated circuit includes architecture to perform the various steps. Generally speaking, these terms may be used interchangeably throughout in general reference to the control system 100. More particularly, the various steps carried out bythe control system 100 (e.g., the processor 104) may include generating integral images 16B from the captured images 16A, the integral images 16B include segmentations S each with an input pixel data. The control system 100 (e.g., the processor 104) may be further configured to review the integral images 16B for the presence of an obstruction 25 on the lens 15 within the field of view 19.

[0023] With reference now to FIGS. 1 and 2A, the control system 100 (e.g., the processor 104) may be configured to detect various edges of objects 26 within the images 16A, 16B and generate alerts 28 in different circumstances that will be described in greater detail below. In the depicted implementation, the imager module 14 is depicted as being located on the rear end 30 of the vehicle 12. However, it should be appreciated that the imager module 14 may be located anywhere on or in the vehicle 12 and oriented to capture images 16A of the environment 18 exterior to the vehicle 12. Further, it should be appreciated that the imager module 14 may include one, two, three, or more imager modules 14, each with an associated lens 15. For example, the imager module 14 may be located around an interior cabin 32 or the exterior of the vehicle 12. For example, the at least one imager module 14 may be located on the rear 30 of the vehicle 12, side mirrors 34 of the vehicle 12, a top surface 36 of the vehicle 12, a full display mirror assembly 38 (e.g., a full display rearview mirror assembly) of the vehicle 12, and / or combinations thereof. In some implementations, the display module 20 may be located in the full display mirror assembly 38. In other implementations, the display module 20 may be located anywhere within the vehicle 12 (e.g., a center module, a heads-up display, a mobile device, and / or the like). Regardless of the locations of the at least one imager module 14 and the display module 20, the control system (e.g., the processor 104) may be configured to perform the functions described herein.

[0024] Generally speaking, when the imager module 14 is located on the exterior of the vehicle 12, the imager module 14 (e.g., the lens 15) is exposed to a variety of environmental conditions that can develop on the lens 15 and obstruct the imager module 14. For example, dirt, mud, ice, leaves, litter, and the like can stick on and / or develop over the lens 15. As a result, portions of the environment 18 intended to be captured by the images 16A can be obscured, reducing the efficiency of object 26 edge detection, recognition, and other functionalities of the control system 100 and also negatively impact a user's (e.g., a driver's) ability to comprehensively obtain environmental information fromthe display module 20. For example, the environmental monitoring system 10 may detect one or more edges of objects 26 in blind spots when the vehicle 12 is being reversed and the object 26 is in the line of vehicular travel, and in some scenarios, an alert 28 may be generated to notify the driver. In some implementations, the field of view 19 is largerthan the outer graphic boundary 24 and, therefore, any automated object 26 edge detection and / or recognition functionalities may include portions of images 16A not visible to the user (e.g., the driver).

[0025] In some embodiments, the alert 28 may be audible, for example, via a sound module in the full display mirror assembly 38 or a vehicular audio system. In some embodiments, the alert 28 may be haptic, for example, via vibration of a steering wheel, a seat, or the full display mirror assembly 38. In some embodiments, the alert 28 may be visual, for example, via generating a graphic on the display module 20. For example, the alert 28 graphic may be generated on a side of the display module 20 associated in a direction of relative positioning of the object 26. In some embodiments, the alert 28 may include two or more of audible, haptic, and visual cues. In some embodiments, the alert 28 may include an initial alert (e.g., a first audible, haptic, and / or visual cue) when the object 26 edges are detected and a second or subsequent alert (e.g., a second audible, haptic, and / or visual cue) if the object 26 is moving into the path of travel of the vehicle 12 and / or the vehicle 12 moves within a predetermined distance of the object 26.

[0026] With reference to FIGS. 2A and 2B, in addition to the automated functionalities of the control system 100, the display module 20 is also configured to be viewed by the user (e.g., the driver) during operation of the vehicle 12. In the depicted implementation, the display module 20 is located in the full display mirror assembly 38 (e.g., the full display rearview mirror assembly) and operates as a traditional rearview mirror, but may selectively or continuously display the graphic 22 (e.g., a reproduction of the captured images 16A) rather than relying solely on reflections from a reflective surface. However, the full display mirror assembly 38 is depicted as including a full display mode wherein the graphic 22 of the images 16A is generated and a reflective mode wherein the full display rearview mirror 38 is reflective and operates in a traditional manner. It should be appreciated that the graphic 22 (e.g., the display module 20) may be substantially equal to the reflective element or smaller than the reflective element (e.g., within only a portion of the viewing surface of the rearview mirror assembly 38). More particularly, theenvironmental monitoring system 10 (e.g., the full display rearview mirror 38) may include an electro-optic device 40 that is switchable between a reflective state and a transmissive state. For example, the electro-optic device 40 may include a liquid crystal medium, an absorbative polarizer, and a reflective polarizer. The liquid crystal medium contains liquid crystals that change a polarization of light based on an applied voltage. However, other types of reflective elements, including transflective or other selective and / or dynamic reflective technologies, may be utilized. In this manner, as will be described in greater detail below, once an obstruction 25 is identified, the control system 100 (e.g., the processor 104) may be configured to switch the mode of the full display rearview mirror 38, to the reflective mode.

[0027] With reference now to FIGS. 3A and 3B, to save processing power and increase the efficiency of the control system 100, the integral image 16B can be generated and reviewed (e.g., via the control system 100) for increased computational efficiency. The image 16A may be a monochrome image with measurable image intensity. The integral image 16B is derived (e.g., via the control system 100) from the image 16A and generally contains the same information of the image 16A that can, therefore, be utilized (e.g., via the control system 100) to recreate the image 16A. Under the principles of integral imaging, the image 16A can be broken down into individual segments S. The segments S can be any shape (e.g., pixel by pixel dimensions or x, y, or dimensions in the integral image x', y') and some varieties of which will be explained in further detail below. The input pixel values, input pixel intensities, input pixel luminance, and / or pixel values of each segment are populated in a summed area table (e.g., over the entire integral image 16B or on a segment-by-segment basis), where each segment S holds the sum of all pixel values to the left and top of the segment S and the segment S itself. Further, utilizing the summed area table, any subset of segments S can be calculated and monitored. In this manner, object 26 edge detection on any portion of the integral image 16B can be accomplished by utilizing, for example, four or fewer (e.g., three or fewer) sum of pixel values of any given segment S, rather than computing each individual segment S. Each sum of all pixel values associated with a segment S can be represented as motion vectors that can be quickly summed (e.g., with four or fewer sum pixel values) without having to quantify each segment S individually. It is important to note that the computational steps (e.g., steps required by the control system 100) to detect the object 26 edges are largely independentof the size of the integral image 16B, but may be affected, although significantly less than relying simply on the image 16A, by the number of segments S and the number of pixels (i.e., image resolution). Each segment S may be associated with a plurality of cells in a grid, and each cell may be associated with a single pixel.

[0028] FIG. 4A is a partially schematic front view of the integral image 16B segmented into a grid pattern with the segments S. The size and number of segmentations may vary based on the total number of available pixels. For example, each segment S may include 50 pixels in the vertical direction and 100 pixels in the horizontal direction. Using for example, left- to-right segments S, each segment S holds the sum of all pixel values (e.g., of segments) of the segments S to the left. Utilizing the integral image 16B the control system 100 may only need to calculate (e.g., two additions and one subtraction) of adjacent segments S. Because the segments S depicted in FIG. 3A may be columns (i.e., greater pixel count from an up-down direction rather than a left-right direction), the segments S may be at least partially overlapped. The integral image 16B can be generated for incoming frames (i.e., images 16A) of data, utilizing the linearized, white-balanced, Bayer pixel values as they exit the color-balance operation. The control system 100 utilizes the integral image 16B with luminance values as input, so a simple bilinear demosaic operation on the linearized pixels can be measured and obtained prior to the computation of the integral image 16B. Once the bilinear-interpolated pixels are obtained, the integral image 16B will line buffer the bilinear-interpolated pixels as the difference equation s(x,y-l) - s(x-l,y-l) (e.g., for input to the line buffer). The control system 100 utilizes this information on the next line (before overwriting with new data) by adding in i(x,y) and s(x-l,y), to create a final s(x,y) integral image pixel or segment S.

[0029] With reference now to FIG. 4B, utilizing the sum of all pixel values in each segment S, motion vectors (e.g., changes to or differences in the sum of all pixel values in select segments S) can be quickly ascertained to detect the object 26 edges and movement of the object 26 based on correlative calculations. Likewise, the usage of motion vectors can be utilized to determine which segments S have pixel data that remains relatively static and thus indicate the possibility of an obstruction 25 on the lens 15.

[0030] Under a first operating principle, the environmental monitoring the control system 100 (e.g., the processor 104) may be configured to review the integral images 16B for the presence of an obstruction 25 on the lens 15 and may further include reviewing thesegmentations S in each of a plurality of sequentially captured integral images 16B for input pixel data that remains static within a predetermined threshold. More particularly, when an obstruction 25 is located on the lens 15, the pixel data associated with one or more obscured segments S remains static as changes to or differences in the sum of all pixel values in select segments S remain relatively static over periods of time (e.g., between sequential images 16A, 16B). In this manner, the control system 100 (e.g., the processor 104) may be configured to compare changes to input pixel data of each segmentation, detect one or more segmentations S that include pixel data that remains relatively static relative to other ones of the segmentations S, and assign the one or more segmentations S with the presence of an obstruction 25 on the lens 15 within the field of view 19. For example, the predetermined threshold may be associated with a profile saved in memory 106 and / or via a comparison with motion vectors proximate segments S, averages of proximate segments S, and / or the like. For example, the predetermined threshold may be associated with motion vectors that change within 30% or less, 20% or less, 15% or less, 10% or less, 5% or less of the pre-saved profiles and / or via a comparison with motion vectors of the proximate segments S.

[0031] The control system 100 (e.g., the processor 104) may be configured to review the integral images 16Bfora predetermined amount of time priorto assigningthe one or more segments S with the presence of an obstruction 25. More particularly, any snapshot in time or review of a particular frame or image 16B will not provide motion vectors as pixel data for any given moment is completely static. In this manner, the predetermined amount of time may be associated with a passage of time (e.g., 20 seconds or less, 15 seconds or less, 10 seconds or less, 5 seconds or less) or a predetermined amount of frames (e.g., subsequent images 16B). The predetermined amount of frames may be dependent on frame rate and may be, for example, every 1000 frames or less, every 500 frames or less, every 400 frames or less, every 300 frames or less, every 200 frames or less, or every 100 frames or less. In FIG. 4B, the segmentations with pixel data that has remained static over the predetermined threshold are highlighted gray.

[0032] With reference to FIG.4C, under a second mode of operation, which may be utilized concurrently or in sequence with the first mode of operation, the environmental monitoring system 10 may be configured to, upon detecting edges of an object 26 within the environment 18, sharpen the edges of the objects 26 for both automated systems ofthe control system 100 and for improving the viewing and recognition of the detected object 26 edges on the display module 20. The number of sharpened pixels can then be monitored to detect behaviors indicating an obstruction 25. More particularly, the control system 100 (e.g., the processor 104) may be configured to detect edges of objects 26 within the captured images 16A, sharpen the detected edges 26, and quantify a number of sharpened pixels within each of the segmentations S. The control system 100 (e.g., the processor 104) may then be configured to detect one or more segmentations S that include a low number of sharpened pixels for a period of time, and assign the one or more segmentations S with the presence of an obstruction 25 on the lens 15 within the field of view 19. More particularly, the control system 100 (e.g., the processor 104) may be configured to detect a direction that an associated vehicle 12 is turning, monitor the quantity of the number of sharpened pixels within each of the segmentations S as the associated vehicle 12 is turning, and estimate a migration profile of the quantity of the number of sharpened pixels between the segmentations S as the associated vehicle 12 turns and changes the orientation of the field of view 19. The control system 100 (e.g., the processor 104) is further configured to assign the one or more segmentations S with the presence of an obstruction 25 on the lens 15 within the field of view 19 when the one or more segments are outside of the migration profile by a predetermined threshold. For example, as the vehicle 12 turns, experiences changes in yaw, and / or otherwise moves the field of view 19, the quantity of sharpened pixels is expected to migrate to proximate segmentations S. In FIG.4C, the segmentations S with sharpened pixels that have remained static over the predetermined threshold are highlighted gray.

[0033] With reference to FIG. 4D, under a third mode of operation, the environmental monitoring system 10 may be configured to operate under both the first and second modes (e.g., simultaneously or in sequence). In this manner, the control system 100 (e.g., the processor 104) may be configured to compare changes to input pixel data of each segmentation S, and detect one or more segmentations S that include pixel data that remains relatively static relative to other ones of the segmentations S. The control system 100 (e.g., the processor 104) may be further configured to assign the one or more segmentations S with the presence of an obstruction 25 on the lens 15 within the field of view 19 only upon detecting both the pixel data that remains relatively static relative to other ones of the segmentations S and detecting that the same segmentations S include alow number of sharpened pixels for a period of time. In FIG. 4D the darkest segmentations S include those segmentations S that meet the first and second mode of operation criteria, the middle darkened segmentations S include only segmentations that meet the first mode of operation criteria, and the lightest darkened segmentations S include only segmentations that meet the second mode of operation criteria.

[0034] With reference to FIG. 5, the control system 100 is associated, for example, with the one or more components of the environmental monitoring system 10, such as the full display mirror assembly 38, such as the electro-optic assembly 40, the imager module 14, and the display module 20. The control system 100 may include an electronic control unit (ECU) 102. The ECU 102 may include the processor 104 and the memory 106. The processor 104 may include any suitable processor 104. Additionally, or alternatively, the ECU 102 may include any suitable number of processors (e.g., located in one or more components of the full display mirror assembly 38 and / or the vehicle 12), in addition to or other than the processor 104. The memory 106 may comprise a single disk or a plurality of disks (e.g., hard drives), and includes a storage management module that manages one or more partitions within the memory 106. In some embodiments, the memory 106 may include flash memory, semiconductor (solid state) memory or the like. The memory 106 may include Random Access Memory (RAM), a Read-Only Memory (ROM), or a combination thereof. The memory 106 may include instructions that, when executed by the processor 104, cause the processor 104 to, at least, perform the functions and method steps as described herein. The imager module 14 and the display module 20 may, therefore, be controlled, receive inputs, and / or transmit inputs in relation to the ECU 102. The ECU 102 may receive and / or the memory 106 may save software 108, an integral image module 110, image data 112 (images 16A) captured by the imager module 14, an object sharpening module 114, and an alert module 116. The software 108 may include application data upgradable via user inputs or a remote device, the integral image module 110 may include instructions for performing the integral image calculations described in reference to FIGS.3A-3D, the image data 112 may include an image 16A or a series of images 16A (e.g., a video) captured by the imager module 14, and the alert module 116 may include instructions for conditions, such as the presence of the obstruction 25 under one of the first through third modes of operation associated with generating the alert 28 as described in the proceeding paragraphs and depicted in FIG. 2A. The control system 100 may be inoperable communication with a vehicular control system 150, for example, the audio system for generating the alert 28, the steering wheel for generating the alert 28, and a speed and movement detection system for detecting relative changes in the field of view 19. In some implementations, the environmental monitoring system 10 may, in addition to or alternatively from generating alerts 28, generate signals for performing other automated functionalities. For example, the alert module 116 may, upon assigning the one or more segmentations S with the presence of an obstruction 25, implement (e.g., automatically or via a recommendation to the user) a lens 15 cleaning or washing process, where water and / or a different cleaning solution or wiper cleans the lens 15. Likewise, the alert module 116 may, upon assigning the one or more segmentations S with the presence of an obstruction 25, implement (e.g., automatically or via a recommendation to the user) switch from the full display mode to the reflective mode.

[0035] In further implementations, the control system 100 may include other configurations, in addition to or alternatively from the above, for performing the methods and functionalities described herein. For example, the control system 100 may include and operate under the principles of microprocessors, microcontrollers, application-specific integrated circuits (ASIC), or other circuitry configured to perform instructions, computations, and control various input / output signals. For example, in some implementations, the control system 100 may include one or more Field Programmable Gate Arrays ("FPGAs"), which are a type of programmable integrated circuits capable of performing the methods and functionalities described herein. In some implementations, the modules (e.g., the integral imager module 110 and / or the alert module 116) may be configured as integrated circuits, for example, in communication with other components of the control system 100. More particularly, in some embodiments, the integral image 16B is generated (e.g., calculated) by an integrated circuit, such as one or more FPGAs.

[0036] The disclosure herein is further summarized in the following paragraphs and is further characterized by combinations of any and all of the various aspects described therein.

[0037] According to one aspect of the present disclosure, an environmental monitoring system for a vehicle is provided, the environmental monitoring system includes lens obstruction detection. The environmental monitoring system includes at least one imager module having a lens defining a field of view and that is configured to capture a series ofimages of an environment defined by the field of view. A display module is configured to generate a graphic of the images defining an outer graphic boundary. A processor and a memory are provided. The memory includes instructions that, when executed by the processor, cause the processor to generate integral images from the captured images, the integral images including segmentations each with an input pixel data. The processor is further configured to review the integral images for the presence of an obstruction on the lens within the field of view.

[0038] According to another aspect, reviewing the integral images for the presence of an obstruction on the lens further includes reviewing the segmentations in each of a plurality of sequentially captured integral images for input pixel data that remains static within a predetermined threshold.

[0039] According to yet another aspect, a processor is configured to compare changes to input pixel data of each segmentation, detect one or more segmentations that include pixel data that remains relatively static relative to other ones of the segmentations, and assign the one or more segmentations with the presence of an obstruction on the lens within the field of view.

[0040] According to still another aspect, a processor is configured to detect edges of objects within the captured images, sharpen the detected edges, and quantify a number of sharpened pixels within each of the segmentations.

[0041] According to another aspect, a processor is configured to detect one or more segmentations that include a low number of sharpened pixels for a period of time, and assign the one or more segmentations with the presence of an obstruction on the lens within the field of view.

[0042] According to yet another aspect, a processor is configured to detect a direction that an associated vehicle is turning, monitor the quantity of the number of sharpened pixels within each of the segmentations as the associated vehicle is turning, and estimate a migration profile of the quantity of the number of sharpened pixels between the segmentations as the associated vehicle turns and changes the orientation of the field of view. The processor is further configured to assign the one or more segmentations with the presence of an obstruction on the lens within the field of view when the one or more segments are outside of the migration profile by a predetermined threshold.

[0043] According to still another aspect, each segmentation includes a grid of cells and each cell associated with a plurality of pixels that are quantified for the input pixel data in the integral image.

[0044] According to still yet another aspect, a processor is configured to generate a notification to a user after assigning the one or more segmentations with the presence of an obstruction on the lens within the field of view.

[0045] According to another aspect, a processor is configured to generate a signal to automatically wash the lens after assigning the one or more segmentations with the presence of an obstruction on the lens within the field of view.

[0046] According to still another aspect, the display module is located in a full display rearview mirror switchable between a full display mode wherein the graphic of the images is generated and a reflective mode wherein the full display rearview mirror is reflective.

[0047] According to still yet another aspect, a processor is configured to generate a signal to automatically switch from the full display mode to the reflective mode after assigning the one or more segmentations with the presence of an obstruction on the lens within the field of view.

[0048] According to another aspect of the present disclosure, an environmental monitoring system for a vehicle is provided, the environmental monitoring system includes lens obstruction detection. The environmental monitoring system includes at least one imager module having a lens defining a field of view and that is configured to capture a series of images of an environment defined by the field of view. A display module is configured to generate a graphic of the images defining an outer graphic boundary. A processor and a memory are provided. The memory includes instructions that, when executed by the processor, cause the processor to generate integral images from the captured images, the integral images including segmentations each with an input pixel data. The processor is further configured to detect edges of objects within the captured images, sharpen the detected edges, and quantify a number of sharpened pixels within each of the segmentations. The processor is further configured to detect one or more segmentations that include a low number of sharpened pixels for a period of time, and assign the one or more segmentations with the presence of an obstruction on the lens within the field of view.

[0049] According to another aspect, a processor is configured to detect a direction that an associated vehicle is turning, monitor the quantity of the number of sharpened pixels within each of the segmentations as the associated vehicle is turning, and estimate a migration profile of the quantity of the number of sharpened pixels between the segmentations as the associated vehicle turns and changes the orientation of the field of view. The processor is further configured to assign the one or more segmentations with the presence of an obstruction on the lens within the field of view when the one or more segments are outside of the migration profile by a predetermined threshold.

[0050] According to still another aspect, reviewing the integral images for the presence of an obstruction on the lens further includes reviewing the segmentations in each of a plurality of sequentially captured integral images for input pixel data that remains static within a predetermined threshold.

[0051] According to still yet another aspect, a processor is configured to compare changes to input pixel data of each segmentation, and detect one or more segmentations that include pixel data that remains relatively static relative to other ones of the segmentations. The processor is further configured to assign the one or more segmentations with the presence of an obstruction on the lens within the field of view only upon detecting pixel data that remains relatively static relative to other ones of the segmentations and detecting that the same segmentations include a low number of sharpened pixels for a period of time.

[0052] According to another aspect, the period of time is associated with a number of frames captured by the at least one imager module.

[0053] According to one aspect of the present disclosure, an environmental monitoring system for a vehicle is provided, the environmental monitoring system includes lens obstruction detection. The environmental monitoring system includes at least one imager module having a lens defining a field of view and that is configured to capture a series of images of an environment defined by the field of view. A display module is configured to generate a graphic of the images defining an outer graphic boundary. A processor and a memory are provided. The memory includes instructions that, when executed by the processor, cause the processor to generate integral images from the captured images, the integral images including segmentations each with an input pixel data. The processor is further configured to review the segmentations in each of a plurality of sequentiallycaptured integral images for input pixel data that remains static within a predetermined threshold.

[0054] According to another aspect, processor is further configured to assign the one or more segmentations with the presence of an obstruction on the lens within the field of view, and generate an alert.

[0055] According to still another aspect, the display module is located in a full display rearview mirror within the vehicle.

[0056] According to yet another aspect, the full display rearview mirror includes a reflective element switchable between a reflective state where the display module is obscured and a transmissive state where the display module is visible.

[0057] It will be understood by one having ordinary skill in the art that construction of the described disclosure and other components is not limited to any specific material. Other exemplary embodiments of the disclosure disclosed herein may be formed from a wide variety of materials, unless described otherwise herein.

[0058] For purposes of this disclosure, the term "coupled" (in all of its forms, couple, coupling, coupled, etc.) generally means the joining of two components (electrical or mechanical) directly or indirectly to one another. Such joining may be stationary in nature or movable in nature. Such joining may be achieved with the two components (electrical or mechanical) and any additional intermediate members being integrally formed as a single unitary body with one another or with the two components. Such joining may be permanent in nature or may be removable or releasable in nature unless otherwise stated.

[0059] As used herein, the term "about" means that amounts, sizes, formulations, parameters, and other quantities and characteristics are not and need not be exact, but may be approximate and / or larger or smaller, as desired, reflecting tolerances, conversion factors, rounding off, measurement error and the like, and other factors known to those of skill in the art. When the term "about" is used in describing a value or an end-point of a range, the disclosure should be understood to include the specific value or end-point referred to. Whether or not a numerical value or end-point of a range in the specification recites "about," the numerical value or end-point of a range is intended to include two embodiments: one modified by "about," and one not modified by "about." It will be further understood that the end-points of each of the ranges are significant both in relation to the other end-point, and independently of the other end-point.

[0060] The terms "substantial," "substantially," and variations thereof as used herein are intended to note that a described feature is equal or approximately equal to a value or description. For example, a "substantially planar" surface is intended to denote a surface that is planar or approximately planar. Moreover, "substantially" is intended to denote that two values are equal or approximately equal. In some embodiments, "substantially" may denote values within about 10% of each other, such as within about 5% of each other, or within about 2% of each other.

[0061] It is also important to note that the construction and arrangement of the elements of the disclosure, as shown in the exemplary embodiments, is illustrative only. Although only a few embodiments of the present innovations have been described in detail in this disclosure, those skilled in the art who review this disclosure will readily appreciate that many modifications are possible (e.g., variations in sizes, dimensions, structures, shapes and proportions of the various elements, values of parameters, mounting arrangements, use of materials, colors, orientations, etc.) without materially departing from the novel teachings and advantages of the subject matter recited. For example, elements shown as integrally formed may be constructed of multiple parts, or elements shown as multiple parts may be integrally formed, the operation of the interfaces may be reversed or otherwise varied, the length or width of the structures and / or members or connectors or other elements of the system may be varied, and the nature or number of adjustment positions provided between the elements may be varied. It should be noted that the elements and / or assemblies of the system may be constructed from any of a wide variety of materials that provide sufficient strength or durability, in any of a wide variety of colors, textures, and combinations. Accordingly, all such modifications are intended to be included within the scope of the present innovations. Other substitutions, modifications, changes, and omissions may be made in the design, operating conditions, and arrangement of the desired and other exemplary embodiments without departing from the spirit of the present innovations.

[0062] It will be understood that any described processes or steps within described processes may be combined with other disclosed processes or steps to form structures within the scope of the present disclosure. The exemplary structures and processes disclosed herein are for illustrative purposes and are not to be construed as limiting.

[0063] It is also to be understood that variations and modifications can be made on the aforementioned structures and methods without departing from the concepts of the present disclosure, and further it is to be understood that such concepts are intended to be covered by the following claims unless these claims by their language expressly state otherwise.

Claims

What is claimed is:

1. An environmental monitoring system for a vehicle with lens obstruction detection, the environmental monitoring system comprising:at least one imager module having a lens defining a field of view and configured to capture a series of images of an environment defined by the field of view;a display module configured to generate a graphic of the images defining an outer graphic boundary;a processor; anda memory including instructions that, when executed by the processor, cause the processor to:generate integral images from the captured images, the integral images including segmentations each with an input pixel data; andreview the integral images for the presence of an obstruction on the lens within the field of view.

2. The environmental monitoring system of claim 1, wherein reviewing the integral images for the presence of an obstruction on the lens further includes reviewing the segmentations in each of a plurality of sequentially captured integral images for input pixel data that remains static within a predetermined threshold.

3. The environmental monitoring system of claim 2, wherein the processor is further configured to:compare changes to input pixel data of each segmentation;detect one or more segmentations that include pixel data that remains relatively static relative to other ones of the segmentations; andassign the one or more segmentations with the presence of an obstruction on the lens within the field of view.

4. The environmental monitoring system as in one of claims 1-3, wherein the processor is further configured to:detect edges of objects within the captured images;sharpen the detected edges; andquantify a number of sharpened pixels within each of the segmentations.

5. The environmental monitoring system of claim 4, wherein the processor is further configured to:detect one or more segmentations that include a low number of sharpened pixels for a period of time; andassign the one or more segmentations with the presence of an obstruction on the lens within the field of view.

6. The environmental monitoring system as in claim 5, wherein the processor is further configured to:detect a direction that an associated vehicle is turning;monitor the quantity of the number of sharpened pixels within each of the segmentations as the associated vehicle is turning;estimate a migration profile of the quantity of the number of sharpened pixels between the segmentations as the associated vehicle turns and changes the orientation of the field of view; andassign the one or more segmentations with the presence of an obstruction on the lens within the field of view when the one or more segments are outside of the migration profile by a predetermined threshold.

7. The environmental monitoring system as in one of claims 1-3, wherein each segmentation includes a grid of cells and each cell associated with a plurality of pixels that are quantified for the input pixel data in the integral image.

8. The environmental monitoring system as in one of claims 1-3, wherein the processor is further configured to:generate a notification to a user after assigning the one or more segmentations with the presence of an obstruction on the lens within the field of view.

9. The environmental monitoring system as in one of claims 1-3, wherein the processor is further configured to:generate a signal to automatically wash the lens after assigning the one or more segmentations with the presence of an obstruction on the lens within the field of view.

10. The environmental monitoring system as in one of claims 1-3, wherein the display module is located in a full display rearview mirror switchable between a full display mode wherein the graphic of the images is generated and a reflective mode wherein the full display rearview mirror is reflective.

11. The environmental monitoring system of claim 10, wherein the processor is further configured to:generate a signal to automatically switch from the full display mode to the reflective mode after assigning the one or more segmentations with the presence of an obstruction on the lens within the field of view.

12. An environmental monitoring system for a vehicle with lens obstruction detection, the environmental monitoring system comprising:at least one imager module having a lens defining a field of view and configured to capture a series of images of an environment defined by the field of view;a display module configured to generate a graphic of the images defining an outer graphic boundary;a processor; anda memory including instructions that, when executed by the processor, cause the processor to:generate integral images from the captured images, the integral images including segmentations each with an input pixel data; anddetect edges of objects within the captured images;sharpen the detected edges;quantify a number of sharpened pixels within each of the segmentations; detect one or more segmentations that include a low number of sharpened pixels for a period of time; andassign the one or more segmentations with the presence of an obstruction on the lens within the field of view.

13. The environmental monitoring system of claim 12, wherein the processor is further configured to:detect a direction that an associated vehicle is turning;monitor the quantity of the number of sharpened pixels within each of the segmentations as the associated vehicle is turning;estimate a migration profile of the quantity of the number of sharpened pixels between the segmentations as the associated vehicle turns and changes the orientation of the field of view; andassign the one or more segmentations with the presence of an obstruction on the lens within the field of view when the one or more segments are outside of the migration profile by a predetermined threshold.

14. The environmental monitoring system as in claim 12 or claim 13, wherein reviewing the integral images for the presence of an obstruction on the lens further includes reviewing the segmentations in each of a plurality of sequentially captured integral images for input pixel data that remains static within a predetermined threshold.

15. The environmental monitoring system of claim 14, wherein the processor is further configured to:compare changes to input pixel data of each segmentation;detect one or more segmentations that include pixel data that remains relatively static relative to other ones of the segmentations; andassign the one or more segmentations with the presence of an obstruction on the lens within the field of view only upon detecting pixel data that remains relatively static relative to other ones of the segmentations and detecting that the same segmentations include a low number of sharpened pixels fora period of time.

16. The environmental monitoring system of claim 15, wherein the period of time is associated with a number of frames captured by the at least one imager module.

17. An environmental monitoring system for a vehicle with lens obstruction detection, the environmental monitoring system comprising:at least one imager module having a lens defining a field of view and configured to capture a series of images of an environment defined by the field of view;a display module configured to generate a graphic of the images defining an outer graphic boundary;a processor; anda memory including instructions that, when executed by the processor, cause the processor to:generate integral images from the captured images, the integral images including segmentations each with an input pixel data; andreview the segmentations in each of a plurality of sequentially captured integral images for input pixel data that remains static within a predetermined threshold.

18. The environmental monitoring system of claim 17, wherein the processor is further configured to:assign the one or more segmentations with the presence of an obstruction on the lens within the field of view; andgenerate an alert.

19. The environmental monitoring system of claim 17 or claim 18, wherein the display module is located in a full display rearview mirror within the vehicle.

20. The environmental monitoring system of claim 17 or claim 18, wherein the full display rearview mirror includes a reflective element switchable between a reflective state where the display module is obscured and a transmissive state where the display module is visible.

Images