Compact integrated occupant monitoring camera

Abstract

Described herein is a vehicle monitoring camera (100) for monitoring an interior of a vehicle (200). The camera (100) comprises a substrate (102) configured to house electronic components. At least one light source (104) is mounted to the substrate (102) and configured to emit light in the infrared wavelength range onto a scene within the vehicle (200). An image sensor (106) is mounted to the substrate (102) and comprises a two dimensional array of sensing elements configured to sense light in at least the infrared wavelength range from the scene. An imaging metasurface (108) is mounted onto or in front of the image sensor (106) and has surface properties configured to define a wide field of view of light received at the image sensor (106) and to focus light onto the image sensor (106) to capture images of the scene.

Description

COMPACT INTEGRATED OCCUPANT MONITORING CAMERAFIELD OF THE INVENTION

[0001] The present application relates to cameras and in particular to vehicle monitoring cameras for monitoring an interior of a vehicle.

[0002] Embodiments of the present invention are particularly adapted for occupant monitoring cameras for monitoring occupants and drivers of vehicles. However, it will be appreciated that the invention is applicable in broader contexts and other applications.BACKGROUND

[0003] Vehicle occupant monitoring systems (OMS) are systems designed to monitor and assess the status and behaviour of vehicle occupants. This includes the vehicle driver and other passengers.

[0004] Utilizing a combination of sensors, cameras, and advanced algorithms, OMS can detect driver drowsiness, fatigue, and distractions by analyzing eye movements, blink rates, and head positions. These systems also monitor seat occupancy and detect the presence of children left unattended in the vehicle, thereby optimizing airbag deployment and seatbelt reminders while preventing potential hazards like heatstroke. Additionally, biometric monitoring features measure vital signs such as heart rate and respiration, enabling the detection of medical emergencies and the analysis of stress levels to recommend necessary breaks or adjustments.

[0005] Beyond safety, OMS also enhance comfort and convenience by personalizing settings such as seats, mirrors, climate control, and infotainment based on occupant preferences. Gesture control allows for easy operation of various vehicle functions, reducing the need for physical controls. In emergency situations, OMS can automatically notify emergency services and provide critical information about the occupants' conditions. Adaptive restraint systems modify airbag deployment and seatbelt tensioners based on the size, position, and posture of the occupants, optimizing protection during collisions. These comprehensive functionalities make OMS an important component in modern automotive technology, significantly contributing to safer and more comfortable driving experiences.

[0006] Occupant monitoring systems are beneficial to image in both the visible and infrared ranges for enhanced functionality. Imaging in the infrared is ideal for performing occupantmonitoring and object identification while imaging in the visible range can be beneficial for obtaining videos of the occupants such as for video calls.

[0007] RGB-1 R image sensors have recently been adopted which are capable of imaging in both the visible (RGB) and infrared (IR) wavelength ranges. However, these devices have difficulty in clearly filtering IR from visible and vice versa. As such, the image quality is generally lower than other sensors that image in a single wavelength domain (visible or infrared). However, incorporating two entirely separate image sensors for imaging in visible and infrared wavelengths adds cost and size to an occupant monitoring camera.

[0008] Any discussion of the background art throughout the specification should in no way be considered as an admission that such art is widely known or forms part of common general knowledge in the field.SUMMARY OF THE INVENTION

[0009] In accordance with a first aspect of the present invention, there is provided a vehicle monitoring camera for monitoring an interior of a vehicle, the camera comprising: a substrate configured to house electronic components; at least one light source mounted to the substrate and configured to emit light in the infrared wavelength range onto a scene within the vehicle; an image sensor mounted to the substrate and comprising a two dimensional array of sensing elements configured to sense light in at least the infrared wavelength range from the scene; and an imaging metasurface mounted in front of the image sensor and having surface properties configured to define a wide field of view of light received at the image sensor and to focus light onto the image sensor to capture images of the scene.;

[0010] In some embodiments, the camera further comprises a processor mounted to the substrate and configured to process images sensed by the image sensor to perform occupant and / or object monitoring.

[0011] In some embodiments, the imaging metasurface has surface properties configured to bandpass filter incident light to only pass light within a predefined range of infrared wavelengths.

[0012] In some embodiments, the imaging metasurface has surface properties configured to polarize incident light into one or more predetermined polarization states.

[0013] In some embodiments, the camera further comprises a light projecting metasurface disposed in front of the light source.

[0014] In some embodiments, the light projecting metasurface has surface properties configured to direct the emitted infrared light uniformly across a wide field of view onto the scene. In some embodiments, the light projecting metasurface has surface properties configured to project a structured light pattern onto the scene. In some embodiments, the light projecting metasurface and the imaging metasurface comprise a single metasurface material.

[0015] In some embodiments, the imaging metasurface defines a field of view that is wide enough to cover all of the seats within the vehicle. In some embodiments, the imaging metasurface defines a field of view of 140 degrees horizontal angle or greater relative to a normal of the substrate. In other embodiments, the imaging metasurface defines a field of view of greater than 60 degrees, 80 degrees, 100 degrees, greater than 120 degrees or greater than 160 degrees.

[0016] In some embodiments, the light source comprises a vertical cavity surface emitting laser (VCSEL) or array of VCSELs.

[0017] In some embodiments, the camera comprises a polarizing metasurface disposed in front of the light source and having surface properties configured to polarize the light emitted from the light source into a predetermined polarization state.

[0018] In some embodiments, the camera comprises a transmissive diffractive mask element disposed in front of the image sensor for encoding received light phase differences into light received at the image sensor.

[0019] In some embodiments, the camera comprises a second image sensor mounted to the substrate and comprising a two dimensional array of sensing elements configured to sense light in the visible wavelength range from a scene within the vehicle.

[0020] In some embodiments, the camera comprises a second imaging metasurface mounted in front of the second image sensor and having surface properties configured to define a wide field of view of light received at the second image sensor.

[0021] In some embodiments, the first and second imaging metasurfaces comprise a single metasurface material.

[0022] In some embodiments, the camera comprises two or more light sources mounted to the substrate, each light source configured to emit light in the infrared wavelength range.

[0023] In some embodiments, the two or more light sources are selectively controllable to be activated or deactivated.

[0024] In some embodiments, the camera comprises: a first light projecting metasurface disposed in front of a first of the two or more light sources and having surface properties configured to direct the emitted infrared light uniformly across a wide field of view onto the scene; and a second light projecting metasurface disposed in front of a second of the two or more light sources and having surface properties configured to project a structured light pattern of the infrared light onto the scene.

[0025] In some embodiments, components of the occupant monitoring camera are incorporated onto the substrate as a System on Chip device.

[0026] In some embodiments, the camera is integrated into a vehicle at a horizontally symmetric central location between left and right seats of the vehicle. In some embodiments, the camera is integrated into a rearview mirror of the vehicle. In other embodiments, the camera is integrated into a central information display, a bezel of a display within the vehicle, an instrument cluster or center console of the vehicle.

[0027] In some embodiments, the camera is integrated into a central rear region of the vehicle on or adjacent a ceiling of the vehicle and oriented to view the interior of the vehicle in a forward direction.

[0028] In accordance with a second aspect of the present invention, there is provided a method of monitoring an interior of a vehicle cabin using the camera of the first aspect, the method comprising the steps of: illuminating the scene from the at least one light source; capturing images from the image sensor; and processing the images to perform occupant and / or object monitoring of the scene.

[0029] In some embodiments, the occupant monitoring comprises determining at least one of the following: a) occupant position and limb movement;b) occupant facial expressions, head pose, and eye gaze; and / or c) occupant drowsiness, intoxication, or impairment.

[0030] In some embodiments, the images are processed by a processor integrated with the substrate. In other embodiments, the images are processed by a processor separate to the camera.

[0031] In accordance with a third aspect of the present invention, there is provided a vehicle comprising a camera according to the first aspect, wherein the camera is integrated into one of a rearview mirror, central information display or a bezel of a display within the vehicle.BRIEF DESCRIPTION OF THE FIGURES

[0032] Example embodiments of the disclosure will now be described, by way of example only, with reference to the accompanying drawings in which:Figure 1 is a schematic view of an interior of a vehicle illustrating a camera according to an embodiment of the invention integrated into a rearview mirror;Figure 2 is a plan view of the vehicle of Figure 1 illustrating the field of view of the camera relative to vehicle occupants;Figure 3 is a schematic plan view of a camera according to a first embodiment of the present invention;Figure 4 is a schematic side view of the camera of Figure 3;Figure 5 is a schematic plan view of a camera according to a second embodiment of the present invention; andFigure 6 is a schematic plan view of a camera according to a third embodiment of the present invention.DESCRIPTION OF THE INVENTION

[0033] Embodiments of the invention will be described herein with reference to the application of a vehicle cabin monitoring system. This includes vehicle driver and occupant monitoring systems. Vehicles may include cars, trucks, aircraft and vehicle training simulation systems. However, it will be appreciated that embodiments of the invention may haveapplications in other fields such as in surveillance systems, webcams, mobile device cameras or other imaging systems.System overview

[0034] Referring initially to Figures 1 and 2, there is illustrated a vehicle monitoring camera 100 for monitoring an interior of a vehicle 200 in the form of a car. In the illustrated embodiment, camera 100 is integrated into a rearview mirror 202 of vehicle 200 so as to be able to view the interior of vehicle 200 along a rearward direction. However, more generally, camera 100 may be integrated into vehicle 200 at other appropriate locations and orientations so as to be able to image the interior of vehicle 200 including a driver 210 and occupants 212, 214 and 216 of vehicle 200. By way of example, camera 100 may be integrated into a central information display, a bezel of a display within the vehicle, an instrument cluster or center console of vehicle 200. In another example, camera 100 may be integrated into a central rear region of vehicle 200 on or adjacent a ceiling of vehicle 200 and oriented to view the interior of vehicle 200 in a forward direction. Where camera 100 is adapted to perform vehicle occupant monitoring, preferably, camera 100 is located within vehicle 200 at a horizontally symmetric central location between left and right seats of the vehicle.

[0035] Rearview mirror 202 is mounted in the conventional location within vehicle 200 at a central upper region of the front windshield. Mirror 202 includes a substantially horizontally elongate body 204 mounted to vehicle 200 at one or more mounting points 206, as shown in Figure 2. Mounting point 206 may be adapted to allow mirror 100 to be pivotally moveable. Body 204 preferably takes the form of a protective housing formed of a rigid material such as a plastics material. Body 204 supports an electrically controllable reflective device such as an electrochromic element that is adapted to selectively filter and reflect light incident onto mirror 202 in a manner known in the art with auto dimming mirrors.

[0036] As will be described below, camera 100 is adapted to be integrated partially or wholly onto a single electronics substrate to reduce form factor and overall cost. Preferably, every element of camera 100 is integrated onto a single substrate. However, in some embodiments, an external processor may be leveraged for processing images captured by the camera. The use of a substrate layer enables precise alignment for simplified manufacturing and eliminates the need for post-assembly calibration.

[0037] Referring now to Figures 3 and 4, camera 100 comprises a substrate 102 configured to house electronic components. Substrate 102 may be any material suitable for forming conventional electronics chips or circuit boards such as silicon or ceramics based materials. Substrate 102 may have the dimensions of a conventional printed circuit board (PCB) such as 100 mm x 100 mm or 200 mm x 300 mm. Alternatively substrate 102 may have custom dimensions.

[0038] It will be appreciated that the relative size and location of components mounted to substrate 102 illustrated in Figure 3 are exemplary only and may be varied in different embodiments.

[0039] At least one light source is mounted to substrate 102 and configured to emit light in at least the infrared wavelength range onto a scene within the vehicle. In the illustrated embodiment, a single light source in the form of a Vertical Cavity Surface Emitting Laser (VCSEL) 104 or array of VCSELs is mounted to substrate 102. VCSEL 104 is a specific form of semiconductor laser diode that emits light perpendicular to the surface of the diode wafer. VCSELs are typically smaller in size than edge emitting laser and typically draw lower power also.

[0040] VCSEL 104 may be mounted to substrate 102 by one of many known bonding techniques such as wire bonding, epoxy adhesive bonding, Cu-Cu Thermo-Compression Bonding, hybrid bonding (combining oxide and Cu bonding), soldering or thermo-compression bonding. VCSEL 104 is mounted such that its active surface is directed upwards so that the infrared light is directed perpendicularly away from substrate 102. To achieve a compact device, Through-Silicon Vias (TSVs) may be used for vertical electrical interconnection between stacked layers. This eliminates the need for wire bonding on the final module and enables the compact form factor. These fabrication processes may be used for mounting the other components described below.

[0041] It will be appreciated that, in other embodiments, VCSEL 104 can be replaced with one or more other light sources such as LEDs or other types of laser diodes. Preferably the one or more light sources are configured to emit light in the near infrared wavelength range, with wavelengths between 750 nm and 2.5 pm. Optionally, the one or more light sources may be configured to also emit light in other wavelength ranges in addition to this range, such as in the visible wavelength range.

[0042] In some embodiments, VCSEL 104 may comprise an array or plurality of individual light sources co-mounted on substrate 102 and configured to emit infrared light and / or visible light at different wavelengths. The different constituent light sources within VCSEL 104 may be individually electrically addressable such that different combinations of light sources can be activated or deactivated. By way of example, VCSEL 104 may be controlled to emit light in the infrared wavelength range only or controlled to illuminate in both the infrared and visible wavelength ranges. Typically, VCSEL 104 will always be controlled to emit light in the infrared wavelength range so that an image sensor (described below) is able to image in that wavelength range. However, in some embodiments, it may be advantageous for VCSEL 104 to only emit light in the visible wavelength range.

[0043] An infrared image sensor 106 is also mounted to substrate 102. Infrared image sensor 106 comprises a two dimensional array of sensing elements (or pixels) configured to sense light in at least the infrared wavelength range from the scene. As with conventional image sensors, each of the sensing elements are individually electrically addressable to sense incident light.

[0044] Image sensor 106 may be mounted to substrate 102 via methods known in the art. This includes preparing the local area of the substrate by creating the necessary circuitry, adhering image sensor components to the substrate and aligning the elements, forming the electrical connections via wire bonding, soldering or the like, encapsulation for protection and integration with other components such as lenses and filters. In some embodiments, image sensor 106 is fabricated with integrated optics such as microlenses and / or optical filters.

[0045] An imaging metasurface 108 is mounted onto or in front of at least image sensor 106. Imaging metasurface 108 is transmissive and configured to modify light that is incident onto image sensor 106. The relative position of metasurface 108 on image sensor 106 can be seen in Figure 4. When viewed in a horizontal configuration, metasurface 108 is mounted vertically on top of image sensor 106. More generally, metasurface 108 is disposed partially or wholly over image sensor 106. An alignment process may be applied to align metasurface 108 with image sensor 106.

[0046] Metasurfaces are engineered surfaces typically formed from a substantially planar substrate, which may be fabricated from materials such as glass, silicon, or polymers. The surface of the substrate has surface properties from sub-wavelength features (called “metaatoms”) that can manipulate light in specific ways, such as focusing, filtering, or polarizinglight. Each of the sub-wavelength meta-atoms impart a specific phase shift, amplitude or polarization change to incident light. The collective effects of the surface properties act to reshape the incident wavefront of light in various ways, similar to other optical elements such as lenses, filters, polarizers and diffraction gratings. The meta-atoms can be composed of various materials, including but not limited to metals, dielectrics, or semiconductors, chosen based on the desired optical properties and the operational wavelength range. By way of example, the meta-atoms may comprise high-aspect-ratio posts made of a high-refractive- index, low-loss dielectric material (e.g., Titanium Dioxide or Silicon Nitride in the relevant IR and visible wavelengths.

[0047] Metasurfaces can be fabricated from a number of techniques including electron beam lithography, photolithography and nanoimprint lithography. By way of example, US Patent 10,795,168 to Metalenz Inc. entitled “Transmissive metasurface lens integration" describes a process for fabricating metasurface elements on systems. The contents of US Patent 10,795,168 is herein incorporated by way of cross reference. Various material deposition techniques such as physical vapor deposition or chemical vapor deposition may be used in combination with etching and lift off processes to form the meta-atoms and define the surface properties of the metasurface. Initially the desired surface profile of the metasurface (comprising an array of surface elements or meta-atoms) is typically designed in computer software to achieve the desired effects such as focussing, beam steering, filtering etc. Computational tools such as finite-difference time-domain (FDTD) or finite element method (FEM) simulations may be used to model the electromagnetic response of the meta-atoms. The output of these simulations may be fed to a fabrication tool to produce the desired metasurface material.

[0048] In some embodiments, imaging metasurface 108 is fabricated directly onto image sensor 106. However, more commonly, imaging metasurface 108 is fabricated on a separate material such as a glass substrate, which is bonded to an upper surface of image sensor 106. Use of the terms ‘mounted onto or in front of’ is intended to cover both fabrication scenarios. Although not shown, the glass substrate or other intermediate substrate material may be included in the optical stack.

[0049] Metasurfaces are highly sensitive to tilt and lateral shift. As such, during the fabrication process, there is a need for active alignment or high-precision passive alignmenttools to achieve alignment tolerances, preferably in the order of 1 um or less to maintain the optical performance defined by the phase profile.

[0050] Imaging metasurface 108 has surface properties configured to define a wide field of view of light received at the image sensor and to focus light onto image sensor 106 to capture images of the scene. That is, imaging metasurface 108 functions as a wide field of view lens to capture a broad scene or image area such as the vehicle cabin interior. At the same time, imaging metasurface 108 comprises a short focal length (e.g. 35 mm or less) to focus light from the scene onto image sensor 106. In more compact embodiments, the focal length may be smaller, such as 3.5 mm or less.

[0051] By way of example, imaging metasurface 108 may define a field of view of 140 degrees horizontal angle or greater relative to a normal of substrate 102. In other embodiments, imaging metasurface 108 defines a field of view of greater than 60 degrees horizontal angle, 80 degrees horizontal angle, 100 degrees horizontal angle, 120 degrees horizontal angle, 160 degrees horizontal angle or 180 degrees horizontal angle. Preferably, imaging metasurface 108 defines a field of view that is wide enough to cover all of the seats within the vehicle.

[0052] Imaging metasurface 108 can be constructed to function as a wide field of view lens by designing the meta-atoms such that they produce a spatially varying phase profile across the metasurface. The spatially varying phase profile should have a gradient such that the metasurface can focus light from a wide range of incident angles in a similar manner to a lens. This is achieved by designing the distribution of meta-atoms to be varied in shape, size, orientation, and material composition to produce the desired phase profile. Before manufacture, the metasurface can be modelled in simulation software to produce a desired phase surface and then fabricated using known methods.

[0053] When functioning as a lens, imaging metasurface 108 offers significant advantages over traditional refractive or diffractive lenses, including reduced thickness, lighter weight, and the potential for integration into compact optical systems.

[0054] Imaging metasurface 108 may be integrated directly onto an upper surface of image sensor 106 in a similar manner to forming a metasurface directly on a substrate. In other embodiments, image metasurface 108 is mounted proximal to but directly above imagesensor 106 by one or more mounting formations or support structures such as a glass substrate that attach to image sensori 06 and / or substrate 102.

[0055] In some embodiments, imaging metasurface 108 has surface properties configured to also bandpass filter incident light to only pass light within a predefined range of infrared wavelengths. A metasurface can function as a bandpass filter by selectively allowing electromagnetic waves within a specific wavelength range to pass through, while blocking or attenuating wavelengths outside that range. This functionality is achieved through the careful design and arrangement of the meta-atoms or nano-structures that make up the metasurface (to form the surface properties). The bandpass filter may be matched to the VCSEL wavelength and temperature variations. This can ensure optimal infrared image quality across a wide temperature range (-40 to +80 degrees Celsius).

[0056] By way of example, meta-atoms can be designed to have sizes, shapes, arrangements and material compositions so as to resonate at specific wavelengths of light (a filter passband). In some embodiments, the arrangement of meta-atoms can be designed so as to create constructive interference for a desired wavelength range (e.g. infrared wavelengths) and to create destructive interference at other wavelengths (e.g. non infrared wavelengths). Wavelengths that do not satisfy the resonant or constructive interference conditions are attenuated while those that do, are passed through the metasurface to be received at image sensor 106.

[0057] While the pixels of image sensor 106 must be sensitive to infrared wavelengths, they need not be sensitive to only infrared wavelengths. With appropriate filtering from imaging metasurface 108, unwanted wavelengths in the visible and other wavelength ranges can be filtered from reaching image sensor 106 thereby improving the signal to noise ratio.

[0058] In some embodiments, imaging metasurface 108 also has surface properties configured to polarize incident light into one or more predetermined polarization states. To achieve this, the meta-atoms may be designed (via their shape and material properties) so as to form anisotropic resonators that interact differently with different polarizations of light. For example, an array of rectangular or elliptical resonators can preferentially absorb or transmit light polarized along one axis while reflecting or blocking light polarized along the orthogonal axis. The meta-atoms may also be designed to impart different phase shifts to different polarizations of light. By controlling the phase response, a metasurface can be designed toconvert an incident unpolarized or mixed polarization light into a single desired polarization state such as linear polarized light.

[0059] In some embodiments, imaging metasurface 108 may be used in conjunction with one or more other polarizing elements to perform polarization manipulation in a manner described in US Patent Application Publication 2021 / 0176403 to Seeing Machines Limited entitled “High Performance Imaging System Using a Dielectric Metasurface”. The contents of this specification are incorporated herein by way of cross reference.

[0060] Camera 100 may further comprise a processor 110 mounted to substrate 102 and configured to control the various components of camera 100 and optionally to also process images sensed by image sensor 106 to perform occupant and / or object monitoring. As its primary function, processor 110 is adapted to provide control of components such as image sensor (e.g. frame rate, exposure time etc.) and VCSEL 104 (e.g. pulse time and power level). Processor 110 may also be configured for processing captured images of a vehicle driver and / or occupants during operation of a vehicle 200. Processor 110 is further adapted for performing various image processing algorithms on the captured images such as facial detection, facial feature detection, facial recognition, facial feature recognition, facial tracking or facial feature tracking, such as tracking a person’s eyes. Example image processing routines are described in US Patent 7,043,056 to Edwards et al. entitled “Facial Image Processing System” and assigned to Seeing Machines Pty Ltd (hereinafter “Edwards et al ”), the contents of which are incorporated herein by way of cross-reference.

[0061] By way of example, processor 110 may be configured to perform one or more of the following operations based on the captured images: Occupant Health and Safety:• Detecting distracted driving, drowsiness, or other forms of impairment.• Detecting if the driver of the vehicle is touching or holding the steering wheel in a manner that allows for safe control of the vehicle.• Correct deployment of restraints, airbags and other crash protection devices, through accurate sensing of the 3D positions of the occupants just prior to the crash event.• Reliable detection of cabin seating positions and seatbelt use by occupants.• Reliable detection of unknown and unsecured objects within the cabin which may injure the occupants if high g forces are encountered.• Detection of occupant biometric vital signs through measurements of respiration and heart rate, achieved via an accurate measurement of 3D chest movements as well as blood volume in the face witnessed in the infrared image.• Detection of living creatures in the cabin while the vehicle is parked and locked to prevent injury or death from heat, cold or poor air quality. Comfort and Convenience:• Monitoring occupant preferences for personalized climate control or entertainment.• Interpreting occupant command intent via observation of hand, head, eye combined dynamic gesture actions, optionally combined with voice, requiring reliable real-time 3D measurements of head, eye, mouth, hand and finger states and positions.• Detecting if the vehicle cabin needs cleaning or a person has accidentally left behind a possession. Security:• Detecting the identity of the occupants using their appearance and / or the 3D shape of their faces and / or bodies.• Monitoring for unauthorized access, suspicious or high risk activities inside the vehicle.

[0062] In other embodiments, camera 100 includes only a basic device controller and image processing is performed by a separate processor not formed on substrate 102. In some embodiments, processor 110 is adapted to perform limited image pre-processing but the primary image processing is performed by a separate processor.

[0063] As illustrated in Figures 3 and 4, camera 100 may further comprise a light projecting metasurface 112 that is mounted such that it is disposed in front of VCSEL 104. Light projecting metasurface 112 is transmissive and configured to modify light that is emitted from VCSEL 104. It will be appreciated that, while light projecting metasurface 112 is illustrated as being separate to imaging metasurface 108, in some embodiments, the functions of bothmetasurfaces 108 and 112 (and other metasurfaces described below) may be incorporated into a single unitary metasurface formed from a single metasurface material. This may be achieved by fabricating different arrays of meta-atoms at different locations on a single metasurface material or substrate. It will be appreciated that, in other embodiments, any subset of metasurfaces may be co-fabricated on a single metasurface material in any combination and independently of one another.

[0064] Light projecting metasurface 112 preferably has surface properties configured to project a flood illumination of light onto the scene. This includes projecting an ambient or uniform beam of light onto the scene such that the light is spread uniformly or substantially uniformly across the desired illuminated area. This may comprise projecting the light at a wide or solid angle that matches or closely matches the receiving angle of imaging metasurface 108. By way of example, light projecting metasurface 112 may define a beam having a horizontal angle in the range of 60 to 180 degrees. Preferably, light projecting metasurface 112 projects a light beam that is wide enough to cover all of the seats within the vehicle 200.

[0065] In some embodiments, light projecting metasurface 112 has surface properties configured to project a structured light pattern onto the scene. Structured light refers to light with a controlled spatial distribution of its contrast, phase, amplitude, and / or polarization to create specific patterns or distributions. Structured light illumination can enhance contrast and resolution in imaging systems by exploiting the specific interaction of structured light with the scene objects. In particular, projecting a structured light pattern onto a scene can allow extraction of depth information about objects within the scene. With the ability to sense depth to objects, camera 100 is able to perform three dimensional sensing or imaging of a scene.

[0066] Light projecting metasurface 112 can be configured to project structured light by designing the array of meta-atoms to perform one or more of phase shifts, amplitude modulation and polarization control of light passing through metasurfacel 12 such that it forms the light into a predefined pattern. Example structured light patterns that can be generated by light projecting metasurface 112 include grids of dots or lines, high contrast point arrays, Airy function intensity profiles, concentric ring patterns, spiral patterns, polarization varying patterns and intensity modulating patterns.

[0067] Each of these patterns exhibit light that is highly varying in contrast and this contrast creates strong edges in the received images. Edges have large phase differences, which iswhere the three dimensional information that can be extracted is most accurate. This structured light projection is not essential because natural edges can appear in a scene (such as along the edges of a seatbelt across the chest). However, projecting additional edges makes the 3D data more deterministic and reliable.

[0068] In some embodiments, a single light source such as a VCSEL array has distinct addressable regions or subarrays, each aligned to different metasurface pattern regions (flood vs. structured), enabling electronic pattern switching without moving parts.

[0069] Light projecting metasurface 112 may also have surface properties configured to polarize the light emitted from the light source into a predetermined polarization state. Alternatively or in addition, a separate polarizing metasurface (not shown) may be disposed in front of VCSEL 104 which has surface properties configured to polarize the light emitted from the light source into a predetermined polarization state. By way of example, light projecting metasurface112 may be configured to polarize light into a linear, elliptical or circular polarized state. In some embodiments, the desired polarization state may be the same as that provided by imaging metasurface 108. In other embodiments, the polarization state may be orthogonal to that provided by imaging metasurface 108.

[0070] In some embodiments, it may be advantageous to switch between illuminating the vehicle cabin with an evenly spread flood illumination or a structured light illumination (or both). Referring now to Figure 5, there is illustrated a second embodiment camera 500 that can achieve this. Camera 500 comprises all of the components of camera 100, which are designated with the same reference numerals as camera 100. In addition, camera 500 comprises a VCSEL array 104 that comprises two VCSEL light sources 104A and 104B that are independently controllable. In some embodiments, VCSEL light sources 104 and 104B are replaced with other types of light source such as LEDs or other laser diodes.

[0071] Light sources 104A and 104B may be configured to emit light at the same or different wavelengths in the infrared wavelength range and optionally also in the visible range. Furthermore, two light projecting metasurfaces 112A and 112B are disposed in front of respective VCSEL light sources 104A and 104B. First metasurface 112A has surface properties configured to project a uniform flood illumination of light onto the scene while second metasurface 112B has surface properties configured to project structured light onto the scene. By selectively activating or deactivating light sources 104A and 104B, illumination can be switched between ambient flood light illumination and structured light illumination. Byactivating both light sources 104A and 104B, ambient and structured light can be projected onto the scene.

[0072] Although illustrated as two separate elements, it will be appreciated that light projecting metasurfaces 112A and 112B may be formed as a single metasurface element and may also be integrated together with other metasurfaces such as imaging metasurface 108.

[0073] Referring again to Figure 4, camera 100 may also comprise a transmissive diffractive mask (TDM) element 114 disposed on or in front of image sensor 106 for encoding received light phase differences into light received at the image sensor. TDM 114 may equivalently be incorporated into camera 500 of Figure 5. The TDM 114 may be incorporated as a layer of image sensor 106 for encoding phase information to facilitate 3D depth sensing. Although illustrated as being above imaging metasurface 108, in some embodiments, TDM 114 may be deposited between image sensor 106 and imaging metasurface 108 such as directly mounted to image sensor 106. TDMs manipulate light passing through them via diffraction. These masks are physical structures that have a patterned surface that alters the phase, amplitude, or direction of the incoming light to produce specific output patterns or beam shapes.

[0074] The surface of TDM 114 is formed by engraving or etching a base material with micro- or nanostructures, which can be designed to create complex diffraction patterns. The diffractive elements may take the form of grooves, slits, ridges, or other geometric configurations, and are characterized by dimensions on the order of the wavelength of the light intended for use. The TDM 114 may be configured to operate over a specified range of wavelengths, such as wavelengths in the infrared or visible regions. The patterns can be periodic, aperiodic, or even random, depending on the desired optical effect. The base material of TDM 114 is typically made from substantially transparent materials such as glass, quartz, or polymers. TDMs can be configured to change the phase of the transmitted light by varying the thickness or refractive index of the material. TDMs can also be configured to modulate the intensity of the transmitted light by varying the transparency of the material. Thus, TDM 114 may be configured to perform one or more predetermined optical effects, such as phase modulation, amplitude modulation, or angular redirection.

[0075] Although illustrated in combination with device 100, it will be appreciated that TDM 114 may be implemented into other embodiments such as camera 500 and camera 600 described below.

[0076] Cameras 100 and 500 described above are adapted to image in the infrared wavelength range. In some instances, it may be advantageous to image the scene in the visible wavelength range, such as to provide video conferencing in vehicle 200. Referring now to Figure 6, there is illustrated a further embodiment camera 600 that can image in both the visible and infrared wavelength ranges. Camera 600 comprises all of the components of camera 500, which are designated with the same reference numerals as camera 500. However, camera 600 further comprises a second image sensor 120 mounted to substrate 102 and comprising a two dimensional array of sensing elements configured to sense light in the visible wavelength range from a scene (e.g. within vehicle 200). Second image sensor 120 may be fabricated onto substrate 102 in a similar manner to that described above in relation to infrared image sensor 106.

[0077] Camera 600 also comprises a second imaging metasurface 122 mounted in front of visible light image sensor 120. Metasurface 122 has surface properties configured to define a wide field of view of light received at the second image sensor in a similar manner to that of imaging metasurface 108 described above. Metasurface 122 may comprise a standalone metasurface element or may be integrated into a single metasurface material with other metasurface elements described above. As the different elements are spatially separated across substrate 102, the different metasurfaces may be formed on a single metasurface layer or material with the different metasurface properties coinciding with the different components below.

[0078] Although not illustrated, camera 600 may comprise additional light sources that are adapted to illuminate the scene with visible light that can be reflected from scene objects and captured by visible light image sensor 120. Additional metasurfaces may be placed in front of these visible light sources to form the visible light into an appropriate beam and to optionally apply other characteristics such as polarization. In some embodiments, camera 600 may utilize metasurfaces to selectively polarize the visible and / or infrared light to form a visible light shutter as described in US Patent Application Publication 2022 / 0252924 to Seeing Machines Limited entitled “Cabin Monitoring With Electrically Switched Polarization”. The contents of this document are incorporated herein by way of cross reference.

[0079] In some embodiments, the components of cameras 100, 500 or 600 are incorporated onto substrate 102 as a System on Chip (SoC) device. An SoC device is an integrated circuit (IC) that consolidates multiple components of a computer or other electronicsystems onto a single chip. In these embodiments, processor 110 acts as a central processing unit (CPU) to control the various components and optionally perform digital signal processing and image processing. Although not illustrated, the SoC device may also comprise one or more memory (RAM, ROM) elements, input / output (I / O) ports, secondary storage (such as flash memory) communication modules (e.g., Wi-Fi, Bluetooth).

[0080] VCSELs and other lights sources are heat sources that add heat to the device. Components such as image sensor 106 and metasurfaces 108, 112 and 122 are sensitive to temperature changes. Further, temperature changes can cause wavelength drift in a VCSEL. To reduce heat within camera 100, a Thermal Interface Material (TIM) may be incorporated between VCSEL 104 and substrate 102 to efficiently transfer heat away. Substrate 102 may also be bonded to a metallic heat spreader or heat sink, particularly if processor 110 performs intensive image processing.

[0081] Although not illustrated, camera 100, 500 or 600 described above may comprise one or more communications interfaces for communicating with other devices such as a central control system of vehicle 200. These communications interfaces may comprise electrical ports for connecting data cables such as Ethernet, FlexRay, Local Interconnect Network (LIN Bus) or Controller Area Network (CAN Bus). These communications interfaces may comprise wireless transceivers for communicating wirelessly via protocols such as Wi-Fi, Bluetooth, Zigbee, Z-Wave, NFC, RFID, LoRa or cellular network communications.

[0082] Finally, to withstand environmental effects, camera 100, 500 or 600 may be housed within a protective casing. The casing may provide hermetic sealing and may comprise a protective cover glass. The cover glass may comprise an anti-reflection coating to prevent damage and reduce surface reflections that could degrade image quality.

[0083] It will be appreciated that embodiments of the invention described above provide a compact camera module optimized for applications such as vehicle cabin monitoring. The camera module integrates infrared and optionally visible image sensors (optionally with transmissive diffractive masks), VCSEL-based infrared illumination sources, and custom metasurface optical lenses to capture both visible and infrared imagery for comprehensive occupant and cabin state analysis. The module enables detection of occupant position, limb movement, facial expressions, head pose, eye gaze, drowsiness, intoxication, and other relevant conditions. Metasurface lens technology miniaturizes the module, facilitating integration into vehicle displays or rearview mirrors. Structured light projection, along withoptional transmissive diffractive masks on the image sensors, enables robust 3D depth sensing for enhanced analysis.

[0084] As the metasurface lens elements are small, this allows the light projector metasurface to be placed very close to the sensor (due to there being no mechanical housings). For some structured light techniques the closer the projector is to the lens optical axis, the more accurate the 3D measurements can be. Thus, embodiments of the invention can provide more accurate 3D measurements.

[0085] The nature of the device allows it to be produced at volume in High-Volume Manufacturing (HVM) methods. Since a primary application of the camera is for automotive (mass-market) use, a HVM process such as Nanoimprint Lithography (NIL) technique may be implemented for cost-effective, high-volume manufacturing of the metasurface structure, rather than slower e-beam techniques.

[0086] Common substrate mounting facilitates precise alignment, reducing assembly complexity and enabling mass production. Metasurface lenses are able to be etched or printed in high volumes with far simpler assembly and alignment onto the camera module compared to traditional diffractive lens components which require a whole additional mechanical sub-assembly and 6-axis alignment process.INTERPRETATION

[0087] The term “light” is used in this specification to generally refer to electromagnetic radiation that is in the visible and / or infrared wavelength ranges. However, it will be appreciated that, depending on the context, the term “light” may apply to electromagnetic radiation in other regions of the electromagnetic spectrum such as the ultraviolet range.

[0088] The term “infrared” is used throughout the description and specification. Within the scope of this specification, infrared refers to the general infrared area of the electromagnetic spectrum which includes near infrared, infrared and far infrared frequencies or light waves. By way of example, the infrared wavelength range may extend from wavelengths of about 750 nm to 1 ,100 nm. However, it will be appreciated that these are not hard boundaries and the definitions of infrared wavelength ranges vary from source to source.

[0089] The terms “visible” in the sense of visible wavelength or visible light are intended to refer to electromagnetic radiation in the visible wavelength range that is visible to the human eye. This generally comprises wavelengths ranging from about 380 nm to about 750 nm.However, it will be appreciated that these are not hard boundaries and the definitions of infrared wavelength ranges vary from source to source.

[0090] Unless specifically stated otherwise, as apparent from the following discussions, it is appreciated that throughout the specification discussions utilizing terms such as "processing," "computing," "calculating," “determining”, analyzing” or the like, refer to the action and / or processes of a computer or computing system, or similar electronic computing device, that manipulate and / or transform data represented as physical, such as electronic, quantities into other data similarly represented as physical quantities.

[0091] In a similar manner, the term “controller” or "processor" may refer to any device, portion of a device or plurality of devices that processes electronic data, e.g., from registers and / or memory to transform that electronic data into other electronic data that, e.g., may be stored in registers and / or memory. A “computer” or a “computing machine” or a "computing platform" may include one or more co-located or distributed processors.

[0092] Reference throughout this specification to “one embodiment”, “some embodiments” or “an embodiment” means that a particular feature, structure or characteristic described in connection with the embodiment is included in at least one embodiment of the present disclosure. Thus, appearances of the phrases “in one embodiment”, “in some embodiments” or “in an embodiment” in various places throughout this specification are not necessarily all referring to the same embodiment. Furthermore, the particular features, structures or characteristics may be combined in any suitable manner, as would be apparent to one of ordinary skill in the art from this disclosure, in one or more embodiments.

[0093] As used herein, unless otherwise specified the use of the ordinal adjectives "first", "second", "third", etc., to describe a common object, merely indicate that different instances of like objects are being referred to, and are not intended to imply that the objects so described must be in a given sequence, either temporally, spatially, in ranking, or in any other manner.

[0094] In the claims below and the description herein, any of the terms “comprising”, “comprised of’, “which comprises” or similar are open terms that mean including at least the elements / features that follow, but not excluding others. Thus, the term “comprising” and its variations, when used in the claims or description, should not be interpreted as being limitative to the means or elements or steps listed thereafter. For example, the scope of the expression a device comprising A and B should not be limited to devices consisting only of elements Aand B. Similarly, any of the terms “including”, “which includes”, “that includes” or similar as used herein are also open terms that also mean including at least the elements / features that follow the term, but not excluding others. Thus, “including” is synonymous with and means “comprising”.

[0095] It should be appreciated that in the above description of exemplary embodiments of the disclosure, various features of the disclosure are sometimes grouped together in a single embodiment, Fig., or description thereof for the purpose of streamlining the disclosure and aiding in the understanding of one or more of the various inventive aspects. This method of disclosure, however, is not to be interpreted as reflecting an intention that the claims require more features than are expressly recited in each claim. Rather, as the following claims reflect, inventive aspects lie in less than all features of a single foregoing disclosed embodiment. Thus, the claims following the Detailed Description are hereby expressly incorporated into this Detailed Description, with each claim standing on its own as a separate embodiment of this disclosure.

[0096] Furthermore, while some embodiments described herein include some but not other features included in other embodiments, combinations of features of different embodiments are meant to be within the scope of the disclosure, and form different embodiments, as would be understood by those skilled in the art. For example, in the following claims, any of the claimed embodiments can be used in any combination.

[0097] In the description provided herein, numerous specific details are set forth. However, it is understood that embodiments of the disclosure may be practiced without these specific details. In other instances, well-known methods, structures and techniques have not been shown in detail in order not to obscure an understanding of this description.

[0098] Similarly, it is to be noticed that the term coupled, when used in the claims, should not be interpreted as being limited to direct connections only. The terms "coupled" and "connected", along with their derivatives, may be used. It should be understood that these terms are not intended as synonyms for each other. Thus, the scope of the expression a device A coupled to a device B should not be limited to devices or systems wherein an output of device A is directly connected to an input of device B. It means that there exists a path between an output of A and an input of B which may be a path including other devices or means. "Coupled" may mean that two or more elements are either in direct physical, electricalor optical contact, or that two or more elements are not in direct contact with each other but yet still co-operate or interact with each other.

[0099] Embodiments described herein are intended to cover any adaptations or variations of the present invention. Although the present invention has been described and explained in terms of particular exemplary embodiments, one skilled in the art will realize that additional embodiments can be readily envisioned that are within the scope of the present invention.

Claims

What is claimed is:

1. A vehicle monitoring camera for monitoring an interior of a vehicle, the camera comprising: a substrate configured to house electronic components; at least one light source mounted to the substrate and configured to emit light in the infrared wavelength range onto a scene within the vehicle; an image sensor mounted to the substrate and comprising a two dimensional array of sensing elements configured to sense light in at least the infrared wavelength range from the scene; and an imaging metasurface mounted onto or in front of the image sensor and having surface properties configured to define a wide field of view of light received at the image sensor and to focus light onto the image sensor to capture images of the scene.

2. The vehicle monitoring camera according to claim 1 further comprising a processor mounted to the substrate and configured to process images sensed by the image sensor to perform occupant and / or object monitoring.

3. The vehicle monitoring camera according to claim 1 or claim 2 wherein the imaging metasurface has surface properties configured to bandpass filter incident light to only pass light within a predefined range of infrared wavelengths.

4. The vehicle monitoring camera according to any one of the preceding claims wherein the imaging metasurface has surface properties configured to polarize incident light into one or more predetermined polarization states.

5. The vehicle monitoring camera according to any one of the preceding claims further comprising a light projecting metasurface disposed in front of the light source.

6. The vehicle monitoring camera according to claim 5 wherein the light projecting metasurface has surface properties configured to direct the emitted infrared light uniformly across a wide field of view onto the scene.

7. The vehicle monitoring camera according to claim 5 wherein the light projecting metasurface has surface properties configured to project a structured light pattern onto the scene.

8. The vehicle monitoring camera according to claim 5 wherein the light projecting metasurface and the imaging metasurface comprise a single metasurface material.

9. The vehicle monitoring camera according to any one of the preceding claims wherein the imaging metasurface defines a field of view that is wide enough to cover all of the seats within the vehicle.

10. The vehicle monitoring camera according to any one of the preceding claims wherein the imaging metasurface defines a field of view of 140 degrees horizontal angle or greater relative to a normal of the substrate.11 . The vehicle monitoring camera according to any one of the preceding claims wherein the light source comprises a vertical cavity surface emitting laser (VCSEL) or array of VCSELs.

12. The vehicle monitoring camera according to any one of the preceding claims comprising a polarizing metasurface disposed in front of the light source and having surface properties configured to polarize the light emitted from the light source into a predetermined polarization state.

13. The vehicle monitoring camera according to any one of the preceding claims comprising a transmissive diffractive mask element disposed in front of the image sensor for encoding received light phase differences into light received at the image sensor.

14. The vehicle monitoring camera according to any one of the preceding claims comprising a second image sensor mounted to the substrate and comprising a two dimensional array of sensing elements configured to sense light in the visible wavelength range from a scene within the vehicle.

15. The vehicle monitoring camera according to claim 14 comprising a second imaging metasurface mounted in front of the second image sensor and having surface properties configured to define a wide field of view of light received at the second image sensor.

16. The vehicle monitoring camera according to claim 15 wherein the first and second imaging metasurfaces comprise a single metasurface material.

17. The vehicle monitoring camera according to any one of the preceding claims comprising two or more light sources mounted to the substrate, each light source configured to emit light in the infrared wavelength range.

18. The vehicle monitoring camera according to claim 17 wherein the two or more light sources are selectively controllable to be activated or deactivated.

19. The vehicle monitoring camera according to claim 18 comprising: a first light projecting metasurface disposed in front of a first of the two or more light sources and having surface properties configured to direct the emitted infrared light uniformly across a wide field of view onto the scene; and a second light projecting metasurface disposed in front of a second of the two or more light sources and having surface properties configured to project a structured light pattern of the infrared light onto the scene.

20. The vehicle monitoring camera according to any one of the preceding claims wherein the components of the occupant monitoring camera are incorporated onto the substrate as a System on Chip device.

21. The vehicle monitoring camera according to any one of the preceding claims integrated into a vehicle at a horizontally symmetric central location between left and right seats of the vehicle.

22. The vehicle monitoring camera according to any one of claims 1 to 20 integrated into a rearview mirror of the vehicle.

23. The vehicle monitoring camera according to any one of claims 1 to 20 integrated into an instrument cluster or center console of the vehicle.

24. The vehicle monitoring camera according to any one of claims 1 to 20 integrated into a central rear region of the vehicle on or adjacent a ceiling of the vehicle and oriented to view the interior of the vehicle in a forward direction.

25. A method of monitoring an interior of a vehicle cabin using the camera of any one of the preceding claims, the method comprising the steps of: illuminating the scene from the at least one light source;capturing images from the image sensor; and processing the images to perform occupant and / or object monitoring of the scene.

26. The method of claim 25 wherein the occupant monitoring comprises determining at least one of the following: a) occupant position and limb movement; b) occupant facial expressions, head pose, and eye gaze; and / or c) occupant drowsiness, intoxication, or impairment.

27. The method of claim 25 or claim 26 wherein the images are processed by a processor integrated with the substrate.

28. The method of claim 25 or claim 26 wherein the images are processed by a processor separate to the camera.

29. A vehicle comprising a camera according to any one of claims 1 to 24, wherein the camera is integrated into one of a rearview mirror, central information display or a bezel of a display within the vehicle.

Images