Spectroscopic device

Abstract

The invention is in the field of spectroscopic Devices. The invention relates to a spectroscopic device for determining a concentration of a body substance of a person comprising: a) a hyperspectral camera for recording hyperspectral data of the person, b) a processor for (i) determining region data from the hyperspectral data, wherein region data is indicative for the suitability of a region of the body for determining the concentration of the body substance, (ii) determining the concentration of the body substance of the person using the hyperspectral data and the region data, and c) an output for outputting the concentration of the body substance of the person.

Description

[0001] 240507W001

[0002] 1

[0003] Spectroscopic Device

[0004] The invention is in the field of spectroscopic devices. The invention relates to a spectroscopic device for determining a concentration of a body substance of a person, a vehicle comprising the spectroscopic device, a method for determining a concentration of a body substance of a person, a use of the concentration of the body substance of the person for granting a user access to a device or application, and a non-transient computer-readable medium including instructions that, when executed by one or more processors, cause the one or more processors to perform a method for determining a concentration of a body substance of a person.

[0005] Background

[0006] The measurement of body substances of a person plays a role in many different areas. Examples are the medical sector, for example diabetes patients need to regularly measure their blood glucose level. Another example is the vehicle security sector in which intoxicants like alcohol or drug blood concentrations of the driver need to be measured. Spectroscopic measurement is an attractive method for determining body substances as it is non-invasive and is hence well accepted by users. However, many body substances have a low concentration, hence they are difficult to measure accurately due to occlusion.

[0007] US 6072 180 discloses a spectrometer for measuring substances which are present in the tissue, such as glucose, by means of thermal gradient spectra by inducing a transient thermal gradient in human tissue. However, such measurement requires quite some time equilibrate. In addition, the occlusion problem is not overcome due the low concentration of body substances in the uppermost skin layers.

[0008] WO 2016 / 036314 A1 discloses a method for spectroscopic determination of in vivo tissue. The measurement of the most suitable body part depends on the operator and is thus subject errors and thus inaccurate results.

[0009] US 10560643 discloses a method of hyperspectral medical imaging for diagnosing a medical condition of a subject. Images are generated to support diagnosis of a medical doctor, but no concentration of a body substance is determined.

[0010] EP 2319406 A1 discloses hyperspectral imaging for detecting the state of systemic physiology. The measurement may be limited to a region of interest. However, the region of interest must be selected manually, it is not extracted from the hyperspectral data.

[0011] US 2015 / 0044098 A1 discloses a hyperspectral imaging system to determine biological data. A region of interest such as face may be used for the measurement, however, the region of interest is not selected from the hyperspectral data.

[0012] It was hence the object of the present invention to provide a reliable and comfortable solution to determining a concentration of a body substance of a person including body substances of low concentration. 240507W001

[0013] 2

[0014] Summary

[0015] In one aspect the invention relates to a spectroscopic device for determining a concentration of a body substance of a person comprising:

[0016] a) a hyperspectral camera for recording hyperspectral data of the person,

[0017] b) a processor for

[0018] (i) determining region data from the hyperspectral data, wherein region data is indicative for the suitability of a region of the body for determining the concentration of the body substance, (ii) determining the concentration of the body substance of the person using the hyperspectral data and the region data, and

[0019] c) an output for outputting the concentration of the body substance of the person.

[0020] In one aspect the invention relates to a spectroscopic device for determining a concentration of a body substance of a person comprising:

[0021] a) a hyperspectral camera for recording hyperspectral data of the person,

[0022] b) a processor for determining the concentration of the body substance of the person using the hyperspectral data, and

[0023] c) an output for outputting the concentration of the body substance of the person.

[0024] In another aspect the invention relates to a vehicle comprising the spectroscopic device according to the invention.

[0025] In another aspect the invention relates to a method for determining a concentration of a body substance of a person comprising:

[0026] a) receiving hyperspectral data of the person from a hyperspectral camera,

[0027] b) determining region data from the hyperspectral data, wherein region data is indicative for the suitability of a region of the body for determining the concentration of the body substance,

[0028] c) determining the concentration of the body substance of the person using the hyperspectral data and the region data, and

[0029] d) outputting the concentration of the body substance of the person.

[0030] In another aspect the invention relates to a method for determining a concentration of a body substance of a person comprising:

[0031] a) receiving hyperspectral data of the person from a hyperspectral camera,

[0032] b) determining the concentration of the body substance of the person using the hyperspectral data, and c) outputting the concentration of the body substance of the person.

[0033] In another aspect the invention relates to a use of the concentration of the body substance of the person obtained from the method of any of the previous claims for granting a user access to a device or application.

[0034] In another aspect the invention relates to a non-transient computer-readable medium including instructions that, when executed by one or more processors, cause the one or more processors to perform a method comprising: a) receiving hyperspectral data of the person from a hyperspectral camera, 240507W001

[0035] 3

[0036] b) determining region data from the hyperspectral data, wherein region data is indicative for the suitability of a region of the body for determining the concentration of the body substance,

[0037] c) determining the concentration of the body substance of the person using the hyperspectral data and the region data, and

[0038] d) outputting the concentration of the body substance of the person.

[0039] In another aspect the invention relates to a non-transient computer-readable medium including instructions that, when executed by one or more processors, cause the one or more processors to perform a method comprising: a) receiving hyperspectral data of the person from a hyperspectral camera,

[0040] b) determining the concentration of the body substance of the person using the hyperspectral data, and c) outputting the concentration of the body substance of the person.

[0041] The advantage of the present invention is that by using spectra primarily from suitable regions of the body, the concentration of the body substance can be determined more accurately without the need for any particular skill of the person using the spectroscopic device. By appropriate selection, disturbing factors like occlusion, for example by body hair, or unsuitable regions, for example corneal skin regions, the spectra contain more information about the body substance of interest. In this way, body substances of low concentration can be measured which are otherwise inaccessible. Further, the approach is more flexible as the selection of suitable regions can be easily combined with metadata, for example personal data, for example sex or ethnicity, or data relating to the surroundings, for example weather conditions. Depending on such influences, different regions may be better suited. In this case, the selection can be implemented in software, so there is no need for changes to hardware or new instructions to the person handling the spectroscopic device. It is also possible to measure the concentration of multiple different body substances from different body parts from one single measurement making the measurement more comfortable and quicker.

[0042] The spectroscopic device may be integrated into a vehicle including cars, motorcycles, buses, trucks, trams, trains or even airplanes, hence a vehicle may comprise a spectroscopic device. The spectroscopic device may be suitable for integration into a vehicle. The spectroscopic device may be attached to the vehicle, or it may be integrated as component or as part of a component of a vehicle, for example as part of a display in the dashboard, an entertainment control system, or loudspeakers. The spectroscopic device can be placed at various places, for example in the steering wheel and its periphery, such as the steering wheel rim, the steering wheel column, or the steering wheel center behind or besides the emblem; in the dashboard, such as in the instrument cluster bezel or its surrounding, the dashboard button panel or in or around frequently used buttons like the infotainment control button or engine start button, the touchscreen display in the center display of the infotainment system; the overhead and A-pillars, such as in the overhead console behind the light sensor, nestled behind the light sensor housing in the overhead console, the A-pillar trim on the driver side placed behind the A-pillar trim panel; the center console, such as the cup holder insert incorporated within a removable cup holder insert, the gear shift knob positioned on top or on the side of the gear shift knob, the arm rest, the parking break button.

[0043] The spectroscopic device may be integrated into consumer electronics products such as smartphones, tablets or, in particular, consumer electronic products which are worn on the body, for example a smartwatch, head phones, hearing aids, continuous glucose monitoring (CGM) systems, wearable medical devices, such as blood pressure, temperature, and oxygen levels, virtual reality headsets, smart glasses, skin patches, sleep trackers. The spectroscopic 240507W001

[0044] 4

[0045] device may be integrated into apparel, for example in a hat, a helmet, a shirt, a scarf, a belt, underwear, shoes, or socks.

[0046] The term "spectroscopic device” may refer to an apparatus which is capable of recording spectroscopic data of a person. The spectroscopic device may be a spectrometer or a device into which a spectrometer is integrated. The spectroscopic device may be portable or stationary, for example a laboratory device. A portable spectroscopic device may be a hand-held spectrometer or a module which is integrated into a portable device like a smartphone, a tablet or a wearable like a smartwatch. A portable spectroscopic device may be communicatively coupled to a computer device, for example a cloud computer or a smartphone. Such computer device may be configured to execute a chemometric model. The computer device may further be configured to receive spectroscopic data from the spectroscopic device. The computer device may store such spectroscopic data, or send it to a system for determining concentration of a body substance.

[0047] The spectroscopic device comprises a hyperspectral camera. The term "hyperspectral camera” may refer to a device or component which is capable of recording hyperspectral data of a scene. The term "hyperspectral data” may refer to a data structure comprising images of the scene for multiple wavelength channels. An image may comprise pixels representing the intensity for a wavelength channel at a certain spatial position in the scene. Hyperspectral data may comprise a three-dimensional data cube of the form (x, y, A), where x and y represent two spatial dimensions of the scene, and A represents a wavelength or a range of wavelength dimension of the electromagnetic spectrum.

[0048] Hyperspectral data may comprise values for at least five wavelength channels, for example at least ten, at least 20, at least 50 or at least 100, for example 5 to 1000 or 10 to 200. A wavelength channel may represent intensities of a particular wavelength or a wavelength range, for example ± 10 % of indicated wavelength or ± 5 % of indicated wavelength. Recording hyperspectral data may also be referred to as imaging spectroscopy or 3D spectroscopy. Hyperspectral data may comprise data from the entire body or parts of the body, for example the face.

[0049] The hyperspectral camera may record hyperspectral data of a scene by spatial scanning, spectral scanning, spatiospectral scanning or non-scanning. A hyperspectral camera recording hyperspectral data of a scene by spatial scanning may comprise a spectrometer for recording one spectrum at a time and scanning optics to direct light beams from different spatial positions of the scene onto the spectrometer. In this way, spectral data is successively recorded for each spatial position of the scene. A hyperspectral camera recording hyperspectral data of a scene by spectral scanning may comprise an array of light sensitive sensors and optics to direct light from different spatial positions of the scene into different sensors. In addition, such hyperspectral camera may comprise optics to direct different wavelengths onto the sensor array at different points in time, for example tunable wavelength filters. In this way, the hyperspectral camera successively records an image, wherein each image is at a particular wavelength at a particular point in time. A hyperspectral camera recording hyperspectral data of a scene by spatiospectral scanning may combine the approaches of spatial and spectral scanning, i.e. at any point in time recording only parts of the scene at a certain wavelength and systematically changing the part of the scene and the wavelength by respective optics. A hyperspectral camera recording hyperspectral data of a scene by non-scanning or snapshot may record the full hyperspectral data simultaneously, for example by optics which split the light beam for the entire scene into different wavelengths hitting a large sensor array, wherein each sensor is hit by light of a particular wavelength from a particular part of the scene. 240507W001

[0050] 5

[0051] The hyperspectral camera may comprise:

[0052] - a dispersive optical element for separating incident light into constituent wavelength components, - optionally a scanning optical element for directing light from a spatial location of the scene onto the photodetector,

[0053] - a photodetector for generating a photodetector signal dependent on an illumination intensity, and

[0054] - a controller for controlling the photodetector and collecting data from the photodetector.

[0055] The term "dispersive optical element” may refer to an element for influencing light depending on the wavelength. The optical element may be configured for at least one of at least partially dispersing the light, at least partially filtering the light, at least partially reflecting the light, e.g. diffusely or directly, at least partially deflecting the light, at least partially transmitting the light and at least partially absorbing the light. The dispersive optical element may comprise at least one of a prism, a grating, a beam splitter, or an interferometer, for example a Michelson interferometer. The dispersive optical element may be configured for being used in mobile applications, for example for being used in handheld spectrometer devices and / or in spectrometer devices comprised by electronic communication devices, such as a smartphone or a tablet. As another example, the dispersive optical element may comprise at least one optical filter element. The optical filter element may be configured for filtering the light or more specifically at least one selected spectral range of the light. The optical filter element may specifically be positioned in a light path before the photodetector. The photodetector may comprise an array of pixels, wherein the optical filter element may filter the incoming light in a way that different wavelengths or wavelengths regions impinge on different pixels. This can be achieved by a linear variable filter or by a set of filters filtering light at different wavelengths, wherein each filter is placed before a different part the photodetector with regard to the path of the incoming light. Alternatively, a tunable optical filter can be used, for example an acousto-optic tunable filter, Liquid Crystal Tunable Filter, or a tunable filter based on micro-electro-mechanical systems (MEMS).

[0056] The term "scanning optical element” may refer to an element for influencing the direction of light. A scanning optical element may be used by a hyperspectral camera which records hyperspectral data of a scene by spatial scanning. An example for a scanning optical element is a movable mirror.

[0057] The term "photodetector” may refer to an element for generating hyperspectral data from the incoming light. The photodetector may comprise at least one photosensitive region, in particular a line or an array of photosensitive regions, for example an array comprising 10 to 10000 photosensitive regions such as 100 to 1000 photosensitive regions. The photosensitive region may be configured for receiving the light from the dispersive optical element. The photodetector may be configured for generating at least one photodetector signal dependent on an illumination of the photosensitive region by the light. The photodetector may comprise a charge-coupled devices (CCD), for example a silicon-based CCD, or a complementary metal oxide semiconductor (CMOS) device, for example a silicon-based CMOS.

[0058] The term "controller” may refer to an electronic device or component which controls the photodetector and collects the hyperspectral data from the photodetector. Controlling may include setting parameters which affect the acquisition of the hyperspectral data, for example setting exposure times, readout rates, or gain settings. Controlling may further include handling a calibration process. A calibration process may ensure accurate spectral calibration for subsequent data analysis. A calibration process may comprise capturing reference spectra, correcting for sensor 240507W001

[0059] 6

[0060] response, compensating for any optical aberrations or distortions. Controlling may comprise synchronizing the photodetector readout with the scanning mechanism to ensure proper spatial and spectral registration. Collecting hyperspectral data may comprise readout of the intensity values from the photodetector and associating corresponding wavelengths, for example including the setting of tunable filters or the location of pixels with regard to dispersive optical elements. Collecting hyperspectral data may comprise data storage and transfer, such as storing the acquired data in an appropriate format or data structure, handling data compression or preprocessing tasks to optimize storage efficiency and transferring the data to a storage medium or processor for further processing.

[0061] The hyperspectral camera may comprise a polarizer. A polarizer may remove any light that does not have a selected polarization. In particular if used together with a light source emitting polarized light, the polarizer may help in removing stray light and thus increase a signal-to-noise ratio. A polarizer may be a polarizing beamsplitter, a thin film polarizer or a rotating polarizer.

[0062] The spectroscopic device may comprise a light emitting element for emitting illumination light for illuminating the person in order to generate detection light from the person. The light emitting element may comprise an incandescent lamp, for example a tungsten filament lamp or a tungsten halogen lamp, a light-emitting diode (LED), a laser diode, a gas-discharge lamp, for example a xenon lamp, a mercury vapor lamp, or a deuterium lamp. Examples for lasers comprise semi-conductor laser, double heterostructure laser, external cavity laser, separate confinement heterostructure laser, quantum cascade laser, distributed Bragg reflector laser, polariton laser, hybrid silicon laser, extended cavity diode laser, quantum dot laser, volume Bragg grating laser, Indium Arsenide laser, Gallium Arsenide laser, transistor laser, diode pumped laser, distributed feedback laser, quantum well laser, interband cascade laser, semiconductor ring laser, vertical cavity surface emitting laser (VCSEL). The light emitting element may comprise a plurality of said light emitters, for example a VCSEL array.

[0063] The light emitting element may emit light of a narrow wavelength distribution, i.e. the difference between the lowest and highest wavelength for which the light emitting elements emits at least 50 % of its maximum intensity, for example a spectral bandwidth of 1 to 20 nm. The light emitting element may emit light of a broad wavelength distribution, for example a spectral bandwidth of 50 to 2500 nm, such as 100 to 1000 nm. The light emitting element may emit the same wavelength range over time, or it may change its wavelength range, for example with a tunable wavelength filter. The light emitting element may be a pattern light emitting element, a floodlight light emitting element or both either simultaneously or the light emitting element may repeatedly switch from illuminating patterned light to floodlight. The term "floodlight” may refer to a light beam with essentially uniform illumination intensity.

[0064] The term "pattern light emitting element” may refer to a device configured for generating or providing at least one light pattern, in particular at least one infrared light pattern. The term "light pattern” may refer to at least one pattern comprising a plurality of light spots. The light spot may be at least partially spatially extended. At least one spot or any spot may have an arbitrary shape. In some cases, a circular shape of at least one spot or any spot may be preferred. The spots may be arranged by considering a structure of a display comprised by a device that is further comprising the optoelectronic apparatus. Typically, an arrangement of an OLED-pixel-structure of the display may be considered. The term "infrared light pattern” may refer to a light pattern comprising spots in the infrared spectral range. The infrared light pattern may be a near infrared light pattern. The infrared light may be coherent. The infrared light pattern may be a coherent infrared light pattern. 240507W001

[0065] 7

[0066] Patterned coherent light may relate to a plurality of light beams of coherent light. Patterned coherent light may comprise a plurality of light beams, e.g. at least two light beams, preferably at least two light beams. Projection of a light beam of the patterned coherent light onto a surface may result in a light spot. Followingly, projecting a plurality of light beams of the patterned coherent light onto the object may result in a plurality of light spots on the object(s). The number of light spots may be equal to the number of light beams associated with the patterned coherent light. The one or more light spots may be shown in the speckle image. A projection of patterned coherent light onto a regular surface may result in a light spot projected onto the regular surface independent of speckle. A projection of patterned coherent light onto a regular surface may result in a light spot projected onto the irregular surface comprising at least one speckle, preferably a plurality of speckles. Object may be associated with an at least partially irregular surface. Therefore, the speckle image may comprise a plurality of speckles. If the object may comprise at least partially skin, a plurality of speckle is formed due to the interference of the coherent light. Followingly, a light spot may comprise zero, one or more speckle depending on the surface the patterned coherent light is projected on. Skin may have an irregular surface. Hence the projection of patterned coherent light may result in the formation of speckle within the one or more light spots. A light spot may be a result of the projection of a light beam associated with the patterned coherent light. A light spot refers to an arbitrarily shaped spot of coherent light. A light spot may refer to a contiguous area illuminated with coherent light. Projecting coherent light on an irregular surface may result in the formation of speckle. Followingly, the light spot may comprise one or more speckle(s). A light spot may have a diameter between 0.5 mm and 5 cm, preferably 0.6 mm and 4 cm, more preferably, 0.7 mm and 3 cm, most preferably 0.4 and 2 cm.

[0067] The pattern light emitting element may be configured for emitting monochromatic light, e.g. in the near infrared region. The term "monochromatic” may refer to light with a wavelength accuracy of less or equal to ± 2 % or less or equal to ± 1 %. The wavelength accuracy may be the maximum difference of emitted wavelength relative to the mean wavelength. In other embodiments, the pattern light emitting element may be adapted to emit light with a plurality of wavelengths, e.g. for allowing additional measurements in other wavelengths channels.

[0068] The infrared light pattern may comprise at least one regular and / or constant and / or periodic pattern such as a triangular pattern, a rectangular pattern, a hexagonal pattern or a pattern comprising further convex tilings. For example, the infrared light pattern is a hexagonal pattern, preferably a hexagonal infrared light pattern. The illumination pattern may comprise a number of rows on which the illumination features are arranged in equidistant positions with distance d. The rows may be orthogonal with respect to the epipolar lines. A distance between the rows may be constant. A different offset may be applied to each of the rows in the same direction. The offset may result in that the illumination features of a row are shifted. The offset 5 may be 5= a / b, wherein a and b are positive integer numbers such that the illumination pattern is a periodic pattern. For example, 5 may be 1 / 3 or 2 / 5. Using a periodical pattern with said offset can allow distinguishing between artefacts and usable signal.

[0069] The light pattern may comprise less than 4000 spots, for example less than 3000 spots or less than 2000 spots or less than 1500 spots or less than 1000 spots. The light pattern may comprise patterned coherent infrared light of less than 4000 spots or less than 3000 spots or less than 2000 spots or less than 1500 spots or less than 1000 spots.

[0070] At least one of the infrared light spots may be associated with a beam divergence of 0.2° to 0.5°, preferably 0.1° to 0.3°. The term "beam divergence” may refer to at least one measure of an increase in at least one diameter and / or at least one diameter equivalent, such as a radius, with a distance from an optical aperture from which the beam 240507W001

[0071] 8

[0072] emerges. The measure may be an angle or an angle equivalent. In the context of the present invention, typically, a beam divergence may be determined at 1 / e2.

[0073] The pattern light emitting element may comprise at least one optical element configured for increasing, e.g. duplicating, the number of spots generated by the pattern light emitting element. The pattern light emitting element, particularly the optical element, may comprises at least one diffractive optical element (DOE) and / or at least one meta surface element. The DOE and / or the meta surface element may be configured for generating multiple light beams from a single incoming light beam. Further arrangements, particularly comprising a different number of projecting VCSEL and / or at least one different optical element configured for increasing the number of spots may be possible. Other multiplication factors are possible. For example, a VCSEL or a plurality of VCSELs may be used and the generated laser spots may be duplicated by using at least one DOE.

[0074] The pattern light emitting element may comprise at least one transfer device. The term "transfer device”, also denoted as "transfer system” may refer to one or more optical elements which are adapted to modify the light beam, particularly the light beam used for generating at least a portion of the infrared light pattern, such as by modifying one or more of a beam parameter of the light beam, a width of the light beam or a direction of the light beam. The transfer device may comprise at least one imaging optical device. The transfer device specifically may comprise one or more of: at least one lens, for example at least one lens selected from the group consisting of at least one focus-tunable lens, at least one aspheric lens, at least one spherical lens, at least one Fresnel lens; at least one diffractive optical element; at least one concave mirror; at least one beam deflection element, preferably at least one mirror; at least one beam splitting element, preferably at least one of a beam splitting cube or a beam splitting mirror; at least one multi lens system; at least one holographic optical element; at least one meta optical element. Specifically, the transfer device comprises at least one refractive optical lens stack. Thus, the transfer device may comprise a multilens system having refractive properties.

[0075] The pattern light emitting element may be configured for emitting modulated or non-modulated light. In case a plurality of emitters is used, the different emitters may have different modulation frequencies, e.g. which can be used for distinguishing the light beams.

[0076] The light beam or light beams generated by the pattern light emitting element may propagate parallel to an optical axis. The pattern light emitting element may comprise at least one reflective element, preferably at least one prism, for deflecting the illuminating light beam onto the optical axis. As an example, the light beam or light beams, such as the laser light beam, and the optical axis may include an angle of less than 10°, preferably less than 5° or even less than 2°. Other embodiments, however, are feasible. Further, the light beam or light beams may be on the optical axis or off the optical axis. As an example, the light beam or light beams may be parallel to the optical axis having a distance of less 10 than 10 mm to the optical axis, preferably less than 5 mm to the optical axis or even less than 1 mm to the optical axis or may even coincide with the optical axis.

[0077] The term "flood light emitting element” may refer to at least one device configured for providing substantially continuous spatial illumination. The flood light emitting element may illuminate a measurement area, such as a user, a portion of the user and / or a face of the user, with a spatially constant or essentially constant illumination intensity. The term "flood light” may refer to substantially continuous spatial illumination, in particular diffuse and / or uniform illumination. The flood light has a wavelength in the infrared range, in particular in the near infrared range. The flood 240507W001

[0078] 9

[0079] light emitting element may comprise at least one least one VCSEL, preferably a plurality of VCSELs, for example an array of VCSELs. The term "substantially continuous spatial illumination” may refer to uniform spatial illumination, wherein areas of non-uniform are possible.

[0080] A relative distance between the flood light emitting element and the pattern light emitting element may be below 3.0 mm. The relative distance between the flood light emitting element and the pattern light emitting element may be below 2.5 mm, preferably below 2.0 mm. The pattern light emitting element and the flood light emitting element may be combined into one module. For example, the pattern light emitting element and the flood light emitting element may be arranged on the same substrate, in particular having a minimum relative distance. The minimum relative distance may be defined by a physical extension of the flood light emitting element and the pattern light emitting element. Arranging the pattern light emitting element and the flood light emitting element having a relative distance below 3.0 mm can result in decreased space requirement of the two light emitting elements. In particular, said light emitting elements can even be combined into one module. Such a reduced space requirement can allow reducing the transparent area(s) in a display necessary for operation of the light emitting element(s) behind the display.

[0081] The pattern light emitting element and the flood light emitting element may comprise at least one VCSEL, preferably a plurality of VCSELs, for example an array of VCSELs. The pattern light emitting element may comprise a plurality of first VCSELs mounted on a first platform. The flood light emitting element may comprise a plurality of second VCSELs mounted on a second platform. The second platform may be beside the first platform. The optoelectronic apparatus may comprise a heat sink. Above the heat sink a first increment comprising the first platform may be attached. Above the heat sink a second increment comprising the second platform may be attached. The second increment may be different from the first increment. Thus, the first platform may be more distant to the optical element configured for increasing, e.g. duplicating, the number of spots. The second platform may be closer to the optical element. The beam emitted from the second VCSEL may be defocused and thus, form overlapping spots. This leads to a substantially continuous illumination and, thus, to flood illumination.

[0082] The term "light” may refer to electromagnetic radiation in one or more of the infrared, the visible and the ultraviolet spectral range. The term "ultraviolet spectral range” may refer to electromagnetic radiation having a wavelength of 1 nm to 380 nm, preferably of 100 nm to 380 nm, for example 280 nm to 315 nm (UV-B) or 315 nm to 380 nm (UV-A). Further, in partial accordance with standard ISO-21348 in a valid version at the date of this document, the term "visible spectral range” may refer to a spectral range of 380 nm to 760 nm. The term "infrared spectral range” (IR) may refer to electromagnetic radiation of 760 nm to 1000 pm, wherein the range of 760 nm to 1.5 pm is usually denominated as "near infrared spectral range” (NIR) while the range from 1.5 p to 15 pm is denoted as "mid infrared spectral range” (MidlR) and the range from 15 pm to 1000 pm as "far infrared spectral range” (FIR). Preferably, light used for the typical purposes of the present invention is light in the infrared (IR) spectral range, more preferred, in the near infrared (NIR) and / or the mid infrared spectral range (MidlR), especially the light having a wavelength of 750 nm to 2.5 pm, for example 780 nm to 1.4 pm or 1.4 pm to 2.5 pm. These wavelength regions are particularly suitable for obtaining material properties of a person.

[0083] The spectroscopic device may contain or be placed behind a transparent display. The term "display” may refer to an arbitrary shaped device configured for displaying an item of information. The item of information may be arbitrary information such as at least one image, at least one diagram, at least one histogram, at least one graphic, text, 240507W001

[0084] 10

[0085] numbers, at least one sign, or an operating menu. The display may be or may comprise at least one screen. The display may have an arbitrary shape, e.g. a rectangular shape. The display may be a front display of a device.

[0086] The display may be or may comprise at least one organic light-emitting diode (OLED) display. The term "organic light emitting diode” may refer to a light-emitting diode (LED) in which an emissive electroluminescent layer is a film of organic compound configured for emitting light in response to an electric current. The OLED display may be configured for emitting visible light. The display, particularly a display area, may be covered by glass. In particular, the display may comprise at least one glass cover.

[0087] The transparent display may be at least partially transparent. The term "at least partially transparent” may refer to a property of the display to allow light, in particular of a certain wavelength range, e.g. in the infrared spectral region, in particular in the near infrared spectral region, to pass at least partially through. For example, the display may be semitransparent in the near infrared region. For example, the display may have a transparency of 20 % to 50 % in the near infrared region. The display may have a different transparency for other wavelength ranges. For example, the display may have a transparency of > 80 % for the visible spectral range, preferably > 90 % for the visible spectral range. The transparent display may be at least partially transparent over the entire display area or only parts thereof. Typically, it is sufficient if only those parts of the display area are at least partially transparent trough which light needs to pass from the projector or to the camera.

[0088] The display may comprise a display area. The term "display area” may refer to an active area of the display, in particular an area which is activatable. The display may have additional areas such as recesses or cutouts. The display may have a first area associated with a first pixel per inch (PPI) value and a second area associated with a second PPI value. The first PPI value may be lower than the second PPI value, preferably first PPI value is equal to or below 400 PPI, more preferably the second PPI value may be equal to or higher than 300 PPI. The first PPI value may be associated with the at least one continuous area being at least partially transparent.

[0089] The spectroscopy module may be positioned such that it can illuminate the person with light through the transparent display. The transparent display may hence be positioned between the light emitting element and the person. The spectroscopy module may be positioned such that it can receive light from the person through the transparent display. The transparent display may hence be positioned between the hyperspectral camera and the person. Light reflected or refracted from the person firstly crosses the transparent display before it impinges on the hyperspectral camera of the spectroscopy module. From the person's view, the light emitting element and the hyperspectral camera may be placed behind the transparent display.

[0090] The term "body substance” may refer to any chemical substance which can be found in a human body, in particular in the skin, blood or interstitial fluid of a human body. The body substance may be indicative of the person's fitness to drive a vehicle, the body substance may, for example, reduce the concentration of a person or may be a metabolite of such substance. The body substance may be indicative for a health or fitness condition which compromises the person's fitness, for example a low hydration level or an irregular blood glucose concentration. Body substance may comprise proteins, such as enzymes, antibodies, or hormones; carbohydrates, such as glucose, glycogen, or fructose; lipids, such as triglycerides, cholesterol, and phospholipids; water; nucleic acids, such as DNA or RNA; amino acids, such as alanine, glutamic acid, cysteine; neurotransmitters, such as dopamine, serotonin, and acetylcholine; hormones, such as insulin, estrogen, or testosterone; electrolytes, such as sodium, potassium, or 240507W001

[0091] 11

[0092] calcium ions; vitamins, such as ascorbic acid, calciferol, cobalamin; metabolites, such as lactate, urea, and creatinine.

[0093] Body substance may be an intoxicant or its metabolite including ethanol, opioids, such as heroin, morphine, fentanyl; stimulants, such as amphetamine, methylphenidate, cocaine; benzodiazepines, such as diazepam, or alprazolam; cannabinoids, such as tetrahydrocannabinol (THC); barbiturates, such as phenobarbital; hallucinogens, such as lysergic acid diethylamide (LSD) or psilocybin; antihistamines, such as diphenhydramine; antipsychotics and antidepressants, such as fluoxetine or amitriptyline; muscle relaxants, such as carisoprodol or cyclobenzaprine; pain killers, such as tramadol, codeine, ibuprofen, naproxen, cyclobenzaprine, or methocarbamol.

[0094] The hyperspectral data may be received from the spectroscopic device of the present invention. The hyperspectral data may be received directly from a spectroscopic device or indirectly, i.e. from a storage device to which the hyperspectral data have been stored after the measurement. A spectroscopic measurement may be triggered by a predefined event, for example when the person seeks access to a device or vehicle or when the vehicle is switched on, before the engine is started, or after a certain period of time. A spectroscopic measurement may be triggered when a measurement trigger event occurs. A measurement trigger event may be a situation in which an indicator indicates the necessity for a spectroscopic measurement necessary. A measurement trigger event may occur when an indicator indicates that the person's fitness to drive is potentially compromised, for example due to intoxicants such as alcohol or drugs, due to a health problem, for example low sugar concentration of a diabetes patient, or due to fitness problems like dehydration. The measurement trigger event may be determined using person data and / or environmental data. For example, the person data and / or environmental data may indicate an increased likelihood that the person's fitness to drive the vehicle are compromised, such as slow pupil reflex recorded by an optical camera, unusual movement patterns recorded by a pressure sensor, or certain voice characteristics recorded by a microphone. Triggering a spectroscopic measurement in such cases may be particularly useful if the body substance is used for access control of the vehicle, for example to keep drunk persons from driving without burdening obviously sober persons with a measurement.

[0095] Region data is determined from the hyperspectral data. The term "region data” may refer to data which is indicative for the suitability of a region of the body for determining the concentration of the body substance. The suitability may refer to the probability that an accurate concentration of the body substance can be determined from the region of the body. The body substance may not be evenly distributed over the body, so regions in which a higher concentration of the body substance can be expected may be more suitable than regions in which a lower concentration of the body substance can be expected. Hence, the suitability may correspond to the expected concentration of the body substance at a region of the body. For example, if the body substance can be found in the blood, body parts containing a high number of blood vessels close to the surface of skin may have a higher suitability for determining the concentration of the body substance. Hence, the lips may be more suitable than the nose. In addition, certain regions of the body may be partially or fully covered, for example by hair, by clothing, glasses, rings or masks. The suitability may hence refer to the probability that the region exposes a tissue containing the body substance, for example skin. The region data may comprise the position of a body part, for example the lips. For example, the hyperspectal data may comprise data of the entire face. This hyperspectral data may be used to determine the position of the lips, for example by segmenting the hyperspectral data of the face. 240507W001

[0096] 12

[0097] The region data may for example comprise a vector or an array of values indicative for the suitability of a region of the body for determining the concentration of the body substance. A region may refer to one pixel or to a range of pixels in the image, in particular a contiguous area of adjacent pixels in the image. Generally, a region may be smaller than the body in the image. The value may be a Boolean value or a float value, for example a float value between 0 and 1, wherein 0 may indicate complete unsuitability for determining the concentration of the body substance and a 1 may indicate perfect suitability for determining the concentration of the body substance.

[0098] Alternatively, the region data may comprise a list of those regions for which the suitability for determining the concentration of the body substance exceeds a threshold value. Hence, the presence in the list indicates for the respective region a high probability and the absence a low probability. In case of more than one body substance, the region data may comprise a matrix, wherein a row may refer to a body substance and the column may refer to the region, wherein each entry indicates the suitability for determining the concentration of the body substance.

[0099] The determination of the region data may comprise an image segmentation algorithm. Image segmentation may comprise a global or local thresholding algorithm, i.e. replacing each pixel in an image with a black pixel if the image intensity is less than a threshold value or white otherwise. An example for a thresholding algorithm is the Otsu's method. Image segmentation may comprise clustering methods, for example K-means clustering based on pixel color, intensity, texture, and location, or a weighted combination of these factors. Image segmentation may comprise motion and interactive segmentation, i.e. by comparing a pair of images and identifying a segment based on the same differences. Image segmentation may comprise a compression-based method, i.e. a segmentation by minimizing the coding length of the image data. Image segmentation may comprise a histogram-based method, i.e. using peaks and valleys in the histogram to locate segments in the image. Image segmentation may comprise edge detection, for example the Canny edge detection algorithm. Image segmentation may comprise isolated point detection, for example using the Laplacian operator. Image segmentation may comprise a dual clustering method, a region-growing method, a partial differential equation-based method like parametric methods, level-set methods or fast marching methods, a variational method like the Potts model, a graph partitioning method like Markov random fields, normalized cuts, random walker, minimum cut, isoperimetric partitioning, minimum spanning tree-based segmentation, or segmentation-based object categorization, a watershed transformation, a model-based segmentation, a multi-scale segmentation, semi-automatic segmentation, trainable segmentation like a Kohonen map, pulse-coupled neural networks, convolutional neural networks, autoencoders. Image segmentation may comprise an active appearance model or an active shape model. Specific examples include YOLO, UNet, Mask R-CNN or models derived therefrom.

[0100] The determination of the region data may comprise attributing the regions an indicator indicating suitability of a region of the body for determining the concentration of the body substance. Determination of the region data may comprise determining the concentration of the body substance of interest for pixels or groups of pixels and selecting those pixels or group of pixels for which the concentration satisfies a preset criterion, for example the determined concentration exceeds a certain threshold or is within an expected range. Determination of the region data may comprise using a body topology map which indicates the suitability of regions of the body for the determination of the concentration of a body substance. The body topology map may comprise multiple suitability values for a region for different body substances. For example, for a face, the body topology map may represent a high suitability of a region of the body for determining the concentration of the body substance for the front, the cheeks and the lips, but a low suitability for the nose, the eyebrows or the chin. The body topology map may comprise multiple suitability values for a region of the body for different body substances. 240507W001

[0101] 13

[0102] Region data is determined from hyperspectral data. An image may be extracted from the hyperspectral data, for example intensity values for one wavelength channel or an average of parts or all of the wavelength channels. The wavelength cannels may be selected depending on the spectra of the body substance to be determined. Selection may be effected by multiplying the hyperspectral data with a weight matrix or vector, wherein each element corresponds to a desired contribution of a wavelength channel, followed by averaging the resulting data with regard to the wavelength channels.

[0103] Hyperspectral data may be recorded using the region data. The hyperspectral camera may be triggered to record hyperspectral data of only the region or regions indicated by the region data. For example, the optics may zoom into the scene to only capture the regions indicated by the region data or they may be recorded in higher resolution or with more wavelength channels. Alternatively, or additionally the hyperspectral camera may be triggered to record hyperspectral data of the person while the light emitting element is focused on the region indicated by the region data, for example to illuminate the regions indicated by the region data with higher intensity. The hyperspectral data recorded using the region data may be used for determining the concentration of the body substance, for example by inputting such hyperspectral data to a chemometric model. Hence, first hyperspectral data may be used to determine region data, second hyperspectral data may be recorded using the region data, and the concentration of the body substance may be determined from the second hyperspectral data.

[0104] The concentration of a body substance of the person is determined using the hyperspectral data and the region data. A subset of the hyperspectral data may be determined according to the region data. For example, a subset may contain only some pixels of the hyperspectral data selected according to the region data. The concentration of the body substance of the person may be determined using the subset of the hyperspectral data, for example by inputting the subset of the hyperspectral data into a chemometric model. Alternatively, a chemometric model may be parametrized to receive the hyperspectral data and the region data as input and output the concentration of the body substance of the person. Alternatively, the hyperspectral data may be aggregated into a spectrum using the region data. Such aggregated spectrum may be input to a chemometric model which outputs the concentration of the body substance. For example, the subset of the hyperspectral data described above may be averaged or the hyperspectral data may be weight-averaged according to the region data.

[0105] The term "chemometric model” may refer to a model which is parameterized to receive hyperspectral data as input and output the concentration of a body substance. The chemometric model may be parameterized to receive spectroscopic data and person data as input and output the concentration of a body substance. The chemometric model may be parameterized to receive spectroscopic data and environmental data as input and output the concentration of a body substance. The chemometric model may be parameterized to receive spectroscopic data, person data and environmental data as input and output the concentration of a body substance. The chemometric model may be parameterized to receive spectroscopic data as input and output an intermediate concentration of a body substance. The intermediate concentration of a body substance may be adjusted or corrected using the person data and / or the environmental data, for example by employing a refining model. The refining model may be a data-driven model which may be trained with historic data for adjusting or correcting the intermediate concentration of a body substance. A refining model may be a multivariate linear or polynomial regression model, or it may be an artificial neural network. 240507W001

[0106] 14

[0107] A chemometric model may comprise a pre-processing method and a machine learning model to obtain the concentration of a body substance. A chemometric model may comprise a pre-processing method, a feature selection filter and a machine learning model. If the chemometric model comprises two or more partial chemometric models, each partial chemometric model may comprise a separate pre-processing method, a feature selection filter and a machine learning model. Alternatively, the partial models may use the same pre-processing method or feature selection filter.

[0108] The term "pre-processing” may refer to a method to reduce or eliminate interferences from a spectrum such as stray light, noise or baseline drift to enhance the subsequent machine learning. Hence, the pre-processing method may be applied before the machine learning method. Pre-processing may include one or more of baseline correction, scatter correction, smoothing, scaling, aggregation.

[0109] The term "machine learning method” may refer to a model which translates spectra into corresponding person data. The machine learning method hence may use a spectrum as input and derive person data therefrom. The machine learning method may be considered as an integral part of the chemometric model. Machine learning methods may be supervised, semi-supervised or unsupervised. Machine learning methods may include multivariate calibration, classification, pattern recognition, clustering, ensemble methods, neural nets and deep learning, or multivariate curve resolution.

[0110] The term "feature selection filter” may refer to a method to select those parts of the spectrum with a correlation to the person data. A feature selection filter may facilitate the machine learning method of the chemometric model and thus avoid overfitting and reduce the number of required training datasets. A feature selection filter may use a spectrum as input, remove all unselected parts and output a spectrum with only the selected parts left. Hence, the output of the feature selection filter may be a spectrum in form of a vector of lower dimensionality than the input vector. The output of the feature selection filter can be used as input for the machine learning method. Hence, the feature selection filter may be applied before the machine learning method. The input of the feature selection filter may be the received spectrum or it may be the pre-processed spectrum, preferably the pre-processed spectrum. Hence, the feature selection filter may be applied after the pre-processing method.

[0111] A chemometric model may be or may contain a data-driven model. The chemometric model may be a trained data-driven model. Training may comprise adjusting parameters of the chemometric model such that the output of the chemometric model most closely fits to the provided training data. Often, training comprises minimizing a loss or cost function, for example a least mean square value of chemometric model output to provided training data. The complete set of training data may be used for training or parts thereof. Parts of the received training data may be used for training and the remainder may be used for determining the prediction accuracy of the trained chemometric model. Alternatively, cross-validation can be applied, for example K-fold cross-validation, leave-one-out cross-validation, stratified cross-validation.

[0112] The concentration of a body substance of the person may be determined using the hyperspectral data, the region data and the person data. The concentration of a body substance of the person may be determined using the hyperspectral data, the region data and the environmental data. The concentration of a body substance of the person may be determined using the hyperspectral data, the region data, the person data and the environmental data. The concentration may be a numeric value, such as mass ratio or a volume ratio. The ratio may relate to the whole body 240507W001

[0113] 15

[0114] or parts thereof, for example the skin or the blood. For example, in case of alcohol the blood alcohol concentration may be determined. The concentration may be a categoric value, for example indicating the presence of the body substance or certain value ranges, for example none, low, medium, high.

[0115] The term "person data” may refer to data associated with a characteristic of the person such as a physical or chemical characteristic of the person. Person data may refer to any data associated with a characteristic of the person which has been obtained with a method other than spectroscopy. Person data may correlate with the alcohol level of the person. Person data may be personalized data, i.e. specific for a particular person, or it may be data associated with a certain group of people, for example female persons of age 25 to 30. Physical characteristics may comprise thermal characteristics, for example the body temperature, the thermal conductivity or the specific heat capacity of the skin; mechanical characteristics, for example pressure exerted on the spectrometer, compressibility or mechanical elasticity of the skin; optical characteristics, for example the color, refractive index, optical conductivity or absorption coefficients of the skin; electro-magnetic characteristics, for example electrical conductivity, dielectric constant, radio frequency-based permittivity, microwave complex permittivity, millimeter wave complex permittivity, magnetic permittivity or susceptibility of the skin. Chemical characteristics of a person typically refer to the chemical composition of some body tissue like skin, blood or sweat, for example the type and the concentration of certain chemical compounds such as the water content.

[0116] The person data may contain or may be a biomarker. The term "biomarker” may refer to a measurable substance, process or characteristic that is indicative of a biological state or condition. A biomarker may refer to a specific molecule, protein, genetic sequence, or other measurable feature that is associated with a particular disease, condition or treatment response. Examples for biomarkers are body dimensions such as size, head circumference, chest girth, abdominal girth, crotch length, arm length; body weight or body mass index; body topology such as face topology, iris structure, finger print, palm topology; muscle measures like muscular strength, muscular endurance, muscular agility and speed, balance, coordination; cardio-vascular measures such as heart rate, heart rate variability, electrocardiogram, blood pressure, blood oxygen; skin measures such as skin conductance, skin impedance, skin moisture level, skin sebum level, skin roughness, skin elasticity, skin pH, skin blood flow, skin sweat rate; blood metabolites such as blood glucose, blood cholesterol, blood triglycerides, blood urea, blood creatinine, blood lactate, blood bilirubin, blood pH; urine metabolites such as urine glucose, urine urea, urine creatinine, urine ketones, urine pH, urine protein content; hormone levels such as thyroid hormone level, insulin level, growth hormone level, cortisol level, estrogen level, progesterone level, testosterone level, prolactin level; drug levels or levels of drug metabolites such as alcohol, amphetamines, opioids, cocaine, marijuana, benzodiazepines, barbiturates.

[0117] Person data may be received from sensors other than a spectrometer or a hyperspectral camera, for example a thermometer, a scale, a balance, an optical camera, an optical 3D scanner system, conductance or impedance gauge such as a corneometer, a sweat rate monitor or sweat patch, a liquid or gas chromatograph, a mass spectrograph, a nuclear magnetic spectrometer or imager, an electrochemical sensor, an immunoassay, a polymerase chain reaction apparatus. The spectroscopic device may be integrated into a portable device which further comprises sensors from which at least parts of the person data is received. Person data may be the temperature measured by the temperature sensor of the spectroscopic device.

[0118] Person data may also be received from a storage device, or it can be obtained from a user interface, for example a graphical user interface, to which a user can enter person data, for example from observations. Person data may 240507W001

[0119] 16

[0120] comprise human characteristics like age, sex, origin, ethnicity; medical history including current and former medications; nutrition such as vegetarian or vegan diet; consumption of stimulants such as caffeine, alcohol, tobacco products, drug; physical activity level such as type of profession, i.e. office job or physically demanding job, kind of sports, average duration of sports, average sleeping hours.

[0121] The term "environmental data” may refer to data associated with a characteristic of the surrounding of the person, for example a physical or chemical characteristic of the surrounding of the person. The characteristic of the surrounding of the person may have an influence on the spectroscopic measurement of the person or on the characteristic of the person such as the physical or chemical characteristic of the person. However, environmental data may not comprise an intrinsic characteristic of the person.

[0122] Environmental data may comprise sensor data from sensors other than a spectrometer or a hyperspectral camera. Environmental data may comprise the location of the person, for example the geolocation such as the GPC coordinates, the height above see level, distance to a reference point such as the spectrometer, acceleration, orientation with regard to gravity; weather conditions such air temperature, air pressure, air humidity, wind speed, wind direction, ambient light intensity; time or date; air pollutant levels like CO2 concentration, CO concentration, ozone concentration, nitrogen oxide concentration, sulfur dioxide concentration, fine dust concentration, volatile organic compounds level.

[0123] Sensor data may have been recorded by a sensor capable of determining the sensor data. The sensor may be integrated into the spectroscopic device. The spectroscopic device may be integrated into a portable device which further comprises sensors from which at least parts of the environmental data is received. The sensor may be communicatively coupled to the spectroscopic device, for example via a wireless communication or via internet. Examples for sensors may be a GPC receiver, an accelerometer, a gyroscope, an altimeter, a goniometer, a distance sensor like a time-of-flight sensor, a radar or a LiDaR, a pressure sensor such as a MEMS sensor, a piezo sensor or a capacitive sensor, a magnetometer, a barometer, a light sensor, a thermometer, a gas sensor.

[0124] Environmental data may comprise data associated with the spectroscopic device, for example a spectroscopic device ID, a version number of the spectroscopic device, the spectroscopic device settings, the temperature of the spectroscopic device, the age of the spectroscopic device, time since the last calibration was performed, age of the illumination source, number of measurements the spectroscopic device has already performed in its lifetime or within a certain time such as the last week or the last month. Environmental data may further comprise data associated with the spectroscopic measurement of the person, for example the sampling time, the illumination strength with which the spectrometer illuminates the person, or the distance of the person to the spectroscopic device.

[0125] Environmental data may be received from a data storage medium. The data storage medium may be part of the spectroscopic device, or it may be a remote storage device, for example a computer system or a cloud system. Environmental data may be received from a database, for example from a database on a remote storage system, in response to a request containing time and / or geographic location. A remote storage system may refer to a system which is far from the person of the measurement, for example a cloud server or a database server. For example, a request containing the GPS coordinates of the person and the time of the spectroscopic measurement may be sent to a cloud server having a weather database. The cloud server may in response to the request send weather data corresponding to the time and location of the request. 240507W001

[0126] 17

[0127] Hyperspectral data may be used for biometric recognition of the person. The term "recognition” or "identification” may refer to identity check and / or verifying an identity of the person. Biometric recognition may comprise analyzing a flood image. Biometric recognition may comprise analyzing one or more than one image of the hyperspectral data. The image may be an image of one wavelength channel or an average of images of different wavelength channels, for example a weight-average images of different wavelength channels. Biometric recognition may comprise performing a face verification of the imaged face to be the user's face. Biometric recognition may comprise matching the flood image, e.g. showing a contour of parts of the user, in particular parts of the user's face, with a template. Determining if the imaged face is the face of the user may comprise identifying the user, in particular determining if the imaged face corresponds to at least one image of the user's face stored in at least one memory, e.g. of the device.

[0128] Biometric recognition may comprise one or more of the following: a filtering; a selection of at least one region of interest; a formation of a difference image between the flood image and at least one offset; an inversion of flood image; a background correction; a decomposition into color channels; a decomposition into hue; saturation; and brightness channels; a frequency decomposition; a singular value decomposition; applying a Canny edge detector; applying a Laplacian of Gaussian filter; applying a Difference of Gaussian filter; applying a Sobel operator; applying a Laplace operator; applying a Scharr operator; applying a Prewitt operator; applying a Roberts operator; applying a Kirsch operator; applying a high-pass filter; applying a low-pass filter; applying a Fourier transformation; applying a Radon-transformation; applying a Hough-transformation; applying a wavelet-transformation; a thresholding; creating a binary image. The region of interest may be determined manually by a user or may be determined automatically, such as by recognizing the user within the image. In particular, Biometric recognition using a flood image may comprise using at least one image recognition technique, in particular a face recognition technique. An image recognition technique comprises at least one process of identifying the user in an image. The image recognition may comprise using at least one technique selected from the technique consisting of: color-based image recognition, e.g. using features such as hue, saturation, and value (HSV) or red, green, blue (RGB); template matching, for example as illustrated on https: / / www.mathworks.com / help / vision / ug / pattern-matching.html; image segment and / or blob analysis e.g. using size, color, or shape; machine learning and / or deep learning e.g. using at least one convolutional neural network. The neural network may be trained by the user, such as in a training procedure, in which the user is indicated to take at least one or a plurality of pictures showing himself.

[0129] Biometric recognition may comprise determining a plurality of facial features. Biometric recognition may comprise comparing, in particular matching, the determined facial features with template features. The template features may be features extracted from at least one template. The template may be or may comprise at least one image generated in an enrollment process, e.g. when initializing the authentication system. Template may be an image of an authorized user. The template features and / or the facial feature may comprise a vector. Matching of the features may comprise determining a distance between the vectors. The identifying of the user may comprise comparing the distance of the vectors to a least one predefined limit, wherein the user is successfully identified in case the distance is smaller than or equal to the predefined limit at least within tolerances. The user declining and / or rejected otherwise.

[0130] For example, biometric recognition may comprise using at least one model, in particular a trained model comprising at least one face recognition model. Biometric recognition may be performed by using a face recognition system, such as FaceNet, e.g. as described in Florian Schroff, Dmitry Kalenichenko, James Philbin, "FaceNet: A Unified 240507W001

[0131] 18

[0132] Embedding for Face Recognition and Clustering”, arXiv: 1503.03832. The trained model may comprises at least one convolutional neural network. For example, the convolutional neural network may be designed as described in M. D. Zeller and R. Fergus, "Visualizing and understanding convolutional networks”, CoRR, abs / 1311.2901, 2013, or C. Szegedy et al., "Going deeper with convolutions”, CoRR, abs / 1409.4842, 2014. For more details with respect to convolutional neural network for the face recognition system reference is made to Florian Schroff, Dmitry Kalenichenko, James Philbin, "FaceNet: A Unified Embedding for Face Recognition and Clustering”,

[0133] arXiv: 1503.03832. As training data labelled image data from an image database may be used. Specifically, labeled faces may be used from one or more of G. B. Huang, M. Ramesh, T. Berg, and E. Learned-Miller, "Labeled faces in the wild: A database for studying face recognition in unconstrained environments”, Technical Report 07-49, University of Massachusetts, Amherst, October 2007, the Youtube® Faces Database as described in L. Wolf, T. Hassner, and I. Maoz, "Face recognition in unconstrained videos with matched background similarity”, in IEEE Conf, on CVPR, 2011, or Google® Facial Expression Comparison dataset. The training of the convolutional neural network may be performed as described in Florian Schroff, Dmitry Kalenichenko, James Philbin, "FaceNet: A Unified Embedding for Face Recognition and Clustering”, arXiv: 1503.03832.

[0134] Image artifacts caused by diffraction of the light when passing the transparent display may be corrected. The term "correct” may mean partially or fully remove the artifacts or tag them so they can be excluded from further processing, in particular from determine if the imaged user is an authorized user. Correcting image artifacts may take into account the information about the transparent display, in particular the dimensions of the pixels or the distance of repeating features to each other. This information can facilitate identifying artifacts as diffraction patterns can be calculated and compared to the image. Correcting image artifacts may comprise identifying reflection features, sorting them by brightness and selecting the locally brightest features. For determining a distance around a feature in the image which qualifies as local, the information of the transparent display may be used, in particular a distance in the image by which a light beam may be displaced by diffraction on the transparent display may be calculated based on the information about the transparent display. This method can be particularly useful for pattern images. Further details are disclosed in WO 2021 / 105265 A1.

[0135] Personalized person data may be obtained using the identity of the person obtained from the biometric recognition. In particular, a personalized reference spectrum, i.e. a reference spectrum which is specific for the identified person, may be obtained from a database using the identity of the person. The personalized reference spectrum may be used to determine the concentration of the body substance.

[0136] Hyperspectral data may be used for biometric authentication of the person. The term "biometric authentication” may refer to any procedure which uses a characteristic of a human to identify if a real human is present, i.e. in front of the biometric capture device, or not. The characteristic of the human may be a material. The material may be any material which is found in a human body, for example skin, hair, or cornea tissue. Biometric authentication may be combined with biometric recognition, i.e. the determination which person is present.

[0137] The material may be determined using a material model. The term "material model” may refer to a model which uses image data as input and outputs a predicted class label indicating the material of the object. The material model may be or may comprise an artificial neural network, in particular a convolutional neural network (CNN), a support vector machine (SVM), a random forest, a Gaussian mixture model (GMM), a hidden Markov model (HMM) or a conditional random field (CRF). 240507W001

[0138] 19

[0139] The material model may comprise an encoder and a material classifier. The term "encoder” may refer to an algorithm which uses the image data as input and output a feature vector representing features in the image data. The encoder may be or may comprise a convolutional neural network (CNN), an autoencoder, a histogram of oriented gradients (HOG), a scale-invariant feature transform (SIFT), speeded-up robust features (SURF) or bag-of-visual-words (BoVW). The image data may comprise multiple images, for example a set of partial images, wherein each partial image is obtained by cropping a pattern image around a pattern feature. The encoder may encode each partial image into a corresponding feature vector, so multiple feature vectors are obtained.

[0140] The term "material classifier” may relate to an algorithm which uses a feature vector as input and outputs a predicted class label. The material classifier may be or may comprise a support vector machine (SVM), a random forest, a K-nearest neighbors (KNN), naive Bayes, decision tree, a gradient boosting model. The encoder may have generated multiple feature vectors. The material classifier may determine a predicted class label for each feature vector. Hence, the material classifier may determine multiple predicted class labels. The material classifier may aggregate the multiple predicted class labels into one aggregated predicted class label, for example by averaging or by weighted averaging. The weights of the weighted averaging may be variable parameters of the material model which are adjusted during training the material model. For example, the material classifier may classify the material from the hyperspectral data as skin or non-skin.

[0141] The material model may comprise a mechanism for positional encoding. In case the image data comprises a set of partial images, the partial images may be encoded according to their position in the original image. In this way the material model may be enabled to use correlations between partial images, i.e. their spatial relationships, for the material determination. Positional encoding may be implemented by adding or multiplying the partial images with a value characteristic for the position, for example sine or cosine functions. The positional encoding may involve region data, for example by adding a value according to the segment in addition to the position in the original image.

[0142] The material model may comprise a self-attention mechanism. The self-attention mechanism may enable identifying correlations between partial images. For example, the material model may comprise a vision transformer. In particular, a patterned image may be cropped into partial images according to the pattern, for example each partial image comprises a pattern feature and at least parts of its nearest neighbors. These partial images may be subject to positional encoding and a self-attention mechanism. In this way, the material can be more reliably determined, in particular for complex objects like a face in which different parts are of different skin or hair types.

[0143] The material model may be trained using a training dataset, for example by minimizing a loss function indicative for how well the predicted class labels match the labels of the training dataset. The loss function may be minimized by gradient descent including stochastic gradient descent (SGD), mini-batch gradient descent, or adaptive gradient descent algorithms like Adam or RMSprop. The loss function may be a cross-entropy loss function, binary crossentropy loss, triplet loss, contrastive loss, center loss, margin loss, focal loss, or dice loss.

[0144] In one aspect the invention relates to a computer-implemented method for authentication a person comprising: a. receiving hyperspectral data comprising a flood image of the person under flood illumination and a pattern image of the object under pattern illumination,

[0145] b. recognizing the person from the flood image, 240507W001

[0146] 20

[0147] c. determine the material from the pattern image, and

[0148] d. authenticate the person based on the recognition and the material determination.

[0149] Biometric authentication may comprise or may be based on one or more than one concentration of a body substance. The person may be authenticated if the concentration of a body substance is within a predetermined range, for example a range typical for human skin. For example, the concentration of water and keratin may be determined using a chemometric model as described above. If both the concentration of water and keratin are within a range typical for human skin, the person may be authenticated. Biometric authentication may comprise material detection using a material model and determination of the concentration of a body substance using a chemometric model. The person may be authenticated if both models yield values indicating the presence of a real human.

[0150] The concentration of a body substance may be used to verify the determined material from the pattern image. For example, a material classifier may have classified the material as skin. This classification may be verified by determining the concentration of a body substance which the classified material is expected to contain, for example water or keratin for skin. In this way, classification errors may be determined adding further security to the authentication.

[0151] Hyperspectral data may be used for determining a surface roughness measure of the person, in particular the skin roughness. For this purpose, hyperspectral data may comprise an image of the person under illumination of pattern coherent light, also referred to as speckle image. The term "surface roughness measure” may relate to a measure suitable for quantifying the surface roughness. Surface roughness measure may be related to the speckle pattern. Preferably, the surface roughness measure may be suitable for describing the vertical and lateral surface features. Surface roughness measure may comprise a value associated with the surface roughness. Surface roughness measure may refer to a term of a quantity for measuring the surface roughness and / or to the values associated with the quantity for measuring the surface roughness. Surface roughness measure may include at least one of a fractal dimension, speckle size, speckle contrast, speckle modulation, roughness exponent, standard deviation of the height associated with surface features, lateral correlation length, average mean height, root mean square height or a combination thereof. There may be various methods and parameters used to measure surface roughness. A parameter may be the Ra value (arithmetical mean roughness). Ra is calculated as the average of the absolute values of the height deviations from the mean line of the surface profile within a specified sampling length. It represents the average roughness of the surface and is expressed in units of length, such as micrometers (pm) or inches (in). Other surface roughness measure(s) may include Rz (maximum height of the roughness profile), Rq (root mean square roughness), Rt (total height of the profile), and Rmax (maximum peak-to-valley height). Each parameter may provide different information about the surface roughness and can be used depending on the specific requirements of the application.

[0152] Speckle pattern(s) may relate to a distribution of the plurality of speckles. The distribution of the plurality of speckles may relate to a spatial distribution of at least one of the plurality of speckles and / or a spatial distribution of at least two of the plurality of speckles in relation to each other. Spatial distribution of the at least one of the plurality of speckles may relate to a spatial extent of the at least one of the plurality of speckles. Spatial distribution of the at least two of the plurality of speckles may relate to a spatial extent of the first speckle of the at least two speckles in relation to the second speckle of the at least two speckles and / or a distance between the first speckle of the at least two speckles and the second speckle of the at least two speckles. 240507W001

[0153] 21

[0154] A roughness model may be used for determining a surface roughness measure from a speckle image. The roughness model may be a data-driven model which represents a correlation between the surface roughness measure and the speckle image. The data-driven model may obtain the correlation between surface roughness measure and speckle image based on a training data set comprising a plurality of speckle images and a plurality of surface roughness measures. A data-driven model may be parametrized according to a training data set to receive the speckle image and provide the surface roughness measure. The data-driven model may be trained with a training data set. The training data set may comprise one or more speckle image(s) and one or more corresponding surface roughness measure(s). The training data set may comprise a plurality of speckle images and a plurality of surface roughness measures. Training the model may include parametrizing the model. The data-driven model may be parametrized and / or trained to provide the surface roughness measure(s) based on the speckle image, in particular receiving the speckle image. Determining the surface roughness measure(s) based on the speckle image may comprise providing the speckle image to a data-driven model and receiving the surface roughness measure from the data-driven model. The roughness model may be or may comprise a neural network, in particular a feedforward neural network such as a CNN, or a recursive neural network (RNN). The roughness model may be or may comprise classification model such as a SVM classifier.

[0155] The roughness model may be a deterministic model, i.e. reflect physical phenomena in mathematical form, e.g., including first-principles models. A deterministic model may comprise a set of equations that describe an interaction between the object and the coherent light thereby resulting in a surface roughness measure. The physical model may be based on at least one of a fractal dimension, speckle size, speckle contrast, speckle modulation, roughness exponent, standard deviation of the height associated with surface features, lateral correlation length, average mean height, root mean square height or a combination thereof. In particular, the physical model may comprise one or more equations relating the speckle image and the surface roughness measure based on equations relating to the fractal dimension, autocorrelation, speckle size, speckle contrast, speckle modulation, roughness exponent, standard deviation of the height associated with surface features, lateral correlation length, average mean height, root mean square height or a combination thereof.

[0156] The roughness model may reflect physical phenomena in mathematical form, e.g., including second order statistical measure(s). A model may be using Grey-Level Co-occurrence matrix (GLCM). A GLCM may be used as a feature extractor. A GLCM is a matrix that may be used to describe the relationship between at least two adjacent pixels in a digital image. GLCM is a statistical method that may be used in image processing to analyze texture features of an image. In a GLCM, element(s) may represent the number of times a particular combination of pixel intensities may occur in the image(s). The matrix is based on the idea that the relative positions of pixels in an image may provide data about the texture of the image. By analyzing the patterns and relationships between pixel values in an image, the GLCM may be used to extract texture metrics of the image, such as contrast, homogeneity, and entropy.

[0157] The determination of the concentration of the body substance may use a surface roughness measure determined from the hyperspectral data, for example as additional input to a chemometric model or as input to a correction model modifying the chemometric data output by a chemometric model according to the surface roughness measure. The surface roughness may influence the absorption of the light, so determining the surface roughness from a speckle image may enhance the accuracy of the concentration of the body substance determined from a spectrum. For 240507W001

[0158] 22

[0159] example, the skin moisture content may be corrected by the skin surface roughness to yield a more reliable value for the skin moisture content.

[0160] The invention further relates to a method for granting a user access to a device or application. A device can be a mobile device, for example smartphone, a tablet computer, a laptop computer or a smartwatch, or it can be a stationary device such as a payment terminal or an access control system, for example to control access to a building, a subway train station, an airport gate, a production facility, a car rental site, an amusement park, a cinema, or a supermarket for registered customers. The access control system may further be integrated into a vehicle, for example a car, a train, an airplane, or a ship. An application may refer to a local program, for example installed on a smartphone or a laptop, or a remote service, for example a service on a cloud system to be accessed via internet. The application may serve several purposes, for example to authorize a payment, identify the user for a transaction with the public administration, for example to renew a driver's license, or authorize the user for high-security communication. For example, a request for access to a device of application may be submitted, for example by a user of the device or application. In response to the request for access, the user may be authenticated with the method of the invention. The authentication method may output an authentication signal indicative for the authenticity of the user. Access to the device or application may be granted based on the authentication signal, i.e. if the user is authenticated. Otherwise, access may be denied.

[0161] The concentration of a body substance determined by the chemometric model may be output. The term "outputting” may refer to writing the concentration of a body substance to a non-transitory data storage medium, for example into a file or database, display it on a user interface, for example a screen, or both. Outputting may further mean to forward the concentration of a body substance to a computer system for further processing, for example an electronic control unit (ECU) or the on-board computer system. It is also possible to output the concentration of a body substance through an interface to a cloud system for storage and / or further processing.

[0162] The concentration of the body substance may be used to determine the person's fitness to drive a vehicle. The processor of the spectroscopic device may be configured to determine the person's fitness to drive a vehicle. The board computer of the vehicle or an ECU may be configured to receive the concentration of the body substance and to determine the person's fitness to drive a vehicle using the concentration of the body substance. The determination may involve determining if the concentration of the body substance exceeds or falls below a threshold. The threshold may be given by law, for example for the blood alcohol concentration or the THC concentration. The threshold may also be specific for a certain group of persons, for example a glucose level for patients suffering from type 1 diabetes. The threshold may be specific for a specific person, i.e. a personal threshold, for example for medical conditions like dehydration which may depend on the specific skin type of a person. Person-specific thresholds may be determined using the person identification described above.

[0163] The concentration of a body substance of the person may be used for controlling a functionality of the vehicle. A control signal may be generated using the concentration of the body substance. The control signal may be usable to control a vehicle access control system, for example to fully exclude a person from using a vehicle with an ignition interlock if the concentration of a body substance is above a threshold or to partially exclude the person if the concentration of a body substance is within a certain range, for example by restricting certain functionalities of a vehicle like the engine power, the maximum achievable speed or the entertainment system. 240507W001

[0164] 23

[0165] The control signal may be a Boolean value indicating whether the access can be granted or not. The control signal may be a numeric value, for example classifier indicating the extent of access which can be granted to the person. The control signal may be generated by determining if the concentration of a body substance is above or below a preset threshold. The determination of the control signal may involve region-specific settings, for example a country or state-specific concentration of a body substance threshold. The region-specific settings may be obtained from a storage medium taking into account the geographic location of the vehicle, for example obtained from a GPS system. The determination of the control signal may involve person data, for example the person's age to determine an agespecific threshold of blood alcohol concentration. The determination of the control signal may involve personalized person data, for example a personalized threshold of blood alcohol concentration which may be lower than the general threshold, for example due to a court order as a consequence of a prior driving under the influence.

[0166] Personalized person data may be selected from a database using the person identity obtained from person identification as described above.

[0167] The control signal may be used for geofence lockout, for example prevent the vehicle from leaving a designated area, for example a home or highways, if a preset concentration of a body substance is exceeded; for passive alert, for example discreetly notify emergency contacts or roadside assistance if a preset concentration of a body substance is exceeded; for adapting autonomous driving functionality, for example, increase distance kept to vehicles driving in front and increase break system pressure to allow for more effective breaking and avoid accidents due to reduced reaction time if the concentration of a body substance is within a preset range; for data logging, for example maintain a discreet log of concentration of a body substance readings for personal health tracking or potential use by law enforcement; for determining eligibility, for example restrict driving privileges based on concentration of a body substance for individuals with prior driving under the influence convictions or for novice persons; for insurance premium adjustments, for example to adjust insurance premiums based on concentration of a body substance measurement history to encourage responsible driving behavior; for emergency response decisions, for example to improve decision making of law enforcement and medical personnel, taking into account concentration of a body substance levels of individuals involved in accidents or medical emergencies; for real-time fleet monitoring, for example an alert fleet managers to elevated concentration of a body substance readings, allowing for immediate intervention such as contacting the person, dispatching a replacement, in particular for commercial vehicles; for route restriction, for example automatically reroute vehicles driven by someone with a detected concentration of a body substance to avoid high-risk areas or congested roads; for remote engine disable, for example in extreme cases such as very high concentration of a body substance to allow fleet managers to remotely disable the vehicle to prevent accidents; for person rewards or penalties, for example to implement incentive programs for maintaining clear person records and penalties for violations.

[0168] The present invention further relates to a non-transient computer-readable medium including instructions that, when executed by one or more processors, cause the one or more processors to perform the method according to the present invention. The term "computer-readable data medium” may refer to any suitable data storage device or computer readable memory on which is stored one or more sets of instructions (for example software) embodying any one or more of the methodologies or functions described herein. The instructions may also reside, completely or at least partially, within the main memory and / or within the processor during execution thereof by the computer, main memory, and processing device, which may constitute computer-readable storage media. The instructions may further be transmitted or received over a network via a network interface device. Computer-readable data medium include hard drives, for example on a server, USB storage device, CD, DVD or Blue-ray discs. The computer 240507W001

[0169] 24

[0170] program may contain all functionalities and data required for execution of the method according to the present invention or it may provide interfaces to have parts of the method processed on remote systems, for example on a cloud system.

[0171] Brief Description of the Figures

[0172] Figure 1 illustrates an example for a spectroscopic device.

[0173] Figure 2 illustrates two examples for the method of the invention.

[0174] Figure 3 illustrates an example for hyperspectral data.

[0175] Figure 4 illustrates an example of using the method for authentication.

[0176] Figure 5 illustrates two examples of how a concentration of a body substance can be determined from hyperspectral data, object data and environmental data.

[0177] Figure 6 illustrates the determination of a concentration of a body substance from hyperspectral data including surface roughness determination.

[0178] Description of Embodiments

[0179] Figure 1 illustrates an example for a spectroscopic device. The spectroscopic device 100 may comprise a hyperspectral camera 110, a light emitting element 120, a processor 130, and a communication interface 140. The light emitting element 120 may emit a light beam 150 to a person 160. The light beam 150 may be reflected by the person 160 and impinge on the hyperspectral camera 110. The light emitting element 120 may be communicatively coupled with the processor 130. The processor may hence control the light emitting element 120, for example by switching it on or off or change settings of the light emitting element, for example the type of light to be emitted or its intensity. The light emitting element 120 may comprise a flood light emitting element 122 for emitting flood light, for example an incandescent lamp or a VCSEL array. Additionally, or alternatively, the light emitting element 120 may comprise a pattern light emitting element 121 for emitting patterned light, for example a regular dot pattern. The pattern light emitting element 121 may be a laser, such as a VCSEL array, combined with a dispersive optical element, for example a meta lens. The light emitting element 120 may comprise both a pattern light emitting element 121 and a flood light emitting element 122. These may emit light both at the same time or one after the other. The processor 130 may be able to switch the pattern light emitting element 121 and a flood light emitting element 122 on and off independent of each other, so one of them is on at a certain point in time and the other is off.

[0180] The hyperspectral camera 110 may comprise a scanning optical element 111, in particular if the hyperspectral camera 110 records hyperspectral data of a scene by spatial scanning. The scanning optical element 111 may be a movable mirror directing the light beam 150 from a certain direction corresponding to a position of the person 160 to the photodetector 113. The hyperspectral camera 110 may comprise a dispersive optical element 112 to direct different wavelengths components of the light beam 150 onto different parts of the photodetector 113. A dispersive optical element 112 may be an optical grating or a linear variable filter. The photodetector 113 may convert the incoming light of the light beam 150 into photodetector signals, for example an electric signal corresponding to the intensity of the incoming light. The photodetector 113 may be a CCD or a CMOS. The photodetector 113 may be communicatively coupled with controller 114. The controller may receive the detector signals and convert them into hyperspectral data. Such conversion may include analog-to-digital conversion, applying calibration coefficients, assigning wavelength data corresponding to the region of the photodetector from where the detector signal is obtained. The 240507W001

[0181] 25

[0182] controller 114 may further be communicatively coupled with the scanning optical element 111. The controller 114 may set the position of the scanning optical element 111 and use such positional information for associating the sensor signal and the spatial information to generate the hyperspectral data. The controller 114 may be communicatively coupled with processor 130. The controller 114 may send the hyperspectral data to the processor 130 for further processing, for example for determining region data and applying a chemometric model in order to determine the concentration of the body substance. The processor 130 may be communicatively coupled with a communication interface 140. The communication interface 140 may be a cable connection, for example USB, or a wireless connection, for example wireless LAN, or a telecommunication interface. The communication interface 140 may send data, for example the concentration of the body substance, to a different computer system, for example the board computer of a car or a cloud server. The communication interface 140 may also receive a request to perform a measurement. The communication interface 140 may forward such request to processor 130 which in response may trigger the light emitting element 120 and the hyperspectral camera 110 for a measurement.

[0183] Figure 2 illustrates two examples for the method of the invention. In Figure 2a, hyperspectral data 211 is fed into a region model 212 to obtain region data 214. Hyperspectral data 211 may comprise a data cube, wherein two dimensions are associated with the spatial position in the scene and one dimension with the wavelength channel. Hyperspectral data 211 may hence comprise an image of the scene for each wavelength channel. The images may have a resolution of 500 to 10000 pixels, the number of wavelength channels may be 12 to 256. The wavelength channels may correspond to wavelengths or wavelength ranges, for example in the rear infrared spectral range, such as 800 to 2500 nm. The region model 212 may receive the hyperspectral data 211 and a body topology map 213 as input and output region data 214. The body topology may comprise values for different parts of the body, wherein each value indicates the suitability of the body part for a spectroscopic determination of the body substance. The region model 212 may use one image of the hyperspectral data 211 or an average image of the hyperspectral data 211 for an image segmentation. For example, the region model 212 may comprise a convolutional neural network and a classifier for assigning each pixel of the image to a body part class. The region model 212 may, for example, determine the elements of a face, such as the eyes, the nose, the cheeks, the lips and the ears. The region model 212 may assign each pixel in the image a value corresponding to the classified body part, wherein the value may be obtained from a corresponding entry in the body topology map 213. Both hyperspectral data 211 and region data 214 may be input to chemometric model 215. The chemometric model 215 may be parametrized to determine the concentration of the body substance 216 from the hyperspectral data 211 and the region data 214. For example, the chemometric model 215 may determine a concentration of the body substance for each pixel and weight average according to the suitability values of the region data 214 to arrive at the concentration of the body substance 216. Alternatively, the chemometric model 215 may generate one spectrum by weight-averaging the pixels according to the suitability values of the region data 214.

[0184] In figure 2b, region data 224 may be determined from hyperspectral data 221 and body topology map 223 using a region model 222 as described for figure 2a. The region data 224 may comprise a matrix of Boolean values indicating if a pixel shall be included in the determination of the body substance or not. A selection procedure 225 may select from the hyperspectral data those pixels for which the region data 224 contains a true value. In this way, a selected hyperspectral data 226 is obtained which may be feed into chemometric model 227. Chemometric model 227 may average all pixels of the selected hyperspectral data 226 to generate one spectrum for all selected pixels. This spectrum may be used to determine the concentration of the body substance 228. Alternatively, a concentration of 240507W001

[0185] 26

[0186] the body substance may be determined for each pixel in the selected hyperspectral data 226. All concentrations may be averaged to arrive at the concentration of the body substance 228.

[0187] Figure 3 illustrates an example for hyperspectral data. A person 310 may be under illumination, for example under near infrared floodlight illumination. A hyperspectral camera may record hyperspectral data of the person's 310 face. Parts of the hyperspectral data around the lips of the person 310 is displayed in the center of the figure. An image segmentation algorithm may detect the contour of the lips to generate region data 321. A topology map may indicate that the lips shall be selected from the hyperspectral data, so region data 321 indicating a selection of the lips may be generated. The hyperspectral camera may be triggered to capture hyperspectral data based on the region data 321, i.e. of the lips, for example with illumination focused on the lips or with higher resolution of the lips. By using the region data 321 those pixels of the lips may be selected to generate selected hyperspectral data 320 indicated in grey in the figure. Each pixel of the selected hyperspectral data 320 may be associated with a selected spectrum 331, 332, 333. The selected spectrum 331, 332, 333 may be used to determine the concentration of the body substance, for example by inputting each spectrum into a chemometric model and averaging the results, or by averaging the spectra and input the averaged spectrum into the chemometric model.

[0188] Figure 4 illustrates an example of using the method for authentication. Hyperspectral data 400 may be obtained from a hyperspectral camera. Hyperspectral data 400 may comprise flood images 401 of the person and a pattern image 402 of the person in multiple wavelength channels. The flood images 401 may be used to determine region data 411, for example by image segmentation. One flood image out of the flood images 401 may be selected for image segmentation or an average image, for example a weight-averaged image, may be used for image segmentation. The region data 411 may be used to determine the concentration of the body substance 412 from the hyperspectral data 400, for example from the flood images 401.

[0189] The hyperspectral data 400 may be used to identify the person 421. Identification may be performed using a convolutional neural network which extracts a feature vector representing features of the person, for example facial features. Identification may further be accomplished by comparing the feature vector with a reference feature vector, for example obtained from a database or an enrollment process. If the two feature vectors are similar enough, for example their cosine value is above a certain value, the corresponding persons may be found to be identical.

[0190] The pattern images 402 may be cropped into partial images 422, for example with a pattern in the center and some of the neighboring patterns around the central pattern. A partial image may be annotated 423 with region data 411 , for example a partial image may be located in a segment. The material may be classified for each annotated partial image 424, for example by a material model comprising an encoder, for example a convolutional neural network, and a classifier, for example a fully connected neural network. In this way, for each partial image a predicted material class may be obtained. These may be aggregated 425, for example by weight averaging, wherein the weight may be chosen according to the probability that the object exposes an expected material at the position of the partial image. The aggregation 425 may yield an aggregated predicted material class.

[0191] An authentication signal 430 may be generated using the concentration of the body substance 412, the identified person 421 and the aggregated material class 425. For example, the authentication signal 430 may be used to start a vehicle. The authentication signal may thus only be sent if the correct person has been identified 421, it has been 240507W001

[0192] 27

[0193] verified that it is really a person by classifying the material 425 as skin and the concentration of the blood alcohol may have been determined 412 to be below a threshold, such as below 0.05 %.

[0194] Figure 5 illustrates two examples of how a concentration of a body substance can be determined from hyperspectral data, object data and environmental data. In Figure 5a hyperspectral data 511 may be input to a chemometric model comprising pre-processing 521, feature selection 522 and a machine learning model 523. Spectroscopic data 511 may comprise a spectrum, for example a near infrared spectrum, obtained from a measurement with a spectrometer. Pre-processing 521 may comprise baseline correction, for example first-order derivation, scatter correction, for example standard normal variate, smoothing, for example moving average filtering, scaling, for example Pareto scaling, aggregation, for example spatial median. The pre-processed hyperspectral data may subsequently undergo feature selection 522. Feature selection 522 may reduce the dimensionality of the hyperspectral data 511, so training the machine learning model 523 requires less training data. Feature selection 522 may for example involve principle component regression (PCR). The thus pre-processed and feature-selected hyperspectral data may be passed as input to a machine leaning model 523, for example an artificial neural network. The machine leaning model 523 may be parametrized to further receive the object data 512 and the environmental data 513 as further input. The object data 512 and the environmental data 513 may be pre-processed before inputting into the machine learning model 513, for example to adjust the format and the units of the data. The machine leaning model 523 may be trained with historic data comprising spectra, object data and environmental data. The machine leaning model 523 may output a concentration of a body substance 515. Using a chemometric model which uses hyperspectral data, object data and environmental data as input has the advantage that complex interplays between hyperspectral data, object data and environmental data can be taken into account.

[0195] Figure 5b shows an alternative example for determination of a concentration of a body substance from hyperspectral data, object data and environmental data. Spectroscopic data 511 may be input to a chemometric model 531 which outputs intermediate concentration of a body substance 532. The chemometric model 531 may not be parametrized to take object data 512 and environmental data 513 into account. Hence, the intermediate concentration of a body substance 532 only depends on the hyperspectral data 532. In order to obtain the desired concentration of a body substance 534, a refining model 533 may be employed. The refining model 533 may be parametrized to receive the intermediate concentration of a body substance 532, the object data 512 and the environmental data 513 and to output the concentration of a body substance 534. The refining model 533 may comprise to sub-models, one which processes the object data 512 and one which processes the environmental data 513. For example, the first sub-model may receive the intermediate concentration of a body substance 532 and the object data 512 as input and output a refined concentration of a body substance. A second sub-model may use the refined concentration of a body substance and the environmental data 513 as input and output the concentration of a body substance 534. The refining model 533 may be a multivariate polynomial regression model which adjusts the intermediate concentration of a body substance 532 according to the object data 512 and the environmental data 513 to arrive at the concentration of a body substance 534. A refining model 533 has the advantage that the chemometric model 531 does not need a retraining for new object data types or environmental data types.

[0196] Figure 6 illustrates the determination of a concentration of a body substance from hyperspectral data including surface roughness determination. Hyperspectral data 601 may be obtained from a hyperspectral camera, for example as described for figure 1 and / or 2. The hyperspectral data 601 may comprise an image or images of a scene under illumination of coherent patterned light, for example in the range of 780 to 1500 nm. Hyperspectral data 601 or parts 240507W001

[0197] 28

[0198] thereof, for example a speckle image at a particular wavelength, for example 940 nm, may be input to a roughness model 602. The roughness model may be or comprise a convolutional neural network which has been trained with training data comprising speckle images labelled with a surface roughness measure, for example the arithmetical mean roughness of the skin of a person. The roughness model 602 may hence output the surface roughness measure 603. A chemometric model 604 may receive hyperspectral data 601 and the surface roughness measure 603 as input. The chemometric model 604 may be as described in figure 5, wherein the surface roughness takes a similar role as the object data in figure 5. The chemometric model 604 may output the concentration of a body substance 610, for example skin hydration. The concentration of a body substance 610 may be determined more accurately by taking into account the skin roughness.

[0199] The present disclosure has been described in conjunction with preferred embodiments and examples as well.

[0200] However, other variations can be understood and effected by those persons skilled in the art and practicing the claimed invention, from the studies of the drawings, this disclosure and the claims.

[0201] Any steps presented herein can be performed in any order. The methods disclosed herein are not limited to a specific order of these steps. It is also not required that the different steps are per-formed at a certain place or in a certain computing node of a distributed system, i.e. each of the steps may be performed at different computing nodes using different equipment / data processing.

[0202] As used herein ..determining" also includes ..initiating or causing to determine", "generating" also includes ..initiating and / or causing to generate" and "providing” also includes "initiating or causing to determine, generate, select, send and / or receive”. "Initiating or causing to perform an action” includes any processing signal that triggers a computing node or device to perform the respective action.

[0203] In the claims as well as in the description the word "comprising” does not exclude other elements or steps and the indefinite article "a” or "an” does not exclude a plurality. A single element or other unit may fulfill the functions of several entities or items recited in the claims. The mere fact that certain measures are recited in the mutual different dependent claims does not indicate that a combination of these measures cannot be used in an advantageous implementation. In the claims as well as in the description the word "comprising” or "including” or similar wording does not exclude other elements or steps and shall not be construed limiting to the elements or steps lined out. The indefinite article "a” or "an” does not exclude a plurality. A single element or other unit may fulfill the functions of several entities or items recited in the claims. The mere fact that certain measures are recited in the mutual different dependent claims does not indicate that a combination of these measures cannot be used in an advantageous implementation or further elements may be included.

[0204] Providing in the scope of this disclosure may include any interface configured to provide data. This may include an application programming interface, a human-machine interface such as a display and / or a software module interface. Providing may include communication of data or sub-mission of data to the interface, in particular display to a user or use of the data by the receiving node, entity or interface.

[0205] Various units, circuits, entities, nodes or other computing components may be described as "configured to” perform a task or tasks. Configured to shall recite structure meaning "having circuitry that” performs the task or tasks on operation. The units, circuits, entities, nodes or other computing components can be configured to perform the task 240507W001

[0206] 29

[0207] even when the unit / circuit / component is not operating. The units, circuits, entities, nodes or other computing components that form the structure corresponding to "configured to” may include hardware circuits and / or memory storing program instructions executable to implement the operation. The units, circuits, entities, nodes or other computing components may be described as performing a task or tasks, for convenience in the description. Such descriptions shall be interpreted as including the phrase "configured to.” Any recitation of "configured to” is expressly intended not to invoke 35 U.S.C. § 112(f) interpretation.

[0208] In general, the methods, apparatuses, systems, computer elements, nodes or other computing components described herein may include memory, software components and hardware components. The memory can include volatile memory such as static or dynamic random-access memory and / or nonvolatile memory such as optical or magnetic disk storage, flash memory, programmable read-only memories, etc. The hardware components may include any combination of combinatorial logic circuitry, clocked storage devices such as flops, registers, latches, etc., finite state machines, memory such as static random-access memory or embedded dynamic random-access memory, custom designed circuitry, programmable logic arrays, etc.

[0209] Any disclosure and embodiments described herein relate to the methods, the systems, apparatuses, devices, chemicals, materials, computer program elements lined out above and vice versa. Advantageously, the benefits provided by any of the embodiments and examples equally apply to all other embodiments and examples and vice versa. All terms and definitions used herein are understood broadly and have their general meaning.

Claims

240507W00130Claims1. A spectroscopic device for determining a concentration of a body substance of a person comprising:a) a hyperspectral camera for recording hyperspectral data of the person,b) a processor for(i) determining region data from the hyperspectral data, wherein region data is indicative for the suitability of a region of the body for determining the concentration of the body substance, (ii) determining the concentration of the body substance of the person using the hyperspectral data and the region data, andc) an output for outputting the concentration of the body substance of the person.

2. The spectroscopic device according to claim 1, wherein the region data comprises the position of the lips.

3. The spectroscopic device according to claim 1 or 2, wherein the hyperspectral camera is placed behind a transparent display.

4. The spectroscopic device according to any of the claims 1 to 3, wherein the spectroscopic device further comprises light emitting element to illuminate the person.

5. The spectroscopic device according to claim 4, wherein the light emitting element is able to illuminate the person with flood light and with patterned light.

6. The spectroscopic device according to any of the claims 1 to 5, wherein the spectroscopic device is integrated in a consumer electronic product.

7. A vehicle comprising the spectroscopic device according to any of the claims 1 to 6.

8. A method for determining a concentration of a body substance of a person comprising:a) receiving hyperspectral data of the person from a hyperspectral camera,b) determining region data from the hyperspectral data, wherein region data is indicative for the suitability of a region of the body for determining the concentration of the body substance,c) determining the concentration of the body substance of the person using the hyperspectral data and the region data, andd) outputting the concentration of the body substance of the person.

9. The method according to claim 8, wherein first hyperspectral data is used to determine region data, second hyperspectral data is recorded using the region data, and the concentration of the body substance is determined from the second hyperspectral data.

10. The method according to claim 9, wherein the second hyperspectral data is recorded while the light emitting element is focused on the region indicated by the region data.

11. The method according to any of the claims 8 to 10, the hyperspectral data is further used for biometric recognition of the person.240507W0013112. The method according to any of the claims 8 to 11, the hyperspectral data is further used for biometric authentication of the person, wherein biometric authentication comprises a material classifier for classifying the material from the hyperspectral data as skin or non-skin.

13. The method according to any of the claims 8 to 12, wherein the concentration of the body substance of the person is determined further using person data associated with a characteristic of the person or environmental data associated with a characteristic of the surrounding of the person14. Use of the concentration of the body substance of the person obtained from the method of any of the previous claims for granting a user access to a device or application.

15. A non-transient computer-readable medium including instructions that, when executed by one or more processors, cause the one or more processors to perform a method comprising:a) receiving hyperspectral data of the person from a hyperspectral camera,b) determining region data from the hyperspectral data, wherein region data is indicative for the suitability of a region of the body for determining the concentration of the body substance,c) determining the concentration of the body substance of the person using the hyperspectral data and the region data, andd) outputting the concentration of the body substance of the person.

Images