Enhancement of semantic perception in an image by controlling vehicle illumination

Abstract

The invention relates to a method comprising, on obtaining (500; 501) a first image representative of a scene facing the vehicle, determining (502) a light intensity map, on the basis of a light intensity map generation model and of the first image obtained, the light intensity map indicating light intensity values for controlling lighting elements of a matrix source of the illumination module of the vehicle. The determined light intensity map is transmitted (503) to the illumination module for the projection of a pixelated illuminating light beam onto the scene facing the vehicle. A second image representative of the scene facing the vehicle is obtained following the projection of the pixelated illuminating beam, and an image processing operation is carried out on the second image obtained, the image processing operation being a semantic perception image processing operation.

Description

[0001] Description

[0002] Title: Improving semantic perception in an image by controlling vehicle lighting

[0003] The present invention relates to the field of controlling a lighting module in a motor vehicle. More specifically, the invention concerns a method and a module for controlling a motor vehicle lighting module, to improve a semantic perception function, for example, object detection or semantic segmentation of an image captured in the motor vehicle.

[0004] Most motor vehicles are now equipped with a driver assistance module, also called ADAS, for “Advanced Driver-Assistance Systems”, capable of implementing at least one driver assistance function, allowing the driver to be assisted in driving or to control, in an automated manner, without driver contribution, certain driving parameters of the vehicle.

[0005] Such functions are based on data captured by sensors on the vehicle, such as lidar, radar, one or more cameras, etc.

[0006] Several driver assistance functions implemented by ADAS modules use images captured by a camera capable of obtaining representative images of a scene facing the vehicle. However, some ADAS functions require that these images be processed before they can be used. Such processing can include, in a known manner:

[0007] - Semantic segmentation aims to segment each captured image into several pixel regions, each region being labeled with a class from a set of predefined classes. In automotive applications, the following classes might be used: car, pedestrian, sign, road, etc. A segmentation score can be associated with the image segmentation, or with each segment identified within the image, the score representing the degree of certainty associated with the segmentation; and / or

[0008] - Object detection aims to detect one or more objects in the scene represented by each image, to identify the category of each detected object from among several predefined categories, and to determine the position of each detected object (or a part of the image in which the object is located). Each detected object is further associated with a detection score representing the certainty associated with the detection.

[0009] However, such processing is less effective in low or very low ambient light, such as when the camera captures images representative of a night scene. In such situations, the segmentation or detection score is significantly lower than in daytime driving conditions, which can prevent the implementation of certain driver assistance functions and even lead to safety issues.

[0010] There is therefore a need to improve the accuracy associated with semantic segmentation or object detection in a scene facing a motor vehicle, when ambient light is low.

[0011] The present invention improves the situation.

[0012] A first aspect of the invention relates to a method for controlling a vehicle lighting module, the method comprising, during a typical phase, the following steps:

[0013] - obtaining a first representative image of a scene facing the vehicle;

[0014] - determination of a light intensity map, from a light intensity map generation model and at least the first image obtained, the light intensity map generation model being configured to receive as input at least the first image representative of the scene and to generate as output the light intensity map, the light intensity map indicating light intensity values ​​to control light elements of a matrix source of the vehicle's lighting module; - transmission of the determined light intensity map to the lighting module, for projection of a pixelated lighting beam into the scene facing the vehicle;

[0015] - obtaining a second image representative of the scene facing the vehicle, following the projection of the pixelated lighting beam;

[0016] - application of image processing to the second image obtained, the image processing being a semantic perception processing.

[0017] Thus, the invention makes it possible to project a beam of light to obtain a second image representative of a scene. This second image has a greater capacity than the first image, obtained before the lighting was controlled, to be processed by semantic perception, for example, by semantic segmentation or object detection, particularly in night scenes. As a result, semantic perception processing is improved. Indeed, the light intensity map generation model can be designed, for example, trained by machine learning, particularly supervised learning, specifically to improve the performance of the image processing implemented in the vehicle. That is, to ensure that a certain level of performance of the semantic perception processing (for example, an object detection score or a semantic segmentation score) is higher for the second image than for the first.Moreover, a high degree of responsiveness is allowed for lighting control (represented by the time between obtaining the first image and the second image), since the light intensity map generation model is capable of directly producing a usable output command to control the matrix source of the lighting module.

[0018] According to some embodiments, the second image processed by the image processing can be transmitted to a vehicle driving assistance module, capable of implementing at least one driving assistance function based on said second processed image.

[0019] Thus, improved image processing performance enables enhanced driver assistance functions based on such image processing. This, in turn, improves vehicle safety and / or driving comfort. Depending on the embodiment, the light intensity map generation model may have one of the following structures:

[0020] - a convolutional neural network;

[0021] - an artificial neural network of the auto-encoder or variational auto-encoder type;

[0022] - a self-aware or transformative model; or

[0023] - a network generating a system of antagonistic generative networks.

[0024] Such model structures are particularly well-suited for receiving an image as input and generating an output command in the form of a light intensity map. Furthermore, such structures can be used for machine learning.

[0025] According to embodiments, the method may further include a training phase of the light intensity map generation model, comprising a modification of at least one parameter of the light intensity map generation model as a function of a loss evaluated from the output of a training image processing algorithm applied by a training image processing module, the training image processing being a semantic perception processing.

[0026] Thus, the light intensity map generation model is specifically trained to produce a light intensity map for controlling lighting that improves the performance of an image processing algorithm, which is a semantic perception algorithm. The training image processing is at least of the same type of semantic perception algorithm as the image processing of the current phase: for example, if the image processing of the current phase is object detection, the training image processing is also object detection.

[0027] In addition, the image processing algorithm used for training the training phase can be identical to the image processing applied during the current phase. Thus, the image generation model is specifically trained to improve the performance of the image processing used during the current phase of the process, thereby optimizing the performance associated with semantic perception.

[0028] In addition or as an alternative, the training phase may include the following steps:

[0029] - obtaining an association of training data, the association comprising a representative training image of a scene and reference data;

[0030] - application of the light intensity map generation model to the training image, to obtain a training light intensity map;

[0031] - obtaining a synthetic image representative of the scene of the training image into which is projected a beam of lighting obtained according to the training light intensity map;

[0032] - application of the training image processing algorithm to the synthetic image, to obtain a processed training image;

[0033] - evaluation of a loss by comparing the processed training image with the reference data;

[0034] - modification of at least one parameter of the light intensity map generation model based on the evaluated loss.

[0035] Such training steps enable supervised learning of the light intensity map generation model when these steps are repeated until a predefined convergence criterion is reached. The resulting light intensity map generation model is optimal for improving image processing performance, as the loss function depends on the performance of the training image processing algorithm.

[0036] In addition, the training phase may also include a modification of at least one parameter of the training image processing algorithm, and the image processing applied during the current phase corresponds to the training image processing algorithm at the end of the training phase.

[0037] Thus, the training phase allows for the joint optimization of the training image processing algorithm and the light intensity map generation model, which improves the ability of the light intensity map generation model to optimize lighting for better image processing performance during the current phase.

[0038] According to some embodiments, the matrix source of the lighting module may comprise electroluminescent semiconductor elements of submillimeter dimensions, epitaxially mounted directly on a common substrate.

[0039] Such a lighting module allows the projection of a high-resolution pixelated beam of light, which enhances the ability of the light intensity map generation model to optimize lighting to improve image processing performance.

[0040] A second aspect of the invention relates to an assembly comprising:

[0041] - a camera arranged to obtain a first image representative of a scene facing the vehicle;

[0042] - at least one lighting module comprising a matrix source comprising a plurality of individually controllable light elements;

[0043] - a control device configured to determine a light intensity map, from a light intensity map generation model and at least the first image obtained, the light intensity map generation model being configured to receive as input at least the first image representative of the scene and to generate as output the light intensity map, the light intensity map indicating light intensity values ​​to control the light elements of the matrix source of the vehicle lighting module.

[0044] The control device is capable of transmitting the determined light intensity map to the lighting module, for projection of a pixelated lighting beam into the scene facing the vehicle and, upon obtaining a second image representative of the scene facing the vehicle by the camera, following the projection of the pixelated lighting beam, an image processing module is configured to apply image processing to the second image obtained, the image processing being a semantic perception processing.

[0045] A third aspect of the invention relates to a vehicle comprising an assembly according to the second aspect of the invention and further comprising a driver assistance module, in which the driver assistance module is configured to implement at least one driver assistance function based on the second image processed by the processing module.

[0046] Other features and advantages of the invention will become apparent upon examination of the detailed description below, and the accompanying drawings in which:

[0047] [Fig 1] illustrates a vehicle according to embodiments of the invention;

[0048] [Fig 2] illustrates a vehicle lighting module according to embodiments of the invention;

[0049] [Fig 3] illustrates an image processed by a vehicle object detection module according to embodiments of the invention;

[0050] [Fig 4] illustrates a drive system for a light intensity map generation module to control a lighting module, according to embodiments of the invention;

[0051] [Fig 5] illustrates the steps of a common phase of a method for controlling a vehicle lighting module, according to embodiments of the invention;

[0052] [Fig 6] illustrates the steps of a training phase of a light intensity map generation model, of a method for controlling a vehicle lighting module, according to embodiments of the invention;

[0053] [Fig 7] illustrates the structure of a control device for a vehicle lighting module, according to embodiments of the invention. The description focuses on the features that distinguish the module and the method from those known in the prior art.

[0054] Figure 1 illustrates a vehicle 100 according to embodiments of the invention.

[0055] The vehicle 100 according to the invention includes at least one lighting module 120 comprising a matrix light source, capable of forming a pixelated beam of light projected outwards from the vehicle so as to illuminate a scene facing the vehicle, particularly at night or when the ambient light is low.

[0056] The 120 lighting module is integrated into a 110 lighting device such as a vehicle's front headlight.

[0057] In some embodiments, the lighting device 110 may include at least one other lighting module, in addition to the lighting module 120, to perform a different or complementary lighting function to the lighting function implemented by the lighting module 120. Alternatively, the lighting module 120 may be the only module of the lighting device 110 that performs a lighting function. In some embodiments, the lighting device 110 may also include a signaling module capable of implementing at least one vehicle signaling function.

[0058] In the example shown, the lighting device 110, in which the light module 120 is integrated, can be either the right or left headlight. Alternatively, a first light module 120 is arranged in the right headlight and a second light module 120 is arranged in the left headlight of the vehicle 100.

[0059] The vehicle 100 according to the invention further includes a control device 130 capable of sending lighting commands to the light module 120, based on images acquired by a camera 140, to which a model 131 of generating a lighting command is applied.

[0060] According to the invention, the lighting control is a light intensity map indicating a light intensity for each light element of the matrix source of the lighting module 120, which is described below with reference to Figure 2.

[0061] For this purpose, the control device 130 is capable of executing a light intensity map generation model 131, described in detail below.

[0062] The control of the lighting module 120 is thus said to be adaptive, in that it is determined from the external scene facing the vehicle, represented by the images acquired by the camera 140.

[0063] Thus, the vehicle 100 according to the invention further comprises the camera 140, or a plurality of cameras 140, capable of acquiring an image, or a series of images, of the scene into which the lighting module 120 is capable of projecting the pixelated light beam. Thus, when the lighting module 120 is mounted in the front headlight of the vehicle 100, the lighting module 120 projects the pixelated light beam into a scene facing the vehicle: in this case, the camera 140 is arranged in the vehicle 100 so as to acquire an image, or a series of images, representative of the scene facing the vehicle. For example, as illustrated in Figure 1, the camera 140 can be arranged in a central position at the top of the windshield of the vehicle 100.

[0064] More generally, the 140 camera is capable of acquiring an image of a scene in which the 120 lighting module is capable of projecting the pixelated light beam.

[0065] The vehicle 100 includes an image processing module 150, capable of applying at least one image processing operation to the images captured by the camera 140, before transmitting the processed images to a driver assistance module 160, referred to as the ADAS module 160 in the following. The ADAS module 160 is capable of implementing at least one vehicle driver assistance function.

[0066] According to the invention, the ADAS 160 module uses images processed by the image processing module 150 to implement at least one driver assistance function. No restrictions are attached to this at least one driver assistance function, which may include pedestrian avoidance, emergency braking assistance, adaptive cruise control (ACC), etc. According to the invention, the image processing module 150 is a semantic perception processor. Thus, the image processing module 150 can be:

[0067] - an object detection module 150, capable of detecting one or more objects in an image or a series of images, of associating a category with each detected object from a predefined set of categories, of determining the position of each detected object in the image, and of determining a detection score for each detected object; or

[0068] - a semantic segmentation module capable of segmenting each captured image into several pixel regions, each region being labeled with a class from a set of predefined classes (car, pedestrian, sign, road, etc.). The semantic segmentation module can determine a segmentation score associated with the image segmentation, or with each segment identified within the image; or

[0069] - a module capable of performing another semantic perception task, for example panoptic segmentation.

[0070] The object detection module 150 can implement an object detection algorithm, such as an available object detection algorithm not covered by the present invention. For example, a well-known YOLO (You Only Look Once) type algorithm can be implemented by the object detection module 150. Alternatively, the object detection algorithm implemented by the object detection module 150 is an object detection model derived from a training phase of the process according to the invention, described below with reference to Figure 6.

[0071] The semantic segmentation module 150 may implement a semantic segmentation algorithm, such as a known semantic segmentation algorithm not covered by the present invention. Alternatively, the semantic segmentation algorithm implemented by the semantic segmentation module 150 is a segmentation model derived from a training phase of the method according to the invention, described below with reference to Figure 6. The control device 130 may be a centralized vehicle control device, called a “Body Controller,” capable of implementing a plurality of vehicle functions, including the control function of the lighting module 120. Alternatively, the control device 130 is dedicated to generating light intensity maps to control the lighting module 120.

[0072] Figure 2 illustrates the structure of a lighting module 120 with a matrix light source 200, according to embodiments of the invention.

[0073] The 120 lighting module includes:

[0074] - a control unit 201, also called a pilot unit or “driver” in English, of the matrix light source 200;

[0075] - the matrix light source 200;

[0076] - a projection optic 202 for the light from the matrix light source 200 to project a pixelated beam of light outwards from the vehicle, into a scene facing the vehicle 100, towards which the previously described camera 140 is directed. No restrictions are attached to the projection optic, which may comprise any set of optical elements.

[0077] The matrix light source 200 comprises a plurality of light elements 210 that can be individually controlled by the control unit 201. The control unit 201 can thus individually control the light elements 210, and can thus control a pattern projected in the pixelated light beam by activating some light elements 210 and deactivating other light elements 210, and by controlling the light intensity of each light element 210 according to the received light intensity map.

[0078] Note that the light intensity map can directly include the light intensity values ​​for all the light elements 210 of the matrix source 200, but more generally includes information enabling the control unit 201 of the lighting module 120 to determine the light intensities of the light elements 210 of the matrix source 200.

[0079] No restrictions are placed on the number of light elements 210 in the matrix light source 200. Preferably, the matrix light source 200 is a high-definition light source, that is, one capable of projecting a beam of light comprising more than one hundred pixels, preferably more than 1000 pixels. According to embodiments of the invention, the matrix light source 200 can project a beam of light comprising more than 10,000 pixels.

[0080] The 210 light elements can be electroluminescent.

[0081] For example, each 210 light element can be a light-emitting diode of the LED type, for “Light Emitting Diode” in English, the plurality of LEDs forming a matrix network of LEDs.

[0082] In some embodiments, the matrix light source 200 can be monolithic. A so-called "monolithic" source can have a particularly high density of light elements 210, making it especially attractive for a wide range of applications. A monolithic source involves a plurality of submillimeter-sized electroluminescent semiconductor elements 210, epitaxially bonded directly onto a common substrate, the substrate generally being made of silicon.Unlike conventional LED arrays, where each individual light-emitting element is a separate electronic component mounted on a substrate such as a printed circuit board (PCB), a monolithic LED source is considered a single electronic component. During its production, multiple areas of light-emitting semiconductor junctions are generated on a common substrate, forming an array. This manufacturing technique allows for the creation of closely spaced areas of light-emitting elements, each acting as a single light-emitting element. The gaps between these elements can be submillimeter in size. One advantage of this manufacturing technique is the high pixel density that can be achieved on a single substrate.

[0083] The individual light elements can be controlled individually by the control unit 201, which is capable of receiving control signals from the control device 201 in the form of a light intensity map. The light intensity map is then interpreted by the control unit 201 to control the light intensity of each light element 210. For example, the light intensity map includes a light intensity value for each light element 210 (when the light intensity map and the matrix source have the same resolution) or to control the light intensity of a group of light elements 210 (when the matrix source 200 has a higher resolution than the light intensity map).

[0084] Individual control of each light element 210 can include controlling the power supply provided to the individual light element 210 by pulse width modulation, or PWM. The control unit 201 can, for example, be an ASIC (Application Specific Integrated Circuit), which offers the advantage of being very compact and can be integrated into the monolithic light source.

[0085] Alternatively, the matrix light source 210 comprises a light source and a micromirror array, each micromirror forming a light element 210 by reflecting light rays from the light source. The micromirror array 210 is also called a DMD (Digital Micromirror Device), and the micromirrors 210 can be individually activated to reflect light from the light source towards the projection optics 202, thus forming a pixelated beam of light, the pattern of which is controllable by controlling the micromirrors 210.

[0086] Figure 3 illustrates an example of an image 300 after processing by the object detection module 150, according to embodiments of the invention. The image 300 was previously captured by the camera 140, as explained above.

[0087] Image 300 represents a scene facing the vehicle, which could include a scene in a night driving situation.

[0088] In the example shown in Figure 3, the image can detect a first object 301 in a first part 301 of the image 300, a second object 302 in a second part 302 of the image 300 and a third object 303 in a third part 313 of the image 300.

[0089] A road 304 on which vehicle 100 is traveling can also be detected as an object in image 300.

[0090] The outlines of the attributes are sharp in Figure 3, for illustrative purposes. However, it should be understood that the outline of objects and the parts of the image in which they are contained may be less clearly defined, particularly in night driving situations. Furthermore, in practice, the 300 image comprises grayscale or color pixels, and not sharp outlines formed by black lines as shown, for simplification, in Figure 3.

[0091] As described previously, the object detection module 150 can further assign a category to each identified object, for example, the category “car” to the first and second objects 301 and 302 and the category “pedestrian” to the third object 303. Parts 311 to 313 allow the position of the objects to be determined in the image 300.

[0092] Inaccuracies may occur in object detection, in determining its category and / or position. As described previously, the degree of certainty with which an object, its category, and its position are determined is expressed by a detection score calculated by the object detection module 150. Note that an object may be associated with a high detection score while another object is associated with a lower confidence score, even within the same image 300, due to uneven lighting across the scene.

[0093] As described below, the invention enables the lighting module to control the illumination, thereby improving the performance of the processing module 150, and thus of semantic perception processing, particularly object detection or object segmentation. Such performance improvement may include an improvement in detection scores associated with object detection in the image 300, or an improvement in the segmentation score(s), and / or a reduction in errors (relative to ground truth) during the detection, categorization, and / or localization of objects in the scene. In particular, the lighting control according to the invention can enable the detection of objects in a night scene that would not have been detected without such lighting control.

[0094] Figure 3 illustrates the result of image processing applied by the object detection module 150. In what follows, the processing module 150 is considered, for illustrative purposes, to be an object detection module: thus, the lighting control according to the invention improves the performance associated with object detection. However, as explained previously, the processing module 150 can more generally be a semantic perception module, for example, a semantic segmentation module, in which case the lighting control according to the invention improves the performance associated with semantic perception in general, for example, the performance associated with semantic segmentation.

[0095] Figure 4 illustrates a drive system for the light intensity map generation model 131 implemented in the control device 130 of the vehicle 100 according to embodiments of the invention.

[0096] Such a drive system 400 is external to the vehicle 100 and is capable of implementing the drive phase of the process according to the invention, which is described later with reference to Figure 6. The drive phase precedes a current phase during which the control device 130 controls the lighting of the lighting module 120 during a driving situation of the vehicle 100. The drive phase may in particular be part of the design and manufacturing process of the control device 130 (or even of the processing module 150 in the second embodiment described below), before its integration into the vehicle 100.

[0097] The 400 training system includes a 401 training database, capable of storing training data associations, each training data association comprising:

[0098] - a training image representative of a night driving scene;

[0099] - Ground truth, which includes reference data corresponding to object detection (or semantic segmentation when the lighting control according to the invention aims to improve the performance of semantic segmentation) in the scene represented by the training image. Ground truth includes reference data describing all objects actually present in the scene, their respective actual categories, and their precise locations in the training image. In the case of semantic segmentation, the reference data includes the actual image segments and the category of each segment.

[0100] Preferably, the associations stored in the training database are varied, that is to say, they were obtained in driving situations varying according to several criteria: number of objects included in the scene, number of categories of objects in the scene, distribution of objects in the scene, level of brightness of the scene, weather conditions, etc.

[0101] In addition, the 401 training database includes more than one hundred training data associations, and preferably several thousand or even tens of thousands of training data associations.

[0102] There are no restrictions on how the 401 training database was constructed. The 401 training database can be obtained from images captured by a fleet of vehicles in real-world night driving situations (preferably varied according to several criteria as described previously), with a human operator or additional sensors providing ground truth for each captured image. Alternatively, the training data associations can be synthetic.

[0103] The training system 400 further includes the light intensity map generation model 131 to be trained. The light intensity map generation model 131 is structurally capable of generating a light intensity map of a given resolution, determined from the matrix source 200 with which the control device 130 is associated in the vehicle 100 during the current phase, from at least one representative image of a scene, an image which may be a training image during the training phase, and which is an image captured in real time by the camera 140 during the current phase.

[0104] According to some embodiments, the light intensity map generation model 131 is capable of receiving as input, in addition to the representative scene image, at least one other piece of information. In this case, each image stored in the training database associates a training image with reference data and at least one other piece of training information. The other information may be:

[0105] - a depth map representing the same scene as the input image. In this case, each association in the training database combines a training image, reference data, and a training depth map (representing the same scene as the training image); and / or

[0106] - an initial light intensity map (which has been previously determined), corresponding to the scene lighting in the input image. In this case, each association in the training database combines a training image, reference data, and an initial training light intensity map (corresponding to the lighting in the training image). The model aims to improve this light intensity map compared to the initial one, resulting in better semantic perception performance.

[0107] In what follows, it is assumed for the sake of simplification that the light intensity map generation model takes as input only an image of a scene, which can be a color image, for example an RGB image for “Red Green Blue” in English.

[0108] For this purpose, the light intensity map generation model 131 can have one of the following structures:

[0109] - a convolutional neural network, such as an ll-Net type network for example; - an artificial neural network of the autoencoder or variational autoencoder type, also called VAE in English;

[0110] - a generating network of a system of generative adversarial networks, also called GAN, for “Generative Adversarial Networks” in English;

[0111] - any other generative model structure capable of receiving as input an image representing a scene, and of producing as output a light intensity map in a format capable of controlling the lighting module 120 previously described.

[0112] The 400 training system also includes a synthesis module 402, capable of producing a synthetic image representative of a scene lit by a lighting module controlled by the light intensity map produced at the output of the light intensity map generation model 131. Thus, the synthesis module 402 makes it possible to simulate the capture by a camera of the scene of the training image, into which would be projected a beam of lighting corresponding to the light intensity map at the output of the light intensity map generation model 131.

[0113] The 400 training system further includes a 403 training image processing module, which is a semantic perception processing module, for example a training object detection module or a training semantic segmentation processing module.

[0114] In accordance with the example described, the 403 training image processing module is considered to be a training object detection processing module in the following.

[0115] The training image processing module 403 is preferably identical to the image processing module 150 integrated into the vehicle 100 during the current phase of the process according to the invention, and is at least of the same type of semantic perception processing (for example, object detection or semantic segmentation). The light intensity map generation module 131 is thus specifically trained to improve the performance of the image processing module 150 with which it is mounted in the vehicle 100 during the current phase. According to a first embodiment, the training image processing module 403 is already optimized, meaning that the parameters of its image processing algorithm do not change during the training phase.

[0116] According to a second embodiment, the training image processing module 403 is trained at the same time as the light intensity map generation module 131, and the training image processing module 403 from the training phase can be implemented in the vehicle 100 for the current phase as a processing module 150.

[0117] The training image processing module 403, considered as a training object detection module 403 in this illustrative example, receives the synthetic image from the synthesis module 402, and detects one or more objects in the synthetic image, as well as a category, a location and a detection score for each detected object.

[0118] The 400 training system also includes a 403 loss assessment module, which is suitable for:

[0119] - to assess a loss by comparison between the ground truth reference data from the 401 training database and the outputs of the training object detection module (objects detected, categories, locations, and detection scores), and / or based on the output detection score(s);

[0120] - to modify one or more parameters of the light intensity map generation model 131 according to the evaluated loss, and according to a predefined training strategy, and optionally (according to the second embodiment) to modify one or more parameters of the image processing algorithm of the training image processing model 403.

[0121] No restrictions are attached to the loss function used, nor to its application to the comparison between the ground truth reference data and the outputs of the training image processing module 403. Such a training system 400 is thus capable of implementing supervised learning of the light intensity map generation model 131, and optionally of the training image processing algorithm of the training image processing module 403 (in the second embodiment).

[0122] Alternatively, the light intensity map generation model 131 can be derived from a machine learning method other than supervised learning. For example, the light intensity map generation model 131 can be optimized by reinforcement learning, with a reward determined based on the performance of a training processing module implementing a semantic perception training algorithm. Thus, in general, the light intensity map generation model 131 can be trained by machine learning to improve the performance of a semantic perception processing method applied to an image captured after the lighting module has been controlled based on the light intensity map generated by the light intensity map generation model 131.

[0123] Figure 5 illustrates the steps of a common phase of a control method for the lighting module 120 of the vehicle 100, according to embodiments of the invention.

[0124] The current phase of the process according to the invention can be implemented by the components of vehicle 100 described above with reference to figure 1.

[0125] The control process for the lighting module 120 is implemented when the lighting module 120 is activated, i.e. in driving situations where the brightness is low, in particular below a predefined brightness threshold, for example at night.

[0126] At step 500, camera 140 captures a first representative image of a scene facing vehicle 100. Alternatively, camera 140 captures several first images and obtains a first stabilized image. Optionally, the first image thus captured is transmitted to the image processing module 150 at step 505, which applies its implemented image processing algorithm, thus obtaining a first processed image at the end of step 505. The first processed image is:

[0127] - a first image in which objects have been detected, categorized and located, along with a detection score for each of them; or

[0128] - a first segmented image, including the location and classification of each segment, and one or more segmentation scores.

[0129] The first image thus processed is transmitted by the image processing module 150 to the ADAS module 160 at a step 506, for implementation of at least one ADAS function from the processed image.

[0130] Note that during the first iteration, the lighting module's illumination is not yet controlled, and thus, the image processing module 150 processes the image with a first level of performance, which can be represented by the object detection scores, the segmentation score(s), and the detection or segmentation errors or lack thereof. This first level of performance is not determined during the current phase but is mentioned in this description to clarify the technical effect associated with implementing the lighting control according to the invention, as described below.

[0131] At a step 501 following step 500, the first image captured by camera 140 is received by control device 130.

[0132] At a step 502, the device 130 applies the light intensity map generation model to the first image received at step 501, and obtains a light intensity map as output.

[0133] At step 503, the light intensity map obtained in step 502 is transmitted to the lighting module 120 for execution of the light intensity map.

[0134] Upon receiving the light intensity map, the previously described control unit 201 drives the matrix source 200 according to the light intensity map, to project a beam of light in front of the vehicle, according to the light intensities of the pixels of the light intensity map.

[0135] Following the projection of the lighting beam according to the light intensity map determined by the control device 120, step 500 is repeated to capture a second image representative of the scene facing the vehicle 100.

[0136] The second image representing the scene facing vehicle 100 is transmitted from camera 140 to processing module 150, and processing module 150 applies the object detection or semantic segmentation algorithm to the second image, at a second iteration of step 505.

[0137] The second image processed is:

[0138] - a second image in which objects have been detected, categorized, and located, along with a detection score for each object; and / or

[0139] - a second segmented image, including the location and classification of each segment, and one or more segmentation scores.

[0140] The second processed image is transmitted by the processing module 150 to the ADAS module at a step 506 for implementation of at least one ADAS function based on the second processed image.

[0141] The second iteration following the capture of the second image follows step 504, which involves projecting a light beam determined by the control device according to the invention. During this second iteration, the image processing module 150 processes the image with a second performance level that is superior to the first performance level, since the light intensity map generation model 131 has been previously trained to improve the processing performance applied by the processing module 150. Like the first performance level, the second performance level is not determined during the current phase but is mentioned in this description to explain the improvement in image processing performance through the control of the lighting module.Note that steps 505 and 506 may not be implemented during the first iteration, in which case only the second image processed with the second performance level is transmitted to the ADAS module during step 506. Thus, only an image processed with a high performance level is transmitted to the ADAS module.

[0142] In some embodiments, steps 501 to 504 are also applied to the second image captured during the second iteration of step 500. This allows the pixelated lighting beam to be controlled again, tracking the evolution of the scene facing the vehicle. In these embodiments, a third image is captured at a third iteration of step 500, and steps 505 and 506 are applied to it (to update the semantic perception), while steps 501 to 504 are applied to it as well (to continue adapting the lighting to the evolving scene facing the vehicle). The steps of the current phase are thus repeated continuously, for successive iterations, during the vehicle's operation.

[0143] According to alternative embodiments, the lighting beam is adapted only once based on the first captured image, and the processing of step 505 is applied only to the second image. Such embodiments can be provided, in particular, when the current phase is triggered by an event. No restrictions are attached to the event, which can be an event associated with a predetermined condition. The event could, for example, be that data captured by a sensor on the vehicle, separate from the camera 140, satisfies a predefined activation condition. For example, upon detection of an object on the road by another sensor on the vehicle, such as a lidar, the current phase of the method according to the invention can be triggered to verify the detection of the same object by the processing of step 505 applied to the second image. This makes it possible to ensure redundancy in the data captured by another sensor on the vehicle.

[0144] Improving image processing for semantic perception, for example semantic segmentation or object detection, thus allows for better reliability of the processed images transmitted to the ADAS 506 module, and thus improves the accuracy and reliability of at least one ADAS function implemented by the ADAS module based on the images processed by the processing module 150.

[0145] Figure 6 illustrates the steps of a training phase of a light intensity map generation model, of a method for controlling a vehicle lighting module, according to embodiments of the invention.

[0146] As previously explained, the training phase of the light intensity map generation model 131 can be implemented in the training system 400 described with reference to Figure 4.

[0147] At step 600, the training database 401 obtains a training data association as previously described, comprising a training image of a scene facing a vehicle and reference data representative of the ground truth. According to embodiments described previously, additional training information is further obtained. In what follows, for the sake of simplicity, it is assumed that only the training image is submitted as input to the light intensity map generation module 131.

[0148] At a step 601, the obtained training image is submitted as input to the light intensity map generation model 131 (without the ground truth reference data), and the light intensity map generation model 131 generates a light intensity map based on the obtained training image.

[0149] The light intensity map is transmitted from the light intensity map generation model 131 to the synthesis module 402, which also receives the training image from the training database 400. At a step 602, the synthesis module 402 determines a synthetic image representative of the scene of the training image, illuminated by a pixelated lighting beam obtained from the light intensity map produced as output from the light intensity map generation model 131. The synthetic image is transmitted to the training image processing module 403, which applies processing to the synthetic image at a step 603, the processing being semantic perception processing, for example semantic segmentation or object detection.

[0150] The image processed in step 603 is transmitted from the training image processing module 403 to the loss evaluation module 404. At a step 604, the loss evaluation module 404 evaluates a loss based on the ground truth reference data received from the training database 401, and based on the processed image received from the training image processing module.

[0151] At a step 605, the loss evaluation module 404 can further determine whether a predefined convergence criterion is met or not, based in particular on the loss evaluated at step 605, and optionally on losses evaluated during previous iterations of steps 600 to 604.

[0152] If the convergence criterion is not met, the loss evaluation module 404 modifies at least one parameter of the light intensity map generation model at a step 606, depending on the evaluated loss, according to a predefined training strategy.

[0153] Furthermore, in the second embodiment described above, the loss evaluation module 404 can modify at least one parameter of the image processing algorithm implemented by the training image processing module 403, at a step 607, as a function of the evaluated loss, according to a predefined training strategy.

[0154] Following step 606, or following step 607 in the second embodiment, the process returns to step 600 to repeat steps 600 to 605 on the basis of a new association of training data from training database 401.

[0155] The light intensity map generation model 131 can thus be trained to convergence. Alternatively, according to the second embodiment, the light intensity map generation model 131 and the image processing algorithm implemented by the training image processing module 403 are trained to convergence.

[0156] At step 608, when the loss evaluation module 404 determines that the convergence criterion has been met, the training phase is completed and the light intensity map generation model can be implemented in the control device 130 for implementation of step 502 of the current phase of the process according to the invention. In the second embodiment, the image processing algorithm thus trained can be implemented in the vehicle's processing module 150.

[0157] Figure 7 illustrates the structure of a control device 130 according to embodiments of the invention.

[0158] The control device 130 includes a processor 701 configured to communicate unidirectionally or bidirectionally, via one or more buses or via a direct wired connection, with a memory 702 such as Random Access Memory (RAM), Read Only Memory (ROM), or any other type of memory (Flash, EEPROM, etc.). Alternatively, the memory 702 comprises several memories of the aforementioned types.

[0159] The memory 702 is capable of storing, permanently or temporarily, at least some of the data used and / or resulting from the implementation of the steps of the current phase of the process according to the invention illustrated with reference to Figure 5. In particular, the memory 702 can store the light intensity map generation model 131, and can temporarily store the first image received at each step 501 of the current phase of the process according to the invention.

[0160] The processor 701 is capable of executing instructions, stored in memory 702, for the implementation of step 502 described above. Alternatively, the processor 701 can be replaced by a microcontroller designed and configured to perform step 502 described above. The control device 130 includes a first interface 703 capable of receiving, in step 501 described above, a first image captured by the camera 140.

[0161] The control device 130 includes a second interface 704 capable of transmitting the light intensity map to the lighting module 120 during step 503 described previously.

[0162] The present invention is not limited to the embodiments described above by way of example; it extends to other variants.

Claims

Demands 1. Method for controlling a vehicle (100) lighting module (120), the method comprising, during a typical phase, the following steps: - obtaining (500: 501) a first representative image of a scene facing the vehicle; - determination (502) of a light intensity map, from a light intensity map generation model and the first image obtained, the light intensity map generation model (131) being configured to receive as input at least the first image representative of the scene and to generate as output the light intensity map, the light intensity map indicating light intensity values ​​to control light elements (210) of a matrix source (200) of the vehicle lighting module; - transmission (503) of the determined light intensity map to the lighting module, for projection of a pixelated lighting beam into the scene facing the vehicle; - obtaining a second image representative of the scene facing the vehicle, following the projection of the pixelated lighting beam; - application (505) of an image processing to the second image obtained, the image processing being a semantic perception processing of the image.

2. Method according to claim 1, wherein the second image processed by the image processing is transmitted (506) to a driver assistance module (160) of the vehicle, capable of implementing at least one driver assistance function based on said second processed image.

3. A method according to claim 1 or 2, wherein the light intensity map generation model (131) has one of the following structures: - a convolutional neural network; - an artificial neural network of the autoencoder or autoencoder type variational; - a self-aware or transformative model; or - a network generating a system of antagonistic generative networks.

4. A method according to any one of the preceding claims, further comprising a training phase of the light intensity map generation model, comprising a modification (606) of at least one parameter of the light intensity map generation model as a function of an evaluated loss (604) from an output of a training image processing algorithm applied by a training image processing module (403), the training image processing being a semantic perception processing.

5. Method according to claim 4, wherein the image processing algorithm for training the training phase is identical to the image processing applied during the current phase.

6. A method according to claim 4 or 5, wherein the training phase comprises the following steps: - obtaining (600) an association of training data, the association comprising a representative training image of a scene and reference data; - application (601) of the light intensity map generation model to the training image, to obtain a training light intensity map; - obtaining (602) a synthetic image representative of the scene of the training image into which is projected a beam of lighting obtained according to the training light intensity map; - application (603) of the image processing algorithm for image training synthetic, to obtain a processed training image; - evaluation (604) of a loss by comparison of the processed training image with the reference data; - modification (606) of at least one parameter of the intensity light map generation model (131) as a function of the evaluated loss.

7. Method according to claim 6, wherein the training phase further comprises a modification (607) of at least one parameter of the training image processing algorithm, and wherein the image processing applied during the current phase corresponds to the training image processing algorithm at the end of the training phase.

8. A method according to any one of the preceding claims, wherein the matrix source (200) of the lighting module (120) comprises electroluminescent semiconductor elements (210) of submillimeter dimensions, epitaxially mounted directly on a common substrate.

9. Vehicle assembly (100) comprising: - a camera (140) arranged to obtain a first representative image of a scene facing the vehicle; - a lighting module (120) comprising a matrix source (200) comprising a plurality of individually controllable light elements (210); - a control device (130) configured to determine a light intensity map, from a light intensity map generation model (131) and at least the first image obtained, the light intensity map generation model being configured to receive as input at least the first image representative of the scene and to generate as output the light intensity map, the light intensity map indicating light intensity values ​​to control the luminous elements of the matrix source of the vehicle lighting module; the control device being capable of transmitting the determined light intensity map to the lighting module, for projection of a pixelated lighting beam into the scene facing the vehicle; wherein, upon obtaining a second image representative of the scene facing the vehicle by the camera, following the projection of the pixelated lighting beam, an image processing module (150) is configured to apply image processing to the second image obtained, the image processing being semantic perception processing of the image.

10. Vehicle (100) comprising an assembly according to claim 9, further comprising a driver assistance module (160), wherein the driver assistance module is configured to implement at least one driver assistance function based on the second image processed by the processing module.

Images