Method for detecting at least one object, and corresponding electronic device, system, computer program product and medium

Abstract

The invention relates to a method for detecting an object, implemented in an electronic device, the method comprising: • Obtaining multimodal images representing the same real and / or virtual spatio-temporal scene, • Obtaining an overall image from the multimodal images, by applying an attention model; • Detecting an object present in the scene by means of inference by an artificial intelligence model using the overall image and / or a tensor representing the overall image as input. The invention also relates to the corresponding electronic device, computer program product and medium.

Description

[0001] DESCRIPTION

[0002] Title of the invention: Method for detecting at least one object, electronic device, system, product, computer program and corresponding medium.

[0003] 1. Technical field

[0004] This application relates to the field of automatic object detection (sometimes called "Computer Vision") in a real or virtual scene, such as detection techniques implementing neural network architectures for deep learning (or "Deep Learning" in English).

[0005] This application relates in particular to a method for detecting at least one object, as well as a corresponding electronic device, system, computer program product and information storage medium.

[0006] 2. State of the art

[0007] For several decades, we have witnessed a rise in automated systems in various fields such as industry, services, transportation, agriculture, and, more recently, in the private and public spheres (smart homes, smart offices, smart cities). These automated systems often rely on automatic detection of objects present in their environment.

[0008] This automatic detection is often performed using computer vision techniques, which implement automatic image analysis of an environment via detector or classifier neural networks (sometimes called automatic object recognition). When applied to images, classifier or detector neural networks assign (during an inference phase) a probability that an image (or portion of an image) represents a certain class of objects from among a plurality of candidate object classes. Detector neural networks also specify the location within the image of the portion (often called the bounding box) that contains this class of object.The inference phase is preceded by a learning phase during which the neural network learns to distinguish the characteristics of image parts corresponding to these candidate object classes from examples. For example, in the case of supervised learning of a "detector" type neural network, the examples provided are pairs, each associating an image with a list of bounding boxes within that image, each containing an object (called an object of interest) to which an object class has already been assigned. The reliability of the results obtained depends greatly on the training performed, particularly on the number and diversity of the examples provided.Good learning often requires supplying the neural network with thousands, or even millions, of examples, which implies a significant amount of time for collecting and annotating (often manually) these examples, as well as significant memory and processing capacity and processing time for these examples for the neural network being trained.

[0009] Learning a neural network is therefore a very time- and resource-intensive task.

[0010] However, the reliability of detection systems based on such neural network architectures can also vary greatly depending on the context in which these images are captured.

[0011] Thus, it may be difficult, or even impossible, to automatically and accurately detect objects in color images taken by a camera (such as a webcam) in the dark, whereas such objects will be more easily detectable on an image acquired, under the same conditions of darkness, but with an infrared camera.

[0012] Solutions based on the use of multimodal images have been developed. These solutions leverage the advantages of capturing different modalities for better adaptation to various capture contexts. However, there is a need for an object detection solution to further improve the reliability of automatic object detection systems.

[0013] The purpose of this application is to propose improvements to at least some of the drawbacks of the state of the art.

[0014] 3. Description of the invention

[0015] This application aims to improve the situation using an object detection method comprising:

[0016] • Obtaining multimodal images representing the same real and / or virtual spatio-temporal scene,

[0017] • Obtaining a global image from said multimodal images, by applying at least one attention model;

[0018] • Detection of at least one object present in the scene by inference using an artificial intelligence model that takes as input the global image and / or a tensor representing the global image. As described in more detail later, multimodal images are defined as images acquired from diverse imaging sources (i.e., different types of sensors) and representing the same portion of the scene. The diversity of imaging sources allows the multimodal images to complement each other in their representation (description) of the common portion of the scene.

[0019] The object detection process can be implemented at least partially in an electronic device.

[0020] Depending on the implementation of the solution described in this request, it may involve various objects, connected or not, such as vehicles, people, animals, robots, and / or machines.

[0021] For the sake of brevity, the term "image" will be used in the remainder of the application to refer either to an image itself, or to a tensor of the same dimensions as that image (in terms of pixels) representing it.

[0022] In some embodiments, said artificial intelligence model is adapted for object detection in a single-mode image.

[0023] In some embodiments, said global image is obtained by applying at least one first attention model to an image resulting from a fusion of said multimodal images.

[0024] In some embodiments, at least one of the merged multimodal images is pre-filtered by applying at least one second attention model.

[0025] In some embodiments, at least one second of the merged multimodal images is pre-filtered by applying said second attention model. In some embodiments, all of said merged multimodal images are pre-filtered by applying said second attention model. In some embodiments, at least one second of the merged multimodal images is pre-filtered by applying at least a third attention model, different from said second attention model.

[0026] In some embodiments, the process includes obtaining an overall image from each video channel of said multimodal images.

[0027] In some embodiments, the process includes obtaining an overall image from each component of said multimodal images.

[0028] In some embodiments, the process includes a joint rendering of at least one geometric shape encompassing said at least one object detected in said global image and of a label representative of the classification of said object detected by said artificial intelligence model.

[0029] In some embodiments, the process includes joint learning of said attention models and said artificial intelligence model.

[0030] The features presented individually in this application in connection with certain embodiments of the method described in this application may be combined in other embodiments of this method. In another aspect, this application also relates to an electronic device adapted to implement the method described in this application in any of its embodiments.

[0031] For example, the present application relates to an electronic device comprising at least one processor configured for object detection comprising:

[0032] • Obtaining multimodal images representing the same real and / or virtual spatio-temporal scene,

[0033] • Obtaining a global image from said multimodal images, by applying at least one attention model;

[0034] • Detection of at least one object present in said scene by inference of an artificial intelligence model taking as input said global image and / or a tensor representing said global image.

[0035] According to another aspect, the present application also relates to a system comprising at least one electronic device adapted to implement the process of the present application in any of its embodiments.

[0036] For example, this application relates to a system comprising at least one electronic device including at least one processor configured to:

[0037] • Obtaining multimodal images representing the same real and / or virtual spatio-temporal scene,

[0038] • Obtaining a global image from said multimodal images, by applying at least one attention model;

[0039] • Detection of at least one object present in said scene by inference of an artificial intelligence model taking as input said global image and / or a tensor representing said global image.

[0040] This application also relates to a computer program product comprising instructions for implementing the various embodiments of the above-mentioned method, when the program is executed by a processor, and to an information storage medium, readable by a processor of an electronic device and on which the computer program is stored. For example, this application thus relates to a computer program product comprising instructions for implementing, when the program is executed by a processor of an electronic device, an object detection method comprising:

[0041] • Obtaining multimodal images representing the same real and / or virtual spatio-temporal scene,

[0042] • Obtaining a global image from said multimodal images, by applying at least one attention model;

[0043] • Detection of at least one object present in said scene by inference of an artificial intelligence model taking as input said global image and / or a tensor representing said global image.

[0044] For example, this application also relates to a processor-readable recording medium of an electronic device on which is recorded a computer program comprising instructions for implementing, when the program is executed by the processor, an object detection method comprising:

[0045] • Obtaining multimodal images representing the same real and / or virtual spatio-temporal scene,

[0046] • Obtaining a global image from said multimodal images, by applying at least one attention model;

[0047] • Detection of at least one object present in said scene by inference of an artificial intelligence model taking as input said global image and / or a tensor representing said global image.

[0048] The programs mentioned above may use any programming language, and be in the form of source code, object code, or code intermediate between source code and object code, such as in a partially compiled form, or in any other desirable form.

[0049] The information storage media referred to in this application may be any entity or device capable of storing the program. For example, a medium may include a storage means, such as a ROM, for example a CD-ROM or a microelectronic circuit ROM, or a magnetic recording means.

[0050] Such a storage medium could be, for example, a hard drive, flash memory, etc. Furthermore, an information carrier could be a transmissible medium such as an electrical or optical signal, which can be transmitted via an electrical or optical cable, by radio, or by other means. A program according to the invention can, in particular, be downloaded from a network such as the Internet.

[0051] Alternatively, an information carrier may be an integrated circuit in which a program is incorporated, the circuit being adapted to execute or to be used in the execution of any of the embodiments of the method which is the subject of this patent application.

[0052] 4. Brief description of the drawings

[0053] Other features and advantages of the invention will become more apparent upon reading the following description of particular embodiments, given by way of simple illustrative and non-limiting examples, and the accompanying drawings, among which: [Fig. 1] presents a simplified view of a system, cited by way of example, in which at least some embodiments of the method of the present application can be implemented,

[0054] Figure 2 presents a simplified view of a device adapted to implement at least some embodiments of the process described in this application.

[0055] Figure 3 presents an overview of the process of this application, in some of its embodiments.

[0056] Figure 4 shows an example of the logical architecture of a set of modules suitable for implementing at least partially the process described in this application. Figure 5 provides a summary of the layers of the modules shown in Figure 4.

[0057] Figure 6 presents in more detail the layers of the modules shown in Figure 4, in an example implementation.

[0058] Figure 7 presents multimodal images acquired in an example implementation.

[0059] Figure 8 shows the merged image obtained from the multimodal images acquired in Figure 7 and a rendering of encompassing geometric shapes and labels of detected and classified objects, based on this merged image, in this implementation example. 5. Description of embodiments

[0060] The present application proposes a solution based on an analysis of a plurality of multimodal images, each representing the same spatio-temporal scene, to detect objects in that scene.

[0061] In the detailed examples below, the scene to be monitored might be a portion of an outdoor urban environment, for example, a street intersection, in which various objects (such as pedestrians and vehicles) are moving or stationary (as in the multimodal images illustrated in Figure 7). It is clear, however, that certain embodiments of the method that is the subject of this application can be implemented in other environments (in particular for other types of scenes to be monitored (for example, indoor scenes such as a portion of a factory, shop, or dwelling, or outdoor scenes such as a garden or a highway)) for the detection of various objects (robots, vehicles, humans, animals, etc.).

[0062] By "images representing the same spatio-temporal scene", we mean images containing visual elements representative of a common portion of a real or virtual environment.

[0063] This could be, for example, images captured from the same viewpoint, or from similar viewpoints, with identical or at least partially overlapping shooting angles (one being wider than the other, for example), these captured images each representing the same common portion (or scene) of this environment.

[0064] These images are captured within a common time interval, for example, images captured simultaneously or nearly simultaneously. The size of the common time interval can depend on the embodiment, and in particular on the nature of the objects whose presence is to be detected (for example, their average speed). For example, a time interval of a few milliseconds could be chosen in embodiments adapted for detecting vehicles and pedestrians. Multimodal images are images acquired by different types of sensors (implementing different acquisition modalities) and used for the same processing.

[0065] For example, the plurality of multimodal images may include color images, and / or black and white images, and / or infrared and / or thermal images, and / or depth or reflectance maps, and / or radar images, and / or stereovision images, and / or maps obtained from point clouds acquired by distance sensors (such as LIDAR (for "Light Detection And Ranging")).

[0066] The present application proposes to apply to at least some of these images, and / or to their fusion, an attention model (preceded and / or followed, optionally, by the application of at least one convolution), before a phase of object detection from the resulting image by a "detector" type neural network taking as input a monomodal image (or a tensor representative of such a monomodal image).

[0067] This solution allows users to benefit from the image feature correlation advantages of attention models while also enabling the use of a third-party "detector" neural network, including pre-trained detectors with a large training database. The detector can be a model specifically designed for detecting objects in 2D images.

[0068] Depending on the embodiment, the detector can be designed specifically for images (for example with a deep learning model based on convolution layers) or be more general (i.e. not limited to images), for example with a deep learning model based on layers of "Transformers".

[0069] Using a deep learning model designed for images can help limit the amount of data used for training.

[0070] Using a more general deep learning model (therefore without inductive bias) can lead to better performance.

[0071] The detector used can employ a model known for its high performance in terms of both relevance and execution speed, such as a YOLO© model (for "You Only Look Once") described by Redmon et al. in the article "You Only Look Once: Unified, Real-Time Object Detection") or an RT-DETR© model (for "Real-Time Detection Transformer"). The solution presented in this application thus offers numerous advantages over other existing multimodal image fusion techniques.

[0072] In the field of multimodal image computer vision, different types of fusion can be distinguished: upstream fusion (or "early fusion"), middle fusion (or "middle fusion") and downstream fusion (or "late fusion").

[0073] Upstream fusion consists of merging the different multimodal images acquired before providing the result of this fusion to a neural network (of detector type for example) with a single input (only one monomodal image being processed at a time by this neural network).

[0074] Upstream fusion has the advantage of allowing the reuse of neural network models designed for single-modal images. It also has the advantage of learning the relationships between multimodal images from their low-level features. However, because fusion precedes processing by the neural network, this technique does not allow the neural network to separately distinguish the intrinsic features of each multimodal image, which can impair the accuracy of object detection, for example.

[0075] In the downstream fusion, each image is processed separately by a different unimodal neural network. The position and / or object class proposals (and their associated probabilities) provided by the different unimodal neural networks are merged to produce a single proposal.

[0076] Downstream fusion allows the extraction of intrinsic features from each of the multimodal images via each neural network. However, it does not leverage the learning capabilities of the neural networks to relate (correlate) the features of different images.

[0077] Mid-frame (or "intermediate") fusion requires a multi-input neural network (detector, classifier, etc.) with one input for each multimodal image, whose architecture has been specialized for processing multimodal images (the architecture may depend in particular on the nature of the modalities and / or the number of multimodal images to be processed for the same spatiotemporal scene). It takes place within the "feature extractor" (or "backbone") of this neural network and consists of merging (by addition or concatenation, for example) the intermediate features, in the latent space, of each of the multimodal images.

[0078] Latent space (also called "latent feature space" or "integration space") refers to an abstract, multidimensional space that encodes an internal representation (the intermediate or latent features) of each input image to the model. The distance between the internal representations of two images in this latent space is a function of the similarity between those images. The internal representation of an image can, for example, be obtained from the output of layers of a neural network (such as a combination of convolution and polling layers) taking that image as input. In particular, the internal representation of the images can be of lower dimension than the input images (or a tensor representing them).

[0079] Among the various types of fusion introduced, mid-frame fusion is the most widely used because it generally provides the best performance. It allows both the retrieval of high-level features from each of the multimodal images (in the neural network's input layers) and the pooling of their latent spaces (taking into account their low-level features). However, implementing this type of network is more complex, as it requires creating an architecture and a model dedicated to the different image modalities to be processed.

[0080] In the fusion process, the linking of latent features related to different multimodal images within the shared latent space (for example, during the latent feature extraction phase) sometimes involves attentional mechanisms. Attentional mechanisms are machine learning methods that determine the relative importance of each element in a set of elements compared to the other elements in that set.

[0081] Attention mechanisms allow the neural network model used in midframe fusion to identify and prioritize certain information present in the latent space that is useful for the desired outcome (here, the location and class of objects present in multimodal images), thus improving the performance of multimodal image fusion models in terms of the relevance of the results. In particular, they can contribute to a better understanding of the overall context of the information to be processed and make the systems more robust to noise and adaptable.

[0082] The present application proposes a solution to benefit from the advantages of attention mechanisms, without resorting to fusion in the middle, in order to allow in particular the reuse of state-of-the-art neural network models (classifier and / or detector for example) designed for single-modal images.

[0083] Thus, unlike fusion in the middle, the solution of the present request allows the use of neural network models dealing with single-modal images, and in particular existing models from the literature (for example models recognized for their efficiency).

[0084] Moreover, unlike upstream or downstream fusion, the solution in this application allows, at least in certain embodiments, for the correlation of the intrinsic characteristics of the different multimodal images through the use of attention mechanisms, before the use of these neural networks for monomodal images.

[0085] This connection can prove important when the neural network architecture is complex, or considered as a black box.

[0086] The present application is now described in more detail with reference to Figure 1. Figure 1 represents a telecommunications system 100 in which the method of the present application can be implemented, in at least some of its embodiments. The system 100 comprises one or more electronic devices, at least some of which can communicate with each other via one or more communication networks 120, possibly interconnected, such as a local area network (LAN) and / or a wide area network (WAN). For example, the network may include a corporate or home LAN and / or a WAN of the internet, or cellular, GSM (Global System for Mobile Communications), UMTS (Universal Mobile Telecommunications System), Wi-Fi (Wireless), etc. type.

[0087] As illustrated in Figure 1, the system 100 may also include several electronic devices, such as a communication terminal 110 (such as a computer, for example a laptop, a smartphone, a tablet), sensors 130, and / or a server 140, for example a server adapted to process (analyze for example) data obtained via at least one of the electronic devices of the system 100, a storage device 150, for example a storage device adapted to store data obtained via at least one of the electronic devices of the system 100. The system may also include management and / or network interconnection elements (not shown).

[0088] In particular, some of the sensors 130 can be adapted to provide images in various forms (or modalities). These may include, for example, cameras (such as an optical camera 132, an infrared camera 134, a thermal camera 136), radar, or distance sensors 138).

[0089] Alternatively, the sensors 130 illustrated in Figure 1 can be integrated into the device 110. Figure 2 illustrates a simplified structure of an electronic device 200 of the system 100, for example the terminal 110 or the server 140 of Figure 1, adapted to implement the principles of this application.

[0090] Device 200 includes, in particular, at least one memory M 210. Device 200 may include, in particular, a buffer memory, volatile memory (e.g., RAM, for "Random Access Memory"), and / or non-volatile memory (e.g., ROM, for "Read Only Memory"). Device 200 may also include a processing unit UT 220, equipped, for example, with at least one processor P 222, and driven by a computer program PG 212 stored in memory M 210. At initialization, the code instructions of the computer program PG are, for example, loaded into RAM before being executed by the processor P.Said at least one processor P 222 of the processing unit UT 220 can in particular implement, individually or collectively, any of the embodiments of the method of the present application (described in particular in relation to figure 3), according to the instructions of the computer program PG.

[0091] The device may also include, or be coupled to, at least one LO 230 input / output module, such as a COM communication module 232, allowing, for example, the device 200 to communicate with other devices (e.g., sensors 130) of the system 100, via wired or wireless communication interfaces, and / or such as an ACQ acquisition module 234 enabling the device to acquire data (including images) representative of an environment to be monitored. Note that the device 200 can, for example, acquire certain multimodal images via the acquisition module 234 and also receive, via the communication module 232, other images from a sensor 130 of the system 100 (or information (such as point clouds) allowing the construction of other images; these images (or information) can optionally be time-stamped, so as, for example, to be temporally synchronized with each other).

[0092] An input / output module LO 230 of device 200 can be an interface module with a user of the device (also referred to more simply in this application as a "user interface").

[0093] The device user interface refers, for example, to an interface integrated into the device 200, or to a part of a third-party device connected to this device by wired or wireless means. For example, it could be a secondary display of the device or a set of speakers connected wirelessly to the device.

[0094] A user interface can be, in particular, an "output" user interface adapted for rendering (or controlling the rendering) of an output element of a computer application used by the device 200, for example, an application running at least partially on the device 200 or an "online" application running at least partially remotely, for example, on the server 140 of the system 100. Examples of output user interfaces for the device include one or more screens, including at least one graphics screen (touchscreen, for example), one or more speakers, and a connected headset. The interface of the device 200 can, for example, be adapted for rendering the images illustrated in one of Figures 4, 5, 7, and 8.

[0095] By rendering, we mean here a display (or "output" according to English terminology) on at least one user interface, in any form, for example including text, audio and / or video components, or a combination of such components.

[0096] Furthermore, a user interface can be an "input" user interface adapted for acquiring a command from a user of device 200. This may include an action to be performed in connection with a returned item, and / or a command to be transmitted to a computer application used by device 200, for example an application running at least partially on device 200 or an "online" application running at least partially remotely, for example on server 140 of system 100. Examples of input user interfaces for device 200 include audio acquisition means (microphone for example), and / or video acquisition means (such as cameras), radar acquisition means (such as the LIDAR 138 already mentioned), a keyboard, a mouse.

[0097] As indicated above, the device may include or be coupled (via its communication means) to at least one acquisition module, such as at least one sensor (hardware and / or software) coupled to said device (for example, sensors 132, 134, 136, and 138 already introduced), enabling the acquisition of multimodal images of the scene to be monitored. Examples of sensors include an optical camera, enabling the acquisition of so-called "visible" images (color or black and white), an infrared camera, and distance sensors (such as a LiDAR (Light Detection and Ranging) sensor or a sonar).

[0098] The device may also include or be coupled with other types of sensors, capable, for example, of communicating additional information about the physical environment being monitored (a positioning module providing GPS coordinates, for example) and / or about at least one acquisition module (for example, the position and / or orientation of a camera monitoring the scene). The information acquired via the acquisition and input / output modules can, for example, be transmitted to the processing module 210.

[0099] For example, said at least one processor of device 200 can be adapted, in particular, for object detection including:

[0100] • Obtaining multimodal images representing the same real and / or virtual spatio-temporal scene,

[0101] • Obtaining a global image from said multimodal images, by applying at least one attention model;

[0102] • Detection of at least one object present in said scene by inference of an artificial intelligence model taking as input said global image and / or a tensor representing said global image.

[0103] Some of the input / output modules described above are optional and may therefore be absent from device 200 in certain embodiments. In particular, while this application is sometimes detailed in relation to a device communicating with at least one other device of system 100, the process can also be implemented locally by device 200, without requiring communication with another device, in certain embodiments. For example, in some embodiments, the device can use images / information acquired by sensors internal to the device, perform local object detection in these images, and display the result of this detection locally on a screen local to the device.

[0104] Thus, at least some of the embodiments of the method described in this application offer a solution that can help analyze temporally and spatially correlated multimodal images locally, without requiring these images to be sent to a remote server or the cloud. Such embodiments can help preserve the confidentiality of the processed data (particularly when it comes to detecting people or objects to identify them). These embodiments may also not require internet access and are therefore suitable for mobile situations (such as crossing a "white zone" without network coverage, or roaming in a geographical area where network access may entail significant costs for a user).

[0105] In some of its embodiments, on the contrary, the process can be implemented in a distributed manner between at least two devices 110, 130, 140 and / or 150 of the system 100. For example, the device 200 can obtain, via a communication module of the device, information acquired by sensors 130 external to the device and temporarily stored in the storage device 140 (before being transmitted to the device 200), perform a detection based on artificial intelligence modules previously downloaded (after their training for example) from the server 140 and return the results of this detection locally and / or to a third device via a communication module of the device.

[0106] The terms "module," "component," or "element" of the device refer to a hardware element, particularly a wired one, a software element, or a combination of at least one hardware element and at least one software element. The method according to the invention can therefore be implemented in various ways, including in wired and / or software form.

[0107] The procedure of the present application is now presented in relation to Figures 3 to 5.

[0108] Figure 3 illustrates some embodiments of method 300 of this application, based on neural network inference, the logical architecture of which is shown in Figure 4. Figures 5 and 6 illustrate in more detail (in white box diagrams) a module (MEFA module) introduced below in connection with Figure 4. As illustrated in Figure 4, according to this application, a module 420 (hereinafter referred to as “MEFA” (for “Module for Early Fusion with Attention”)) comprising, in particular, convolution and attention layers, is used upstream of a generic deep learning model 440 for vision systems, designed to process a single image (e.g., 2D) at a time. This model can be a detection module (or, alternatively, a classification module).

[0109] This MEFA module 420 takes as input several images 410, 412, 414 (for example, 2D images) of the same scene in different modalities (visible, infrared, etc.). The output of this MEFA module is an intermediate image 430, or a tensor of size (w, h, x) representing it, where w and h represent the spatial dimensions (width and height respectively), in pixels for example, of the intermediate image, and where x represents a configurable number of channels, adapted to the use of the generic model. The intermediate image output of the module 420 can, in particular, have the same spatial dimensions as the multimodal images input to the module 420. The generic model can be a one- or two-stage detection model (or alternatively, a model dedicated to a computer vision task other than detection, such as classification). The process 300 can, for example, be implemented by the electronic device 200 illustrated in Figure 2.

[0110] As illustrated in Figure 3, the process 300 can include a multimodal acquisition 310 of a plurality of images. The image acquisition 310 can, for example, include receiving at least one of these images, or a tensor representing one of these images, on a communication interface of the device, and / or reading at least one of these images, or their tensor, from a local or remote storage area (for example, a database as illustrated by element 150 in Figure 1), or reading a buffer accessed for writing by at least one of the sensors monitoring the scene.

[0111] Optionally, the acquisition 310 of an image may also include the acquisition 312 of additional data (or metadata) associated with that image, such as a date and / or time of capture, an identifier of the camera that acquired the image, information relating to the positioning and / or capture of a camera that acquired that image, an identifier of the image, a resolution used for the acquisition of the image, etc.

[0112] As illustrated in Figure 3, the process may include preprocessing of the raw data corresponding to the obtained images. This preprocessing may optionally include applying at least one image processing step to at least one of the obtained images to "homogenize" all the images before their subsequent use. Indeed, some images may exhibit differences in shooting style or resolution. This could involve, for example, cropping some images taken with a wider shot than others, to retain only portions of a scene of interest in the wider-angle images. It could also involve spatial and / or component-level homogenization of the images.For example, 320 preprocessing might involve modifying the image resolution to obtain images of the same size (in terms of the number and distribution of pixels), or filtering and / or transforming these images to obtain the same number of components for each image. For example, color images obtained by an optical camera might have three components: R, G, and B, corresponding to the video channels (red, green, blue) (or Red, Green, Blue (RGB) in English terminology). Grayscale images obtained by an infrared camera can be grouped into batches of images (3, for example), spaced temporally (within the time interval common to all the multimodal images processed together). LiDAR images can be constructed by projecting the point cloud onto each component of the color images.

[0113] In some embodiments, preprocessing may include correcting at least one of the images. This may involve, for example, filtering the images to correct certain isolated pixels (in terms of color, for example) based on neighboring pixels, or conversely, padding to fill in, for example, "missing" pixels in the resulting images.

[0114] Image preprocessing can also yield images 410, 412, 414 (Figures 4, 5, and 6) whose size is consistent with the expected size of the input data for the MEFA module described below and illustrated in Figures 4, 5, and 6. In some embodiments, the expected size of the MEFA module's input data may depend on the expected size of the detector's input data 440 during the detection (350) of the detected objects. In particular, in some embodiments where the MEFA module outputs data 430 of the same size as its own input data, the expected size of the MEFA module's input data may be identical to the expected size of the detector's input data 440. For example, if the detector expects a tensor of size (w, h, x) as input, the sizes of the resulting images can be modified to obtain images of dimension w, h for each of the x channels.The values ​​w, h and / or x can for example be configurable data (step 302), accessible by reading a memory area (such as a configuration file for example).

[0115] For example, when using a MEFA module that produces an output image with the same dimensions as its input images, the resulting images can be scaled and recentered to achieve an input resolution for the MEFA model identical to the resolution of the generic detection model. Consequently, the output image of the MEFA module, concatenating the input images, will also have the resolution expected by the model.

[0116] For example, in the case of using a MEFA module that produces an output image with the same dimensions as the input images of that module, and using, for detection, a model designed to process images with a resolution of 640 x 640 pixels, such as a YOLO© model like YOLOv8, the resulting images can be scaled and recentered to have a resolution of 640 x 640 pixels as input to the MEFA model. As illustrated, the process 300 can also include processing 330 of the images preprocessed by the MEFA module 420. This processing includes, in particular, applying at least one convolution 331 and applying a local attention mechanism 332 to at least one of the images (for example, to the set of images once spatially unified).

[0117] The application of convolution can allow for the creation of a better representation of the latent space.

[0118] Attention mechanisms serve as a guide to select the information needed by the detector used downstream of the MEFA model.

[0119] The local attention mechanism, when applied to a representation of an image, selects the information necessary for the detector within that representation (independently of the other images in the multimodal image plurality). For example, applying a local attention mechanism to a ("visible") image acquired by an optical camera can allow object detection to focus on specific color, shape, and texture information within portions of that "visible" image.

[0120] These convolution and / or attention mechanism applications 331, 332 on at least one of the images may be optional in some embodiments.

[0121] The method 300 can also include a fusion 333 of the features of the different multimodal images. This could, for example, involve a fusion by addition (at the pixel level, for example) of the image features. Such an embodiment offers advantages in terms of memory usage and processing time. It could also, for example, involve a fusion by concatenation of the images (at the level of each channel, for example). Such an embodiment can preserve all the image features without loss of information, and therefore offer advantages in terms of detection reliability.

[0122] As illustrated, process 300 may include an application 334 of at least one convolution and an application 335 of an attention mechanism (“global”) to the merged features.

[0123] These applications 334, 335 of convolution and / or global attention mechanism on the result of the fusion may be optional in certain embodiments. The application 338 of a global attention mechanism can make it possible to identify, in the merged intermediate features, relationships between features originating from at least two of the multimodal images (and in particular between features some of which have been considered important by a local attention mechanism). These relationships can make it possible to emphasize, and / or retain, or not, features of the multimodal images.

[0124] For example, applying a global attention mechanism can fuse shape information from a visible image with depth information from a LiDAR image, thus facilitating the detection of an object in the "pedestrian" class. In one application example, a life-size portrait of a person would not be considered a pedestrian because the depth information provided by the LiDAR indicates that it is a flat object.

[0125] The MEFA module produces as output an intermediate image 430, or the tensor representing it, whose size can be identical, for example, to the size of each of the input images of this module.

[0126] The process 300 may include obtaining 350 at least one position and / or at least one class 450 of objects from the tensor obtained. As illustrated in Figures 4 to 6, this obtaining of position(s) and / or class(es) may include an inference of a detector 440, after supplying this detector with the intermediate image 430 obtained via the MEFA module, or the tensor representing it, to obtain at the output of this detector 440 at least one position and / or at least one class (element 450) of at least one object detected in this intermediate image 430.

[0127] Optionally, the process may include, prior to the implementation of the detector 440, an adaptation 340 of the dimensions of the intermediate image, or of the tensor representing it, to obtain an image, or a tensor, of dimensions adapted to the detector 440 (Figures 4 to 6) in charge of the detection.

[0128] For example, the 340 adaptation may include an application of a convolution to the merged image (or resulting from the application of the global attention model) so as to adapt the dimension, in terms of the number of channels, of the output image of the MEFA 420 model, and to make it directly usable by the 440 detector used.

[0129] Thus, as explained, the tensor, once adapted, may be of size (w, h, x), where (w, h) represents the expected spatial dimension (w: width, h: height), in pixels, for an input image of the detector 440 used during the acquisition step 350 described above, and x represents the number of channels expected for this input image of the generic model of the detector 440. In some embodiments, the process may include a 360 rendering of at least one indication relating to an object detected on a user interface of said device 200.

[0130] For example, as illustrated in Figures 3, 7, and 8, in some embodiments, the 360 ​​rendering may include a rendering 362 of at least one of the multimodal images 410, 412, 414, preprocessed or not, and / or a rendering 364 of an overall (intermediate) image 430 constructed from the merged and possibly post-processed features of the multimodal images. The 360 ​​rendering may further include a rendering 366 of a copy 700 of at least one (410) of the multimodal images, onto which has been inserted at least one encompassing geometric shape representing the location, in that image, of a detected object, in association with a rendering 368 of an alphanumeric (e.g., a label) and / or graphical indication representing the class associated by the artificial intelligence model with the detected object.

[0131] In other embodiments, only the representative indication of the class associated by the artificial intelligence model with the detected object can be rendered. As illustrated in Figure 3, the MEFA 420 module may have been previously parameterized (during an initialization phase, for example) so that its output is adapted to the input of the generic model (in terms of the resolution of the input and / or output images, and the number of input and / or output channels). A default value for the number of output channels of the MEFA model can be defined (making this parameterization optional).

[0132] Similarly, the MEFA 420 module may have been previously configured (during the initialization phase, for example) to set a number of multimodal images (associated with the same spatiotemporal scene) to be considered by the MEFA module. A default number of multimodal images input to the MEFA model can also be defined (for example, 3), making this parameterization optional.

[0133] It should be noted that process 300 can include, prior to inferring the models used (for example, after setting the number of input images, image resolution, and / or the number of input and / or output channels of the model), joint training of the MEFA module (particularly local and / or global attention mechanisms) and the detector-type neural network model used. Indeed, as mentioned above, attention mechanisms are used to identify elements useful to the detector. Joint training of the MEFA module 420 and the detector 440 can therefore help improve the relevance of the attention mechanisms.

[0134] We now present in more detail, in connection with figures 5 and 6, the internal architecture of the MEFA module.

[0135] As illustrated in Figure 5, the MEFA 420 module can include at least one series 4210, 4212, 4214 of one or more convolutional layers, each series being designed to process one of the multimodal images input to the MEFA model. The output of each of these series of convolutional layers can be provided to a local attention model 422 (implementing a local attention mechanism). This local attention model can vary depending on the image modalities.

[0136] Depending on the implementation, these convolutions can be the same for all multimodal images or vary according to the images (in particular according to the modalities of the images), for example in terms of weight, the output tensors being for example all of the same dimension.

[0137] The local attention model may include a layer for merging data relating to each of the images (i.e., data from the convolution layers).

[0138] Note that these convolution series and local attention models may be optional in some embodiments.

[0139] The merged data can be provided to at least one convolution layer 423 and then to a global attention model 424 (implementing a global attention mechanism).

[0140] Note that this at least one convolution layer and this global attention model may be optional in some embodiments.

[0141] In at least some embodiments, the MEFA module can include either local attention models (and associated convolution layers as described above), or a global attention model (and at least one associated convolution layer as described above), or both local attention models and a global attention model (and their convolution layers), as illustrated in Figures 5 and 6.

[0142] The MEFA module can also optionally include at least one convolutional layer 425, allowing the size of the data from the global attention model, if one exists, or failing that, from the fusion model, in order to adapt it to the image size for which the detector was designed. Figure 6 shows the architecture used for the experiment, the results of which are detailed later. The obtained single-mode images 410, 412, 414 are provided as input to convolutional blocks 4210, 4212, 4214, whose outputs are provided to local attention blocks 4220, 4222, 4224. For the experiment, the convolutional blocks are "Inception Block version A" type blocks of the model known as "Inception V3" (Szegedy et al., "Rethinking the Inception Architecture for Computer Vision."). The outputs of the local attention blocks are processed by merging (multiplication operations 4225, 4226, 4227 then concatenation 4228).The result of the fusion is provided as input to a convolution block 423 (of type "Inception Block version A" in the experiment), the output of the convolution itself being provided to global attention blocks 4240, 4242, 4244, whose outputs are merged (multiplication operations 610, 612, 614 and concatenation 616).

[0143] The solution in this application may help detect objects more effectively than some prior art solutions, by exploiting the different types of images provided by the sensors in System 100.

[0144] The process of this application can be implemented, in at least some of its embodiments, in various technical fields, using multimodal images coupled with deep learning models, whether these are designed for images or more general.

[0145] For example, the process can be implemented in the field of medical imaging with multimodal images such as Magnetic Resonance Imaging (MRI) and Positron Emission Tomography (PET). It can also be implemented in the field of satellite imaging, with multimodal images of the visible, infrared, and hyperspectral spectrum. It can also be implemented for surveillance systems and / or for automated driving. This application may find applications in areas where processing time constraints are significant, such as driver assistance systems, for example, autonomous driving (AD) or autonomous driving assisting systems (ADAS). Experimentation

[0146] The following are the experimental results of the MEFA module used by the process of this application.

[0147] For this experiment, training was performed on 10,000 images and 100,000 bounding boxes.

[0148] In this experiment, the MEFA model aims to detect pedestrians and vehicles within a scene (represented by three multimodal images, with respective modalities of "visible," "infrared," and "LIDAR," provided as model inputs) and to indicate the coordinates, in at least one of these images, of bounding boxes representing the detected pedestrians and vehicles. The bounding boxes are labeled according to the detection result (position and / or class). The cameras are placed inside the car, and the scene captured by these cameras is a scene within a driver's field of vision while driving. Since the LIDAR sensor provides a 3D point cloud, the points in the cloud were projected onto the coordinates (in pixels) of the visible image to produce a 2D image superimposed on this visible image (same objects in the same locations).Alternatively, the points of the cloud could have been projected onto the coordinates of an image acquired by another sensor, for example the infrared image.

[0149] The database used for training is the DENSE © database, composed of damages under three different modalities: visible, infrared and LIDAR.

[0150] To verify the quality of the detection performance of the tested models (with and without coupling to the MEFA module), the following protocol was implemented:

[0151] - Search for the best hyperparameters of the generic model being tested, trained on each of the modalities

[0152] - Search for the best hyperparameters of the generic model tested, trained on concatenations of three multimodal images of the same spatio-temporal scene, of respective modalities "visible", "infrared" and "LIDAR".

[0153] - Use the 5-fold cross-validation method (k-folding). This method creates 5 training and validation subsets from the same dataset in order to estimate the model's performance multiple times and thus obtain a more robust estimate (with mean and variance) of the model's performance.

[0154] In the experiment, we: • Trained the tested unimodal models for each of the modalities,

[0155] • Trained the generic model tested on the concatenation of the 3 modalities,

[0156] • Trained the generic model tested combined with the MEFA module using the hyperparameters of the generic model trained on the concatenation of the 3 modalities.

[0157] The evaluation of the models is carried out using the "mean of the mean prediction" (i.e. the metric known as "Mean Average Precision" (mAP) in English) with an interval on the union equal to 0.5 and a confidence parameter of 0.1. A first experiment is carried out on an embodiment where the MEFA module is to the generic YOLOv8 model (as detector).

[0158] In this embodiment, the convolution layers can, for example, be layers of type "Inception Block version A" and the attention model used is the local and global attention model of Chaturvedi et al., 2022 ("Pay attention to adverse weather: Weather-aware attention based object detection. - 2022 - 26th International Conference on Pattern Recognition (ICPR), pages 4573-4579).

[0159] Table 1 below summarizes the results obtained with the generic YOLOv8 model in Nano size.

[0160] [Table 1]

[0161] For this experiment, the YOLOv8 Nano Visible / Infrared / LIDAR models are state-of-the-art YOLOv8 models trained on a 3-channel image. The YOLOv8 Nano Concatenation model is adapted to handle an input resulting from the concatenation of the three modalities (visible, infrared, and LIDAR), i.e., a 9-channel “image”.

[0162] The MEFA model combined with the generic Y0L0v8 model corresponds to the MEFA module without the optional convolution layer after merging (and / or the application of convolution followed by a global attention model). This implies that the Y0L0v8 model has been adapted to handle inputs corresponding to the output of the MEFA module, namely an "image" composed of 256 channels.

[0163] The results of this experiment prove that the MEFA model combined with the generic Y0L0v8 model surpasses in accuracy the Y0L0v8 model adapted for concatenation.

[0164] In a second example, the MEFA model is tested in conjunction with the generic RT-DETR vl model (Zhao et al., "DETRs Beat YOLOs on Real-Time Object Detection") pre-trained on the COCO database. Table 2 below presents the preliminary results obtained on the MEFA + RT-DETR model.

[0165] [Table 2]

[0166] The RT-DETRvl Concatenation model corresponds to a model adapted to support an input resulting from the concatenation of the three modalities (visible, infrared and LIDAR), i.e. a 9-channel “image”.

[0167] The MEFA model combined with the generic RT-DETRvl model corresponds to the MEFA module without the optional convolution layer after merging (and / or the application of convolution followed by a global attention model). This implies that the Y0L0v8 model has been adapted to handle inputs corresponding to the output of the MEFA module, namely an "image" composed of 256 channels.

[0168] The results of this experiment prove that the MEFA model combined with the generic RT-DETRvl model surpasses in accuracy the RT-DETRvl model adapted for concatenation.

Claims

DEMANDS 1. Object detection method comprising: • Obtaining multimodal images representing the same real and / or virtual spatio-temporal scene, • Obtaining a global image from said multimodal images, by applying at least one attention model; • Detection of at least one object present in said scene by inference of an artificial intelligence model taking said global image as input.

2. Object detection method according to claim 1 wherein said artificial intelligence model is adapted for object detection in a single-mode 2D image.

3. Object detection method according to claim 1 or 2 wherein said global image is obtained by applying at least one first attention model to an image resulting from a fusion of said multimodal images.

4. Object detection method according to any one of claims 1 to 3 wherein at least a first of the fused multimodal images is pre-filtered by application of at least a second attention model.

5. Object detection method according to any one of claims 1 to 4 wherein at least one second of the merged multimodal images is pre-filtered by application of said second attention model.

6. Object detection method according to any one of claims 1 to 5 wherein the set of said merged multimodal images is pre-filtered by application of said second attention model.

7. Object detection method according to any one of claims 1 to 4 wherein at least one second of the merged multimodal images is pre-filtered by applying at least one third attention model, different from said second attention model.

8. Object detection method according to any one of claims 1 to 7 wherein the method comprises obtaining an overall image from each component of said multimodal images.

9. Object detection method according to any one of claims 1 to 8 wherein the method comprises a joint rendering of at least one geometric shape encompassing said at least one detected object in said global image and of a label representative of the classification of said detected object by said artificial intelligence model.

10. Object detection method according to any one of claims 1 to 9 wherein the method includes joint learning of said attention models and said artificial intelligence model.

11. Electronic device comprising at least one processor, said at least one processor being configured for object detection comprising • Obtaining multimodal images representing the same real and / or virtual spatio-temporal scene, • Obtaining a global image from said multimodal images, by applying at least one attention model; • Detection of at least one object present in said scene by inference of an artificial intelligence model taking said global image as input.

12. A computer program product comprising instructions for the implementation, when the program is executed by a processor of an electronic device, of an object detection method implemented in an electronic device, said method comprising • Obtaining multimodal images representing the same real and / or virtual spatio-temporal scene, • Obtaining a global image from said multimodal images, by applying at least one attention model; • Detection of at least one object present in said scene by inference of an artificial intelligence model taking said global image as input.

13. A processor-readable recording medium on which is recorded a computer program comprising instructions for the implementation, when the program is executed by a processor of an electronic device, of an object detection method, said method comprising: • Obtaining multimodal images representing the same real and / or virtual spatio-temporal scene, • Obtaining a global image from said multimodal images, by applying at least one attention model; • Detection of at least one object present in said scene by inference of an artificial intelligence model taking said global image as input.

Images