Construction of panoramic images from vehicle-mounted camera images

Abstract

The invention relates to a method for estimating a panoramic image according to a panoramic field from a vehicle and implemented by a processing unit, the processing unit being connected to a camera mounted on the vehicle at a fixed position p1, the camera defining a field narrower than the panoramic field and being configured to capture images in a reference frame R1, the panoramic field being defined from a point at a fixed position p2 on the vehicle, and associated with a reference frame R2, the method comprising the following steps: - obtaining a past image from the camera, capturing a view of a scene according to the first field for a past position of the vehicle; - transforming from R1 to R2 said past image into a transformed image showing a transformed view of the scene, - estimating the panoramic image from the transformed image, showing a panoramic view of the scene.

Description

Description Title: Constructing panoramic images from on-board camera footage from a vehicle technical field

[0001] This disclosure falls within the domain of vehicle driver assistance systems and more specifically the construction of panoramic images of the environment outside the vehicle. Previous technique

[0002] Advanced Driver Assistance Systems (ADAS) for road vehicles have experienced rapid growth in recent years, driven by technological advancements and increased demand for safer and smarter vehicles. The implementation of most vehicle assistance systems relies on the use of multiple on-board cameras with varying characteristics (e.g., different positions and orientations), enabling the implementation of diverse functionalities such as traffic sign recognition, lane keeping assist, obstacle detection, and parking assistance. These different cameras allow for the combined implementation of multiple functionalities for a single vehicle, ensuring comprehensive, reliable, and precise driver assistance.

[0003] For example, in the case of parking assistance, it is common to implement a real-time panoramic (e.g., 360-degree) view of the vehicle's immediate surroundings, generally presented as a 3D model and / or a low-angle view of the vehicle and its surroundings on the driver's dashboard, for example, when the driver is driving at low speed and / or engages reverse gear. This 360° functionality can rely on the real-time capture of panoramic images by several wide-field (or panoramic) cameras, for example, 180-degree cameras, positioned at the front, sides, and / or rear of the vehicle, and / or on other data measured by separate sensors on the vehicle, such as audio data from ultrasonic sensors.The panoramic view is particularly useful when parking or in confined spaces, allowing the driver to easily detect obstacles invisible from inside the passenger compartment.

[0004] However, the cameras used for such 360° functionality, despite their wide field of view, have a limited depth of field, capturing images only a few meters deep. These cameras would therefore not be suitable for other, longer-range ADAS functionalities, such as capturing elements tens or even hundreds of meters away, like obstacle detection, lane keeping assist, or traffic sign recognition. Such other ADAS functionalities rely on the use of different, longer-range cameras, for example, a front-facing camera mounted on the vehicle's rearview mirror specifically designed to capture these distant elements.Conversely, this long-range front camera would not be suitable for 360° functionality because it has a more restricted field of view laterally and accentuates blind spots, which limits its ability to reliably detect the entire environment of a vehicle.

[0005] Thus, in the current state of technology, integrating an optimized set of ADAS functionalities into a vehicle requires the combined use of multiple cameras with parameters specifically adapted to each function. Such a proliferation of cameras on a vehicle leads not only to increased costs but also to design challenges, vehicle weight, the size of the onboard electronics, the complexity of data capture and processing, maintenance, and the reliability of the onboard systems. The use of multiple cameras with specific parameters to enable various assistance functionalities also limits the accessibility of these technologies, particularly for entry-level vehicles.In addition, some assistance features, for example the 360° feature, require the use of cameras and sensors positioned in areas that provide a clear panoramic view of the vehicle's surroundings (such as at the vehicle's bumpers) but which may be exposed areas more susceptible to damage, which can also lead to reliability and maintenance difficulties for the vehicle's equipment.

[0006] There is therefore a need to offer a reliable, precise and efficient implementation of vehicle driver assistance systems while limiting their complexity, cost and bulk on vehicles. Summary

[0007] This disclosure improves the situation.

[0008] A method is proposed for estimating at least one panoramic image according to a field of view called panoramic from a vehicle and implemented by a processing unit, said vehicle being located at a position called of interest associated with a given current instant, the processing unit being connected to a first camera mounted on the vehicle at a first position, fixed in a reference frame of the vehicle, said first camera defining a first field of view narrower than the panoramic field of view and being configured to capture images in a first reference frame, the panoramic field of view being defined from a viewpoint associated with a second position, fixed in the vehicle's frame of reference and distinct from the first position, the panoramic field of view being associated with a second frame of reference, The process includes the following steps: obtain at least one past image acquired by the first camera at a past time prior to the current time, the past image capturing a first view of a scene according to the first field of view for a past position of the vehicle at the past time; transform said past image into a transformed past image by a change of reference frame from the first reference frame to the second reference frame, said transformed past image representing a first transformed view of the scene; estimate the panoramic image, from at least the transformed past image, the panoramic image representing a panoramic view of the scene, estimated from at least the first transformed view.

[0009] A panoramic field of view is defined as a field of view around the vehicle, defined by a viewing angle of more than 180° and up to 360°, defined from the viewpoint associated with the second position located on the vehicle (i.e. having the second position as its origin), and with a depth of field of at least several tens of meters.

[0010] The method described in this disclosure advantageously allows for the simple and optimized processing of one or more past images, acquired from the first camera's narrower field of view than the panoramic field of view, to form a panoramic image of the environment outside the vehicle. The described method thus enables observation of the entire area in front of the vehicle from the panoramic field of view, covering both elements very close to the vehicle, as they would be observed by a front-facing camera, and distant elements as observed by the first camera, as well as intermediate lateral elements.

[0011] At times prior to the current moment, images can be acquired by the first camera, capturing views of the scene, for example, the front of the vehicle. At the current moment, when the vehicle has moved to its position of interest, for example, by moving forward in a straight line, the past image(s) acquired at respective past times have each captured a past view of the scene, set back from each other and from the vehicle's position of interest. The past image(s) thus allow for coverage of all or part of the environment outside the vehicle to estimate the panoramic image, unlike the single current image, which is limited to the view at the current moment from the first camera's initial field of view.This process offers the advantage of eliminating the need for an additional front-facing camera, or "360 front camera," to capture wide-angle frontal images for panoramic imaging. By relying solely on past images acquired by the primary camera, it becomes possible to completely remove this additional 360 camera. This reduces vehicle design and maintenance costs, while also decreasing the number of cables and the overall vehicle weight.

[0012] The first camera can be a dedicated, long-range ADAS camera for detecting road signs, distant obstacles, lane changes, and so on. Such an ADAS camera is generally distinguished by higher resolution, a higher refresh rate, and better sensitivity in low-light conditions, compared to a 360 camera, which, although offering a wider field of view, typically has lower resolution, a lower refresh rate, and reduced light sensitivity. By implementing the described process, it is therefore possible to use higher-quality images to create more detailed panoramic images.

[0013] The process described in this disclosure further enables the processing of at least one image acquired in the past so that it is transformed in the second frame of reference, from a viewpoint of interest, as if it had been acquired by a camera in the second frame of reference. This transformed past image(s) are then compatible with existing functionalities and their dedicated algorithms, including parking and maneuvering assistance functionalities, such as the 360° panoramic viewing functionality for parking assistance (or "360 functionality"). Through the transformation step, it is thus possible to manipulate images actually acquired by the first camera and to "shift" the viewpoint of the scene captured by these images according to the chosen viewpoint of interest, by performing geometric transformations on the images from the first camera.We can thus modify the position, orientation, field of view, projection, to obtain different views from the images acquired by the first camera 1, without needing to use images acquired by another camera positioned at the desired viewpoint.

[0014] The features described in the following paragraphs may optionally be implemented, independently of each other or in combination with each other:

[0015] According to one embodiment, the first position may belong to an upper part of the vehicle and the second position may belong to a lower part of the vehicle.

[0016] The first camera can be mounted in the vehicle, typically near the windshield and rearview mirror, facing forward. The second camera, and its point of interest, can be configured to be placed in a lower position, usually near the license plate or bumpers. The higher position protects the first camera from most impacts and weather, but it also creates blind spots, especially in front of the vehicle. Conversely, a camera physically positioned lower, such as the second camera, would be highly exposed to impacts and weather but would offer a comprehensive view of the immediate surroundings in front of the vehicle with few blind spots.By considering a viewpoint of interest at this second low position, an observation point that meets the limitations of the first camera's viewpoint can be obtained, while avoiding the risks of wear and tear on a physical camera directly positioned at the second position.

[0017] In a production, the first field of view and the panoramic field of view can cover scenes in front of the vehicle.

[0018] Thus, with these respective camera positions and a forward-facing viewpoint, as the vehicle moved forward to its position of interest, the past image(s) acquired at respective past times each captured a past view of the scene that was set back from the vehicle's position of interest. In other words, some past images may capture views to the rear of the vehicle, while the current image is limited to the view to the front of the vehicle at the current time.

[0019] According to one embodiment, the process may further include the following steps: to obtain a plurality of images including the past image and at least one other image from among other past images acquired by the first camera at respective past times prior to the current time and / or a present image capturing a second view of the scene according to the first field of view for the position of interest of the vehicle at the current time, transform the plurality of images obtained into a plurality of images transformed by said change of reference frame, and in which the panoramic image estimation may include: a cutting and selection of parts from the plurality of transformed images, and a mosaicking of said parts to form said panoramic image.

[0020] Thus, in order to estimate a continuous panoramic image, the transformed views can be extended and stitched together. This can be achieved by cutting and generating a mosaic of parts from several transformed past images, taking into account areas of overlap between them.

[0021] According to another embodiment, the cutting, selection and / or mosaicking of said parts may depend on at least one element among: data relating to the image acquisition frequency of the first camera, a movement of the vehicle, an element of interest represented in at least two transformed images.

[0022] These factors can influence the relative arrangement and number of parts in the images, which may overlap from one image to another, or conversely, whether a single image could capture a certain part of a view. The cropping and mosaicking can then be calibrated based on overlapping parts and / or by identifying common elements of interest.

[0023] According to one implementation, the process may also include the following steps, prior to the estimation of the panoramic image: define a validity criterion at least for the transformed past image; Based on the aforementioned validity criterion, determine a data point relating to a confidence index of the transformed past image. and in which the estimation of the panoramic image may depend on said confidence index.

[0024] Evaluating such a validity criterion can be implemented to verify the validity of images acquired over a given period, compared to images acquired later, relative to the current time. This ensures that the processed, older images are still up-to-date and properly refreshed, and verifies, for example, that no new elements or obstacles have appeared in the vehicle's environment between the older images, which would therefore not capture the new element, and the more recent images.

[0025] In one embodiment, the validity criterion may depend on at least one of the following elements: a difference between the past moment and the current moment, data from vehicle sensors, a level of correlation between pixels of the transformed past image and pixels of another transformed image from an image acquired by the first camera at the current time.

[0026] The process described in this disclosure may also include: implement a correction of the pixels of said at least one transformed past image, and in which the confidence index can be determined on the basis of at least one corrected transformed image.

[0027] Depending on the validity of the images, the process may involve correcting past images for parameters that may have changed over time. A confidence level can also be associated with these corrected images, which can then be presented to a vehicle user, such as the driver. Image correction ensures visual continuity and, based on the determined confidence level, can warn the driver of the possibility that an obstacle may have appeared between several images and / or that some images are too old and may be outdated and not reflect the current situation.

[0028] Depending on the implementation, prior to estimating the panoramic image, the following steps can be implemented: obtain at least one additional image captured by at least a third camera, at a third fixed position in the vehicle's reference frame and distinct from the first and second positions, said at least one additional image capturing an additional view of the scene, not disjoint from the first view; and in which the estimation of the panoramic image may include the assembly of at least one transformed image with at least one additional image, said panoramic view being able to be estimated from the first transformed view and at least one additional view.

[0029] Images obtained by one or more other cameras on the vehicle can be implemented to enhance the panoramic image estimation, in particular to supplement any areas less or poorly covered by past images from the first camera.

[0030] According to one implementation, said at least a third camera can be at least one of the following: a side camera of the vehicle, a rear camera of the vehicle.

[0031] This disclosure also relates to a vehicle driver assistance system that can be implemented by a processing unit connected to a first camera mounted on the vehicle; the driver assistance system may include: an estimation of a panoramic image, from at least one image acquired by the first camera, using an estimation method as proposed, and a display of the panoramic image on a display unit of the vehicle, and image processing of images acquired by said first camera to perform at least one of the following: o a lane change detection system for said vehicle, o a traffic sign detection; o obstacle detection.

[0032] The vehicle's processing unit can then implement both the process described in this disclosure and ADAS-type functionalities, as well as any other so-called intelligent vehicle features. The process described in this disclosure can therefore be easily adapted to all vehicles already equipped, in whole or in part, with these ADAS and driver assistance functionalities. It requires only an update of the existing functionalities in the processing unit, without requiring any hardware modifications.

[0033] Furthermore, the processing unit has a predetermined computing power, a portion of which is allocated to processing and controlling the ADAS camera (first camera) while the vehicle is driving. This computing power can also be used during parking or maneuvering phases, in addition to that dedicated to 360° functionalities. This allows the process described in this disclosure to be implemented optimally, without requiring additional hardware resources.

[0034] In another aspect, this disclosure relates to a computer program that may include instructions for implementing the process according to this disclosure, when executed by a processing circuit.

[0035] In another aspect, this disclosure also relates to a processing unit embedded in the vehicle comprising a processing circuit and memory for implementing the process according to this disclosure. Brief description of the drawings

[0036] Other features, details, and advantages will become apparent upon reading the detailed description below and analyzing the attached drawings, on which: Fig. 1

[0037] [Fig. 1] shows a diagram of a vehicle according to one embodiment. Fig. 2

[0038] [Fig. 2] shows a diagram of a vehicle and different fields of view visible from the vehicle according to one embodiment. Fig. 3

[0039] [Fig. 3] shows a diagram of image acquisition of a vehicle moving over time according to one embodiment. Fig. 4

[0040] [Fig. 4] shows an example of estimating a panoramic image according to one embodiment. Fig. 5

[0041] [Fig. 5] shows a succession of steps in a process for estimating a panoramic image according to one embodiment. Description of the implementation methods

[0042] Reference is now made to Figure 1. Figure 1 illustrates an example of a vehicle V considered in this disclosure. Vehicle V can be a motor vehicle. It can be, for example, a passenger vehicle, a commercial vehicle, or an industrial vehicle, and can correspond to a car, van, motorcycle, truck, or bus. Vehicle V can also be a towed vehicle, or a vehicle towing, such as a trailer, semi-trailer, or caravan.

[0043] Vehicle V includes at least one on-board electronic system providing a range of vehicle functions or applications. For example, these functions may include cruise control, power steering, airbag deployment, etc. In addition, vehicle V includes a driver assistance system, comprising ADAS-type features such as obstacle detection, lane change assist, automatic lane change, parking assist, automatic braking, night vision enhancement, blind spot detection, driver fatigue detection, lane departure warning, cruise control, collision warning, etc. Vehicle V may include at least one processing unit 10 implementing one or more of these features.The processing unit 10 is specifically configured to implement a panoramic image estimation process, which will be detailed in Figure 5. The processing unit 10 may include at least one processor and one memory (not shown in the figures) for the implementation of these functionalities and / or the process.

[0044] The vehicle V can be associated with a frame of reference, called the vehicle frame of reference Rv. The vehicle frame of reference Rv can, for example, be defined, as illustrated in Figure 1, by an orthonormal basis (xv, yv, and zv) and an origin point, for example, defined at the center of gravity of the vehicle V. In other embodiments, such an origin point can be defined at any other fixed element of the vehicle V. Thus, the elements on board the vehicle V (e.g., electronic systems, sensors) have a position considered fixed in the vehicle frame of reference Rv.

[0045] The vehicle V can be defined within an environment (e.g., comprising one or more roads, traffic lanes, sidewalks, traffic lights, etc.) associated with a world frame Rm, as illustrated in Figures 1 and 2. The world frame Rm can, for example, be defined by an orthonormal basis (xm, ym, and zm) and an origin point fixed in the environment. The vehicle V can, in particular, evolve within the environment over time, moving from past positions to a position of interest, with the positions and movements of the vehicle V being considered within the world frame Rm (for example, when the vehicle V moves along a traffic lane or when the vehicle moves from a position in a traffic lane to a parked position). Portions of the environment observable from the vehicle V (e.g., by a camera mounted on the vehicle V) are designated as scenes.The scene (or plurality of scenes) can correspond to portions of the environment located outside, in front of, to the sides and / or behind vehicle V, and can include elements of interest in the environment, for example fixed obstacles, such as a median strip, a sidewalk, another stopped vehicle, or road signs and / or moving obstacles, such as pedestrians, animals, moving vehicles, etc. Thus, the scene can correspond to a part of the environment of interest to be observed, particularly by a driver of the vehicle.

[0046] In the following description, a position of vehicle V is understood as a position of vehicle V at a given instant within the environment and considered in the world frame Rm. Such a position of vehicle V can, for example, designate the spatial coordinates in the world frame Rm of a fixed point belonging to vehicle V (e.g., the center of gravity of vehicle V, a point on a wheel, on the bumper, or on the vehicle body) or a set of spatial coordinates in the world frame Rm of a point cloud modeling the occupation of vehicle V in the environment. Such a position can also be understood as including an orientation of vehicle V in the environment within the world frame Rm.

[0047] For the purposes of this disclosure, vehicle V is considered to be positioned at a so-called position of interest at a given present (or current) time. At a time before or after such a present time, vehicle V may have a position different from the position of interest (Le., if the vehicle has moved in the meantime) or a position similar to the position of interest (Le., if the vehicle has remained stationary in the meantime). In particular, the position of interest may remain unchanged over a more or less extended period. For example, when the vehicle is parked, the position of interest may remain unchanged for a few seconds or up to several months. In another example, when the vehicle is temporarily stopped in a traffic lane to comply with traffic regulations (e.g., to stop at a red light or a stop sign), the vehicle's position of interest may remain unchanged for a few seconds or up to a few minutes.

[0048] For the purposes of this disclosure, at least one past position occupied by vehicle V at a time prior to the present time is considered. Such past and present times may or may not be consecutive (e.g., other positions at other intermediate times may or may not be considered between these two times). Such a past position is notably different from the position of interest. In particular, between the past time and the present time, vehicle V may have moved along a given trajectory of interest, for example, along a straight path (e.g., in a traffic lane). More generally, between the past time and the present time, vehicle V may also have moved forward and / or backward (e.g., along the xm axis) and / or laterally (e.g., along the ym axis).Vehicle V may have moved at higher or lower speeds, for example, speeds may be between approximately 0 and 200 km / h for forward and / or forward lateral movement, and between approximately 0 and 20 km / h for backward and / or backward lateral movement, for a conventional motor vehicle.

[0049] Vehicle V may also include a variety of sensors, including one or more onboard cameras, to capture a range of data and information relating to vehicle V and its internal and / or external environment, for the implementation of the vehicle's functionalities. Each of these sensors can be connected to the processing unit 10. In addition to cameras, vehicle V may also include other sensors such as radar, lidar, and speed sensors, for example.

[0050] In particular, the vehicle V includes at least one first camera 1 located at a first position p1 on the vehicle V. Such a first position p1 is considered fixed in the car's frame of reference Rv. The first camera 1 can be associated with a first frame of reference R1, defined by a three-dimensional coordinate system (x1, y1, z1) and an origin, for example at position p1.

[0051] The first position p1 is preferably a high position on a front part of the vehicle V. The high position is defined as a position in the upper part of the vehicle, e.g., the upper part above a mid-plane of the vehicle V, and the front part of the vehicle is defined as a position between the driver and the front end of the vehicle. The first camera 1 can, for example, be located inside the vehicle's passenger compartment, on the windshield, at the level of the interior rearview mirror.

[0052] In the context of the invention, the first camera 1 is configured to capture images according to a first field of view a1, covering a scene of the environment, for example an area in front of the vehicle V, as illustrated in Figures 2 and 3. The first field of view a1 is defined in particular by a predefined angle of view 01 and depth of field d1, as illustrated in Figure 2. Preferably, the horizontal dimension of the angle of view 01 can be between 90 and 120° and the depth of field d1 can typically extend from 5 meters to approximately 150 and 250 meters. In addition, besides its horizontal dimension, the angle of view 01 can have a vertical dimension typically between 60 and 120° (the vertical dimension of 01 is not shown in the figures to simplify the representation of the field of view).In particular, according to the first position p1 of the first camera 1, an area in the immediate vicinity of the vehicle V - for example an area located between 0 and 5 meters in front of the vehicle and / or at the level of the bumper of the vehicle V - may be obscured from the first field of view a1.

[0053] By virtue of its initial position p1 and its depth of field, the first camera 1 can therefore capture distant elements located within the observable area of ​​the first field of view a1, such as road signs, distant obstacles, other vehicles stationary and / or in motion, etc. The first camera 1 can, for example, correspond to a front-facing ADAS camera, enabling the capture of images for the detection of road signs or lane changes, for example.

[0054] However, as mentioned previously, the field of view a1 of the first camera 1 may obscure nearby elements (e.g., less than 5 meters from the first position p1 in the field of view a1), for example, those hidden by the front of the vehicle V. Typically, the field of view a1 of the first camera 1 may obscure low-lying elements or obstacles (e.g., located at the level of the front wheels of the vehicle V) so that they are outside the field of view a1, such as an animal, a bush, a child, etc. Such obstacles may also be outside the field of view of an observer located inside the vehicle V, such as the driver, particularly during maneuvers for parking the vehicle V.Similarly, due to the limited viewing angle 01 of the first field of view a1 of the first camera 1, the first field of view a1 may exclude lateral elements located beyond the viewing angle 01, for example located near the headlights, wheels, or doors of the vehicle V.

[0055] In addition to the first camera 1, the vehicle V may also include additional cameras, for example side and / or rear cameras. Such additional cameras may be positioned on the sides of the vehicle, for example at the sill below the side doors, at the logo or the trunk handle.

[0056] Reference is now made to Figure 5, which presents the successive steps of the method according to this disclosure, for estimating at least one panoramic image IMGp around the vehicle V, from at least one processing of one or more images acquired by the first camera 1. The proposed method then makes it possible, from images captured according to a field of view a1 having a restricted angle of view 01, to observe the entire area at least in front of the vehicle according to a panoramic field of view a2 with a wider angle of view than that of the first field of view a1.In particular, the estimation of such a panoramic image capturing the environment according to the panoramic field of view aims to cover both elements very close to the vehicle (typically located outside the first field of view a1 and as they would be observed by a front 360 camera), distant elements as observed by the first camera 1, as well as intermediate lateral and / or rear elements.

[0057] An optional first initialization step S0, including the definition of a viewpoint, called the point of interest, from which it is desired to observe the scene of the environment, can be implemented.

[0058] The viewpoint can be defined or chosen to be located at a second fixed position p2 relative to vehicle V, distinct from the first position p1. Such a second position p2 can be chosen to allow the inclusion of an area outside the field of view a1 of the first camera 1. Such a second position p2 can also be chosen to allow observation of the scene from a different viewpoint than the first camera 1. For example, if the first camera 1 is located in a high position at the front of vehicle V, the viewpoint chosen in step S0 can be positioned in a low position at the front of vehicle V. For example, the viewpoint can be positioned at the level of the front license plate of vehicle V, to observe the scene located on the ground in front of vehicle V.

[0059] Alternatively, the viewpoint can be from elsewhere on vehicle V, such as on the roof or underneath the vehicle. It is also possible to consider a viewpoint fixed relative to vehicle V but outside the vehicle, for example, above vehicle V, a few tens of centimeters above its roof. In one of these cases, preferably, the second camera 2 is pointed towards the vehicle, from a low angle, allowing visualization of the immediate surroundings.

[0060] The viewpoint can be chosen, for example, by a vehicle manufacturer, an operator, and / or a vehicle user.

[0061] The chosen viewpoint of interest 2 may be associated with a field of view having a depth of field and angle of field different from those 01, d1 of the first camera 1 or not.

[0062] The chosen viewpoint of interest 2 can be associated with a second reference frame R2 defined for example by a coordinate system (x2, y2, z2) and an origin point for example located at the level of the position p2 of the viewpoint, as illustrated in Figure 1. In other words, the movements and positions of the elements of the scene observed from the viewpoint of interest 2 can be expressed in the second reference frame R2.

[0063] The viewpoint of interest is fictitious and not associated with any particular physical camera. It can therefore be modeled using a virtual camera, for example. In other words, viewpoint of interest 2 can be understood as a virtual camera positioned at the second position p2 and associated with the second frame of reference R2. Specifically, viewpoint of interest 2 can be understood as a virtual camera with extrinsic parameters distinct from those of the first camera 1, and intrinsic parameters similar to those of the first camera 1, for example. In the following description, the terms viewpoint 2, viewpoint of interest 2, and virtual camera 2 may be used interchangeably.

[0064] Step S0 can be optional and replaced by an initialization and viewpoint selection step prior to the implementation of the proposed process, before the vehicle starts, or during its factory configuration by an operator, or predefined by a vehicle user later, for example. In particular, it can be stipulated that such a viewpoint can be modified or redefined between two iterations of the proposed process.

[0065] Once viewpoint of interest 2 is defined, steps S1 to S3, and optionally S20, S21, S22 and S23, of the process according to this disclosure can be implemented.

[0066] A step S1 is implemented, which involves obtaining one or more images captured in the past by the first camera 1 of the vehicle V. In other words, each image captured by the first camera 1 is associated with a past time, tn, prior to the current time t. Each past image captures, in the first reference frame R1 of the first camera 1, a view of the scene outside the vehicle. Thus, the past images associated with times t-1, t-2, ..., tn capture views of the scene at times t-1, t-2, ..., tn.

[0067] If the vehicle moves between two time points during the acquisition of two past images, the past images capture geometrically different views of the scene because the vehicle's frame of reference Rv and the first camera's frame of reference R1 are moving relative to the scene's world frame of reference Rm, and the scene evolves over time. In addition to the vehicle's movement, which influences the view captured by the images, the scene itself can also evolve over time. For example, vehicles, people, wind-related movements, and / or environmental changes, such as changes in brightness, climate, time of day, weather, etc., can modify the scene. Thus, even if the vehicle is not moving (and therefore the vehicle's frame of reference Rv and the first camera's frame of reference R1 are fixed relative to the scene's world frame of reference Rm), the views at t-1, t-2, ..., tn captured by past images at times t-1, t-2, ...tn may differ, due to potential changes in the scene between different moments of acquisition.

[0068] The principle of obtaining images captured in the past by the first camera 1 is illustrated for example in Figure 3. In this figure, the vehicle V, which is a car, has moved in the direction shown by the arrow, from a past position (illustrated by the vehicle in transparency) at the past time tn, to its current position (illustrated by the vehicle in solid lines) at the current time t=0. According to this illustrated example, during the movement, the first camera 1 mounted on the vehicle was able to acquire, via the first camera 1, at least one first past image IMGn1 according to the first field of view a1 at time tn, then possibly a second past image IMGn2 according to the first field of view a1 at time t-2, then possibly a third past image IMGn3 according to the first field of view a1 at time t-3, and so on.Images IMGn1 and IMGn2 capture at least partially an object O in the scene outside the vehicle V, unlike more recent past images, such as past image IMGn3, which does not capture object O, as illustrated in the example. Furthermore, depending on the time difference between tn and t-2, object O may have been moved (a vehicle, a person, an obstacle that has been moved, etc.) or may have been visually altered (difference in brightness, dust, damage, etc.).

[0069] The past image(s) IMGn, along with their information (including, among other parameters, the images can be time-stamped and associated with their acquisition time), can be stored, for example, in the memory of the processing unit 10, for a predetermined retention period. Thus, the past images can be used later, and the stored information can indicate changes in the scene and the vehicle over time, from one image to the next.

[0070] In addition to the timestamp of each image captured, other information can also be recorded and associated with the image. For example, this information may include, but is not limited to: the vehicle's speed at the time the image was captured, the image size / number of pixels, the vehicle's GPS location, a set of data relating to other vehicle sensors (temperature, position sonar), automatic labels identifying obstacles, signs, traffic lanes, image metadata, etc.

[0071] For example, the image retention period could be at least one vehicle lifecycle (V), where the vehicle lifecycle could include all stages between two successive starts, including use, shutdown, and subsequent restart. The vehicle lifecycle duration could then be defined as the time between two successive vehicle starts. Images from a previous cycle could then be deleted from memory when the vehicle restarts in a subsequent cycle. Alternatively, images could be deleted after a predefined period, or all images could be stored for the entire vehicle life.

[0072] In one embodiment, the time intervals between the acquisition of two successive images (e.g., between times t-2 and t-1) can be separated by a predefined, constant time step. This time step can, for example, be 33 milliseconds. Alternatively, or in combination, the time step can also be adjusted adaptively during vehicle operation or according to process iterations.

[0073] In another example, the time step can be configured to depend on the speed of vehicle V and / or on the nature of vehicle V's movement - e.g., whether vehicle V is moving forward or backward - and / or to depend on the distance traveled by vehicle V - for example: when the vehicle is moving below a predefined speed, e.g., less than 30 km / h, the time step can be reduced for speeds between 10 and 30 km / h, and increased for speeds between 0 and 10 km / h; The time step can be configured to depend on the distance traveled and / or the trajectory, independently of the speed, particularly when the vehicle is moving at a lower speed, for example a past image is obtained at each meter traveled by the vehicle; A vehicle traveling at more than 30 km / h can be considered as not requiring parking or maneuvering assistance. In these cases, the time interval between two images can therefore be longer, for example, one second. It can also be stipulated that beyond a certain speed, for example 30 km / h, no images are acquired or recorded.

[0074] It is also possible to provide a custom setting, e.g., defined by the user or driver, of the time step between two images passed, for example via an interface on the vehicle's dashboard.

[0075] At an optional step S20, a current image captured by the first camera 1 at the current time t can be obtained, in addition to the past image(s) obtained during step S1.

[0076] The current image IMG can represent a second view of the scene seen by the first camera 1 for the vehicle's position of interest at the current time t. The second view captured by the current image may be different from the first view captured by the past image, and / or from different views captured by past images, in particular if the vehicle has moved between the current time and past times, and / or if the scene has evolved between the current time and past times.

[0077] An example of a current image is illustrated in Figure 3, where, at the current time t=0, the first camera 1 acquires a current image IMG according to the first field of view a1. In the illustrated example, the object O is not captured in the current image IMG at the current time t=0, because it is in a blind spot of the first camera 1.

[0078] The current image IMG can be stored in the memory of the processing unit 10, and can for example become a past image for a future position of the vehicle V considered at a future time (not shown in the figures).

[0079] A step S2 can then be implemented. This step may involve transforming the past image(s) IMGn1, IMGn2, IMGn3 acquired during step S1, and optionally the current image IMG acquired in step S20, into images viewed from the viewpoint of interest 2 defined in step S0. To achieve this, step S2 may include a change of reference frame for the images obtained in steps S1 and S20, from the first reference frame R1 of the first camera 1, to the second reference frame R2 of the viewpoint of interest 2. Each past (and / or current) image transformed according to step S2 in the second reference frame R2 then results in a transformed past (and / or transformed current) image representing a transformed past view of the captured past view of the scene, as if it were viewed from the viewpoint of interest 2.In other words, such a transformed image then corresponds to the estimation of an image that would have been captured by a virtual camera 2, positioned at position p2, e.g., in a low position and at the front of the vehicle, according to the field of view of such a virtual camera 2 (e.g., corresponding to the first field of view a1).

[0080] The transformation implemented in step S2 depends in particular on the relative movements of vehicle V with respect to the world frame Rm, with positions p1 and p2 being fixed positions on vehicle V, and moving relative to the world frame Rm. The geometric transformations performed on the images processed in step S2 then take this relative movement into account in order to preserve the geometry of the scene portion as captured in each processed and / or present image, so as not to distort the views of the scene.

[0081] To perform the change of reference frame, a pinhole camera model can be conventionally used. Such a pinhole camera model, or equivalent, can be described by the following relationships.

[0082] A point p of an object in the scene in the world frame Rm, captured by an image denoted l (past or current) acquired by the first camera 1, can be defined according to the relation: > > Where p(x m0 , y m0 , z m0 ), is the point of the scene object in the world frame Rm; T is the transformation of the point p(x m0 ,y m0 ,z m0 ) from the world frame Rm to the first frame R1 of the first camera 1, to obtain the point p(x ci ,y ci ,z ci ) ; P is the transformation between the camera frame R1 and a sensor frame, where the point p(ci-7ci- z ci) en3D is transformed into a point-image / (xy), with x, y corresponding to coordinates in metric units; E is the transformation between the sensor frame and the image frame, where the image point / (x, ) is converted into an image plane l(u, v), where u, v correspond to discrete coordinates (pixels).

[0083] The transformation by change of reference frame according to step S2 can therefore be based on: - determine the position of each point p(x m0 ,y m0 ,z m0 ) of the scene object in the world frame Rm, by applying the inverse transformations A -1 , P -1 then T -1 on each point defined in the first reference frame R1 captured by each of the past (and / or current) images acquired by the first camera 1; - apply the transformations T, P, A on each point p(x m0 ,y m0 ,z m0) of a determined position, by changing the reference frame, in order to obtain the points p(.x C2 ,y C2 ,z C2 ) defined in the second reference frame R2 (x2, y2, z2) of the second camera 2; and - calculate each of the past images transformed from the points defined in the second reference frame R2, each of the past images thus representing a past transformed view of the object of the scene.

[0084] Optionally, the method may include a step S21 in which one or more additional side and / or rear images are acquired at past and / or current times by at least one additional camera, for example an existing camera mounted on the vehicle V as part of a 360° feature. Each of the additional images captures at least one additional view of the scene, in particular a side and / or rear view of the scene.

[0085] In this step, the additional camera(s) can be other cameras installed in the vehicle, specifically configured to implement driver assistance features and 360° functionality by acquiring panoramic side and / or rear views of the vehicle (V) according to the panoramic field of view, for example. The additional camera(s) are thus placed in fixed positions on the vehicle, within the vehicle's reference frame (Rv), and distinct from the first (p1) and second (p2) positions (not shown). Specifically, the additional camera positions can be on one side of the vehicle, such as on a side or rear door (trunk), or in the lower or upper part of the vehicle (V). Typically, the additional camera(s) are associated with an expanded field of view, usually corresponding to a panoramic field of view of at least 180°.

[0086] The additional images already present a view from a desired viewpoint (e.g., lateral and / or rear), complementary to the viewpoint of interest 2 defined in step S0. They therefore do not require transformation according to step S2.

[0087] Additional images can also be stored, for example in the memory of the vehicle's processing unit 10.

[0088] An optional image validity assessment step (S22) can then be implemented to verify that the processed image(s), particularly past images, are still valid and correctly updated relative to the current time (t). Step S22 can also be used to warn the vehicle user (V) in the event of invalid or inconsistent past images compared to the present time. Over time, some images may become too old to reflect the current time (t), causing inconsistencies between older and more recent images.

[0089] For example, when images were acquired at distant past times, for example more than 10 minutes ago, or if the vehicle has moved a certain distance, such as one kilometer, from its previous position, or when the vehicle is switched off and remains parked for a variable period, for example from a few minutes to several days, some of these past images may no longer accurately represent the current scene, and vehicles parked around vehicle V may have moved or left the scene, or the outside brightness may have changed, going from day to night, or the weather conditions may have changed, etc.

[0090] To assess image validity, step S22 may include defining one or more validity criteria for each past image, for example, in relation to a current and / or more recent image. The validity criterion may also be used to determine the consistency of the evaluated past images with respect to the current and / or more recent image.

[0091] The validity criterion may depend on one or more elements such as: A maximum time difference between a past moment associated with the processed historical image (i.e., a transformed past image IMGn') and the additional past image, and the current moment; data from vehicle sensors, where data recorded at a more recent time, such as the current moment, can be compared to the older past image(s) to verify their adequacy. For example, data from a near-obstacle detection sonar or other object / obstacle identification cameras that detect (or have recently detected) a nearby obstacle not present in one or more images acquired at earlier times can help indicate image validity. A level of correlation between the pixels of the old image(s) and a more recent or current image. The correlation can take into account all data relating to the brightness, contrast, and color of the pixels, with a predetermined tolerance, in order to identify any change, whether it be the appearance / disappearance of an element, or a physical / visual change in the environment.

[0092] Each validity criterion can be compared to a predetermined reference criterion (for example, if the time difference is greater than ten seconds, or if the data from the sensor indicates a nearby obstacle, etc.) and recorded, for example, in the memory of the processing unit 10 of vehicle V.

[0093] Evaluating the validity criterion(as) for each processed image (or each additional processed image) allows us to define a confidence index associated with the image, for example, by assigning scores after each comparison of the validity criterion with the associated reference criterion. The confidence index can, for instance, be calculated using a linear function that varies with time, taking into account the times associated with each of the corrected images considered relative to the current time. The index is then calculated, for example, as a confidence percentage.

[0094] The confidence index can then be added and presented as a value and / or as information to the vehicle user upon restarting, directly integrated into the corrected images, or as additional information, for example, via an audible and / or visual warning on a component of the vehicle's dashboard. For example, if the validity criterion relates to the time difference between images, the confidence index can be presented as a confidence percentage dependent on the time difference in the corrected images. If the validity criterion relates to data from the vehicle's sensors, the confidence index can include text indicating the sensor of interest, and / or add a pictogram to the current image indicating the probability of a potential obstacle.For example, if the validity criterion relates to the level of correlation between images, pixels that are not correlated between two images can be indicated in red, with their confidence index as a percentage and / or a predicted shape of an obstacle detected by another sensor of the vehicle can be added, etc.

[0095] These examples are not exhaustive and any relevant provision can be implemented by the person in the trade to simply and quickly identify information relating to a potential obstacle, or to add a clear warning (audible and / or visual) to the user that the last images processed when the vehicle was switched off are not representative of the current environment when restarting.

[0096] The confidence index can then be used to reject and remove images with a confidence index below a predefined threshold. Such images are identified as outdated (e.g., too old) and / or inconsistent (e.g., an element captured in these images does not correspond to reality at the current time or a later time). To this end, a selection of consistent images, even if older than more recent but inconsistent ones, can be made. This selection can then be used for subsequent panoramic image estimation steps as described later.

[0097] Optionally, when a passed image is rejected following step S22, a correction step S23 of the pixels of the passed images based on the validity criterion can be implemented after step S22, in order to improve the confidence index of the rejected image.

[0098] Pixel correction can, for example, involve adapting the brightness of pixels in older images to the current brightness of pixels at the present time, for instance, using an artificial intelligence algorithm or equivalent. Similarly, image correlation correction, comparing past images with the current image, can also be performed. It is also possible to correct images by modeling predicted and approximate obstacle shapes, identified from sensor data (e.g., sonar data, etc.), and adding them to the images to be corrected.

[0099] The images corrected at the end of step S23 can then be reused for a validity assessment at step S22 to re-evaluate the validity criteria and consistency of the corrected images with respect to the current and / or more recent image. Alternatively, the images corrected at the end of step S23 can be used directly to perform the panoramic image estimation according to step S3. The evaluation of the corrected images at step S22 can then lead to the validation of the corrected images if the new associated confidence level is greater than the confidence threshold. The validity step S22 can also maintain the rejection of the image, even corrected, if the corrected image still has a confidence level below a threshold.

[0100] The transformed images from step S2, and / or optionally, the images considered valid from step S22 (with or without prior correction), and / or optionally the corrected images from step S23, can then be implemented in a final step S3 for image extension and estimation of the panoramic image IMGp. This step aims to obtain, from the transformed or supplementary image(s), initially presenting a field of view a1, a panoramic image with a panoramic field of view exceeding 180° and extending up to 360°. The panoramic image estimated in step S3 is associated with the current time t (and therefore with the vehicle's position of interest V).

[0101] In the case where only a past image IMGn is obtained in step S1 and transformed IMGn' in step S2, the panoramic image IMGp associated with the current time t can be directly estimated as corresponding to this single transformed past image IMGn'. This embodiment can be observed in Figure 3; for example, the past image IMGnl, acquired at tn, captures a panoramic view around the vehicle, notably capturing the object O in its entirety, compared to the current image IMG which only captures the view in front of the vehicle and does not capture the object O.

[0102] In other cases where several past transformed IMGn' images are used, the S3 extension of the transformed images may include: a cutting and selection of parts of each of the past transformed images IMGn'; a mosaic of the selected parts.

[0103] Image segmentation and mosaicking allow for the creation of a comprehensive and continuous view from multiple separate shots. Segmentation involves selecting and extracting relevant portions of the images based on overlapping areas or shared features. These portions are then assembled during mosaicking, aligning and merging the fragments to produce a complete and seamless image.

[0104] According to the example in Figure 3, at least the parts of the image IMGnl at tn that captured the object O can be selected and cropped. These parts can then be combined with other selected and corresponding parts of the other transformed past images, here for example the complementary parts of the images IMGn2 and IMGn3 at t-2 and t-1 respectively, capturing complementary parts of the object O and the scene, in order to enlarge the field of view of the reconstructed image. By mosaicking these parts, the object O can therefore be fully reconstructed, and the panoramic field of view of the mosaicked image is increased, both in front of and behind the vehicle, compared to the field of view a1 of a single image. In Figure 3, two examples of panoramic images IMGp are shown; IMGpl in dotted lines results from a mosaicking of the parts of the past images transformed IMGn', from the past images IMGn acquired at times t-2 and t-1; IMGp2, shown in solid lines, is obtained by mosaicking parts of transformed past images IMGn' from a plurality of past images IMGn (not all of which are shown in Figure 3) in order to construct a 180° panoramic image. With a plurality of past images prior to tn (not shown), it would be possible to construct panoramic images of more than 180°, and up to 360°.

[0105] The segmentation of selected areas within past images can depend on one or more parameters. For example, it can depend on the image acquisition frequency, which influences the amount of overlap between successive views. For instance, the closer the past images are in time, the smaller the segments to be segmented will be. Segmentation can also depend on the movement of vehicle V, whether its speed or trajectory, which can each affect the continuity of visual elements. For example, if the vehicle turns left, the left-hand segments of the images will have greater overlap than the right-hand segments. Segmentation can also be based on a feature of interest present in several images (for example, object O in Figure 3). This feature of interest can serve as a reference to guide segmentation and ensure image continuity.For example, an obstacle automatically identified by a dedicated algorithm can serve as a guide to center images and crop relevant parts.

[0106] The selection and segmentation of parts and the mosaicking of images can rely on image assembly and combination techniques (for example, with stitching algorithms, panographic overlay, etc.). Any suitable method for extending the field of view from one or more transformed past images can be applied.

[0107] To perform mosaicking and to reinforce the construction of a continuous image, it is also possible to associate parts of the past image(s) transformed IMGn' with parts of the current image IMG' transformed from the current image IMG acquired at the current time t=0 by the first camera 1 and / or parts of the additional image(s) acquired at the same past and / or current times.

[0108] Once one or more past images transformed into IMGn', and / or additional images and / or a current image transformed into IMG', have been extended, cropped and mosaicked, the panoramic image IMGp can be estimated: When only one processed image IMGn is obtained in step S1, the panoramic image IMGp includes the transformed processed image IMGn' obtained at the end of step S2; the panoramic image IMGp corresponds to the transformed processed image IMGn'. In Figure 4, a panoramic image IMGp can be estimated, for example, from the processed image at tN only; when several past images IMGn have been acquired at different times t-1, t-2,... tn, there can be several panoramic images IMGp, which are estimated from the extended images of these past images transformed IMGn', by cutting and mosaicking the different parts of the images in step S3. According to the example in figure 4, a panoramic image IMGp can be estimated for example from the images at tN, t-4, t-3, t-2 and at t-1; In addition to the transformed past image(s) IMGn', the current image IMG acquired in step S20 and then transformed into an image IMG' in step S2, and / or the additional image(s) obtained in step S21 and then extended according to step S3, can all, or some, be integrated for the estimation of panoramic images IMGp. As illustrated in Figure 4, a panoramic image IMGp can be estimated, for example, from the images at tN, t-4, t-3, t-2, t-1, and from a current image at t.

[0109] The estimated IMGp panoramic image can then be presented to the user, for example, via a graphical interface on the dashboard, and perhaps with all or part of the relevant information used to create it. For example, the IMGp panoramic image can also be displayed with the confidence index percentage.

Claims

Demands

1. A method for estimating at least one panoramic image according to a field of view said to be panoramic from a vehicle (V) and implemented by a processing unit (10), said vehicle (V) being located at a position said to be of interest associated with a given current instant (t), the processing unit (10) being connected to a first camera (1) mounted on the vehicle (V) at a first position (p1), fixed in a reference frame of the vehicle (Rv), said first camera (1) defining a first field of view (a1) narrower than the panoramic field of view and being configured to capture images in a first reference frame (R1), the panoramic field of view being defined from a viewpoint associated with a second position (p2), fixed in the vehicle's frame of reference (Rv) and distinct from the first position (p1), the panoramic field of view being associated with a second frame of reference (R2), the first position (p1) belonging to an upper part of the vehicle (V) and the second position (p2) belonging to a lower part of the vehicle (V), The process includes the following steps: obtain (S1) at least one past image (IMGn) acquired by the first camera (1) at a past time (tn) prior to the current time (t), the past image (IMGn) capturing a first view of a scene according to the first field of view (a1) for a past position of the vehicle (V) at the past time (tn); transform (S2) said past image (IMGn) into a transformed past image (IMGn') by a change of reference frame from the first reference frame (R1) to the second reference frame (R2), said transformed past image (IMGn') representing a first transformed view of the scene; estimate (S3) the panoramic image (IMGp), from at least the transformed past image (IMGn'), the panoramic image (IMGp) representing a panoramic view of the scene, estimated from at least the first transformed view.

2. A method according to any one of the preceding claims, wherein the first field of view (a1) and the panoramic field of view cover scenes in front of the vehicle (V).

3. A method according to any one of the preceding claims, wherein the method further comprises the following steps: obtain a plurality of images including the past image (IMGn) and at least one other image from among other past images acquired by the first camera (1) at respective past times prior to the current time (t) and / or a present image (IMG) capturing a second view of the scene according to the first field of view (a1) for the position of interest of the vehicle (V) at the current time (t), transform the plurality of images obtained into a plurality of images transformed by said change of reference frame, and in which the panoramic image estimation (IMGp) includes: a cutting and selection of parts of the plurality of transformed images, and a mosaicking of said parts to form said panoramic image (IMGp).

4. A method according to claim 3, wherein the cutting, selection and / or mosaicking of said parts depends on at least one element among: data relating to the image acquisition frequency of the first camera (1), a movement of the vehicle (V), an element of interest represented in at least two transformed images.

5. A method according to any one of the preceding claims, further comprising the following steps, prior to the estimation (S3) of the panoramic image (IMGp): define a validity criterion at least for the transformed past image (IMGn'); based on said validity criterion, determine (S22) a data point relating to a confidence index of the transformed past image (IMGn'), and in which the estimation (S3) of the panoramic image (IMGp) depends on said confidence index.

6. A method according to claim 5, wherein the validity criterion depends on at least one element among: a difference between the past moment and the current moment, data from vehicle sensors (V), a level of correlation between pixels of the transformed past image (IMGn') and pixels of another transformed image from an image acquired by the first camera (a1) at the current time.

7. A method according to any one of claims 5 and 6 further comprising: implement (S23) a correction of the pixels of said at least one transformed past image (IMGn'), and in which the confidence index is determined on the basis of at least one corrected transformed image.

8. A method according to any one of the preceding claims comprising, prior to the estimation of the panoramic image (S3), the following steps: obtain (S21) at least one additional image captured by at least a third camera, at a third fixed position in the vehicle's reference frame (Rv) and distinct from the first (p1) and second (p2) positions, said at least one additional image capturing an additional view of the scene, not disjoint from the first view; and in which the estimation (S3) of the panoramic image includes the assembly (S3) of the at least one transformed image (IMGn') with the at least one additional image, said panoramic view being estimated from the first transformed view and the at least one additional view.

9. A method according to claim 8, wherein said at least one third camera (3) is at least one element among: a side camera of the vehicle (V), a rear camera of the vehicle (V).

10. A method for assisting the driving of a vehicle implemented by a processing unit (10) connected to a first camera (1) mounted on the vehicle (V), comprising: an estimation of a panoramic image, from at least one image acquired by the first camera, by an estimation method according to any one of the preceding claims, and a display of the panoramic image on a display unit of the vehicle, and image processing of images acquired by said first camera (1) to perform at least one of the following: o lane change detection of said vehicle (V), o traffic sign detection; o obstacle detection.

11. Computer program comprising instructions for carrying out the process according to any one of the preceding claims, when executed by a processing circuit.

12. Processing unit (10) mounted in the vehicle comprising a processing circuit and a memory for implementing the process according to any one of claims 1 to 10.

Images