Dynamic reflections on the vehicle model based on camera images in the surround view system

Abstract

The invention is related to a process implemented in a means of transport (10), comprising at least the following steps: a) determining a visual influence (12) of a surrounding (14) of a means of transport (10) on the means of transport (10) itself, b) generating an image (16) of the means of transport (10), in due consideration of the visual influence (12) of the surrounding (14), and c) the method steps a) and b) being repeated at least once.

Description

[0001] R.416501 IP1

[0002] 1

[0003] Description

[0004] Title

[0005] Dynamic reflections on the vehicle model based on camera images in the Surround View System

[0006] Technical Field

[0007] The present invention relates to a process, implemented in a means of transport, comprising at least a determination of a visual influence of a surrounding of a means of transport on the means of transport itself and a generation of an image of the means of transport, in due consideration of the visual influence of the surrounding.

[0008] State of the Art

[0009] Early vehicle parking assistant products use ultrasonic parking sensors and / or a single rear-view camera to view and obtain distances to objects surrounding the vehicle, providing drivers with an audible alarm or rear-view video through a fisheye lens. There are some drawbacks to these early products: the alarm only provides a proximity warning but not the position of the object(s) relative to the vehicle, and the rear-view camera has a limited field of view. Multiple-camera systems overcome these problems and are increasingly available. In most systems, there are four wide-angle cameras: one in the front of the vehicle, one in the back of the vehicle, and one each in the side-mounted rear view mirrors. The four cameras have overlapping fields of view that collectively cover the whole area around the vehicle and serve as an omnidirectional camera. Videos from the cameras are sent to the processor, which synthesizes a bird's-eye view from above the vehicle by stitching the video feeds together, correcting distortion, and transforming the perspective.

[0010] US10 017 114 B2 relates to a vision system of a vehicle that includes a camera, a display unit and a controller. The camera is disposed at a vehicle and has a R.416501 IP1

[0011] 2 field of view exterior of the vehicle. The camera is operable to capture image data. The display unit includes a display screen disposed at a display area at a windshield of the vehicle. The display unit is electrically changeable from a nondisplay state to a display state. When the vehicle is being driven, the controller, responsive to a determination that the gaze of the driver of the vehicle is at the display area for a threshold period of time, changes the display unit from its nondisplay state to its display state. When in its display state, the display unit displays images at said display screen that are derived from captured image data.

[0012] Invention disclosure

[0013] According to the present invention, a process implemented in a means of transport is proposed, wherein the process comprises at least a determination of a visual influence of a surrounding of a means of transport on the means of transport itself, and a generation of an image of the means of transport in due consideration of the visual influence of the surrounding.

[0014] A means of transport is a facility used for carrying either people or cargo or both. The means of transport is not restricted in its way of transportation; hence, the movement of the means of transport can occur by land way, water way or air way.

[0015] A surrounding is understood as the entirety of all elements that are located in such a matter that their visual influence affects the appearance of the means of transport. A surrounding is associated with the physical location of the means of transport and a specific time.

[0016] A visual influence is understood as any kind of optical occurrence, involving the emission of visible light by active and passive sources of light as well as a discontinuity of a light intensity, e.g. a shadow. This comprises sources of natural and artificial lighting, the reflection of visible light by objects with a reflective surface as well as the reflection of objects onto the surface of the means of transport. The image means a photo, captured by a camera system and processed by a computer. In the context of the inventive process, the computer processes the photo in such manner that a model of the physical appearance of the means of transport, depending on the visual influence of the surrounding, is pasted into the photo. The image means a photo, captured by a camera system and processed by a computer. In the context of the inventive process, the computer processes the photo in such manner that a model of the physical appearance of the means of transport, depending on the visual influence of the surrounding, is pasted into the photo.

[0017] In an advantageous embodiment of the inventive process, the visual influence is a reflection on a surface of the means of transport. This comprises visual light being reflected by surfaces of objects, located in the surrounding of the means of transport. Hereby, the source of the visual light that is reflected comprises light of natural origin, light of artificial origin and light that originates from another reflection.

[0018] In another advantageous embodiment of the inventive process, the surrounding comprises at least an object with a reflective surface and / or an object reflected on the surface of the means of transport. These objects comprise other means of transport, static objects, living creatures as well as the surface character of the ground and the sky. Other means of transport can be of the same kind as the means of transport itself, as well as of another kind. Static objects comprise structures, vegetation and traffic related infrastructure. The surface character of the ground comprises paved and unpaved road, depending on their environmental condition, as well as waterways.

[0019] In another advantageous embodiment of the inventive process, either the surrounding comprises at least a source of light, or the means of transport is equipped with at least a source of light. A typical source of light for the means of transport itself may be a headlight, an indicating light system as well as an interior light. A typical source of light for the surrounding includes all sources of natural light, other means of transport, traffic related infrastructure, such as a streetlight, a signaling light, such as a traffic light, a hazard flasher, an aviation obstruction lighting, beacon light and others, and non-traffic related sources of light, such as interior lighting of buildings.

[0020] In another advantageous embodiment of the inventive process, the surrounding of the means of transport is captured by a camera system. The camera system comprises at least one camera and is mounted in such a way that the surrounding of the means of transport itself is covered. R.416501 IP1

[0021] 4

[0022] In another advantageous embodiment of the inventive process, the means of transport is a vehicle, e.g. a car, a truck, a motorcycle, a bus, a utility vehicle, a boat, a ship, a plane or a rolling stock. The camera system covers preferably an environment as large as possible, yet it can cover as well an environment of less than 360 degrees surrounding the means of transport.

[0023] In another advantageous embodiment of the inventive process, the image is a 3D computer-generated image. A computing unit processes the photo, captured by the camera system in the following way: if the camera system comprises two or more independent cameras, the photos are merged in such a way that a continuous image of the surrounding is created. In the following step, the computing unit integrates a computer-generated model of the means of transport itself into the continuous image of the surrounding. Thereby, the computer-generated model of the means of transport itself considers the visual influence of the surrounding of the means of transport on the means of transport itself.

[0024] In another advantageous embodiment of the inventive process, in a first step an image of the surrounding captured with the camera system is projected on a 3D concave vessel and in a second step, the visual influence of the surrounding on the means of transport is determined. The incoming photos coming from the cam- era-system are projected on a three-dimensional concave vessel, e.g. a bowl for creating a continuous image of the surrounding, determining the visual appearance of the means of transport.

[0025] In another advantageous embodiment of the inventive process, the visual influence of the surrounding on an uncovered surface of the means of transport by the camera system is reconstructed by stretching the visual influence of the surrounding of an adjacent captured surface onto the uncovered surface combining it with at least another captured surface. Determining the visual influence on the top surface of the means of transport is made possible by interpolating bordering surface areas. The way these surfaces are interpolated depends on the amount of bordering surfaces. For example, for an uncovered surface with four bordering surfaces, stretching the top of the bordering surfaces onto the uncovered surface in a triangular matter and blurring the edges afterwards brings positive results.

[0026] In another advantageous embodiment of the inventive process, the resolution of the image of the surrounding, captured by the camera system is adjustable. The computing unit that determines the visual influence of the surrounding on the means of transport itself has a finite quantity of operations it can process. Depending on the scale of the visual influence of the surrounding on the means of transport itself and the frequency of the alternation of this visual influence, the processing power of the computing unit may be exceeded. In this case, it is rather preferred to adapt the resolution of the image of the surrounding captured by the camera system in such a way that a smooth display of the moving image is guaranteed than displaying a more detailed moving image with a lower frequency.

[0027] Advantages of the invention

[0028] Conventional Surround View Systems, especially those that are common in the automotive industry use static vehicle models. In this way, the vehicle model in the Surround View System looks always the same. Depending on the environmental conditions, the vehicle model will appear unrealistic, as it is not possible to consider all possible scenarios. The appearance of the vehicle will drastically change depending on whether the vehicle is situated outside or inside or during daytime or during nighttime.

[0029] Therefore, applying the inventive process, the 3D vehicle model will appear more realistic. In an image, where authentic footage and computer-generated imagery are combined, it is crucial to provide consistent lighting and reflections on the computer-generated object for the composition to appear credible. Current technologies are lacking this feature. The more reflective the material on the model, the more apparent this issue becomes.

[0030] A dynamic depiction of the vehicle is made possible by consistently updating a cube map. Each 3D environment, especially the one mentioned earlier, requires different lighting on the model. Without this invention, the model will seem out of place.

[0031] Furthermore, common applied methods struggle with a 360-degree coverage, since the implementation of a camera system that covers the top and bottom of the vehicle comes with a poor cost-benefit ratio. The inventive process solves this by its ability of an approximation of the appearance of surfaces that are uncov- R.416501 IP1

[0032] 6 ered by the camera-system. Thus a credible appearance of the respective surface can be determined without the need of supplementary hardware. For a surface with four bordering surfaces, this is achieved by stretching the top part of the side faces into a triangular shape with the triangle corners representing the two corners of the top face and the midpoint of the top face. Subsequently, a blur is applied to eliminate any sharp edges that may arise at the connecting points of these triangles. The blur is executed in a manner that renders the triangle increasingly transparent as it moves away from the triangle's altitude line. The same technique is applied to the bottom part of the triangle, with the only difference being the stretching of the bottom side of the faces.

[0033] Brief description of the drawings

[0034] Embodiments of the invention will be explained in more detail with reference to the drawings and the following description.

[0035] The drawings show:

[0036] Figure 1 a means of transport that is situated in a surrounding and equipped with a camera system and a computer. The surrounding comprises objects with reflective surfaces, objects reflected on the surface of the means of transport and sources of light.

[0037] Figure 2 a dynamic generated cube with captured and uncaptured surfaces, respectively, and

[0038] Figure 3 a top view of a reconstructed uncaptured surface on top of the dynamic cube according to figure 2.

[0039] Embodiments of the invention

[0040] In the following description of the embodiments of the invention, identical or similar elements are designated by the same reference signs, whereas a repeated description of these elements in individual cases is dispensed with. The figures represent the subject matter of the invention only schematically. R.416501 IP1

[0041] 7

[0042] Figure 1 shows a means of transport 10, located in a surrounding 14. The surrounding 14 comprises objects with a reflective surface 18, objects reflected on the surface of the means of transport itself 19 and at least one source of light 20, whereby the means of transport 10 may be equipped with the source of light 20. The impact of the surrounding 14 on the visual appearance of the means of transport 10 is labeled visual influence 12. Furthermore, the means of transport 10 is equipped with a camera system 22. The coverage of the camera system 22 may vary, i. e. captured surfaces 30 will be distinguished, which means that surfaces of the means of transport 10 that are located in such a manner that the visual influence 12 of the surrounding 14 may be determined by the camera-system as well as uncaptured surfaces 32, i. e. surfaces that are located in such a manner that the location of the sources 18,19, 20 for the visual influence 12 on said surface exceeds the range of the camera system 22, both are visualized. After obtaining an image of the surrounding 24, a computer 26 processes the image of the surrounding 24 in such a way that an image 16 is created comprising a 3D- computer generated depiction of the means of transport 10 in due consideration of its appearance inside a surrounding located inside its surrounding.

[0043] Figure 2 shows a generated dynamic cube 40 having an uncaptured surface 32 on its top. Said uncaptured surface 32 according to figure 2 is a top face 34. Said top face 34 is limited by at least a first top face edge 36 and an adjacently arranged second top face edge 38, respectively.

[0044] As can be derived from figure 2, side faces 42 constitute captured surfaces 30. The top part of said captured surfaces 30, i. e. said side faces 42, i. e. said portions 44 are projected on the uncaptured surface 32, i. e. said top face 34. Thus, the appearance of the uncaptured surface 32, i. e. said top face 34 is estimated by means of the four adjacent captured surfaces 30, i. e. said side faces 42, respectively. Consequently, the top part of each captured surface 30 is projected onto the uncaptured surface 32 and is stretched in a triangular manner.

[0045] Figure 3 shows the pattern of said top face 34 according to figure 2. As can be derived from figure 3, i. e. the top view of the uncaptured surface 32, it is formed by the patterns of the four adjacent side faces 42, i. e. the captured surfaces 30 of the generated dynamic cube 40 as shown in figure 2. The top view 3 is divided by a center line 48 into a bright portion 50 and a more shadowed portion 52. Reference numeral 54 depicts the midpoint of the top face 34 and is located on said R.416501 IP1

[0046] 8 center line 48 dividing the top face 34, i. e. said uncaptured surface 32 into the said bright portion 50 and said shadowed portion 52, respectively.

[0047] As schematically shown in figure 3, each of said captured surfaces 30, i. e. said four side faces 42 of the generated dynamic cube 40 are stretched, i. e. projected forming triangular shaped portions 46 of the uncaptured surface 32 on said top face 34. Thus, said uncaptured surface 32, i. e. said top face 34, is reconstructed by said triangular shapes 46, the projection of said top parts 44 of said side faces 42 as shown in figure 2.

[0048] The invention is not limited to the embodiments described herein and the aspects emphasized therein. Rather, within the range indicated by the claims, a large number of modifications are possible, which are within the scope of skill in the art.

Claims

R.416501 IP19Claims1 . A process implemented in a means of transport (10), comprising at least the following steps: a) determining a visual influence (12) of a surrounding (14) of a means of transport (10) on the means of transport (10) itself, b) generating an image (16) of the means of transport (10), in due consideration of the visual influence (12) of the surrounding (14), and c) the method steps a) and b) being repeated at least once.

2. A process according to claim 1 , wherein the visual influence (12) is a reflection on a surface (30, 32) of the means of transport (10).

3. A process according to claim 1 , wherein the surrounding (14) comprises at least an object with a reflective surface (18) and / or an object reflected on the surface of the means of transport (19).

4. A process according to claim 1 , wherein either the surrounding (14) comprises at least a source of light (20) or the means of transport (10) is equipped with at least a source of light (20).

5. A process according to claim 1 , wherein the surrounding (14) is being captured by a camera system (22).

6. A process according to claim 1 , wherein the means of transport (10) is a vehicle.

7. A process according to claim 1 , wherein the image (16) is a 3D computer-generated image.

8. A process according to claims 1 and 5, wherein the camera system (22) is covering an environment of less than 360 degrees.

9. A process according to claims 1 and 5, wherein in a first step an image of the surrounding (24), captured by the camera system (22) is beingR.416501 IP110 projected on a 3D concave vessel and in a second step, the visual influence (12) of the surrounding (14) on the means of transport (10) is determined.

10. A process according to claims 1 and 7, wherein the 3D computer-generated image (16) of the means of transport (10) is generated by a computer (26) using a cube map.

11. A process according to claims 1 , 5 and 8, wherein the visual influence(12) of the surrounding (14) on an uncaptured surface (32) of the means of transport (10) by the camera system (22) is reconstructed by stretching the visual influence (12) of the surrounding (14) of an adjacent captured surface (30) onto the uncaptured surface (32) combining it with at least another captured surface (30).

12. A process according to claims 1 and 5, wherein the resolution of the image of the surrounding (24) captured by the camera system (22) is adjustable.

Images