Display control device, display control method, and recording medium

Abstract

The present invention provides a display control method performed by a display control device mounted in a vehicle, the method comprising: detecting an obstacle around the vehicle; generating, for any obstacle detected, a three-dimensional image of the vehicle as viewed from a virtual viewpoint located on the opposite side of the obstacle relative to the vehicle, the three-dimensional image generated on the basis of a plurality of captured images; outputting the three-dimensional image to a display 30; and displaying a vehicle image included in the three-dimensional image as a translucent image.

Description

Display control device, display control method, and recording medium ,

[0001] The present invention relates to a display control device, a display control method, and a recording medium.

[0002] Conventionally, an image transmission / reception system that displays an image of a blind spot situation to a driver from before the host vehicle approaches is known. (For example, Patent Document 1). The image transmission / reception system described in Patent Document 1 receives a video captured using a camera installed outside the host vehicle, such as on the road, and imaging parameters related to the video, geometrically transforms the video based on the video parameters, and converts it into a video as seen from a predetermined position outside the host vehicle, and displays it on a display unit for the driver.

[0003] Japanese Patent Application Laid-Open No. 2013-201707 <0​​​​​​​​​​​​Figure 1 is a block diagram showing the configuration of the display control system according to this embodiment. Figure 2 is a plan view showing a part of the instrument panel. Figure 3 is a diagram showing the display screen of the display. Figure 4 is a diagram showing the display screen of the display (skeleton display screen). Figure 5 is a diagram showing the display screen of the display. Figure 6 is a diagram showing the display screen of the display (skeleton display screen). Figure 7 is a diagram showing the display screen of the display. Figure 8 is a diagram showing the display screen of the display (skeleton display screen). Figure 9 is a diagram showing the display screen of the display. Figure 10 is a diagram showing the display screen of the display (skeleton display screen). Figure 11 is a flowchart of the display control method (scene determination process) executed by the controller. Figure 12 is a flowchart of the display control method (view display process) executed by the controller. Figure 13 is a flowchart of the subflow of the control flow (step S15) shown in Figure 12.

[0009] Embodiments of the present invention will be described below with reference to the drawings. In this embodiment, the case in which the display control device according to the present invention is applied to a display control system 1 mounted on a vehicle will be described as an example. The display control system 1 of this embodiment displays images for understanding the situation of the vehicle and the surrounding area on a display viewed by the vehicle operator. Figure 1 is a block diagram showing the configuration of the display control system according to this embodiment. The display control system 1 includes a front camera 21, a right camera 22, a left camera 23, a rear camera 24, an ATCU 30, a display 31, a display 32, and a display control device 100. The display control system 1 is installed in a vehicle. The front camera 21, the right camera 22, the left camera 23, the rear camera 24, the in-vehicle display 30, and the display control device 100 are connected by an in-vehicle communication network such as CAN or LIN.

[0010] The front camera 21, right camera 22, left camera 23, and rear camera 24 are installed on the vehicle and capture images of the area around the vehicle. The front camera 21, right camera 22, left camera 23, and rear camera 24 capture images of the area around the vehicle while the vehicle is in motion. The front camera 21 captures images in front of the vehicle, the right camera 22 captures images to the right of the vehicle, the left camera 23 captures images to the left of the vehicle, and the rear camera 24 captures images behind the vehicle. The front camera 21 is installed on the vehicle's grille, the right camera 22 is installed on the right door mirror, the left camera 23 is installed on the left door mirror, and the rear camera 24 is installed on the tailgate. Note that the various cameras may be installed in other locations on the vehicle. In the following description, the front camera 21, right camera 22, left camera 23, and rear camera 24 will be collectively referred to as camera 20.

[0011] The in-vehicle display 30 is a display included in the IVI (in-vehicle infotainment) system and includes displays 31 and 32. Displays 31 and 32 are provided on the instrument panel of the vehicle. Displays 31 and 32 are touch-panel displays. Figure 2 is a plan view showing a part of the instrument panel. As shown in Figure 2, display 31 is provided in the center of the instrument panel. Display 32 is provided on the instrument panel in front of the driver. Display 31 displays, for example, maps in the navigation system, menu screens (such as selection screens for various applications) and content videos of the entertainment system. Display 32 displays, for example, meters, a range display showing the current range of the shift position, icons indicating the seat belt status, vehicle images, images of the vehicle's surroundings, predicted path, driving route, etc. Displays 31 and / or displays 32 also display display images generated by the display control device 100.

[0012] Displays 31 and 32 have horizontally elongated display screens along the vehicle width direction (left-right direction in the paper of Figure 2). As shown in Figure 2, displays 31 and 32 are arranged side by side along the vehicle width direction, with their display screens close together. Note that the vehicle does not need to have both displays 31 and 32; it may have only one of the displays.

[0013] The display control device 100 comprises a controller 10 and a recording medium 19. The controller 10 has functions for acquiring various data such as camera images and vehicle data, and image processing functions. The controller 10 has an image acquisition unit 11, an obstacle detection unit 12, and an image processing unit 13 as functional blocks. The controller 10 stores programs for realizing various functions in the recording medium 19, and executes the processing flow indicated by the program by having the processor execute the control program. The recording medium 19 that stores the control program may be stored as memory within the controller 10. The location of the recording medium 19 may be inside or outside the controller 10. The recording medium 19 records a display control program, which is a program that causes the controller 10 to execute processing including steps S1 to S9, S11 to S21, and S31 to S36 described later.

[0014] The image acquisition unit 11 acquires camera images from the camera 20, etc. The obstacle detection unit 12 detects obstacles around the vehicle based on detection data from a distance measuring device such as LiDAR or sonar and / or the image captured by the camera 20. The distance measuring device is mounted on the vehicle. Obstacles include moving objects such as pedestrians, bicycles, and vehicles, as well as stationary objects such as parked vehicles and signs installed on the road. Other examples of stationary objects include obstacles at intersections, houses / walls, plants, utility poles, banners, fences, signs, snow walls, and traffic lights. When the obstacle detection unit 12 detects an obstacle around the vehicle, it also detects the position or movement of the obstacle.

[0015] The image processing unit 13 performs image processing on the camera image in order to display the current surrounding image and / or vehicle image of the vehicle on the displays 31 and 32. The image processing unit 13 generates a display image to be displayed on the displays 31 and 32 in a display mode specified by the user or system, and outputs the display image to the displays 31 and 32. There are multiple display modes, for example, a front view or front wide view representing the front of the vehicle, a rear view or rear wide view representing the rear of the vehicle, a top view that gives an overhead view of the vehicle, a 3D view that shows the vehicle or its surroundings from a virtual viewpoint, a skeleton view, etc. In the front view or front wide view, an image is displayed that shows the area in front of the vehicle's grille and the exterior of the vehicle. In the rear view or rear wide view, an image is displayed that shows the area behind the vehicle's back door and the exterior of the vehicle. In the top view (around view), the position directly above the vehicle is used as a virtual viewpoint, and an overhead view image is displayed that looks down on the vehicle from that virtual viewpoint. In the top view, a two-dimensional image is displayed, and the vehicle image is represented as a plan view of the vehicle. If there are tall three-dimensional objects such as walls around the vehicle, an image of the surroundings is displayed as if the three-dimensional objects were toppled over. In the 3D view, the position around the object is used as a virtual viewpoint, and the image is displayed as if the object were viewed from that virtual viewpoint. The object is not limited to the vehicle; it can also be obstacles around the vehicle. The virtual viewpoint is a viewpoint that simulates viewing the object from outside the object. The virtual viewpoint is movable. For example, with the display image (3D image) of the vehicle as seen from the virtual viewpoint displayed on displays 31 and 32, the vehicle image displayed in 3D can be rotated by rotating the virtual viewpoint around the vehicle along a plane. In addition, as other display modes, displays 31 and 32 may display the area around the vehicle in, for example, skeleton view, front wide view, and rear wide view. In the skeleton view, in addition to the current camera image (real image) representing the area around the vehicle, a virtual image representing the underside of the vehicle is displayed. The front wide-view system, for example, when a vehicle is turning left or right at an intersection with poor visibility, such as a T-junction, can display an image of the area in front of the vehicle that would otherwise be the driver's blind spot, using an image from a front camera 21 with a wide-angle lens.

[0016] The display mode can be specified by the user or changed as appropriate by the vehicle system according to the vehicle's driving conditions. For example, when the gear is shifted to reverse in order to park the vehicle, the display mode may automatically switch to top view, rear view, or rear wide view. Alternatively, the display mode may be switched by the driver operating a switch or similar device while the vehicle is in motion.

[0017] The image processing unit 13 can display images in multiple display modes on the display 31 in multiple screens. The image processing unit 13 may display images in different display modes on the display 31 and the display 32, or it may display the same image in the same display mode on both the display 31 and the display 32. The image processing unit 13 may also treat both the display 31 and the display 32 as a single display screen and display images in multiple display modes, or it may display an image in a single display mode. In the following description, the image processing performed by the image processing unit 13 when displaying an image on the display 31 will be described, but when displaying on the display 32, the image processing for the display 31 may be applied to the image display on the display 32.

[0018] The image processing unit 13 generates a display image according to the display mode based on multiple camera images acquired by the image acquisition unit 11, and outputs the display image to the display 31. When displaying an image in front view (or front wide view) or rear view (or rear wide view), the image processing unit 13 generates a display image showing the front or rear of the vehicle based on the camera images of the front camera 21 or the rear camera 24.

[0019] When displaying an image in top view, the image processing unit 13 generates an overhead view image of the vehicle based on multiple camera images captured by the camera 20. The virtual viewpoint of the overhead view image is set at a predetermined position, vertically separated from the vehicle's roof, and is located outside the vehicle. For example, the image processing unit 13 performs viewpoint transformation processing on images captured by the front camera 21, right camera 22, left camera 23, and rear camera 24 to generate viewpoint-transformed images that appear to be viewed from the virtual viewpoint, showing the front, rear, and left and right regions of the vehicle. In other words, the image processing unit 13 transforms camera images corresponding to the view from the camera's position into viewpoint-transformed images viewed from the virtual viewpoint. The viewpoint transformation from camera position to virtual viewpoint is performed by calculation processing using viewpoint parameters, which are stored in the recording medium 19. The image processing unit 13 then applies each of the images after viewpoint transformation processing to the surrounding area of ​​the vehicle. The image processing unit 13 performs blending processing on the overlapping parts of each camera image from camera 20. Blending is a process to maintain continuity between two overlapping images, and methods known at the time of filing this application can be applied. The image after blending becomes the surrounding image showing the area around the vehicle. The image processing unit 13 also composites the vehicle image located in the center of the screen with the surrounding image. The resulting image becomes an overhead view image, and the image processing unit 13 outputs the overhead view image as the display image to the display 31.

[0020] When displaying an image in 3D view, the image processing unit 13 generates a display image of the vehicle as seen from a virtual viewpoint, based on multiple camera images captured by the camera 20. First, the image processing unit 13 sets the virtual viewpoint according to user specifications or driving conditions. The position and direction of the line of sight of the virtual viewpoint can be arbitrarily set based on commands from the user. The user can specify the position and direction of the line of sight of the virtual viewpoint, for example, by touching the display 31. The image processing unit 13 can also set the virtual viewpoint according to driving conditions. For example, when moving forward, the virtual viewpoint can be set behind the vehicle to view the image in front of the vehicle, and when reversing, the virtual viewpoint can be set in front of the vehicle to view the image behind the vehicle. Specifically, the image processing unit 13 sets the position of the virtual viewpoint when it receives shift information indicating that the shift position has entered reverse. Furthermore, the image processing unit 13 can also set the virtual viewpoint according to the vehicle speed. For example, depending on the vehicle speed input from the vehicle speed sensor, the virtual viewpoint may be set to move away from the vehicle when the vehicle speed is above a predetermined value, and to move closer to the vehicle when the vehicle speed is below a predetermined value.

[0021] The image processing unit 13 deforms at least a portion of a reference surface coordinate system having a predefined surface for projecting the camera image, according to the position of the virtual viewpoint. The reference surface coordinate system is stored in the recording medium 19. The reference surface coordinate system is, for example, bowl-shaped, surrounding the vehicle. The reference surface coordinate system has its center on the xy plane parallel to the vehicle's mounting surface (driving surface), and is formed from a surface having a component in the vehicle's height direction (z direction) with curvature from the center or near the center.

[0022] A base surface parallel to the xy plane can be formed near the center of the reference surface coordinate system, and its shape can be any shape such as a rectangle, triangle, ellipse, or circle. Furthermore, the curvature of the surfaces of the reference surface coordinate system does not have to be uniform. For example, the curvature of the surfaces near the base surface may be made relatively large, and the curvature of surfaces further away from the base surface may be made relatively small. The image processing unit 13 may also store in advance multiple reference surface coordinate systems of different shapes depending on the shape (size, form) of the vehicle. When multiple reference surface coordinate systems are stored, the reference surface coordinate system to be used is stored in the recording medium 19 in association with the vehicle speed, the vehicle's shift position, etc., and the image processing unit 13 may select a reference surface coordinate system according to the vehicle speed, the vehicle's shift position, etc.

[0023] The image processing unit 13 may read a pre-stored reference surface coordinate system and deform the shape of the read reference surface coordinate system. The image processing unit 13 may deform the shape of the reference surface coordinate system according to the position of the virtual viewpoint. For example, if the virtual viewpoint is located outside the outer edge of the reference surface coordinate system, a part of the reference surface coordinate system may be deformed so that the virtual viewpoint is located inside the outer edge of the reference surface coordinate system. By deforming a part of the reference surface coordinate system so that the virtual viewpoint is located inside the outer edge of the reference surface coordinate system, distortion of the shape of obstacles with height can be suppressed when obstacles with height are displayed on the screen, and an image without image defects can be displayed on the display 31. Other coordinate system deformation methods known at the time of filing can be used as the method for deforming the reference surface coordinate system.

[0024] The image processing unit 13 projects multiple camera images acquired from the camera 20 onto a reference surface coordinate system. If the reference surface coordinate system is deformed, the image processing unit 13 should project the camera images onto the deformed surface coordinate system (deformable coordinate system). The image processing unit 13 has an image conversion table that associates the coordinates of pixels in the camera image with the coordinates of the coordinate system in order to project the camera image data onto the reference surface coordinate system or the deformed coordinate system. The image conversion table is recorded on the recording medium 19.

[0025] Furthermore, a pre-prepared vehicle image may be superimposed on the reference surface coordinate system. The vehicle image can be created in advance based on the vehicle's design and stored in the recording medium 19. By superimposing the vehicle image onto the reference surface coordinate system, not only the surrounding image of the vehicle but also the vehicle itself can be displayed in three dimensions, which helps in understanding the relationship between the vehicle's position and orientation and the surrounding image. In this way, the image processing unit 13 refers to the image conversion table to associate the coordinates of the pixels included in the camera image with the coordinates of the coordinate system, projects the camera image onto the reference surface coordinate system or the deformed coordinate system, and superimposes the vehicle image onto the projected three-dimensional image to generate a display image (three-dimensional image) in a 3D view as seen from a virtual viewpoint of the vehicle and its surroundings.

[0026] The image processing unit 13 may generate display images from selectable virtual viewpoints, not limited to those specified by the user or set according to the driving conditions. For example, when a display image is shown on the display 31 in 3D view, the user can freely move the position of the virtual viewpoint by touching the display 31. When display in 3D view is selected, the image processing unit 13 generates multiple 3D view display images for each of the multiple virtual viewpoints selectable by the user. For example, when the user moves the virtual viewpoint by touching the display 31, a 3D image is displayed on the display 31 in which the vehicle image rotates at the center point of the curved coordinate system. The image processing unit 13 may also set a virtual viewpoint at any position around the vehicle, not limited to the multiple virtual viewpoints selectable by the user, and generate multiple 3D view display images for each virtual viewpoint. In other words, when a 3D image is displayed on the display 31 in rotational display, the image processing unit 13 generates a continuous 3D image so that the image display is smooth as the virtual viewpoint moves.

[0027] Furthermore, the image processing methods for the display images shown in the front / rear view, top view, and 3D view are not limited to those described above; other image processing methods may be used. Also, for the image processing methods for the display images shown in other display modes such as the skeleton view and front / rear wide view, methods known at the time of filing may be used as appropriate.

[0028] Next, display control by the display control device 100 will be explained along with specific examples of the display screen of the display 31. Figure 3 shows the display screen of the display 31 when the vehicle image is not displayed in skeleton form, and Figure 4 shows the display screen of the display 31 when the vehicle image is displayed in skeleton form. Note that in the top view display screen (left screen of the display 31) in Figures 3 and 4, the top direction on the paper is north. In the examples of Figures 3 and 4, the vehicle is stopped before an intersection (T-junction), the left screen of the display 31 displays the top view image, and the right screen of the display 31 displays the 3D view image. When the controller 10 displays the top view overhead image and the 3D view display image on the display 31, it generates a top view image 37 by superimposing viewpoint icons 33a to 33h and an automatic display start icon 34 onto the overhead view image of the vehicle. The top view image 37 includes an overhead view image of the vehicle from above, viewpoint icons 33a to 33h, and an automatic display start icon 34. Viewpoint icons 33a to 33h are icons that indicate the position and / or direction of the line of sight of a virtual viewpoint, and are selectable by the display control system 1 or the user. Multiple viewpoint icons 33a to 33h are arranged around the vehicle 35, with the vehicle 35 at the center. Viewpoint icon 33a is positioned above the vehicle image, which is located at the center of the overhead view image, while viewpoint icons 33c, 33e, and 33g are positioned to the right, below, and left of the vehicle image, respectively. Viewpoint icons 33b, 33d, 33f, and 33h are positioned in the upper right, lower right, lower left, and lower right directions relative to the vehicle image, respectively. When the display control system 1 or the user selects a viewpoint icon 33a to 33h, the display image (3D image) as seen from the selected viewpoint icon is displayed on the right side of the display 31.

[0029] As shown in Figures 3 and 4, the bicycle 36 is moving from left to right in front of the vehicle, and the bicycle 36 is located in the front left of the vehicle. The image of the bicycle 36 is displayed on the top view screen. When the driver (an occupant sitting in the front right seat) looks at the bicycle 36, it is difficult for the driver to see the bicycle 36 because it is in the blind spot area created by the vehicle's A-pillar. The controller 10 generates a 3D image of the vehicle as seen from a virtual viewpoint located on the opposite side of the bicycle 36, in order to make it easier for the driver to see the presence of the bicycle 36.

[0030] First, the controller 10 sets a virtual viewpoint on the opposite side of the vehicle from the bicycle 36 so that the driver can see the position and direction of the bicycle 36. In the example in Figure 3, since the bicycle 36 is located northwest of the vehicle, the virtual viewpoint is set southeast of the vehicle (the position of viewpoint icon 33d). The virtual viewpoint is set to a position where the vehicle 35 and bicycle 36 are viewed from the right rear of the vehicle. In other words, the position of the virtual viewpoint is set so that the direction the vehicle is viewed from the virtual viewpoint aligns with the direction the vehicle driver sees the obstacle. If the virtual viewpoint moves, the above virtual viewpoint position may be included in a series of movements. Then, the image processing unit 13 generates a 3D image 38 of the vehicle and its surroundings as seen from the virtual viewpoint, based on multiple camera images. The image processing unit 13 outputs the 3D image 38 as a display image to the display 31. The screen on the right side of the display 31 shows the 3D image 38 of the vehicle as seen from the virtual viewpoint corresponding to viewpoint icon 33d. Furthermore, on the left side of the display 31, the controller 10 displays a viewpoint icon 33d as a selection icon to indicate the position of the virtual viewpoint on the top view screen. The controller 10 displays the selection icon, for example, by changing the color of the viewpoint icon 33d. In the example in Figure 3, the selection icon is indicated by making the outer edge of the viewpoint icon 33a thicker. As shown in Figure 3, the display 31 displays the top view image 37 and the 3D image 38 side by side.

[0031] Furthermore, in this embodiment, when displaying the 3D image 38 on the display 3 in the 3D view, if a detected obstacle is hidden by the vehicle, the vehicle image included in the 3D image is displayed semi-transparently (skeleton display). The semi-transparent portion may be applied only to the part where the vehicle and the obstacle overlap and its surroundings. As shown in Figure 3, when the 3D image 38 is displayed on the display 31, the upper body of the rider is visible, but part of the bicycle 36 is hidden by the vehicle image 39. When the driver checks the 3D image 38, there is a risk of overlooking the bicycle 36. Therefore, the controller 10 displays the 3D image 38 on the display 31 with the vehicle image 39 semi-transparent. As shown in Figure 4, by making the vehicle image 39 semi-transparent, the entire image of the bicycle 36 is displayed on the display 31. That is, when the bicycle 36 is detected as a moving object, the controller 10 sets a virtual viewpoint on the opposite side of the vehicle from the bicycle 36 and generates a 3D image of the vehicle as seen from the virtual viewpoint, so at least a part of the image of the bicycle 36 may overlap with the image of the vehicle. When the moving object image and the vehicle image overlap, the moving object is hidden by the vehicle in the 3D view. Therefore, in this embodiment, by displaying the vehicle image semi-transparently, the entire bicycle 36 is displayed on the display 31 without being hidden by the vehicle, thus reducing the possibility that the driver may overlook the bicycle 36 on the 3D view screen.

[0032] The automatic display start icon 34 is an icon used to start automatic display in the 3D view, which automatically switches the 3D image by moving the virtual viewpoint. The automatic display start icon 34 is displayed at the center position of the vehicle 35 included in the top view image 37. When automatic display is executed, predetermined positions around the vehicle become the start and end points of the virtual viewpoint, and the virtual viewpoint moves from the start point to the end point, causing the vehicle image displayed in the 3D view to rotate 360 ​​degrees along a plane parallel to the road surface. In other words, in automatic display, a center point is placed on the vehicle image, and the vehicle image and the image of the area around the vehicle rotate 360 ​​degrees around the center point. For example, in the example in Figure 3, if the viewpoint icon 33a is selected and the user touches the automatic display start icon 34, the selected viewpoint icon 33a becomes the start and end point of the virtual viewpoint. The virtual viewpoint moves sequentially from viewpoint icon 33b to viewpoint icon 33h, rotating clockwise around the vehicle, starting from the position corresponding to viewpoint icon 33a, and returns to the position corresponding to the original viewpoint icon 33a. As the virtual viewpoint rotates, the three-dimensional image 47 in the 3D view rotates. The orientation of the vehicle image 39 included in the three-dimensional image 47 changes in the following order, starting from the front, then the right side, rear, and left side, before returning to the original front. In other words, the display is automatically rotated so that the three-dimensional image appears to rotate 360 ​​degrees.

[0033] Other examples of the screen displayed on the display 31 will be described with reference to Figures 5 and 6. Figure 5 shows the display screen of the display 31 when the vehicle image is not displayed in skeleton form, and Figure 6 shows the display screen of the display 31 when the vehicle image is displayed in skeleton form. Note that in the top view display screens of Figures 5 and 6 (left screen of the display 31), the top of the paper is considered to be north.

[0034] As shown in Figures 5 and 6, pedestrian 40 is walking towards the front of the vehicle on the left side of the vehicle's B-pillar. The image of pedestrian 40 is displayed on the top-view screen. As pedestrian 40 continues walking, pedestrian 40 enters the blind spot area created by the vehicle's A-pillar, making it difficult for the driver to see pedestrian 40 visually. The controller 10 generates a 3D image of the vehicle as seen from a virtual viewpoint located on the opposite side of pedestrian 40, making it easier for the driver to see pedestrian 40.

[0035] First, the controller 10 sets a virtual viewpoint on the opposite side of the vehicle 35 from the pedestrian 40 so that the driver can see the position and direction of the pedestrian 40. In the example in Figure 5, the pedestrian 40 is located to the west of the vehicle 35, and the virtual viewpoint is set to the southeast of the vehicle 35 (the position of the viewpoint icon 33d). Note that the position on the opposite side of the vehicle 35 from the pedestrian 40 does not necessarily have to be directly opposite the pedestrian 40 across the vehicle 35; it is sufficient if a part of the vehicle is located between the pedestrian 40 and the virtual viewpoint. Then, the image processing unit 13 generates a 3D image 38 of the vehicle and its surroundings as seen from the virtual viewpoint, based on multiple camera images. The image processing unit 13 outputs the 3D image 38 as a display image to the display 31. On the right side of the display 31, the 3D image 38 of the vehicle as seen from the virtual viewpoint corresponding to the viewpoint icon 33d is displayed. On the left side of the display 31, the controller 10 displays the viewpoint icon 33d as a selection icon to indicate the position of the virtual viewpoint on the top view screen. Furthermore, on the left side of the display 31, the controller 10 displays a viewpoint icon 33d as a selection icon to indicate the position of the virtual viewpoint on the top view screen. As shown in Figure 5, the display 31 displays the top view image 37 and the 3D image 38 side by side.

[0036] Furthermore, in this embodiment, when displaying the 3D image 38 on the display 3 in the 3D view, if a detected obstacle is hidden by a vehicle, the vehicle image included in the 3D image is displayed semi-transparently (skeleton display). As shown in Figure 5, if the 3D image 38 is displayed as is on the display 31, part of the pedestrian 40 will be hidden by the vehicle image 39. Therefore, as shown in Figure 6, the controller 10 displays the 3D image 38 on the display 31 with the vehicle image 39 semi-transparent. In this embodiment, by displaying the vehicle image semi-transparently, the entire pedestrian 40 is displayed on the display 31 without being hidden by the vehicle, thus reducing the possibility that the driver may overlook the pedestrian 40 on the 3D view display screen.

[0037] Other examples of the screen displayed on the display 31 will be described with reference to Figures 7 and 8. Figure 7 shows the display screen of the display 31 when the vehicle image is not displayed in skeleton form, and Figure 8 shows the display screen of the display 31 when the vehicle image is displayed in skeleton form. Note that in Figures 7 and 7, the top view display screen (left screen of the display 31) is oriented with the top of the paper facing north.

[0038] As shown in Figures 7 and 8, the pedestrian 40 (child) is positioned behind the left side of the vehicle's B-pillar and is walking towards the front of the vehicle. The pedestrian 40 (child) is displayed on the top-view screen. When the driver visually observes the pedestrian 40, it is difficult for the driver to see the pedestrian because the pedestrian 40 is in the blind spot area created by the vehicle's B-pillar. The controller 10 generates a 3D image of the vehicle as seen from a virtual viewpoint located on the opposite side of the pedestrian 40, making it easier for the driver to see the presence of the pedestrian 40.

[0039] First, the controller 10 sets a virtual viewpoint on the opposite side of the vehicle 35 from the pedestrian 40 so that the driver can see the position and direction of the pedestrian 40. In the example in Figure 7, the pedestrian 40 is located southwest of the vehicle 35, and the virtual viewpoint is set northeast of the vehicle 35 (the position of viewpoint icon 33b). The virtual viewpoint is set to a position where the vehicle 35 and the pedestrian 40 are viewed from the front right of the vehicle. In other words, the position of the virtual viewpoint is set so that the direction the vehicle is viewed from the virtual viewpoint aligns with the direction the vehicle driver sees the obstacle. If the virtual viewpoint moves, the above virtual viewpoint position may be included in a series of movements. Then, the image processing unit 13 generates a 3D image 38 of the vehicle and its surroundings as seen from the virtual viewpoint, based on multiple camera images. The image processing unit 13 outputs the 3D image 38 as a display image to the display 31. The screen on the right side of the display 31 shows the 3D image 38 of the vehicle as seen from the virtual viewpoint corresponding to viewpoint icon 33b. Furthermore, on the left side of the display 31, the controller 10 displays a viewpoint icon 33b as a selection icon to indicate the position of the virtual viewpoint on the top view screen. As shown in Figure 7, the display 31 displays the top view image 37 and the 3D image 38 side by side.

[0040] Furthermore, in this embodiment, when displaying the 3D image 38 on the display 3 in the 3D view, if a detected obstacle is hidden by a vehicle, the vehicle image included in the 3D image is displayed semi-transparently (skeleton display). As shown in Figure 7, if the 3D image 38 is displayed as is on the display 31, the entire body of the pedestrian 40 will be hidden by the vehicle image 39. Therefore, as shown in Figure 8, the controller 10 displays the 3D image 38 on the display 31 with the vehicle image 39 semi-transparent. In this embodiment, by displaying the vehicle image semi-transparently, the entire body of the pedestrian 40 is displayed on the display 31 without being hidden by the vehicle, thus reducing the possibility that the driver may overlook the pedestrian 40 on the 3D view display screen.

[0041] Other examples of the screen displayed on the display 31 will be described with reference to Figures 9 and 10. Figure 9 shows the display screen of the display 31 when the vehicle image is not displayed in skeleton form, and Figure 10 shows the display screen of the display 31 when the vehicle image is displayed in skeleton form. Note that in the top view display screens of Figures 9 and 10 (the left screen of the display 31), the top of the paper is considered to be north.

[0042] As shown in Figures 9 and 10, the bicycle 36 is located to the left rear of the vehicle and is walking toward the front of the vehicle. The bicycle 36 is displayed on the top-view screen. The controller 10 generates a 3D image of the vehicle as seen from a virtual viewpoint located on the opposite side of the bicycle 36, making it easier for the driver to confirm the presence of the bicycle 36.

[0043] First, the controller 10 sets a virtual viewpoint on the opposite side of the vehicle 35 from the bicycle 36 so that the driver can see the position and direction of the bicycle 36. In the example in Figure 9, the bicycle 36 is located south-southwest of the vehicle 35, and the virtual viewpoint is set northeast of the vehicle 35 (the position of the viewpoint icon 33b). The virtual viewpoint is set to a position where the vehicle 35 and bicycle 36 are viewed from the front right of the vehicle. In other words, the position of the virtual viewpoint is set so that the direction the vehicle is viewed from the virtual viewpoint aligns with the direction the vehicle driver sees obstacles. Then, the image processing unit 13 generates a 3D image 38 of the vehicle and its surroundings as seen from the virtual viewpoint, based on multiple camera images. The image processing unit 13 outputs the 3D image 38 as a display image to the display 31. The right side of the display 31 screen shows the 3D image 38 of the vehicle as seen from the virtual viewpoint corresponding to the viewpoint icon 33b. On the left side of the display 31 screen, the controller 10 displays the viewpoint icon 33b as a selection icon to indicate the position of the virtual viewpoint on the top view screen. As shown in Figure 9, the display 31 displays the top-view image 37 and the 3D image 38 side by side.

[0044] If the position of the bicycle 36 is as shown in FIG. 9, even if the three-dimensional image 38 is displayed in the 3D view, the bicycle 36 will not be hidden by the vehicle image 39. However, for example, when the vehicle 35 is parked, if the bicycle 36 moves from the position shown in FIG. 9 and enters the blind spot area of the vehicle 35, in the 3D view, the bicycle 36 will be hidden by the vehicle image 39. Therefore, as shown in FIG. 10, the controller 10 may display the three-dimensional image 38 with the vehicle image 39 in a semi-transparent state on the display 31 (skeleton display).

[0045] Specifically, when the obstacle detection unit 12 detects an obstacle around the vehicle, it predicts the relative position of the obstacle with respect to the vehicle. As shown in FIGS. 9 and 10, when the vehicle 35 is stopped and the bicycle 36 is moving, the obstacle detection unit 12 detects the bicycle 36 as an obstacle and predicts the relative position of the bicycle 36 with respect to the vehicle 35 from the change in the position of the bicycle 36. The obstacle detection unit 12 predicts whether the obstacle will enter the blind spot area based on the predicted relative position of the obstacle with respect to the vehicle. For example, the obstacle detection unit 12 predicts the elapsed time until the obstacle enters the blind spot area of the vehicle from the relative position of the obstacle with respect to the vehicle. The blind spot area of the vehicle is the area that is blind to the driver and is caused by the A-pillar and / or B-pillar. And when the predicted elapsed time is less than or equal to a predetermined time threshold, the obstacle detection unit 12 predicts that the obstacle will enter the blind spot area. When the vehicle 35 is running, the obstacle detection unit 12 may also acquire vehicle speed information and predict the elapsed time until the obstacle enters the blind spot area of the vehicle from the vehicle speed and the relative position of the obstacle with respect to the vehicle.

[0046] When it is predicted that the obstacle will enter the blind spot area, the image processing unit 13 displays the three-dimensional image 38 with the vehicle image semi-transparent on the display 31 before the obstacle enters the blind spot area. As shown in FIG. 10, before the bicycle 36 enters the blind spot area, the entire bicycle 36 is displayed on the display 31 without being hidden by the vehicle. Thereby, even if the bicycle 36 moves and enters the blind spot area, the driver can confirm the bicycle 36 on the display screen of the 3D view.

[0047] Next, a display control method by the controller 10 will be described with reference to FIGS. 11 to 13. The processing sequence executed by the controller 10 includes a scene determination process for determining a driving scene, an obstacle detection process for detecting an obstacle, a view display process for selecting a display mode and displaying an image in the selected display mode. FIG. 11 shows a flowchart of the scene determination process, and FIG. 12 shows a flowchart of the view display process. FIG. 13 is a sub-flow of the process of step S15 included in the view display process. Note that the controller 10 executes the obstacle detection process before executing the view display process. The obstacle detection process is executed by the obstacle detection unit 12 and includes a process of detecting an obstacle around the vehicle and a process of detecting the position or movement of the obstacle. Note that the controller 10 executes the following control flow when the main switch of the vehicle (also referred to as a power switch or an ignition switch) is turned on. Also, when the main switch of the vehicle is in the on state, the controller 10 repeatedly executes the following control flow and terminates the following control flow when the main switch of the vehicle is turned off.

[0048] As shown in FIG. 11, at step S1, the controller 10 determines whether the shift position is in the drive range (D range). When the shift position is in the drive range (D range), the controller 10 determines whether the vehicle speed is zero (0 [km / h]). When the vehicle speed is zero, the controller 10 determines whether the brake is off based on the measurement value of a sensor that measures the strength of the depression of the brake pedal (step S3). When the brake is off, the controller 10 determines that it is a starting scene (step S4). The starting scene is a scene in which the vehicle is stopped, the brake is released, and driving is started. Then, the controller 10 ends the scene determination process.

[0049] In the determination process of step S2, if the vehicle speed is greater than zero ("No" in step S2), the controller 10 determines whether the current vehicle speed is less than or equal to a predetermined vehicle speed threshold (e.g., 30 [km / h]) (step S5). If the vehicle speed is less than or equal to the predetermined vehicle speed threshold (30 [km / h]), in step S6 the controller 10 determines whether the distance from the vehicle's current position to the intersection is less than a predetermined distance threshold (e.g., 30 [m]). The controller 10 can determine the location of the intersection from the image captured by the camera 20 or from map information.

[0050] If the distance from the vehicle's current position to the intersection is greater than or equal to a predetermined distance threshold (e.g., 30 m) (No in step S6), the controller 10 determines that it is a straight-ahead scene (step S7). A straight-ahead scene is a scene where the vehicle is traveling on a road that is not at or near an intersection. The controller 10 then terminates the scene determination process. On the other hand, if the distance from the vehicle's current position to the intersection is less than a predetermined distance threshold (e.g., 30 m) (Yes in step S6), the controller 10 determines that it is an intersection scene (step S8). A straight-ahead scene is a scene where the vehicle is traveling on a road that is at or near an intersection. The controller 10 then terminates the scene determination process.

[0051] If, in the control flow of step S1, it is determined that the shift position is not in the drive range (D range) ("No" in step S1), if, in the control flow of step S3, it is determined that the brake is applied ("No" in step S3), or if, in the control flow of step S5, it is determined that the vehicle speed is greater than a predetermined vehicle speed threshold (30 [km / h]) ("No" in step S5), then in step S9, the controller 10 determines that the scene does not apply (step S9). The controller 10 then terminates the scene determination process.

[0052] After executing the scene determination process, the controller 10 performs the following view display process. As shown in Figure 12, in step S11, the controller 10 determines whether the driving scene is an irrelevant scene or not. If it is determined that the driving scene is not an irrelevant scene, the controller 10 determines whether an obstacle is within the detection range or not (step S12). If it is determined that an obstacle is within the detection range, in step S13, the controller 10 determines whether the driving scene is an intersection scene or not. If it is determined that the driving scene is an intersection scene, in step S14, the controller 10 determines whether visibility is poor or not based on the image captured by the camera 20 and / or map information.

[0053] If, in the control flow of step S13, it is determined that the driving scene is not an intersection scene ("No" in step S13), or if, in the control flow of step S14, it is determined that visibility is not poor ("No" in step S14), then in step S15, the controller 10 controls the display 30 to display the image in both the 3D view and the top view. Details of the 3D view display control flow will be described later. Then, the controller 10 terminates the view display process.

[0054] If the control flow in step S14 determines that visibility is poor (Yes in step S4), the controller 10 controls the display 30 in step S16 to display the image in both front wide view and top view. For example, the top view image is displayed on the left side of the display 31, and the front wide view screen is displayed on the right side of the display 31. Then the controller 10 terminates the view display process.

[0055] If it is determined in the control flow of step S12 that there are no obstacles within the detection range, in step S17 the controller 10 determines whether the driving scene is an intersection scene. If it is determined that the driving scene is an intersection scene, in step S18 the controller 10 determines whether visibility is poor based on the image captured by the camera 20 and / or map information. If it is determined that visibility is poor, in step S19 the controller 10 determines whether there are traffic lights at the intersection based on the image captured by the camera 20 and / or map information. If it is determined that there are no traffic lights at the intersection, in step S20 the controller 10 controls the display 30 so that the display image is shown in front wide view. Then the controller 10 terminates the view display process.

[0056] In the control flow of step S11, if it is determined that the driving scene is not an applicable scene ("Yes" in step S11), if it is determined in the control flow of step S17 that the driving scene is not an intersection scene ("No" in step S17), if it is determined in the control flow of step S18 that visibility is not poor ("No" in step S18), or if it is determined that there is a traffic light at the intersection ("No" in step S19), the controller 10 will cancel the view display. The controller 10 will then terminate the view display process.

[0057] In the control flow of step S15, the controller 10 executes the following subflow when displaying an image in the 3D view. In step S31, the image acquisition unit 11 acquires an image captured from the camera 20. In step S32, the image processing unit 13 sets a virtual viewpoint on the opposite side of the vehicle from the obstacle. The obstacle is detected by the obstacle detection process. In step S33, the image processing unit 13 determines whether the detected obstacle is within the vehicle's blind spot. The blind spot may be a pre-set range. For example, an area that is likely to be a blind spot when the driver looks around the vehicle, due to the structure of the body such as the A-pillar and B-pillar, may be pre-set as the blind spot. Alternatively, the driver's eye position, face position, face direction, gaze, and viewpoint position may be detected by an in-vehicle camera, and the blind spot may be dynamically set based on the detected eye position, etc.

[0058] If it is determined that the detected obstacle is outside the vehicle's blind spot (No in step S33), in step S34 the image processing unit 13 determines whether the obstacle is hidden by the vehicle on the display screen of the display 3 when an overhead image of the vehicle viewed from a virtual viewpoint in 3D view is displayed on the display 30. If it is determined that the obstacle is not hidden by the vehicle, the image processing unit 13 executes the control process in step S36.

[0059] If, in the control flow of step S33, it is determined that an obstacle is in a blind spot area (Yes in step S33), or if, in the control flow of step S34, it is determined that an obstacle is hidden by the vehicle (Yes in step S34), the image processing unit sets to skeleton display in step S35. In step S36, the image processing unit 13 generates a 3D image of the vehicle as seen from a virtual viewpoint. When set to skeleton display, the vehicle image included in the 3D image becomes semi-transparent. On the other hand, when not set to skeleton display (when set to normal display), the vehicle image included in the 3D image is not semi-transparent.

[0060] In step S37, the controller 10 outputs a 3D image to the display 30. Specifically, the controller 10 outputs a display control signal to the display 30 that includes the display image to display the 3D image. As a result, the display 30 displays the 3D view image.

[0061] As described above, the display control method and display control device 100 according to this embodiment acquire images from the camera 20, detect obstacles around the vehicle, and, if an obstacle is detected, generate a 3D image of the vehicle as seen from a virtual viewpoint located on the opposite side of the obstacle from the vehicle, based on multiple images, and output the 3D image to the display 30. The vehicle image included in the 3D image is displayed semi-transparently. This allows the positional relationship between the vehicle and the obstacle to be understood on the display screen.

[0062] Furthermore, in the display control method and display control device 100 according to this embodiment, the position of the virtual viewpoint is set such that the direction in which the vehicle is viewed from the virtual viewpoint aligns with the direction in which the obstacle is viewed from the vehicle's driver. This allows the positional relationship between the vehicle and the obstacle to be understood on the display screen.

[0063] Furthermore, in the display control method and display control device 100 according to this embodiment, if the detected obstacle is located in a blind spot area that is a blind spot for the driver, the vehicle image included in the 3D image is displayed semi-transparently. This allows the presence and location of the obstacle to be understood in the 3D view.

[0064] Furthermore, the display control method and display control device 100 according to this embodiment predict the relative position of an obstacle to the vehicle, and if it is predicted that the obstacle will enter the blind spot area, a three-dimensional image of the vehicle with semi-transparent material may be displayed on the display before the obstacle enters the blind spot area. This allows the driver to confirm the obstacle on the 3D view display screen even if the obstacle moves and enters the blind spot area. In addition, it can reduce the risk of the vehicle coming into contact with an obstacle when the vehicle is turning left or right.

[0065] As another modification of this embodiment, the display control method and display control device 100 may enlarge the image of the obstacle included in the 3D image and display it on the display 30 if the size of the detected obstacle is smaller than a predetermined value. For example, as shown in Figures 9 and 10, when a child pedestrian is detected as an obstacle, the obstacle detection unit 12 measures the size of the obstacle from the image captured by the camera 20. If the size of the obstacle is smaller than a predetermined value, the image processing unit 13 enlarges the image of the child and generates a 3D image. As a result, the image of the child is displayed larger on the display 30, allowing the driver to easily identify the obstacle. Note that obstacles smaller than a predetermined value may include not only children but also small animals such as cats and dogs.

[0066] As a second modification of this embodiment, the display control method and display control device 100 may display a 3D image with a higher virtual viewpoint on the display 30 if the size of the detected obstacle is smaller than a predetermined value. This makes it easier to identify small obstacles because the display 30 shows an image viewed from a higher position.

[0067] As a third modification of this embodiment, the display control method and display control device 100 may predict the relative position of the obstacle to the vehicle, and if it is predicted that the obstacle and the vehicle may come into contact, the vehicle image of the part where the obstacle will come into contact with the vehicle may be highlighted and displayed. For example, as shown in Figures 5 and 6, if a pedestrian 40 is walking to the left of the vehicle 35 and the vehicle 35 turns left, there is a possibility that the pedestrian 40 will come into contact with the front or side of the vehicle. The image processing unit 13 displays the vehicle image 39 in skeleton form and generates a three-dimensional image 38 such that the front or side portion of the vehicle image 39 is highlighted. The highlighting of the vehicle image 39 can be done, for example, by changing the color (red, etc.) of the highlighted portion from the rest of the vehicle or by making it blink. The image processing unit 13 then displays the three-dimensional image on the display 30. This allows the driver to confirm on the display screen that there is a possibility that the obstacle will come into contact with the vehicle.

[0068] As a fourth modification of this embodiment, in the display control method and display control device 100, if the vehicle image includes a first image in which the obstacle and the vehicle overlap when viewed from a virtual viewpoint, and a second image in which the obstacle and the vehicle do not overlap, the transparency of the first image may be made higher than the transparency of the second image, thereby emphasizing and displaying the obstacle image.

[0069] As shown in Figures 7 and 8, a modified example of this embodiment, Modification 4, will be described using the case where a child is walking around the vehicle and the pedestrian is detected as an obstacle. On the right side of the display 31, a three-dimensional image 38 of the vehicle viewed from a virtual viewpoint corresponding to the viewpoint icon 33b is displayed. At this time, the child's image overlaps with the vehicle image 39. The part of the vehicle image 39 that overlaps with the child's image corresponds to the first image, and the part of the vehicle image 39 that does not overlap with the child's image corresponds to the second image. When the image processing unit 13 displays the vehicle image 39 on the display 31 in skeleton view, it makes the transparency of the first image higher than that of the second image. This allows the driver to easily confirm the presence of an obstacle when viewing the display screen of the display 31 in 3D view. The image processing unit 13 may also add color (red, etc.) to the outline of the obstacle when displaying the image of the obstacle.

[0070] In this embodiment, the controller 10 does not need to execute all of the control flows shown in Figures 11 to 13, nor does it need to execute each control flow in the order shown in Figures 11 to 13.

[0071] 1 Display control system 10 Controller 11 Image acquisition unit 12 Obstacle detection unit 13 Image processing unit 19 Recording medium 20 Camera 30 Display 100 Display control device

Claims

1. A display control method executed by a display control device mounted on a vehicle, comprising: acquiring images from a plurality of on-board cameras that capture images of the area around the vehicle; detecting obstacles around the vehicle; if an obstacle is detected, generating a three-dimensional image of the vehicle as seen from a virtual viewpoint located on the opposite side of the vehicle from the obstacle, based on the plurality of images; outputting the three-dimensional image to a display; and displaying the vehicle image included in the three-dimensional image semi-transparently.

2. A display control method according to claim 1, wherein the position of the virtual viewpoint is set such that the direction in which the vehicle is viewed from the virtual viewpoint is aligned with the direction in which the obstacle is viewed from the driver of the vehicle.

3. A display control method according to claim 1 or 2, wherein when the detected obstacle is located in a blind spot area that is a blind spot for the driver, the vehicle image included in the three-dimensional image is displayed semi-transparently.

4. A display control method according to claim 3, wherein the relative position of the obstacle with respect to the vehicle is predicted, and when it is predicted that the obstacle will enter the blind spot area, the three-dimensional image of the vehicle, in which the vehicle image is made semi-transparent, is displayed on the display before the obstacle enters the blind spot area.

5. A display control method according to any one of claims 1 to 3, wherein if the size of the detected obstacle is smaller than a predetermined value, the image of the obstacle included in the three-dimensional image is enlarged and displayed.

6. A display control method according to any one of claims 1 to 5, wherein if the size of the detected obstacle is smaller than a predetermined value, the height of the virtual viewpoint is increased and the three-dimensional image is displayed.

7. A display control method according to any one of claims 1 to 3, wherein the relative position of the obstacle with respect to the vehicle is predicted, and when it is predicted that the obstacle and the vehicle may come into contact, the portion of the vehicle image in which the obstacle comes into contact with the vehicle is highlighted and displayed.

8. A display control method according to any one of claims 1 to 7, wherein the vehicle image includes a first image in which the obstacle and the vehicle overlap when viewed from the virtual viewpoint, and a second image in which the obstacle and the vehicle do not overlap, the transparency of the first image is higher than the transparency of the second image, and the obstacle image showing the obstacle is emphasized in the display control method.

9. A display control device comprising a controller for controlling a display mounted on a vehicle, wherein the controller acquires images from a plurality of on-board cameras that capture images of the area around the vehicle, detects obstacles around the vehicle, and, if an obstacle is detected, generates a three-dimensional image of the vehicle as seen from a virtual viewpoint located on the opposite side of the vehicle from the obstacle, based on the plurality of acquired images, outputs the three-dimensional image to a display, and displays the vehicle image included in the three-dimensional image semi-transparently.

10. A recording medium on which a display control program executed by a controller is recorded, wherein the display control program causes the controller to execute a process including: acquiring images from a plurality of on-board cameras that capture images of the area around a vehicle; detecting obstacles around the vehicle; and, if an obstacle is detected, generating a three-dimensional image of the vehicle as seen from a virtual viewpoint located on the opposite side of the vehicle from the obstacle, based on the plurality of images; and outputting the three-dimensional image to a display, wherein the vehicle image included in the three-dimensional image is displayed semi-transparently.

Images