Display control method, display control device, and display control system

Abstract

This display control device acquires a vehicle rearward image obtained by imaging toward the rear of a vehicle in which a user is riding, and controls image display by an information presenting device worn by the user such that the vehicle rearward image is superimposed on the field of vision of the user or a window portion of the vehicle included in an actual image corresponding to the field of vision.

Description

Display Control Method, Display Control Apparatus and Display Control System

[0001] The present invention relates to a display control method, a display control apparatus and a display control system.

[0002] There is known a technology in which a camera is provided at a position similar to that of a mirror-type right outside mirror to capture an image of the right rear of the vehicle, and the captured image is displayed on an image display device provided at the right end of an instrument panel (Patent Document 1).

[0003] Japanese Unexamined Patent Application Publication No. 2018-55075

[0004] However, the technology described in Patent Document 1 has a problem that the image display device needs to be installed in the front of the vehicle interior to display the rear image of the vehicle, which impairs the appearance of the vehicle interior.

[0005] The problem to be solved by the present invention is to provide a display control method, a display control apparatus and a display control system capable of displaying a rear image of a vehicle without impairing the appearance of the vehicle interior.

[0006] The present invention solves the above problem by acquiring a vehicle rear image obtained by capturing an image of the rear of a vehicle on which a user rides, and controlling image display on an information presentation device worn by the user such that the vehicle rear image is superimposed on a window portion of the vehicle included in the user's field of view or a real image corresponding to the user's field of view.

[0007] According to the present invention, a rear image of a vehicle can be displayed without impairing the appearance of the vehicle interior.

[0008] Figure 1 is a block diagram showing one embodiment of a display control system equipped with a display control device according to this embodiment. Figure 2A is a diagram showing an example of an augmented reality space presented to a user via the information presentation device of this embodiment. Figure 2B is a diagram showing an example of an augmented reality space presented to a user via the information presentation device of this embodiment. Figure 2C is a diagram showing an example of an augmented reality space presented to a user via the information presentation device of this embodiment. Figure 3 is a flowchart diagram showing the control procedure of a display control method executed by the display control device according to this embodiment. Figure 4 is a flowchart diagram showing the control procedure of another example of a display control method executed by the display control device according to this embodiment. Figure 5 is a flowchart diagram showing the control procedure of another example of a display control method executed by the display control device according to this embodiment. Figure 6 is a flowchart diagram showing the control procedure of another example of a display control method executed by the display control device according to this embodiment. Figure 7 is a flowchart diagram showing the control procedure of another example of a display control method executed by the display control device according to this embodiment.

[0009] Embodiments of the present invention will be described below with reference to the drawings. Figure 1 is a block diagram showing one embodiment of a display control system equipped with a display control device according to this embodiment. As shown in Figure 1, the display control system 1000 comprises a vehicle 1, an information display device 2, and an external server 3. The display control system 1000 is a system for presenting images to a user to grasp the situation outside and inside the vehicle 1 by controlling the image display of the information display device 2. The user is an occupant of the vehicle 1, such as the driver. The vehicle 1 comprises a display control device 10, an on-board camera 11, a LiDAR 12, an ultrasonic sensor 13, an IVI 14, an IVC 15, and a vehicle ECU 16. Each device provided in the vehicle 1 is connected by an in-vehicle communication network such as CAN or LIN to exchange information with each other. The vehicle 1, the information display device 2, and the external server 3 exchange information with each other via a wireless communication network. The wireless communication network includes, for example, mobile communication networks such as 4G lines, the Internet, Wi-Fi (registered trademark), and the like.

[0010] The display control device 10 is a device that controls the image display of the information display device 2. The information display device 2 is a device worn by the user and presents an augmented reality (AR) space to the user wearing the information display device 2. The augmented reality space is a space constructed based on real space information and virtual space information. Specifically, the augmented reality space is a space in which the real space is visually extended by superimposing virtual objects onto images of the real space. In this embodiment, the display control device 10 controls the image display of the information display device 2 so as to present the user with an augmented reality space in which virtual objects are displayed in the real space, including the outside and inside of the vehicle 1, via the information display device 2 worn by the user. In the following description, the display control device 10 will be described as a device provided in the vehicle 1, but it is not limited to this, and the display control device 10 may be located outside the vehicle 1 or provided in the information display device 2. As an example of the display control device 10 being located outside the vehicle 1, the display control device 10 may be provided in the external server 3. Details of the display control device 10 and the information display device 2 will be described later.

[0011] The on-board camera 11 is installed in the vehicle 1 and captures images of the area around the vehicle 1. The on-board camera 11 includes a front camera 111, a rear camera 112, a right-side camera 113, and a left-side camera 114. The front camera 111 captures images of the front of the vehicle 1, the rear camera 112 captures images of the rear of the vehicle 1, the right-side camera 113 captures images of the right side of the vehicle 1, and the left-side camera 114 captures images of the left side of the vehicle 1. The front camera 111 is installed in the front part of the vehicle 1, for example, in the grille of the vehicle 1. The rear camera 112 is installed in the rear part of the vehicle 1, for example, in the tailgate. The right-side camera 113 is installed in the right-side part of the vehicle 1, for example, in the right-side door of the front part of the vehicle 1. The left-side camera 114 is installed in the left-side part of the vehicle 1, for example, in the left-side door of the front part of the vehicle 1. The various cameras may be installed in other locations on the vehicle 1. These in-vehicle cameras 11 may also be cameras used in an in-vehicle camera monitor system (CMS). In the following description, the front camera 111, rear camera 112, right-side camera 113, and left-side camera 114 will be collectively referred to as in-vehicle cameras 11. The in-vehicle cameras 11 capture images of the area around the vehicle 1 at regular intervals and output the vehicle surrounding images to the display control device 10. The in-vehicle cameras 11 may also output the vehicle surrounding images to the vehicle ECU 16.

[0012] The LiDAR 12 measures the distance and direction from the vehicle 1 to an object, as well as the distance between objects, by irradiating it with laser light and measuring the reflected light from the object. The ultrasonic sensor 13 measures the distance and direction from the vehicle 1 to an object, as well as the distance between objects, by emitting ultrasonic waves and measuring the reflected waves from the object. The measurement information measured by the LiDAR 12 and the ultrasonic sensor 13 is output to the display control device 10. The IVI 14 is a system that integrates a navigation system that provides map information and route guidance, and an entertainment system that provides entertainment elements such as music and video. The IVI 14 controls in-vehicle equipment such as displays and speakers to provide information to the user. The IVI 14 outputs various information received from the external server 3 via the IVC 15 to the display control device 10. The IVC 15 is a communication terminal that connects the vehicle 1 to the outside via a wireless communication network. The IVC 15 can communicate with the external server 3. The IVC 15 receives various information from the external server 3. The information received from the external server 3 includes, for example, entertainment content data such as music and videos, and software update data.

[0013] The vehicle ECU 16 is an electronic control unit mounted on the vehicle 1 and collects vehicle information of the vehicle 1. The vehicle information includes information that includes the state of the vehicle 1, such as the vehicle speed of the vehicle 1, the shift position of the vehicle 1, the energy level of the vehicle 1, the current position of the vehicle 1, user driving operation information, and the functional status of the vehicle 1. The vehicle information may also include images of the vehicle's surroundings acquired by the onboard camera 11. The vehicle ECU 16 acquires vehicle information from other devices such as other ECUs in the vehicle 1 and outputs it to the display control device 10.

[0014] The information display device 2 presents the user with an augmented reality space in which virtual objects are displayed in the real space, including the exterior and interior of the vehicle 1. The virtual objects displayed on the information display device 2 include, for example, an image of the area around the vehicle. The image of the area around the vehicle includes an image of the area behind the vehicle 1. The information display device 2 includes a camera 21, a display 22, and a speaker 23. The camera 21 captures the surrounding environment as seen by the user and acquires a real image corresponding to the user's field of view. When the user is riding in the vehicle 1, the surrounding environment as seen by the user includes the environment outside and inside the vehicle 1. The display 22 is a display that displays a display image including virtual objects, and is, for example, a transparent display that realizes the augmented reality space using an optical see-through method. The display 22 displays the display image based on a control signal related to display control output by the display control device 10. The speaker 23 is a device that outputs audio information to the user. The information display device 2 also includes a CPU, which is a processor, ROM, and RAM. The ROM contains a program for presenting information to the user. This program is executed by the CPU and controls each component of the information presentation device 2 to realize the functions of the information presentation device 2.

[0015] In this embodiment, the information display device 2 is a wearable terminal worn by the user. The information display device 2 is, for example, AR glasses, smart glasses, or a head-mounted display, and the display 22 is positioned in front of the user's eyes when the user is wearing the information display device 2. In the following description, the information display device 2 will be described as AR glasses. That is, the display 22 corresponds to the lens portion of the AR glasses. The camera 21 is located on the frame portion of the AR glasses, outside the left and right lens portions. The speaker 23 is a speaker positioned near the user's left and right ears when the user is wearing the information display device 2.

[0016] If the display 22 is a transparent display, the image behind the display is visible, allowing the user to directly view the image, that is, the image generated by the refraction and reflection of light rays. The information presentation device 2 displays a virtual object on the display 22, so as to superimpose the virtual object onto the image that is directly projected into the user's field of view via the display 22. This presents the user with an augmented reality space that appears as if the virtual object exists in the surrounding environment projected into the user's field of view. In this embodiment, the virtual object displayed on the display 22 is an image of the rear of the vehicle, and the user can see an image through the display 22 of the information presentation device 2 as if an electronic mirror device were installed inside the vehicle 1.

[0017] In the following description, the display 22 is described as a transparent display, but the display 22 may also be an opaque display that realizes an augmented reality space using a video see-through method. If the display 22 is an opaque display, the user's field of view is obstructed by the display 22 placed in front of the user's eyes, so the user cannot directly see the image behind the display 22, and instead sees the actual image displayed on the display 22, which is an image of the user's field of view that has been captured, that is, an image reproduced by electrical conversion. The information presentation device 2 displays a composite image on the display 22, which is created by combining a real image (an image corresponding to the real space) corresponding to the user's field of view captured by the camera 21 and a virtual object, so that the virtual object is superimposed on the real image.

[0018] External server 3 transmits various information to display control device 10 via IVC 15. External server 3 is, for example, a cloud server. The various information is information provided to the user and includes, for example, entertainment content data. External server 3 may also transmit software update data, etc.

[0019] The display control device 10 includes at least a controller 100. The controller 100 controls the image display of the information display device 2. The controller 100 includes a computer having hardware and software, which includes a ROM that stores programs, a CPU that executes the programs stored in the ROM, and RAM that functions as an accessible storage device. The controller 100 includes an image acquisition unit 101, a display control unit 102, and a vehicle state acquisition unit 103 as functional blocks, and each function is executed by the cooperation of software and hardware to realize each of the above functions or to execute each of the processes. In this embodiment, the functions of the controller 100 are divided into three blocks and the functions of each functional block are explained, but the functions of the controller 100 do not necessarily have to be divided into three blocks, and may be divided into two or fewer functional blocks, or four or more functional blocks. Also, in Figure 1, the vehicle 1 is shown to have one controller 100, but it is not limited to this, and the controller 100 may be divided into two or more units. In this case, for example, the controller 100 may be divided into a controller equipped with an image acquisition unit 101 and a controller equipped with a display control unit 102 and a vehicle status acquisition unit 103.

[0020] The image acquisition unit 101 performs an image acquisition process to acquire an image of the area around the vehicle, including an image of the rear of the vehicle, from the on-board camera 11 of the vehicle 1. The image acquisition process is performed at regular intervals. The image acquisition unit 101 may also generate an overhead view image including the area around the vehicle 1 based on the image of the area around the vehicle acquired from the on-board camera 11. The overhead view image is an around-view monitor (AVM) image as viewed from a virtual viewpoint above the vehicle 1, and is generated by combining images captured by multiple cameras included in the on-board camera 11. The image of the rear of the vehicle may be the image of the area around the vehicle itself captured by the rear camera 112, that is, the entire image, or a part of the image. The image of the rear of the vehicle may also include an image of the right rear of the vehicle 1, for example, an image captured by the right-side camera 113, or an image of the left rear of the vehicle 1, for example, an image captured by the left-side camera 114. The image of the rear of the vehicle may also be an image included in the central region of the image of the rear of the vehicle.

[0021] Furthermore, the image acquisition unit 101 may control the on-board camera 11 according to the state of the vehicle 1. For example, the image acquisition unit 101 generates a control command according to the state of the vehicle 1 and transmits the control command to the on-board camera 11. The on-board camera 11 operates according to the control command and takes images of the area around the vehicle 1.

[0022] The image acquisition unit 101 may generate a control command to image the area behind the vehicle 1 so that the rear image of the vehicle is enlarged or reduced depending on the vehicle speed of the vehicle 1. For example, when the shift position of the vehicle 1 is in the drive range, the image acquisition unit 101 determines whether the vehicle speed of the vehicle 1 is above a predetermined speed. If it is determined that the vehicle speed of the vehicle 1 is above the predetermined speed, the image acquisition unit 101 generates a control command to control the onboard camera 11 so that it zooms out. The onboard camera 11 images the area around the vehicle 1 in accordance with the control command, with a wide area around the vehicle 1 included in the field of view of the onboard camera 11. As a result, the image acquisition unit 101 can acquire a reduced rear image of the vehicle. If it is determined that the vehicle speed of the vehicle 1 is not above the predetermined speed, the image acquisition unit 101 generates a control command to control the onboard camera 11 so that it zooms in. In accordance with the control command, the on-board camera 11 captures images of the area around the vehicle 1 within a narrow field of view of the on-board camera 11. As a result, the image acquisition unit 101 can acquire an enlarged image of the rear of the vehicle.

[0023] Furthermore, the image acquisition unit 101 may generate a control command to adjust the angle of the onboard camera 11 so that the imaging direction of the onboard camera 11 faces downwards towards the vehicle 1 when the vehicle 1 is moving backward while parking spaces located around the vehicle 1 have been recognized. For example, when the shift position of the vehicle 1 is in the reverse range, the image acquisition unit 101 determines whether or not parking spaces have been recognized around the vehicle 1 by image recognition processing of the vehicle surrounding image. If it is determined that parking spaces have been recognized around the vehicle 1, the image acquisition unit 101 generates a control command to adjust the angle of the onboard camera 11 so that the imaging direction of the onboard camera 11 faces downwards towards the vehicle 1. The onboard camera 11 adjusts its angle so that the imaging direction faces downwards towards the vehicle 1 in accordance with the control command and images the area around the vehicle 1. As a result, the image acquisition unit 101 can acquire a vehicle surrounding image in which the parking spaces located around the vehicle 1 are clearly visible. If the system determines that no parking spaces are recognized around vehicle 1, the image acquisition unit 101 generates a control command to image the area behind vehicle 1 as usual without adjusting the angle of the onboard camera 11. In this case, the display 22 displays a normal image of the area behind the vehicle.

[0024] The display control unit 102 executes a display control process to control the image display of the information display device 2. In the display control process, the rear image of the vehicle is displayed on the display 22 as a virtual object. That is, the display control unit 102 controls the image display of the information display device 2 so that the rear image of the vehicle is superimposed on the window portion of the vehicle 1 that is included in the user's field of view. Specifically, the display control unit 102 outputs a control signal to the information display device 2 to perform the above display control. The control signal includes the rear image of the vehicle. In this embodiment, the vehicle 1 may be a vehicle in which door mirrors and electronic mirror devices are not provided on the interior and exterior such as the doors of the vehicle 1. Also, the vehicle 1 may be a vehicle in which there is no rearview mirror. As a result, there are no electronic mirror devices in the interior of the vehicle 1, and the appearance of the vehicle interior is not spoiled. Even in such a vehicle, the rear image of the vehicle is displayed in the user's field of view via the information display device 2 worn by the user, so the user can grasp what is behind the vehicle 1.

[0025] If the display 22 of the information display device 2 is a transparent display, first the display control unit 102 acquires video information included in the user's field of view from the information display device 2. The video included in the user's field of view is the captured image taken by the camera 21 of the information display device 2. The user's field of view includes the environment outside and inside the vehicle 1. The display control unit 102 recognizes the interior space of the vehicle 1 included in the user's field of view and determines the display position in the interior space where the rear image of the vehicle will be displayed. In this embodiment, the display control unit 102 recognizes the window portion in the video included in the user's field of view from the video information included in the user's field of view and acquires the position of the window portion. The window portion of the vehicle 1 is the front portion of the vehicle 1, more specifically, the window portion located in front of the driver's seat, and includes the door glass provided in the front door portion of the vehicle 1. The window portion of the vehicle 1 may also be the windshield provided at the front of the vehicle 1. The display control unit 102 identifies the position of the window portion of the vehicle 1 recognized on the captured image as the display position on the display 22 for displaying the rear image of the vehicle.

[0026] The display control unit 102 then uses the position of the window of the vehicle 1 as the display position and performs display control to superimpose the rear image of the vehicle onto the display position on the display 22. Specifically, if the display 22 of the information presentation device 2 is a transparent display, the display control unit 102 outputs a control signal to the information presentation device 2 to display the rear image of the vehicle on the display 22 based on two-dimensional coordinates representing the position of the window on the display 22. Since the position of the window of the vehicle 1 on the transparent display corresponds to the position of the window of the vehicle 1 in the user's field of view, it appears to the user that the rear image of the vehicle is displayed at the position of the window of the vehicle 1. In other words, the information presentation device 2 can provide the user with an augmented reality space in which a virtual electronic mirror device is installed at the position of the window of the vehicle 1. The display control by the display control unit 102 is performed on the user's field of view information acquired at regular intervals. That is, if the user's field of view changes in accordance with the movement of the user's head, the superimposed display of virtual objects, including the rear image of the vehicle, is also updated in accordance with the change in the user's field of view.

[0027] If the display of the information display device 2 is a non-transparent display, the display control unit 102 may control the image display of the information display device 2 so that the rear vehicle image is superimposed on the actual image corresponding to the user's field of view. The display control unit 102 acquires an actual image corresponding to the user's field of view from the camera 21 of the information display device 2. This actual image is an image captured of the exterior and interior of the vehicle 1 as seen by the user. The display control unit 102 uses the position of the window of the vehicle 1 as the display position and generates a display image by superimposing the rear vehicle image onto the window position on the actual image based on the two-dimensional coordinates representing the display position. The display control unit 102 then outputs a control signal to the information display device 2 to display the generated display image on the display 22 of the information display device 2. Note that the generation of the display image is not limited to being performed by the display control device 10; the display control device 10 may transmit information on the display position on the actual image and the rear vehicle image, and the display image may be generated in the information display device 2.

[0028] Here, an example of an augmented reality space presented to the user via the information display device 2 will be explained using Figures 2A, 2B, and 2C. Figures 2A, 2B, and 2C are diagrams showing an example of an augmented reality space presented to the user via the information display device of this embodiment. Figures 2A, 2B, and 2C are images viewed by the user via the display 22 of the information display device 2. The user is seated in the driver's seat and wearing the information display device 2. The display 22 is a transparent display, and the user can directly see the actual environment outside and inside the vehicle 1, except for the rear view image Vm.

[0029] In Figure 2A, the rear vehicle image Vm includes the first rear vehicle image Vm1, the second rear vehicle image Vm2, and the third rear vehicle image Vm3. The first rear vehicle image Vm1 is displayed superimposed on the door glass located on the left side door of vehicle 1. The first rear vehicle image Vm1 is an image of the left rear of vehicle 1. The first rear vehicle image Vm1 virtually displays the left side mirror. The second rear vehicle image Vm2 is displayed superimposed on the door glass located on the right side door of vehicle 1. The second rear vehicle image Vm2 is an image of the right rear of vehicle 1. The second rear vehicle image Vm2 virtually displays the right side mirror. The third rear vehicle image Vm3 is displayed superimposed on the upper front of vehicle 1, above the windshield. The third rear vehicle image Vm3 is an image of the area directly behind vehicle 1, that is, an image included in the central region of the rear vehicle image. The third vehicle rear view image Vm3 is a virtual representation of the rearview mirror.

[0030] In Figure 2B, the rear vehicle image Vm includes the fourth rear vehicle image Vm4. The fourth rear vehicle image Vm4 is displayed superimposed on the upper front of vehicle 1, above the windshield of vehicle 1. The fourth rear vehicle image Vm4 is displayed extending from the left side to the right side of vehicle 1. The fourth rear vehicle image Vm4 is an image created by integrally combining the first rear vehicle image Vm1, the second rear vehicle image Vm2, and the third rear vehicle image Vm3. Also, in Figure 2C, the rear vehicle image Vm includes the fifth rear vehicle image Vm5. The fifth rear vehicle image Vm5 is displayed superimposed on the steering wheel of vehicle 1. The fifth rear vehicle image Vm5 is an image created by integrally combining the first rear vehicle image Vm1, the second rear vehicle image Vm2, and the third rear vehicle image Vm3.

[0031] Furthermore, the display control unit 102 controls the switching between displaying and hiding the rear-view image of the vehicle on the information display device 2 based on the state of the vehicle 1. The state of the vehicle 1 is obtained from the vehicle ECU 16. For example, when the vehicle 1 is moving forward or backward, the display control unit 102 controls the image display of the information display device 2 so that the rear-view image of the vehicle is displayed. Whether or not the vehicle 1 is moving forward or backward is determined according to the speed of the vehicle 1 or the shift position of the vehicle 1. Also, when the vehicle 1 is stopped, when there are no obstacles around the vehicle 1, or when the vehicle 1 is in automatic driving mode, the display control unit 102 controls the image display of the information display device 2 so that the rear-view image of the vehicle is hidden. Obstacles are, for example, pedestrians, other vehicles, or any other object that obstructs the movement of the vehicle 1. The determination of whether or not the vehicle 1 is stopped is performed according to the speed of the vehicle 1 or the shift position of the vehicle 1. The determination of whether or not there are obstacles around the vehicle 1 is performed based on obstacle detection by on-board sensors such as the on-board camera 11, LiDAR 12, and ultrasonic sensor 13. The determination of whether or not vehicle 1 is in autonomous driving mode is performed based on information regarding the autonomous driving status obtained from in-vehicle equipment such as the ECU 16 that implements ADAS, etc.

[0032] Furthermore, in this embodiment, the display control unit 102 controls the image display of the information display device 2 so that the rear view of the vehicle is displayed in the front of the vehicle interior, within the user's field of view, in either the left or right side of the vehicle 1. The left and right sides of the vehicle 1 are the left and right sides of the vehicle centerline that extends in the front-rear direction of the vehicle 1. The display control unit 102 controls the image display of the information display device 2 so that the rear view of the vehicle, captured from one side, is displayed within one of the sides of the area. For example, the display control unit 102 controls the image display of the information display device 2 so that the rear view of the vehicle, captured from the left rear of the vehicle 1, is displayed within the left side of the area.

[0033] Furthermore, in this embodiment, the display control unit 102 may control the image display of the information display device 2 so that the image included in the central region of the rear view of the vehicle is superimposed on the upper front of the vehicle 1 in the user's field of view. The upper front of the vehicle 1 is, for example, the upper part of the front windshield, the so-called rearview mirror position in conventional vehicles.

[0034] Furthermore, in this embodiment, the display control unit 102 may control the image display of the information display device 2 so that when the vehicle 1 is moving backward or when there are obstacles around the vehicle 1, an overhead view image including the area around the vehicle 1 is superimposed on the upper front of the vehicle 1 in the user's field of view. The overhead view image is an AVM image. The determination of whether or not the vehicle 1 is moving backward is performed based on the speed of the vehicle 1 and the shift position of the vehicle 1. The determination of whether or not there are obstacles around the vehicle 1 is performed based on obstacle detection by on-board sensors such as the on-board camera 11, LiDAR 12, and ultrasonic sensor 13. The upper front of the vehicle 1 is, for example, the upper part of the front windshield, the so-called rearview mirror position in conventional vehicles.

[0035] The vehicle status acquisition unit 103 executes a status acquisition process to acquire the status of vehicle 1. In the status acquisition process, the vehicle status acquisition unit 103 acquires the status of vehicle 1 from the vehicle ECU 16. The status of vehicle 1 includes the vehicle speed of vehicle 1, the shift position of vehicle 1, the function status of vehicle 1, the energy level of vehicle 1, warning signals, and location information of vehicle 1.

[0036] Here, an example of the control procedure for the display control method executed by the display control device 10 according to this embodiment will be explained using Figure 3. Figure 3 is a flowchart showing the control procedure for the display control method executed by the display control device according to this embodiment. The controller 100 starts the control flow from step S1 at regular intervals. In step S1, the controller 100 acquires a rear-view image of the vehicle from the onboard camera 11. In step S2, the controller 100 controls the image display of the information display device 2 so that the rear-view image of the vehicle is superimposed on the window portion of the vehicle 1 that is included in the user's field of view. When the controller 100 completes the control process in step S2, it terminates the control flow.

[0037] In this embodiment, the flowchart shown in Figure 3 does not necessarily have to include all of steps S1 to S2; some steps may be omitted, or additional steps may be added as appropriate. For example, it may include a step of determining the display position for displaying the rear vehicle image on the display 22 of the information presentation device 2, or a step of controlling the image display so that the rear vehicle image is hidden when predetermined conditions are met. The processing order of each step in the control flow may also be changed as appropriate.

[0038] Next, another example of a display control method executed by the display control device 10 according to this embodiment will be described using Figure 4. Figure 4 is a flowchart showing the control procedure of another example of a display control method executed by the display control device according to this embodiment. When the controller 100 receives a rear view image of the vehicle from the onboard camera 11, it starts the control flow from step S11. In step S11, the controller 100 determines whether the vehicle speed of the vehicle 1 is 0 km / h or not. If it determines that the vehicle speed of the vehicle 1 is 0 km / h, the controller 100 proceeds to step S12. If it determines that the vehicle speed of the vehicle 1 is not 0 km / h, the controller 100 proceeds to step S14. In step S12, the controller 100 determines whether there are any obstacles around the vehicle 1. If it determines that there are no obstacles around the vehicle 1, the controller 100 proceeds to step S13. If it determines that there are obstacles around the vehicle 1, the controller 100 proceeds to step S14. In step S13, the controller 100 controls the image display of the information display device 2 so that the rear view of the vehicle is hidden. In step S14, the controller 100 controls the image display of the information display device 2 so that the rear view of the vehicle is displayed. Once the controller 100 has completed the processing in step S13 or step S14, it terminates the control flow.

[0039] Next, another example of a display control method executed by the display control device 10 according to this embodiment will be described using Figure 5. Figure 5 is a flowchart showing the control procedure of another example of a display control method executed by the display control device according to this embodiment. The controller 100 starts the control flow from step S21 when the shift position of the vehicle 1 is in the reverse range. In step S21, the controller 100 determines whether or not there are obstacles around the vehicle 1. If it is determined that there are obstacles around the vehicle 1, the controller 100 proceeds to step S22. If it is determined that there are no obstacles around the vehicle 1, the controller 100 proceeds to step S23. In step S22, the controller 100 controls the image display of the information display device 2 so that the around-view monitor (AVM) image is superimposed on the user's field of view. In step S23, the controller 100 controls the image display of the information display device 2 so that the rear view image of the vehicle is superimposed on the user's field of view. When the controller 100 completes the processing in step S22 or step S23, it terminates the control flow.

[0040] Next, another example of a display control method executed by the display control device 10 according to this embodiment will be described using Figure 6. Figure 6 is a flowchart showing the control procedure of another example of a display control method executed by the display control device according to this embodiment. The controller 100 starts the control flow from step S31 when the shift position of the vehicle 1 is in the drive range. In step S31, the controller 100 determines whether the vehicle speed of the vehicle 1 is equal to or greater than a predetermined vehicle speed. If it is determined that the vehicle speed of the vehicle 1 is equal to or greater than a predetermined vehicle speed, the controller 100 proceeds to step S32. If it is determined that the vehicle speed of the vehicle 1 is not equal to or greater than a predetermined vehicle speed, the controller 100 proceeds to step S33. In step S32, the controller 100 controls the image display of the information display device 2 so that a reduced-size image of the rear of the vehicle is superimposed on the user's field of view. In step S33, the controller 100 controls the image display of the information display device 2 so that an enlarged image of the rear of the vehicle is superimposed on the user's field of view. When the controller 100 completes the processing in step S32 or step S33, it terminates the control flow.

[0041] Next, another example of a display control method executed by the display control device 10 according to this embodiment will be described using Figure 7. Figure 7 is a flowchart showing the control procedure of another example of a display control method executed by the display control device according to this embodiment. The controller 100 starts the control flow from step S41 when the shift position of the vehicle 1 is in the reverse range. In step S41, the controller 100 determines whether or not there is a parking space around the vehicle 1. If it is determined that there is a parking space around the vehicle 1, the controller 100 proceeds to step S42. If it is determined that there is no parking space around the vehicle 1, the controller 100 proceeds to step S44. In step S42, the controller 100 transmits a control command to the onboard camera 11 to adjust its angle and take an image. Specifically, the controller 100 adjusts the angle of the onboard camera 11 so that the imaging direction of the onboard camera 11 faces downwards towards the vehicle 1. In step S43, the controller 100 controls the image display of the information display device 2 so that the vehicle surroundings image captured by the angle-adjusted in-vehicle camera 11 is superimposed on the user's field of view. The vehicle surroundings image superimposed on the user's field of view may include not only the rear image of the vehicle, but also a left-side image captured of the left side of the vehicle 1 and a right-side image captured of the right side of the vehicle 1. In step S44, the controller 100 controls the image display of the information display device 2 so that the vehicle rear image captured by the in-vehicle camera 11 without angle adjustment is superimposed on the user's field of view. When the controller 100 completes the processing in step S43 or step S44, it terminates the control flow.

[0042] As described above, in the display control method, display control device, and display control system according to this embodiment, the controller is a controller that controls the image display of an information display device worn by the user. The controller acquires a rear-view image of the vehicle captured from an onboard camera in the vehicle in which the user is riding, and controls the image display of the information display device so that the rear-view image is superimposed on the vehicle window portion included in the user's field of view or the actual image corresponding to the field of view. This makes it possible to display the rear-view image of the vehicle without impairing the appearance of the vehicle's interior.

[0043] Furthermore, in the display control method, display control apparatus, and display control system according to the present embodiment, the controller acquires a state of the vehicle, and controls switching between displaying and not displaying a vehicle rear image in the information presentation device based on the acquired state of the vehicle. This enables switching between displaying and not displaying an image captured of the rear of the vehicle in accordance with the state of the vehicle.

[0044] Furthermore, in the display control method, display control apparatus, and display control system according to the present embodiment, the controller controls image display on the information presentation device such that a vehicle rear image is displayed when the vehicle is traveling forward or backward. This enables an image captured of the rear of the vehicle to be displayed while the vehicle is moving.

[0045] Furthermore, in the display control method, display control apparatus, and display control system according to the present embodiment, the controller controls image display on the information presentation device such that the vehicle rear image is not displayed when the vehicle is stopped, when there are no obstacles around the vehicle, or when the vehicle is in autonomous driving. This enables an image captured of the rear of the vehicle to be displayed when the vehicle is stopped, when there are no obstacles around the vehicle, or when the vehicle is in autonomous driving.

[0046] Furthermore, in the display control method, display control apparatus, and display control system according to the present embodiment, the controller controls image display on the information presentation device such that the vehicle rear image is displayed within either the left region or the right region in front of the vehicle inside the vehicle cabin in a field of view or a real image. This enables an image captured of the rear of the vehicle to be displayed so as to be superimposed on the left or right side inside the vehicle cabin.

[0047] Furthermore, in the display control method, display control apparatus, and display control system according to the present embodiment, the controller controls image display on the information presentation device such that a vehicle rear image, which is an image captured of the rear of the vehicle on one of the left and right sides, is displayed within the region on the one side. This enables an image captured of the left rear or right rear of the vehicle to be displayed so as to be superimposed on the left side or the right side inside the vehicle cabin.

[0048] Furthermore, in the display control method, display control apparatus, and display control system according to the present embodiment, the window portion is a door glass provided in a door portion of a vehicle. Accordingly, an image obtained by capturing an image of the rear of the vehicle can be displayed so as to be superimposed on the door glass of the vehicle.

[0049] Furthermore, in the display control method, display control apparatus, and display control system according to the present embodiment, the information presentation apparatus is AR glasses or smart glasses. Accordingly, an image obtained by capturing an image of the rear of the vehicle can be displayed on a display disposed in front of the user's eyes in a state where the information presentation apparatus is worn by the user.

[0050] Furthermore, in the display control method, display control apparatus, and display control system according to the present embodiment, the controller controls image display by the information presentation apparatus such that an image included in a central region of the vehicle rear image is superimposed on an upper front portion of the vehicle in a field of view or a real image. Accordingly, the central region of the image obtained by capturing the rear of the vehicle can be displayed at the upper front portion of the vehicle.

[0051] Furthermore, in the display control method, display control apparatus, and display control system according to the present embodiment, when the vehicle is traveling rearward or when there is an obstacle around the vehicle, the controller controls image display by the information presentation apparatus such that a bird's-eye view image including the surroundings of the vehicle is superimposed on an upper front portion of the vehicle in a field of view or a real image. Accordingly, when the vehicle is traveling rearward or when an obstacle around the vehicle is detected, the user can check the surroundings of the vehicle.

[0052] Furthermore, in the display control method, display control apparatus, and display control system according to the present embodiment, the controller acquires the vehicle speed of the vehicle, generates a control command for causing an in-vehicle camera to capture an image of the rear of the vehicle such that the vehicle rear image is displayed in an enlarged or reduced size in accordance with the vehicle speed, and transmits the control command to the in-vehicle camera. Accordingly, the image obtained by capturing the rear of the vehicle can be displayed in an enlarged or reduced size.

[0053] Furthermore, in the display control method, display control device, and display control system according to this embodiment, when the vehicle is moving backward while a parking space located around the vehicle is recognized, the controller transmits a control command to the on-board camera to adjust the angle of the on-board camera so that the imaging direction of the on-board camera is directed downwards towards the vehicle. This makes it possible to display an image of the underside of the vehicle.

[0054] Furthermore, in the display control method, display control device, and display control system according to this embodiment, the vehicle is one that does not have door mirrors on the doors. This allows the user to check the rear of the vehicle even in a vehicle that does not have door mirrors.

[0055] In this embodiment of the display control system according to the present invention, the controller 100 may be provided in the information display device 2 instead of the vehicle 1. The functions of the controller 100 are executed in the information display device 2. In this case as well, the description of the configuration and control processing when the controller 100 is provided in the vehicle 1 can be appropriately applied.

[0056] Specifically, the display control unit 102 uses the camera 21 mounted on the information display device 2 to recognize the interior space of the vehicle 1 included in the user's field of view or the actual image corresponding to the user's field of view. The display control unit 102 determines the display position in the interior space where the rear image of the vehicle will be displayed. For example, the display position is the position of the window of the vehicle 1. The display control unit 102 receives an image of the area around the vehicle, including the rear image, from the vehicle 1 via communication with the vehicle 1 and displays the rear image at the display position on the display 22. The display control unit 102 also acquires the state of the vehicle 1 via communication with the vehicle 1 and controls the switching between displaying and hiding the rear image on the display 22 based on the acquired state of the vehicle 1.

[0057] As described above, in the display control method, display control device, and display control system according to this embodiment, the controller is provided in the information display device and uses a camera mounted on the information display device to recognize the interior space of the vehicle included in the field of view or actual image, and determines the display position in the interior space where the rear image of the vehicle is displayed. This makes it possible to determine the position in the information display device where the image captured of the rear of the vehicle is displayed.

[0058] Furthermore, in the display control method, display control device, and display control system according to this embodiment, the controller is provided in the information display device and acquires the status of the vehicle via communication with the vehicle, and controls the switching between displaying and hiding the rear image of the vehicle in the information display device based on the acquired vehicle status. As a result, the information display device can switch between displaying and hiding the image of the rear of the vehicle according to the status of the vehicle.

[0059] The embodiments described above are provided to facilitate understanding of the present invention and are not intended to limit it. Therefore, each element disclosed in the above embodiments is intended to include all design modifications and equivalents that fall within the technical scope of the present invention.

[0060] 1000...Display control system 1...Vehicle 10...Display control device 100...Controller 101...Image acquisition unit 102...Display control unit 103...Vehicle status acquisition unit 11...In-vehicle camera 111...Front camera 112...Rear camera 113...Right side camera 114...Left side camera 12...LiDAR 13...Ultrasonic sensor 2...Information display device 21...Camera 22...Display 23...Speaker 3...External server

Claims

1. A display control method performed by a controller that controls the image display of an information display device worn by a user, wherein the controller acquires a rear-view image of the vehicle captured from an onboard camera of the vehicle in which the user is riding, and controls the image display of the information display device such that the rear-view image of the vehicle is superimposed on the window portion of the vehicle included in the user's field of view or a real image corresponding to the field of view.

2. A display control method according to claim 1, wherein the controller acquires the status of the vehicle and controls the switching of the display and hiding of the rear image of the vehicle in the information display device based on the acquired status of the vehicle.

3. A display control method according to claim 2, wherein the controller controls the image display of the information display device so that an image of the rear of the vehicle is displayed when the vehicle is moving forward or backward.

4. A display control method according to claim 2 or 3, wherein the controller controls the image display of the information display device such that the rear view image of the vehicle is not displayed when the vehicle is stopped, when there are no obstacles around the vehicle, or when the vehicle is in automatic driving mode.

5. A display control method according to any one of claims 1 to 4, wherein the controller controls the image display of the information display device so that the rear view of the vehicle is displayed in the field of view or the actual image in the area in front of the vehicle interior, on either the left side or the right side of the vehicle.

6. A display control method according to claim 5, wherein the controller controls the image display of the information display device so that the rear image of the vehicle, which is an image of the rear of the vehicle on one side, is displayed within the area on one side.

7. A display control method according to any one of claims 1 to 6, wherein the window portion is a door glass provided in the door portion of the vehicle.

8. A display control method according to any one of claims 1 to 7, wherein the information presentation device is AR glasses or smart glasses.

9. A display control method according to any one of claims 1 to 8, wherein the controller controls the image display of the information display device such that an image included in the central region of the rear view of the vehicle is superimposed on the front upper part of the vehicle in the field of view or the actual image.

10. A display control method according to claim 9, wherein the controller controls the image display of the information display device such that when the vehicle is moving backward or when there are obstacles around the vehicle, an overhead image including the area around the vehicle is superimposed on the front upper part of the vehicle in the field of view or the actual image.

11. A display control method according to any one of claims 1 to 10, wherein the controller acquires the vehicle speed of the vehicle, generates a control command to image the area behind the vehicle so that the rear image of the vehicle is enlarged or reduced according to the vehicle speed, and transmits the control command to the on-board camera.

12. A display control method according to any one of claims 1 to 11, wherein the controller transmits a control command to the on-board camera to adjust the angle of the on-board camera so that the imaging direction of the on-board camera faces downwards towards the vehicle when the vehicle is moving backward while a parking space located around the vehicle is recognized.

13. A display control method according to any one of claims 1 to 12, wherein the controller is provided in the information display device, and the controller uses a camera mounted on the information display device to recognize the interior space of the vehicle included in the field of view or the actual image, and determines a display position in the interior space where the rear image of the vehicle is displayed.

14. A display control method according to any one of claims 1 to 13, wherein the controller is provided in the information display device, acquires the status of the vehicle via communication with the vehicle, and controls the switching of the display and hiding of the rear image of the vehicle in the information display device based on the acquired status of the vehicle.

15. A display control method according to any one of claims 1 to 14, wherein the vehicle is a vehicle in which door mirrors are not provided on the door portion.

16. A display control device comprising a controller for controlling the image display of an information display device worn by a user, wherein the controller acquires a rear-view image of the vehicle captured from an onboard camera of the vehicle in which the user is riding, and controls the image display of the information display device such that the rear-view image of the vehicle is superimposed on the window portion of the vehicle included in the user's field of view or a real image corresponding to the field of view.

17. A display control system comprising: a vehicle in which a user rides; an information display device worn by the user; and a controller for controlling the image display of the information display device, wherein the controller acquires a rear-view image of the vehicle captured from an on-board camera of the vehicle, and controls the image display of the information display device so that the rear-view image is superimposed on the window portion of the vehicle included in the user's field of view or a real image corresponding to the field of view.

Images