Display control method, display control device, and display control system
Patent Information
- Application Number
- PCT/JP2025/011850
- Authority / Receiving Office
- WO · WO
- Patent Type
- Applications
- Current Assignee / Owner
- Filing Date
- 2025-03-25
- Publication Date
- 2026-10-01
Smart Images
Figure JP2025011850_01102026_PF_FP_ABST
Abstract
Description
Display control method, display control device, and display control system
[0001] The present invention relates to a display control method, a display control device, and a display control system.
[0002] There is known a control device that causes content related to vehicle driving to be displayed on a lens of a glasses-type device in a state where a wearer wearing the glasses-type device is positioned in the driver's seat of the vehicle (Patent Document 1).
[0003] Japanese Unexamined Patent Application Publication No. 2018-55075
[0004] However, with the control device described in Patent Document 1, if an image of a content list remains displayed on the lens after acceptance of selection of content, there is a problem that the occupant's field of view is obstructed.
[0005] The problem to be solved by the present invention is to provide a display control method, a display control device, and a display control system that prevent the field of view from being obstructed by a content image after receiving an operation input to content.
[0006] The present invention controls image display of an information presentation device such that a content image is superimposed on a specific display surface included in a user's field of view or a real image corresponding to the field of view; accepts an operation input to content included in the content image in a state where the content image is displayed on the specific display surface in a direct-facing display mode; and when the operation input is accepted, changes the display mode from displaying the content image in the direct-facing display mode to displaying the content image in a horizontal display mode, thereby solving the above problem.
[0007] According to the present invention, it is possible to prevent the field of view from being obstructed by the content image after receiving an operation input to the content.
[0008] Figure 1 is a block diagram of the display control system according to this embodiment. Figure 2 is a schematic diagram of the interior when a user is seated in the driver's seat (front right seat) and looking straight ahead. Figure 3 is a diagram showing an example of an augmented reality space presented to the user via the information display device of this embodiment. Figure 4 is a diagram showing an example of an augmented reality space presented to the user via the information display device of this embodiment. Figure 5 is a diagram showing an example of an augmented reality space presented to the user via the information display device of this embodiment. Figure 6 is a diagram showing an example of an augmented reality space presented to the user via the information display device of this embodiment. Figure 7 is a diagram showing an example of an augmented reality space presented to the user via the information display device of this embodiment. Figure 8A is a diagram showing an example of an augmented reality space presented to the user via the information display device of this embodiment. Figure 8B is a plan view of the content image displayed in horizontal display mode (top). Figure 9 is a flowchart diagram showing another example of a control procedure of the display control method executed by the display control device according to this embodiment.
[0009] Embodiments of the present invention will be described below with reference to the drawings. Figure 1 is a block diagram showing one embodiment of a display control system equipped with a display control device according to this embodiment. As shown in Figure 1, the display control system 1000 comprises a vehicle 1, an information display device 2, and an external server 3. The display control system 1000 is a system for presenting images to a user to grasp the situation outside and inside the vehicle 1 by controlling the image display of the information display device 2. The user is an occupant of the vehicle 1, such as the driver. The vehicle 1 comprises a display control device 10, an on-board camera 11, a LiDAR 12, an ultrasonic sensor 13, an IVI 14, an IVC 15, and a vehicle ECU 16. Each device provided in the vehicle 1 is connected by an in-vehicle communication network such as CAN or LIN to exchange information with each other. The vehicle 1, the information display device 2, and the external server 3 exchange information with each other via a wireless communication network. The wireless communication network includes, for example, mobile communication networks such as 4G lines, the Internet, Wi-Fi (registered trademark), and the like.
[0010] The display control device 10 is a device that controls the image display of the information display device 2. The information display device 2 is a device worn by the user and presents an augmented reality (AR) space to the user wearing the information display device 2. The augmented reality space is a space constructed based on real space information and virtual space information. Specifically, the augmented reality space is a space in which the real space is visually extended by superimposing virtual objects onto images of the real space. In this embodiment, the display control device 10 controls the image display of the information display device 2 so as to present the user with an augmented reality space in which content images are displayed in the real space including the outside and inside of the vehicle 1, via the information display device 2 worn by the user. In the following description, the display control device 10 is described as a device provided in the vehicle 1, but it is not limited to this, and the display control device 10 may be located outside the vehicle 1 or provided in the information display device 2. As an example of the display control device 10 being located outside the vehicle 1, the display control device 10 may be provided in the external server 3.
[0011] The on-board camera 11 is installed in the vehicle 1 and captures images of the area around the vehicle 1. The on-board camera 11 includes a front camera 111, a rear camera 112, a right-side camera 113, and a left-side camera 114. The front camera 111 captures images of the front of the vehicle 1, the rear camera 112 captures images of the rear of the vehicle 1, the right-side camera 113 captures images of the right side of the vehicle 1, and the left-side camera 114 captures images of the left side of the vehicle 1. The front camera 111 is installed in the front part of the vehicle 1, for example, in the grille of the vehicle 1. The rear camera 112 is installed in the rear part of the vehicle 1, for example, in the tailgate. The right-side camera 113 is installed in the right-side part of the vehicle 1, for example, in the right-side door of the front part of the vehicle 1. The left-side camera 114 is installed in the left-side door of the front part of the vehicle 1. The various cameras may be installed in other locations on the vehicle 1. These on-board cameras 11 may also be cameras used in an on-board camera monitoring system (CMS). In the following description, the front camera 111, rear camera 112, right-side camera 113, and left-side camera 114 will be collectively referred to as the on-board camera 11. The on-board camera 11 captures images of the area around the vehicle 1 at regular intervals and outputs the vehicle surrounding image to the display control device 10. The on-board camera 11 may also output the vehicle surrounding image to the vehicle ECU 16.
[0012] The LiDAR 12 measures the distance and direction from the vehicle 1 to an object, as well as the distance between objects, by irradiating it with laser light and measuring the reflected light from the object. The ultrasonic sensor 13 measures the distance and direction from the vehicle 1 to an object, as well as the distance between objects, by emitting ultrasonic waves and measuring the reflected waves from the object. The measurement information measured by the LiDAR 12 and the ultrasonic sensor 13 is output to the display control device 10. The IVI 14 is a system that integrates a navigation system that provides map information and route guidance, and an entertainment system that provides entertainment elements such as music and video. The IVI 14 controls in-vehicle equipment such as displays and speakers to provide information to the user. The IVI 14 outputs various information received from the external server 3 via the IVC 15 to the display control device 10. The IVC 15 is a communication terminal that connects the vehicle 1 to the outside via a wireless communication network. The IVC 15 can communicate with the external server 3. IVC15 receives various information from external server 3. The information received from external server 3 includes, for example, entertainment content data such as music and videos, and software update data.
[0013] The vehicle ECU 16 is an electronic control unit mounted on the vehicle 1 that collects vehicle information of the vehicle 1. The vehicle information includes information that includes the state of the vehicle 1, such as the vehicle speed of the vehicle 1, the shift position of the vehicle 1, the energy level of the vehicle 1, and the current position of the vehicle 1. The vehicle information may also include images of the vehicle's surroundings acquired by the onboard camera 11. The vehicle ECU 16 acquires vehicle information from other devices such as other ECUs in the vehicle 1 and outputs it to the display control device 10.
[0014] The information display device 2 presents the user with an augmented reality space in which content images are displayed in the real space, including the exterior and interior of vehicle 1. The content images displayed on the information display device 2 include map images, setting screen images for configuring in-vehicle devices such as audio and air conditioners, and operation screen images for operating entertainment content such as videos and music. The information display device may also present the user with an augmented reality space in which images of the vehicle's surroundings are displayed in the real space, including the exterior and interior of vehicle 1. The vehicle's surroundings images include rearward images of vehicle 1. The information display device 2 may also present the user with an augmented reality space in which content images and vehicle's surroundings images are displayed in the real space. The information display device 2 comprises a camera 21, a display 22, and a speaker 23. The camera 21 captures the surrounding environment in the user's field of view and acquires real images corresponding to the user's field of view. When the user is riding in vehicle 1, the surrounding environment in the user's field of view includes the environment outside and inside vehicle 1. The display 22 is a display that shows display images including content images, and is, for example, a transparent display that realizes an augmented reality space using an optical see-through method. The display 22 displays the display images based on control signals related to display control output by the display control device 10. The speaker 23 is a device that outputs audio information to the user. The information presentation device 2 also includes a CPU, which is a processor, ROM, and RAM. The ROM stores a program for presenting information to the user. This program is executed by the CPU and controls each device of the information presentation device 2 to realize the functions of the information presentation device 2.
[0015] In this embodiment, the information display device 2 is a wearable terminal worn by the user. The information display device 2 is, for example, AR glasses, smart glasses, or a head-mounted display, and the display 22 is positioned in front of the user's eyes when the user is wearing the information display device 2. In the following description, the information display device 2 will be described as AR glasses. That is, the display 22 corresponds to the lens portion of the AR glasses. The camera 21 is located on the frame portion of the AR glasses, outside the left and right lens portions. The speaker 23 is a speaker positioned near the user's left and right ears when the user is wearing the information display device 2.
[0016] If the display 22 is a transparent display, the image behind the display is visible, allowing the user to directly view the image, that is, the image generated by the refraction and reflection of light rays. The information presentation device 2 displays the content image on the display 22, so as to superimpose the content image onto the image that is directly projected into the user's field of view via the display 22. This presents the user with an augmented reality space that appears as if a display screen containing the content image exists in the surrounding environment within the user's field of view. For example, if a map image is displayed on the display 22 as the content image, the user can view the map screen through the display 22 of the information presentation device 2 while viewing the interior and / or exterior of the vehicle (the interior equipment and scenery in the exterior space that the user is aware of).
[0017] In the following description, the display 22 is described as a transparent display, but the display 22 may also be an opaque display that realizes an augmented reality space using a video see-through method. If the display 22 is an opaque display, the user's field of view is obstructed by the display 22 placed in front of the user's eyes, so the user cannot directly see the image behind the display 22, and instead sees the actual image displayed on the display 22, which is an image of the user's field of view that has been captured, that is, an image reproduced by electrical conversion. The information presentation device 2 displays a composite image on the display 22, which is created by combining the actual image (an image corresponding to the real space) corresponding to the user's field of view captured by the camera 21 and the content image so that the content image is superimposed on the actual image.
[0018] External server 3 transmits various information to display control device 10 via IVC 15. External server 3 is, for example, a cloud server. The various information is information provided to the user and includes, for example, entertainment content data. External server 3 may also transmit software update data, etc.
[0019] The display control device 10 includes at least a controller 100. The controller 100 controls the image display of the information presentation device 2. The controller 100 includes a computer having hardware and software, which includes a ROM that stores programs, a CPU that executes the programs stored in the ROM, and RAM that functions as an accessible storage device.
[0020] The controller 100 comprises an information acquisition unit 101, a display control unit 102, and a vehicle status acquisition unit 103 as functional blocks, and each function is executed through the cooperation of software and hardware to realize each function or execute each process. In this embodiment, the functions of the controller 100 are divided into three blocks and the functions of each functional block are explained, but the functions of the controller 100 do not necessarily have to be divided into three blocks, and may be divided into two or fewer functional blocks, or four or more functional blocks. Also, in Figure 1, the vehicle 1 is shown to be equipped with one controller 100, but this is not limited to this, and the controller 100 may be divided into two. In this case, for example, it may be divided into a controller equipped with an information acquisition unit 101 and a controller equipped with a display control unit 102 and a vehicle status acquisition unit 103.
[0021] The information acquisition unit 101 performs an information acquisition process to acquire information necessary for generating content images. The information acquisition process is performed at regular intervals. For example, when displaying a map image as content image, the information acquisition unit 101 acquires map information and location information indicating the vehicle's current position from the IVI 14. The map information includes road information and POI information. When displaying a setting screen image of an in-vehicle device, the information acquisition unit 101 acquires vehicle information indicating the current status of the in-vehicle device from the vehicle ECU 16. For example, when displaying a setting screen image of an air conditioner, the information acquisition unit 101 acquires the current set temperature of the air conditioner, the airflow of the air conditioner, the current temperature inside the vehicle, etc.
[0022] The display control unit 102 executes display control processing to control the image display of the information presentation device 2. During the display control processing, the content image is displayed on the display 22. That is, the display control unit 102 controls the image display of the information presentation device 2 so that the content image is superimposed on a specific display surface included in the user's field of view. The specific display surface represents the display surface on which the content image is displayed in the user's visual space, or in the user's visual space represented by a real image (a space on the image corresponding to real space). The specific display surface is set, for example, in a part of the interior of a vehicle or in a part of the passenger compartment. The specific display surface is set in advance according to the display mode of the content image. The display modes of the content image include a frontal display mode and a horizontal display mode. The frontal display mode is a mode in which the content image is displayed so as to face the user directly. The specific display surface for the frontal display mode is located, for example, in front of the front seats of the vehicle and between the passenger seat and the driver's seat. The specific display surface for the frontal display mode is also arranged so that the angle it makes with the vertical plane is within a predetermined angular range. As another example, the specific display surface in the facing display mode is positioned in front of the driver's seat of the vehicle and arranged so that the angle it makes with the vertical plane is within a predetermined angular range. The specific display surface in the facing display mode may be a surface aligned with the vertical plane, or an inclined surface that is inclined with respect to the vertical plane within a predetermined angular range. The predetermined angular range may be set to, for example, a range from -30 degrees to +30 degrees, preferably a range from -20 degrees to +20 degrees, and more preferably a range from -10 degrees to +10 degrees.
[0023] Horizontal display mode is a mode in which content images are displayed in front of the front seats of the vehicle, in a position that does not obstruct the forward view of the user sitting in the front seat. In horizontal display mode, content images are displayed along a horizontal plane or an inclined surface that is tilted relative to the horizontal plane. The specific display surface for horizontal display mode is located in front of the front seats of the vehicle and is inclined relative to the vertical plane. In addition, the way in which content images are viewed can be changed in horizontal display mode; for example, it is possible to display the content image in a top view from above, or to display the content image in two dimensions at the top of the user's field of view. In the following description, the horizontal display mode in which the content image is viewed in a top view will be referred to as "horizontal display mode (lower)". The horizontal display mode in which the content image is viewed at the top of the user's field of view will be referred to as "horizontal display mode (upper)".
[0024] For example, when set to horizontal display mode (lower), the specific display surface is positioned below the user's viewpoint when the user is seated in the front seat (with the user's face facing the front of the vehicle). When set to horizontal display mode (upper), the specific display surface is positioned above the user's viewpoint when the user is seated in the front seat (with the user's face facing the front of the vehicle). Note that the specific display surface in horizontal display mode (upper) and horizontal display mode (lower) is not necessarily limited to a horizontal plane, but may be an inclined surface tilted within a predetermined angle range relative to the horizontal plane. The predetermined angle range may be, for example, -30 degrees to +30 degrees, preferably -20 degrees to +20 degrees, and more preferably -10 degrees to +10 degrees. The "horizontal" in horizontal display mode is merely a notation to distinguish it from the straight-ahead display mode and does not limit the display surface of the content image to horizontal. Detailed display forms for each display mode will be described later with specific examples. The display control unit 102 outputs a control signal to the information display device 2 that includes at least a content image in order to perform the above-mentioned display control.
[0025] The forward-facing display mode, horizontal display mode (upper), and horizontal display mode (lower) can be selected according to system and / or user operation. The forward-facing display mode displays the content image facing the user directly, and the display of the content image may interfere with the user's driving depending on the size and position of the content image. Therefore, the display control unit 102 may allow image display in the forward-facing display mode when the vehicle is stopped or parked, and prohibit image display in the forward-facing display mode when the vehicle is in motion. Also, if each display mode can be selected by user input, the display control unit 102 may accept input to display an image in the forward-facing display mode when the vehicle is stopped or parked, but may not accept input to display an image in the forward-facing display mode when the vehicle is in motion. In this way, image display in the forward-facing display mode is prohibited while the vehicle is in motion.
[0026] If the display 22 of the information display device 2 is a transparent display, first, the display control unit 102 acquires video information included in the user's field of view from the information display device 2. The video included in the user's field of view is the captured image taken by the camera 21 of the information display device 2. The user's field of view includes the environment outside and inside the vehicle 1. From the acquired information, the display control unit 102 recognizes the interior of the vehicle included in the user's field of view and identifies the position of a specific display surface from the recognized interior. The interior of the vehicle consists of vehicle structures that make up the interior, such as the instrument panel, front windshield, pillars, and vehicle panels. The display control unit 102 sets the position of the specific display surface in relation to the recognized interior of the vehicle.
[0027] When displaying content images in front-facing display mode, the display control unit 102 is located in front of the front seats of the vehicle, and a specific display surface is set at the top of the instrument panel. When displaying content images in horizontal display mode (bottom), the display control unit 102 is located in front of the front seats of the vehicle, and a specific display surface is set in the space further forward of the instrument panel. Also, when displaying content images in horizontal display mode (top), the specific display surface is set at a position above the user's viewpoint when the user is seated in the front seat.
[0028] The display control unit 102 then performs display control to superimpose the content image onto the specific display surface. Specifically, if the display 22 of the information presentation device 2 is a transparent display, the display control unit 102 outputs a control signal to the information presentation device 2 to display the content image on the display 22 based on two-dimensional coordinates representing the position of the specific display surface on the display 22. Since the position of the specific display surface on the transparent display corresponds to the position of the specific display surface in the user's field of view, the content image appears to be displayed along the specific display surface in the user's field of view. In other words, the information presentation device 2 can provide the user with an augmented reality space as if a monitor for displaying information about content such as maps were installed. Furthermore, when the user's face moves left, right, up, or down while wearing the information presentation device 2, the external and internal environments visible from the display 22 change. On the other hand, the relative positional relationship between the specific display surface and the interior of the vehicle remains unchanged. Therefore, when the user views the information presentation device 2 through the display 22, a virtual space as if a display for displaying content images were installed at the position of the specific display surface can be presented to the user. The display control unit 102 performs display control based on user field of view information acquired at regular intervals. In other words, if the user's field of view changes in accordance with the user's head movements, the superimposed display of the content image is also updated in accordance with the change in the user's field of view.
[0029] If the display of the information presentation device 2 is a non-transparent display, the display control unit 102 may control the image display of the information presentation device 2 so that the content image is superimposed on the real image corresponding to the user's field of view. The display control unit 102 acquires a real image corresponding to the user's field of view from the camera 21 of the information presentation device 2. This real image corresponds to the images of the outside and inside of the vehicle 1 as seen by the user. Based on the two-dimensional coordinates representing the position of a specific display surface on the real image, the display control unit 102 generates a display image in which the content image is superimposed on a specific display surface on the real image. The display control unit 102 then outputs a control signal to the information presentation device 2 to display the generated display image on the display 22 of the information presentation device 2. Note that the generation of the display image is not limited to being performed by the display control device 10; the display control device 10 may transmit information on the display position on the real image and the content image, and the display image may be generated in the information presentation device 2.
[0030] The vehicle status acquisition unit 103 executes a status acquisition process to acquire the status of vehicle 1. In the status acquisition process, the vehicle status acquisition unit 103 acquires the status of vehicle 1 from the vehicle ECU 16. The status of vehicle 1 includes the vehicle speed of vehicle 1, the shift position of vehicle 1, the function status of vehicle 1, the energy level of vehicle 1, warning signals, and location information of vehicle 1.
[0031] Next, referring to Figure 2, we will explain the interior of the vehicle's front and the positions of the specific display surfaces in horizontal display mode (upper / lower). Figure 2 is a schematic diagram of the interior when a user is seated in the driver's seat (right front seat) and looking straight ahead. The x-axis represents the direction of travel when the vehicle is moving forward, the y-axis represents the vehicle width direction, and the z-axis represents the vehicle height direction.
[0032] A space 43 is provided between the instrument panel 41 and the front interior section 42. The front interior section 42 is the rear part of the front grille or front bumper. For example, in a front-wheel-drive engine vehicle, the engine is located in the space 43. For example, if the vehicle is an electric vehicle, an engine is not required, so as shown in Figure 2, the space where the engine would normally be can be left empty. In addition, even in engine vehicles and hybrid vehicles, the engine may be located in a part other than space 43, leaving space 43 empty as shown in Figure 2.
[0033] The position of the specific display surface in horizontal display mode (upper part) will be explained. The specific display surface is located at the top of the front windshield 44, or in the front part of the vehicle's ceiling 45 (the end on the side facing the direction of travel when the vehicle is moving forward). The specific display surface may also be at the boundary between the front windshield 44 and the vehicle's ceiling 45. The area P shown in Figure 2 is the position of the specific display surface in horizontal display mode (upper part). Note that the specific display surface does not need to be the entire area P, but may be a part of area P.
[0034] Next, the position of the specific display surface in horizontal display mode (lower part) will be explained. The specific display surface in horizontal display mode (lower part) is the surface along the bottom surface 43a of space 43, or the surface parallel to the bottom surface 43a. The surface parallel to the bottom surface 43a is a surface that is parallel to the bottom surface 43a and is located at a predetermined distance from the bottom surface 43a in the height direction (z-axis direction). When a user is sitting in the front seat and looking straight ahead, the bottom surface 43a of space 43 is located below the vehicle's steering wheel 46. Also, the bottom surface 43a of space 43 is located between the left and right front side members 47a and 47b of the vehicle, and is located below the front side members 47a and 47b. The region Q shown in Figure 2 is the position of the specific display surface in horizontal display mode (upper part). Note that the specific display surface does not have to be the entire region Q, but may be a part of region Q. Therefore, when a content image is displayed in horizontal display mode (lower part), the user will be looking down at the content image. When displaying a map as a content image, it can be either a two-dimensional map image or a three-dimensional image such as a bird's-eye view.
[0035] Furthermore, specific display surfaces in horizontal display mode (upper / lower) may be positioned outside the user's central field of view so as not to obstruct the forward view of a front-seat passenger with the content image display screen. Also, specific display surfaces in horizontal display mode (upper / lower) may be positioned outside the user's effective field of view. The user's central field of view is the range in which characters and object shapes can be clearly distinguished, and is a range of -1 to +1 degree or -3 to +3 degrees in both the horizontal and vertical directions, centered on the point of gaze. The effective field of view is the range in which colors can be distinguished, and is a range of -30 to +30 degrees or -35 to +35 degrees in both the horizontal and vertical directions, centered on the point of gaze. Note that the effective field of view also varies with vehicle speed, becoming narrower as vehicle speed increases. For example, if region U shown in Figure 2 represents the user's central field of vision, then the specific display surface in horizontal display mode (upper) is located above region U, and the specific display surface in horizontal display mode (lower) is located below region U. In other words, when displaying content images in horizontal display mode (upper / lower), the content image display screen is positioned so as not to obstruct the user's forward field of vision, thus minimizing the impact on the user's driving operations even while the vehicle is in motion.
[0036] Referring to Figure 3, the positions of the specific display surfaces in the horizontal display mode (upper / lower) and the specific display surfaces in the frontal display mode will be explained. In Figure 3, region P indicates the position of the horizontal display mode (upper), region Q indicates the position of the horizontal display mode (lower), and region R indicates the position of the frontal display mode. The x-axis direction is the direction of travel when the vehicle is moving forward, the y-axis direction is the vehicle width direction, and the z-axis direction is the vehicle height direction.
[0037] In horizontal display mode (upper), the specific display area (region P) is located above the front windshield 44, and the height of region P is higher than the position of the steering wheel 46. On the other hand, in horizontal display mode (lower), the height of the specific display area (region Q) is lower than the position of the steering wheel 46. Regions P and Q are located between the steering wheel 46 and the front grille (or front bumper) 48 in the direction of vehicle travel (x-axis direction). In front-facing display mode, the specific display area (region R) is in front of the steering wheel 46 and is located between the steering wheel 46 and the lower edge of the front windshield 44 in the direction of vehicle travel (x-axis direction). The height of region R is higher than the steering wheel 46 and lower than the ceiling of the vehicle or the upper edge of the front windshield 44. When the user is seated in the driver's seat (right front seat), the height of the specific display area (region R) in front-facing display mode corresponds to the height of the user's face. Furthermore, the specific display area (region R) in the front-facing display mode corresponds to the position of the driver's seat (right front seat) in the vehicle's width direction (y-axis direction), or is located between the driver's seat and the passenger seat. When a content image is displayed in the front-facing display mode, the content image display screen is included in the user's field of view.
[0038] Next, an example of an augmented reality space presented to the user via the information presentation device 2 will be described. Figures 4 to 7 and 8A are diagrams showing an example of an augmented reality space presented to the user via the information presentation device of this embodiment. Figures 4 to 7 and 8A are images viewed by the user via the display 22 of the information presentation device 2. Figures 4 and 5 are images when the content image is displayed in the frontal display mode. The frontal display mode is the display mode when the content image is displayed on a specific display surface (corresponding to area R shown in Figure 3). Figure 7 is an image when the content image is displayed in the horizontal display mode (lower part). The horizontal display mode (lower part) is the display mode when the content image is displayed on a specific display surface (corresponding to area Q shown in Figures 2 and 3). Figure 8A is an image when the content image is displayed in the horizontal display mode (upper part). The horizontal display mode (upper part) is the display mode when the content image is displayed on a specific display surface (corresponding to area P shown in Figures 2 and 3).
[0039] The user is seated in the driver's seat and wearing the information display device 2. The display 22 is a transparent display, and the user can directly see the actual environment outside and inside the vehicle 1, except for the content image. An example of the augmented reality space presented to the user will be explained with reference to Figures 4 to 7 and Figure 8A, where the content image will be a map image as an example.
[0040] For example, with the vehicle stopped or parked, the user operates the IVI 14 to search for a route on the map and performs an operation to display a content image (map image) in front-facing display mode. The IVI 14 receives the operation input from the user and outputs a control command based on the operation input to the display control device 10. The controller 100 executes display control processing based on the control command and displays the content image on the display 22 in front-facing display mode. When the user looks ahead of the vehicle through the display 22, they can see an image as shown in Figure 4. The content image includes a map image 51 that represents a map in a circular shape, and characters (N, E, S, W) indicating directions around the circular map screen. The content image also includes an icon S indicating the starting point and an icon G indicating the destination on the circular map image 51.
[0041] The controller 100 accepts operation input for the content contained in the content image when the content image is displayed in a frontal display mode. Operation input is performed by tapping a finger on a specific display surface that displays the content image. For example, if the user wants to change the destination, they may tap icon G on the map image 51 shown in Figure 4 and then slide icon G.
[0042] For example, in a state where a departure point (icon S) and a destination (icon G) are set, to set a waypoint (icon W), a user taps one point on the map image 51 shown in FIG. 4. The controller 100 transmits the position information of the tapped position to the IVI 14. The IVI 14 sets the tapped position as the waypoint, calculates a travel route from the departure point to the destination via the waypoint, and outputs the calculation result to the controller 100. The controller 100 displays the travel route on the map image 51 based on the calculation result from the IVI 14.
[0043] FIG. 5 shows an example of a content image obtained by adding a travel route to the content image shown in FIG. 4. A user taps the travel route on the map image 51 shown in FIG. 5 to confirm the travel route. Tapping on the map image 51 for confirming the travel route corresponds to an operation input to the content. As described above, the controller 100 accepts an operation input to the content included in the map image 51 in a state where the content image (map image) is displayed on the specific display surface in the front-facing display mode.
[0044] When an operation input to the content is accepted, the controller 100 changes the display mode of the content image from the front-facing display mode to the horizontal display mode (lower side). The controller 100 may also perform image presentation when changing the display mode.
[0045] An example of an image presentation performed by the controller 100 when the display mode of a content image is changed from a frontal display mode to a horizontal display mode will be explained with reference to Figure 6. Figure 6 is an example of an augmented reality space presented to the user. After receiving an operation input for the content, the controller 100 moves the map image 51 within the user's field of view so that the map image 51 moves forward of the vehicle while gradually tilting from a vertical plane to a horizontal plane. In other words, the controller 100 tilts the map image 51 toward the front of the vehicle so that the map image 51 moves from an upright state to a horizontal state within the user's field of view. As shown in Figure 6, the map image 51 is tilted in the space in front of the steering wheel 46. When the user looks forward of the vehicle through the display 22, they can see the tilted map image 51 and confirm that the display mode of the content image has changed from a frontal display mode to a horizontal display mode (bottom).
[0046] Then, when the image presentation of the map image 51 is finished, the display mode of the content image changes to horizontal display mode (bottom). Figure 7 shows the image when the content image is displayed in horizontal display mode (bottom). The content image includes a map image 51 that represents the map in a circular shape, characters indicating directions (N, E, S, W) around the circular map image 51, icons indicating the current location, waypoints, and destination respectively, and the driving route. When the user looks forward through the display 22, they can see an image in which the map image follows the bottom surface 43a of the space 43 (see Figure 7). In addition, when the user is sitting in the driver's seat (right front seat) and looking straight ahead, the content image is not displayed within a predetermined range that includes the user's central field of vision, or within the user's effective field of vision, thus preventing the display of the content image from interfering with the user's driving.
[0047] Furthermore, when the controller 100 displays the map image 51 in the horizontal display mode (lower portion), the direction of the map image may be aligned with the traveling direction of the vehicle such that the traveling direction of the vehicle on the map image faces the front of the vehicle. For example, suppose that the vehicle makes a right turn after traveling north and then travels east. In such a case, starting from the state where the map image 51 is displayed such that the direction character (N) around the map image 51 faces the front of the vehicle, the map image 51 is rotated 90 degrees counterclockwise in accordance with the right turn, so that the direction character (E) around the map image 51 faces the front of the vehicle. In other words, when the traveling direction of the vehicle changes due to the vehicle turning left or right or traveling on a curved road, the map image 51 rotates on the bottom surface 43a of the space 43. This allows the user to easily check the traveling direction of the vehicle on the map image 51.
[0048] As described above, when an operation input to content is received while the content image is displayed in the front-facing display mode, the display mode of the content image is changed from the display mode for displaying in the front-facing display mode to the display mode for displaying in the horizontal display mode (lower portion). This prevents the content image from obstructing the user's field of view when the vehicle starts traveling after the user has set the content.
[0049] When an operation input to content is received, the change of the display mode of the content image is not limited to changing from the front-facing display mode to the horizontal display mode (lower portion), and may also be changing from the front-facing display mode to the horizontal display mode (upper portion). FIG. 8A is an image when a content image is displayed in the horizontal display mode (upper portion). FIG. 8B is a plan view of the content image displayed in the horizontal display mode (upper portion).
[0050] While the content image is being displayed in the horizontal display mode (upper portion), when the user looks forward of the vehicle through the display 22, the user can check an image as shown in FIG. 8A. The content image includes a rectangular map image 52 representing a map, a vehicle icon V indicating the current position of the vehicle, an icon W indicating an intermediate waypoint, an icon G indicating the destination, a travel route from the current position of the vehicle to the destination, a remaining battery level, and a cruising range.
[0051] The map image 52 is rectangular in shape with the vehicle width direction (y-axis direction) as its longitudinal direction. The driving route extends along the y-axis direction. The length component along the vehicle width direction (y-axis direction) represents the distance from the vehicle's current position to the destination or waypoint. In addition, the background image of the map image 52 displays landmarks such as skyscrapers and radio towers, as well as tourist facilities. Therefore, by checking the map image 52, the user can recognize that there are POIs such as tourist facilities between the vehicle's current position and the destination, and can also understand the distance from the vehicle's current position to the POI. If the user taps a POI on the map image 52, the controller 100 may output detailed information about the POI via a pop-up display or audio output. The height component along the height direction (z-axis direction) represents the elevation. For example, in the example in Figure 8B, the user can confirm that there is a downhill slope with a large difference in elevation between the waypoint W and the destination G.
[0052] When the user looks ahead at the front of the vehicle via the display 22, they can see the map image 52 projected onto the top of the front windshield 44 (see Figure 8A). Furthermore, when the user is seated in the driver's seat (right front seat) and looking straight ahead, the content image is not displayed within a predetermined range that includes the user's central field of vision, or within the user's effective field of vision, thus preventing the display of the content image from interfering with the user's driving.
[0053] Next, an example of the control procedure for the display control method executed by the display control device 10 according to this embodiment will be described using Figure 9. Figure 9 is a flowchart showing the control procedure for the display control method executed by the display control device according to this embodiment. In the following description of the control flow, an example of the content image displayed on the display 22 will be a map image. For example, when a user operates the start button of the navigation system, the IVI 14 outputs a control command to the controller 100 for displaying a map image. The start button is provided, for example, on the steering wheel 46. When the controller 100 receives the control command, it starts the control flow from step S1.
[0054] In step S1, the information acquisition unit 101 of the controller 100 acquires information necessary for generating content images, and the vehicle status acquisition unit 103 acquires information indicating the vehicle's driving status, such as vehicle speed or shift position information.
[0055] In step S2, the controller 100 determines whether the vehicle is stopped or parked based on the vehicle speed or the position of the shift lever. If it determines that the vehicle is in motion (indicated as "No" in the control flow of step S2), the controller 100 terminates the control flow in step S3 without displaying the map image.
[0056] If the controller determines that the vehicle is stopped or parked (Yes in the control flow of step S2), in step S4 the controller 100 displays the map image in front-facing display mode. In step S5, with the map image displayed in front-facing display mode, the controller 100 determines whether or not it has received an operation input for the content included in the map image. Operation input for the content includes setting a destination, selecting a driving route, setting a driving route, etc. If no operation input has been received (No in the control flow of step S5), the controller 100 executes the control flow of step S2.
[0057] If an operation input is received ("Yes" in the control flow of step S5), the controller 100 determines whether to change the forward-facing display mode to the horizontal display mode (top) or to change the forward-facing display mode to the horizontal display mode (bottom). The display mode after the change is determined according to, for example, the user's selection, the vehicle's driving status, or the driving environment. The user's selection may be made by tapping on the content image. For example, if the user taps the top part of the content image while it is displayed in forward-facing display mode, the controller 100 changes the forward-facing display mode to the horizontal display mode (top). Also, if the user taps the bottom part of the content image, the controller 100 changes the forward-facing display mode to the horizontal display mode (bottom). Furthermore, the user's selection is not limited to tapping on the content image; for example, an input image for accepting the selection between the horizontal display mode (top) and the horizontal display mode (bottom), separate from the map image, may be displayed on a specific display screen in forward-facing display mode.
[0058] If the changed display mode is horizontal display mode (top), in step S7 the controller 100 displays the map image in horizontal display mode (top). If the changed display mode is horizontal display mode (bottom), in step S8 the controller 100 displays the map image in horizontal display mode (bottom). Then the controller 100 terminates the control flow.
[0059] As described above, in the display control method, display control device, or display control system according to this embodiment, the controller 100 controls the image display of the information presentation device 2 so that the content image is superimposed on a specific display surface in horizontal display mode (upper / lower) that is included in the user's field of view or the actual image corresponding to the field of view, and controls the image display of the information presentation device 2 so that the content image is superimposed on a specific display surface in frontal display mode that is included in the user's field of view or the actual image corresponding to the field of view, and while the content image is displayed on the specific display surface in frontal display mode, the controller 100 accepts operation input for the content contained in the content image, and when operation input is accepted, changes the display mode from displaying the content image in frontal display mode to displaying the content image in horizontal display mode (upper / lower). The specific display surface in horizontal display mode (upper / lower) is inclined with respect to the vertical plane. The specific display surface in frontal display mode has an angle with respect to the vertical plane within a predetermined angle range. This prevents the field of view from being obstructed by the content image.
[0060] Furthermore, in this embodiment, when the user is seated in the front seat, the specific display surface in horizontal display mode (lower part) may be located below the user's line of sight. The specific display surface in horizontal display mode (lower part) may also be located below the steering wheel 46. This allows the user to view the content image in a top view and grasp the entire content.
[0061] Furthermore, in this embodiment, when the map image included in the content image is displayed on a specific display surface in horizontal display mode (bottom), the direction of travel of the vehicle in the map image is facing forward. As a result, the direction of travel of the vehicle on the map, the actual direction of travel of the vehicle, and the orientation of the map are aligned, allowing the user to easily understand the map content and the direction of travel of the vehicle.
[0062] In this embodiment, when the user is seated in the front seat, the specific display surface in horizontal display mode (upper) may be positioned above the user's line of sight. The specific display surface in horizontal display mode (lower) may be positioned above the front windshield 44. This prevents the view from being obstructed by the content images.
[0063] As a variation of this embodiment, if the vehicle starts moving while the content image is displayed in front-facing display mode, the controller 100 may change the display mode of the content image from the front-facing display mode to the horizontal display mode (top or bottom). This prevents the view from being obstructed by the content image while the vehicle is moving.
[0064] As a variation of this embodiment, when displaying content images in a frontal view mode, the display position of the content images while the vehicle is in motion may be positioned between the lines of sight of multiple users sitting in multiple front seats. This allows the passenger in the front passenger seat to operate the content, while the passenger in the rear seat can view the content images.
[0065] As a variation of this embodiment, the controller 100 restricts the driver's input operations to the content while the vehicle is in motion, while allowing users other than the driver to input the content. This allows the driver to concentrate on driving while the vehicle is in motion, and enables the passenger to select and set the content. Furthermore, the controller may restrict the driver's input operations to the content while the vehicle is in motion, and allow users other than the driver to input the content, not only when the content image is displayed in a frontal display mode, but also when it is in a horizontal display mode.
[0066] In this embodiment, the display control processing by the controller 100 does not need to include processing to display the content image in all modes: horizontal display mode (top), horizontal display mode (bottom), and frontal display mode.
[0067] The controller 100 may perform display control processing to display the content image in a horizontal display mode (top) or horizontal display mode (bottom) instead of in a frontal display mode. For example, if the controller 100 can display the content image in a horizontal display mode (top) or horizontal display mode (bottom), it may start displaying the content image in a horizontal display mode (top) or horizontal display mode (bottom) after the navigation system has performed route calculation and destination setting, triggered by the start of driving or by user operation.
[0068] In the horizontal display mode (upper / lower), a specific display surface corresponds to the "first display surface" of the present invention, and the display form in the horizontal display mode (upper / lower) corresponds to the "first display form" of the present invention. Furthermore, in the frontal display mode, a specific display surface corresponds to the "second display surface" of the present invention, and the display form in the frontal display mode corresponds to the "second display form" of the present invention.
[0069] 1 Vehicle 2 Information display device 3 External server 10 Display control device 11 In-vehicle camera 21 Camera 22 Display 23 Speaker 100 Controller 101 Information acquisition unit 102 Display control unit 103 Vehicle status acquisition unit
Claims
1. A display control method performed by a controller that controls the image display of an information display device worn by a user, wherein the controller controls the image display of the information display device so that a content image is superimposed on a first display surface included in the user's field of view or a real image corresponding to the field of view; controls the image display of the information display device so that a content image is superimposed on a second display surface included in the user's field of view or a real image corresponding to the field of view; accepts an operation input for the content included in the content image while the content image is displayed on the second display surface; and, upon acceptance of the operation input, changes from a second display mode in which the content image is displayed on the second display surface to a first display mode in which the content image is displayed on the first display surface, wherein the first display surface is inclined with respect to a vertical plane, and the angle that the second display surface makes with the vertical plane is within a predetermined angular range.
2. A display control method according to claim 1, wherein when the vehicle starts moving while the content image is displayed in the second display mode, the controller changes the display mode of the content image from the second display mode to the first display mode.
3. A display control method according to claim 1 or 2, wherein the first display surface is located below the user's line of sight when the user is seated in the front seat.
4. A display control method according to any one of claims 1 to 3, wherein the first display surface is located below the steering wheel of the vehicle.
5. A display control method according to any one of claims 1 to 4, wherein when a map image included in the content image is displayed on the first display surface, the direction of travel of the vehicle in the map image is facing forward of the vehicle.
6. A display control method according to claim 1 or 2, wherein the first display surface is positioned above the user's line of sight when the user is seated in the front seat.
7. A display control method according to claim 1 or 2, wherein the first display surface is located on the upper part of the front windshield of a vehicle. Display control method.
8. A display control method according to claim 1 or 2, wherein when the content image is displayed in the second display mode, the display position of the content image while the vehicle is in motion is located between the lines of sight of multiple users sitting in multiple front seats.
9. A display control method according to claim 1 or 2, wherein the controller restricts input operations to the content by the driver while the vehicle is in motion, and permits input operations to the content by a user other than the driver.
10. A display control device comprising a controller for controlling the image display of an information display device worn by a user, wherein the controller controls the image display of the information display device so that a content image is superimposed on a first display surface included in the user's field of view or a real image corresponding to the field of view; controls the image display of the information display device so that a content image is superimposed on a second display surface included in the user's field of view or a real image corresponding to the field of view; accepts operation input for the content included in the content image while the content image is displayed on the second display surface; and, upon acceptance of the operation input, changes from a second display mode in which the content image is displayed on the second display surface to a first display mode in which the content image is displayed on the first display surface, wherein the first display surface is inclined with respect to a vertical plane, and the angle that the second display surface makes with the vertical plane is within a predetermined angular range.
11. A display control system comprising: an information display device worn by a user; and a controller for controlling the image display of the information display device, wherein the controller controls the image display of the information display device so that a content image is superimposed on a first display surface included in the user's field of view or a real image corresponding to the field of view; controls the image display of the information display device so that a content image is superimposed on a second display surface included in the user's field of view or a real image corresponding to the field of view; accepts operation input for the content included in the content image while the content image is displayed on the second display surface; and, upon acceptance of the operation input, changes from a second display mode in which the content image is displayed on the second display surface to a first display mode in which the content image is displayed on the first display surface, wherein the first display surface is inclined with respect to a vertical plane, and the angle that the second display surface makes with the vertical plane is within a predetermined angular range.