Head-mounted display terminal

Abstract

The present invention provides a technology for enabling more appropriate display if a head-mounted display terminal and an information display device are used. A head-mounted display terminal (1) includes a processor (101), a display (131), and a communication interface (160). The processor (101) communicates with an on-board image display device (10) via the communication interface (160), and controls image display of at least one of the display (131) and the on-board image display device (10) so that an image displayed on the display (131) and an image displayed by the on-board video display device (10) are not displayed in an overlapping manner.

Description

Head-mounted display terminal

[0001] The present invention relates to a head-mounted display terminal.

[0002] Patent Document 1 describes a technique for displaying information of an object on a display unit of a wearable terminal when it is determined that the object is present in a direction where it does not appear on a head-up display provided on a vehicle windshield as viewed by an occupant.

[0003] Patent Document 2 also describes a technique for determining content to be superimposed on a transmitted real image in a first display unit fixed to a moving body and a second display unit worn on a human body, and selecting whether to display the content on either the first display unit or the second display unit based on predetermined conditions.

[0004] Japanese Unexamined Patent Application Publication No. 2020-91663, Japanese Unexamined Patent Application Publication No. 2023-35095

[0005] However, since the head-up display and the wearable terminal use the same space for information display, there is a risk that the display areas of both overlap. In that case, the display images of the head-up display and the wearable terminal overlap within the driver's field of vision, and accurate information cannot be transmitted to the driver, which may lead to the driver's incorrect understanding.

[0006] In Patent Documents 1 and 2 described above, the overlap of display images between the head-up display and the wearable terminal within the driver's field of vision is not considered.

[0007] Therefore, an object of the present invention is to provide a technique that enables more appropriate display when using a head-mounted display terminal and an information display device.

[0008] In order to solve the above problems, one representative head-mounted display terminal of the present invention is a head-mounted display terminal including a processor, a display, and a communication interface. The processor communicates with an information display device via the communication interface and controls the video display of at least one of the display and the information display device so that the video displayed on the display and the video displayed by the information display device do not overlap and are not displayed.

[0009] According to the present invention, it is possible to provide a technology that enables more appropriate display when using a head-mounted display terminal and information display device.

[0010] Other issues, configurations, and effects not mentioned above will be clarified by the following description of the embodiments.

[0011] This is a block diagram showing an example of the configuration of a head-mounted display terminal. This is a diagram showing an example of the appearance of a head-mounted display terminal. This is a diagram showing an example of the appearance of a head-mounted display terminal. This is a diagram showing an example of video display by a head-mounted display terminal. This is a diagram showing an example of the configuration of an in-vehicle system including an in-vehicle video display device. This is a diagram showing an example of a functional block configuration related to a controller. This is a schematic diagram showing an example of the configuration of an in-vehicle video display device in Figure 4. This is a diagram showing an example of video display by an in-vehicle video display device. This is a diagram showing an example of video display by an in-vehicle video display device. This is a diagram showing an example of information display in coordination with an in-vehicle video display device on a head-mounted display terminal. This is a flowchart showing an example of coordinated display processing with an in-vehicle video display device performed on a head-mounted display terminal. This is a sequence diagram showing an example of communication between head-mounted display terminal 1 and a vehicle having an in-vehicle video display device. This is a diagram showing an example of display area recognition. This is a diagram showing an example of connection determination processing. This is a flowchart showing an example of processing in display area adjustment. This is a table showing an example of processing for relative coordinate confirmation of a head-mounted display terminal. This is a diagram showing an example of display area adjustment. This is a diagram showing an example of display area adjustment. This is a table showing an example of processing for relative coordinate adjustment of a head-mounted display terminal. This is a table in a different format showing specific examples of relative coordinate adjustment. This is a diagram for explaining the coordinate system of the in-vehicle video display device. This is a flowchart showing an example of processing in display adjustment. This is a flowchart showing an example of processing in display adjustment. This is a table showing an example of display adjustment processing for the head-mounted display terminal and in-vehicle video display device according to Example 1. This is a diagram showing an example of display adjustment. This is a diagram showing an example of display adjustment. This is a diagram showing an example of display adjustment processing for the head-mounted display terminal and in-vehicle video display device according to Example 2. This is a diagram showing an example of video priority. This is a diagram showing an example of video priority. This is a diagram showing an example of display content for the head-mounted display terminal and in-vehicle video display device according to Example 3. This is a diagram showing an example of processing related to the battery of the head-mounted display terminal according to Example 4. This is a diagram showing an example of processing for determining the display termination condition. This is a diagram showing another example of video display by the head-mounted display terminal. This is a diagram showing another example of video display by the head-mounted display terminal.

[0012] Embodiments of the present invention will be described below with reference to the drawings. The embodiments are illustrative examples for explaining the present invention, and have been omitted and simplified as appropriate for clarity of explanation. The present invention can also be implemented in various other forms. Unless otherwise specified, each component may be singular or plural. The position, size, shape, and range of each component shown in the drawings may not represent the actual position, size, shape, and range in order to facilitate understanding of the invention. For this reason, the present invention is not necessarily limited to the position, size, shape, and range disclosed in the drawings. Various types of information may be described using expressions such as "table," "list," and "queue," but various types of information may be represented by other data structures. For example, various types of information such as "XX table," "XX list," and "XX queue" may be referred to as "XX information." When describing identification information, expressions such as "identification information," "identifier," "name," "ID," and "number" are used, and these are interchangeable. When there are multiple components that have the same or similar functions, different subscripts may be attached to the same reference numeral in the description. Also, when it is not necessary to distinguish between these multiple components, the subscript may be omitted in the description. In embodiments, processing performed by executing a program may be described. Here, the computer executes the program using a processor (e.g., CPU, GPU) and performs processing defined by the program using memory resources (e.g., memory) and interface devices (e.g., communication ports). Therefore, the main entity performing the processing by executing the program may be the processor. The processor includes transistors and other circuits and is considered a circuit or processing circuit. Similarly, the main entity performing the processing by executing the program may be a controller, device, system, computer, or node having a processor. The main entity performing the processing by executing the program may be an arithmetic unit and may include dedicated circuits that perform specific processing.Here, a dedicated circuit is, for example, an FPGA (Field Programmable Gate Array), an ASIC (Application Specific Integrated Circuit), or a CPLD (Complex Programmable Logic Device). The program may be installed on the computer from the program source. The program source may be, for example, a program distribution server or a storage medium readable by the computer. If the program source is a program distribution server, the program distribution server includes a processor and storage resources for storing the program to be distributed, and the processor of the program distribution server may distribute the program to other computers. In addition, in this embodiment, two or more programs may be implemented as one program, or one program may be implemented as two or more programs.

[0013] Furthermore, in the drawings, (X, Y, Z) may be used as an explanatory coordinate system or direction. In the drawings, (X, Y, Z) indicates the spatial coordinate system relative to the vehicle. The X-axis and X-direction represent the left-right direction of the vehicle, and if the road surface on which the vehicle travels is a horizontal plane, the X-direction can also be called the first horizontal direction. The Y-axis and Y-direction represent the front-rear direction of the vehicle, and if the X-direction is the first horizontal direction, the Y-direction can also be called the second horizontal direction perpendicular to the X-direction. In addition, the Z-axis and Z-direction represent the up-down direction of the vehicle, and if the X-direction is the first horizontal direction and the Y-direction is the second horizontal direction, the Z-direction can also be called the vertical direction.

[0014] There are in-vehicle video display devices, such as head-up displays, that project images onto the vehicle's windshield, allowing the projected image to be observed as a virtual image through the windshield. Furthermore, technologies are being explored for using head-mounted display terminals, such as AR glasses, to display information while driving a vehicle.

[0015] In-vehicle video display devices have a fixed display surface on the vehicle, allowing for the display of images at a fixed position while driving. However, the field of view of the projected image is limited by the size of the concave mirror. Furthermore, because the position of the display surface and the user's head are not fixed, it is difficult to maintain the user's eye position, the image surface, and the object in a straight line when, for example, displaying an alert by superimposing an image of a vehicle or pedestrian ahead. On the other hand, head-mounted display terminals have a fixed positional relationship between the image display surface and the user's head, so they can only display images in the direction the user is facing. However, they make it easier to maintain the user's eye position, the image surface, and the object in a straight line. Therefore, it is desirable to use both in-vehicle video display devices and head-mounted display terminals in combination to provide more appropriate information display.

[0016] However, when displaying information using a head-mounted display terminal and an in-vehicle video display device, the images displayed by the head-mounted display terminal may interfere with driving. Therefore, this embodiment describes a technology that enables more appropriate display when using a head-mounted display terminal while driving. This makes it possible, for example, to display images that do not interfere with driving.

[0017] Head-mounted display devices include, for example, head-mounted displays and AR glasses. In-vehicle video display devices are devices (information display devices) that are mounted in a vehicle and display images, such as head-up displays, information projection headlights, and instrument panels.

[0018] First, an example of the configuration of a head-mounted display terminal will be explained with reference to Figure 1. Figure 1 is a block diagram showing an example of the configuration of a head-mounted display terminal.

[0019] The head-mounted display terminal 1 is a device capable of displaying information based on AR (Augmented Reality). As shown in Figure 1, this head-mounted display terminal 1 includes a processor 101, a storage device 110, an input I / F 120, a video input / output device 130, an audio input / output device 140, a sensor group 150, a communication I / F 160, an expansion I / F 171, a timer 172, and an actuator 173.

[0020] The processor 101 is configured using a CPU (Central Processing Unit) and the like, and is connected to various configurations via the bus 102.

[0021] The storage device 110 includes a volatile memory 111 and a non-volatile memory 112. The volatile memory 111 is the main memory and is configured using, for example, RAM (Random Access Memory). The processor 101 temporarily stores data such as programs in the volatile memory 111 and performs data processing. The non-volatile memory 112 is an auxiliary storage device that stores data nonvolatility. The non-volatile memory 112 is configured using a non-volatile storage medium and stores data such as programs. The non-volatile memory 112 stores, for example, a basic operation PRG 112a, an AR viewing PRG 112b, and a proximity transmission PRG 112c. The basic operation PRG 112a is, for example, a program related to the OS (Operating System). The AR viewing PRG 112b is a program used for viewing AR content. The proximity transmission PGR112c is a program for transmitting proximity information to the outside using proximity communication. These programs may each be separate programs. However, it is not limited to this, and a single program may include and provide basic operation functions, AR viewing functions, and proximity transmission functions. Alternatively, different divided programs may provide the aforementioned functions. Furthermore, the programs and data stored in these storage devices 110 may be modified or newly added using the communication I / F 160 described later.

[0022] The input I / F 120 is an interface used by the user for operation input, and information that the user wishes to input is input via the input I / F 120. The input I / F 120 may be configured to accept user operation by having the user operate a predetermined button switch 121, for example, a power button, volume buttons, etc. Alternatively, the input I / F 120 may be configured to accept user operation based on the user's gaze detection, the user's hand detection, the user's gesture detection, the detection of a predetermined sound, etc. Furthermore, the input I / F 120 may be configured to accept user operation by having the user operate a pointer on the display. In addition, the input I / F 120 may be configured to accept user operation by having the user operate an operating device connected via the expansion I / F 171, which will be described later.

[0023] The video input / output device 130 comprises a display 131, an image signal processing unit 132, and an out-camera 133. The display 131 is configured to output video. The display 131 is a transparent display, and a user wearing the head-mounted display terminal 1 can view the outside world through the display 131. The display 131 can display, for example, images superimposed on the outside world (AR images), a pointer used for user operation, etc. The image signal processing unit 132 is configured for processing image signals and can be configured using, for example, VRAM (video RAM), a dedicated image processing circuit (LSI), an image (video) signal processor, software, etc. The rear camera 133 is configured to capture images of the outside world and is a camera unit that inputs image data of subjects in the outside world by converting light input from the lens into an electrical signal using an electronic device such as a CCD (Charge Coupled Device) or CMOS (Complementary Metal Oxide Semiconductor) sensor. The rear camera 133 may capture images of the user's hands, gestures, etc., and the head-mounted display terminal 1 may acquire the image data captured by the rear camera 133 as instruction information for input operations, etc., and perform predetermined processing. The display 131 is an opaque display and may be configured to allow the user to see the outside world by displaying images from cameras that capture images of the outside world, such as the rear camera 133. In addition, although not shown in the figure, an in-camera that captures the user's eyes may also be provided.

[0024] The audio input / output device 140 includes a speaker 141, an audio signal processing unit 142, and a microphone 143. The speaker 141 outputs, for example, the audio (sound) of the content played during the experience of the head-mounted display terminal 1. The speaker 141 can also inform the user of various notification information by voice. The audio signal processing unit 142 is a configuration used for processing audio signals and can be configured using, for example, a circuit board, an audio signal processor, software, etc. The microphone 143 collects the user's own voice, external sounds, etc., and converts them into audio data. The user may speak voice instructions such as input operations, and the head-mounted display terminal 1 may acquire the audio data collected by the microphone as instruction information such as input operations and execute predetermined processing.

[0025] The sensor group 150 may include, for example, a positioning sensor 151, a geomagnetic sensor 152, a distance measuring sensor 153, an acceleration sensor 154, a gyroscope sensor 155, a gaze detection sensor 156, and the like. The sensor group 150 may also include different types of sensors than those listed above. Furthermore, the sensor group 150 may be configured in which the above sensors are appropriately omitted. In addition, the sensor group 150 may be omitted from the head-mounted display terminal 1, and sensors used for processing may be appropriately provided.

[0026] The positioning sensor 151 is a GNSS sensor. The positioning sensor 151 is a device that receives signals from GNSS (Global Navigation Satellite System) satellites in the sky and is used to detect the current position of the head-mounted display terminal 1. The head-mounted display terminal 1 can use the positioning sensor 151 to detect its own position (in other words, the position of the user wearing the head-mounted display terminal 1).

[0027] The geomagnetic sensor 152 is a sensor that detects the Earth's magnetic field and the direction in which the head-mounted display terminal 1 is facing. By using a three-axis type sensor that detects the geomagnetic field in the vertical direction in addition to the front-back and left-right directions, it is also possible to detect the movement of the head-mounted display terminal 1 by capturing the changes in the geomagnetic field in response to the movement of the head-mounted display terminal 1. This makes it possible to detect the posture of the user wearing the head-mounted display terminal 1.

[0028] The distance measuring sensor 153 is a sensor that measures the distance from the head-mounted display terminal 1 to an object, measures the position of the object, and can capture the shape of an object as a three-dimensional object. Examples of distance measuring sensors 153 include LiDAR (Light Detection and Ranging), which irradiates an object with laser light such as infrared light and measures the scattered light that reflects back; TOF (Time Of Flight), which measures the reflection time of pulsed light irradiated onto a subject for each pixel; and millimeter-wave radar, which emits millimeter-wave radio waves and captures the reflected waves. Furthermore, the distance measuring sensor 153 may be a sensor that measures based on the angle at which the reflected light reflected from the object is received. In other words, the distance measuring sensor 153 may be a triangulation-type sensor. Also, in the head-mounted display terminal 1, the distance measuring sensor 153 may be configured as a stereo camera that performs measurements based on a parallax image.

[0029] The acceleration sensor 154 is a sensor that detects acceleration, which is the change in velocity per unit time, and can capture movement, vibration, shock, etc. The acceleration sensor 154 can detect the tilt and direction of the head-mounted display terminal 1 worn by the user. The gyro sensor 155 is a sensor that detects angular velocity in the rotational direction, and can capture the vertical, horizontal, and diagonal orientation. Therefore, the orientation, such as the tilt and direction of the head-mounted display terminal 1, can be detected using the acceleration sensor 154 and the gyro sensor 155.

[0030] The gaze detection sensor 156 may include a left-eye gaze sensor for detecting the gaze of the left eye and a right-eye gaze sensor for detecting the gaze of the right eye, and can detect the movement and direction of the left and right eyes to capture the user's viewpoint, which is the target of their gaze. The head-mounted display terminal 1 may perform eye tracking using the gaze detection sensor 156 and, for example, acquire the detection result as input information. The head-mounted display terminal 1 may then execute predetermined processing based on this input information. For example, the head-mounted display terminal 1 may detect the user's gaze toward a predetermined display on the display and acquire the detection result as input information to execute predetermined processing corresponding to that display. Alternatively, the gaze may be detected from an image of the user's eyes acquired by an in-camera. The in-camera may be a camera that captures images in the visible light range or a camera that captures images in the infrared light range. If it is a camera that captures images in the infrared light range, an infrared LED may be provided as illumination.

[0031] The communication interface 160 is an interface used for communication and includes, for example, a LAN communication interface 161, a short-range wireless communication interface 162, and a telephone network communication interface 163.

[0032] The LAN communication interface 161 is an interface used for communication over a LAN (Local Area Network). The LAN communication interface 161 is connected to the network via an access point (AP) device, for example, by a wireless connection such as Wi-Fi (registered trademark), and transmits and receives data with other devices on the network.

[0033] The short-range wireless communication interface 162 is a communication interface for short-range wireless communication with devices within short-range wireless communication range. Short-range wireless communication is performed, for example, using electronic tags, but is not limited to this. If the head-mounted display terminal 1 is near a device and this device is at least capable of wireless communication, short-range wireless communication may be performed using Bluetooth®, IrDA (Infrared Data Association®), Zigbee®, HomeRF (Home Radio Frequency®), or wireless LAN (IEEE 802.11a, IEEE 802.11b, IEEE 802.11g).

[0034] Telephone network communication I / F163 is used for third-generation mobile communication systems (hereinafter referred to as "3G") such as GSM (Registered Trademark) (Global System for Mobile Communications), W-CDMA (Wideband Code Division Multiple Access), CDMA2000, and UMTS (Universal Mobile Telecommunications System), or communication methods called LTE (Long Term Evolution), fourth generation (4G), and fifth generation (5G). It connects to the communication network through base stations using the mobile communication network and transmits and receives information with servers on the communication network.

[0035] The expansion I / F 171 is an interface for extending the functionality of the head-mounted display terminal 1, and is, for example, a USB device connection terminal. Various devices can be connected to the expansion I / F 171, and it is used for purposes such as transmitting and receiving data and charging. For example, an operating device separate from the head-mounted display terminal 1, such as a keyboard, key buttons, or touch keys, may be connected to the expansion I / F 171. The head-mounted display terminal 1 may then acquire the content input to the operating device as instruction information such as input operations and execute predetermined processing. The expansion I / F 171 can be configured to allow connection of devices via wired or wireless connections.

[0036] Timer 172 is a timer that holds the current time in the real world, and holds a time such as Coordinated Universal Time (UTC) as the current time. The timer may be configured as software or using Real Time Clock (RTC). Actuator 173 transmits physical movements such as vibrations to the user. The actuator may be electrically operated, or it may utilize piezoelectric elements, magnetism, or hydraulics, as long as it generates a physical force.

[0037] Although not shown in Figure 1, the head-mounted display terminal may also be equipped with a battery. Furthermore, some of the components shown in the block diagram of Figure 1 may be provided in a separate device connected to the head-mounted display terminal by wire or wireless connection. For example, the head-mounted display terminal may be connected to a separate housing containing heat-generating components such as a battery. Alternatively, it may be connected to a mobile device such as a smartphone, allowing the mobile device to perform some or all of the processing performed by the processor 101.

[0038] Next, an example of the configuration of a head-mounted display terminal will be described with reference to Figures 2A and 2B.

[0039] Figures 2A and 2B show an example of the external appearance of a head-mounted display terminal. In this example, the head-mounted display terminal 1 is a glasses-type HMD (head-mounted display) and has the same configuration as in Figure 1. As shown in Figure 2A, an out-camera is located at the front of the glasses-type frame 193. Specifically, an out-camera 133L is located on the front left side of the frame 193, and an out-camera 133R is located on the front right side of the frame 193. In addition, an out-camera 133F and a distance measuring sensor 153 are located on the front center side of the frame. Furthermore, a left display 131L, which is a transmissive display on the left side, and a right display 131R, which is a transmissive display on the right side, are located at the front of the frame 193.

[0040] In the example shown in Figure 2A, the left display 131L and the right display 131R are transparent displays, but the system is not limited to this. For example, Figure 2B shows an example of a head-mounted display terminal using a virtual image method. As in this example, the display 131 may be built into the upper part of the glasses within the frame 193, and the image from the display 131 may be reflected by the glasses to allow the user to view the image as a virtual image.

[0041] In the example shown in Figure 2B, the left display 131L and the right display 131R themselves may be opaque displays. Furthermore, the display 131 is not limited to the top of the frame 193, but may be built into other locations within the frame 193, as long as the reflected light from the display 131 can be seen. The generation of the virtual image in this example may be achieved by providing a prism between the glasses and the pupil to control the polarization of the light emitted from the display 131, or by reflecting or totally internalizing the light emitted from the display 131 within the glasses and guiding it to the pupil.

[0042] Furthermore, a microphone 143 and a left speaker 141L are located on the left side of frame 193, while a right speaker 141R, a sensor group 150, a communication interface 160, and a control device 192 are located on the right side of frame 193. Here, the control device 192 is a device that implements a processor 101, a storage device 110, an input interface 120, an expansion interface 171, a timer 172, and an actuator 173. In addition, a gaze detection sensor 156 is located on the inside of the front of frame 193. Although not shown, a battery may be built into frame 193 or the like, or it may be connected to an external battery. Alternatively, as shown in Figure 2B, it may be connected to another housing 900 such as a mobile terminal and powered by it.

[0043] Regarding the head-mounted display terminal 1 shown in Figure 2A, the out-cameras (133L, 133R, 133F) are related to the out-camera 133. The left display 131L and the right display 131R are related to the display 131. The speakers (141L, 141R) are related to the speaker 141.

[0044] Next, an example of video display within the user's field of view will be described while referring to FIG. 3. FIG. 3 is a diagram showing an example of video display by a head-mounted display terminal.

[0045] A user wearing the head-mounted display terminal 1 on the head can visually recognize the scenery in front of the user through the display 131 of the head-mounted display terminal 1. Further, the user can visually recognize an AR video displayed on the display 131 of the head-mounted display terminal 1 and superimposed on the scenery in front of the user.

[0046] Here, the head-mounted display terminal 1 displays, for example, navigation information to a destination set by the user, information about the vehicle, and information about detected surrounding objects. Also, the head-mounted display terminal 1 displays, for example, the video of music content viewed using the head-mounted display terminal 1 and the video used for operations (such as playback, rewind, fast forward, stop, etc.) of the music content.

[0047] FIG. 4 is a diagram showing a configuration example of an in-vehicle system including an in-vehicle video display device. As shown in FIG. 1, in the vehicle 11, an in-vehicle video display device 10 and a controller 100 are mounted as an in-vehicle system 300. That is, the in-vehicle system 300 mounted on the vehicle 11 is configured to include the in-vehicle video display device 10 and the controller 100. In the present embodiment, the in-vehicle video display device 10 is, for example, a HUD (Head Up Display) device.

[0048] The in-vehicle video display device 10 provides a virtual image of the video to the user's field of view through the projection area (also referred to as the display area) 3 of the windshield (also referred to as the front glass) 2. The in-vehicle video display device 10 includes a video display device described later, and reflects the video displayed by this video display device in the direction of the user in the projection area 3 of the windshield 2. As a result, from the driver of the vehicle 11, the video displayed by the video display device is visually recognized as a virtual image formed in front of the windshield 2.

[0049] The controller 100 is a control device / control unit, also known as an ECU (Electronic Control Unit), which controls all or part of the in-vehicle system 300. The in-vehicle video display device 10 and the controller 100 are connected by an information transmission path 200, which consists of an interface such as a CAN (Controller Area Network), and are able to communicate with each other.

[0050] The controller 100 acquires vehicle information 4 using various sensors, measuring devices, communication devices, etc., as shown in Figure 5. The controller 100 then controls the in-vehicle video display device 10 via an information transmission path 200 such as a CAN, causing the video based on the vehicle information 4 to be displayed as a virtual image in the projection area 3 of the windshield 2 as seen by the driver. In other words, the in-vehicle video display device 10 acquires video data based on the vehicle information 4 from the controller 100 via an information transmission path 200 such as a CAN, and emits video light based on the acquired video data. Alternatively, the in-vehicle video display device 10 generates video data based on the vehicle information 4 etc. acquired and input from the controller 100, and emits video light based on the generated video data.

[0051] Figure 5 shows an example of a functional block configuration for the controller 100. As shown in Figure 5, various sensors and devices connected to the controller 100 include, for example, a vehicle speed sensor 201, a shift position sensor 202, a steering angle sensor 203, a headlight sensor 204, an illuminance sensor 205, a chromaticity sensor 206, a distance sensor 207, an infrared sensor 208, an engine start sensor 209, an acceleration sensor 210, a gyro sensor 211, a temperature sensor 212, a wireless transceiver for vehicle-to-infrastructure communication 213, a wireless transceiver for vehicle-to-vehicle communication 214, a camera (in-vehicle camera) 215, a camera (out-of-vehicle camera) 216, a GPS receiver 217, and a VICS (Vehicle Information and Communication System, registered trademark) receiver 218. The sensors and devices connected to the controller 100 are not limited to those listed above, and can be added, deleted, or replaced as needed.

[0052] The vehicle speed sensor 201 detects the speed of the vehicle 11 and generates speed information as the detection result. The shift position sensor 202 detects the current gear and generates gear information as the detection result. The steering angle sensor 203 detects the current steering angle and generates steering angle information as the detection result. The headlight sensor 204 detects whether the headlights are ON / OFF and generates lamp illumination information as the detection result. The illuminance sensor 205 and the chromaticity sensor 206 detect ambient light (for example, sunlight) and generate ambient light information as the detection result.

[0053] The distance measuring sensor 207 detects the distance between the vehicle 11 and external objects and generates distance information as a detection result. The infrared sensor 208 detects the presence and distance of objects in close proximity to the vehicle 11 and generates infrared information as a detection result. The engine start sensor 209 detects the ON / OFF status of the engine and generates ON / OFF information as a detection result. The acceleration sensor 210 and gyro sensor 211 detect the acceleration and angular velocity of the vehicle 11 and generate acceleration gyro information representing the attitude and behavior of the vehicle 11 as a detection result. The temperature sensor 212 detects the temperature inside and outside the vehicle 11 and generates temperature information as a detection result.

[0054] The vehicle-to-infrastructure wireless transceiver 213 generates vehicle-to-infrastructure communication information through vehicle-to-infrastructure communication between the vehicle 11 and roads, signs, traffic lights, etc. The vehicle-to-vehicle wireless transceiver 214 generates vehicle-to-vehicle communication information through vehicle-to-vehicle communication between the vehicle 11 and other surrounding vehicles.

[0055] The in-vehicle camera 215 generates in-vehicle camera video information by photographing the interior of the vehicle 11. The exterior camera 216 generates exterior camera video information by photographing the exterior of the vehicle 11. The in-vehicle camera 215 may also capture the driver's posture, eye position, and movement, and may constitute a Driver Monitoring System (DMS). The exterior camera 216 photographs the surrounding environment, such as the area in front of the vehicle 11. By analyzing the video information captured by the exterior camera 216, it is possible to determine the presence or absence of other vehicles or people around the vehicle 11, buildings and terrain, road surface conditions such as rain, snow, ice, and unevenness, and road signs. The exterior camera 216 may also function as a drive recorder, recording the vehicle 11's driving conditions in video.

[0056] The GPS receiver 217 generates GPS information by receiving GPS signals from GPS satellites. GPS information includes information such as the current time, latitude, and longitude. The VICS receiver 218 generates VICS information obtained by receiving VICS signals. The GPS receiver 217 and the VICS receiver 218 may be provided as part of a navigation system.

[0057] Figure 6 is a schematic diagram showing an example of the configuration of the in-vehicle video display device in Figure 4. As shown in Figure 6, the in-vehicle video display device 10 is housed in the dashboard 5 of the vehicle 11. The in-vehicle video display device 10 has a video forming unit (also called a video forming unit) 12 and a video projection unit. The video forming unit 12 is, for example, a projector or a liquid crystal display (LCD), and displays an image based on input video data and emits video light of the displayed image. In this embodiment, the video forming unit 12 more specifically comprises a light source device 13, optical components 14, and a display panel 15. Alternatively, the video forming unit 12 may consist only of a display panel 15, such as a micro-LED display composed of an array of tiny LEDs that become individual pixel elements. The video projection unit has reflective mirrors M1 and M2, and the reflective mirror M1 may be omitted depending on the design.

[0058] The light source device 13 includes, for example, an LED (Light Emitting Diode) light source or a laser light source, and illuminates the display panel 15 with light. In this example, the light source device 13 is positioned opposite the display panel 15 when viewed in the Z direction. When the light source device 13 is controlled to be ON, it turns on the backlight, that is, it turns on the light source, and when it is controlled to be OFF, it turns off the backlight. As will be described in more detail later, the light source device 13 has multiple light sources, and the turning on and off of these multiple light sources is controlled individually.

[0059] The optical component 14 is, for example, a lens for the light source, which adjusts the optical path of the backlight from the light source device 13 so that the backlight is uniformly illuminated over a predetermined area of ​​the display panel 15. The display panel 15 is typically a liquid crystal panel. The display panel 15 displays the image by modulating the backlight from the light source device 13 according to the input video data, specifically by modulating the transmittance for each pixel.

[0060] The reflective mirrors M1 and M2 are positioned in the optical path of the image light emitted from the image forming unit 12, and project the reflected image light onto the projection area 3 of the windshield 2. Reflective mirror M1 is, for example, a planar mirror and is provided in the optical path of the image light between the image forming unit 12 and reflective mirror M2, and reflects the image light toward reflective mirror M2. Reflective mirror M2 is, for example, a concave mirror (magnifying mirror) and is provided in the optical path of the image light between the display panel 15 and the projection area 3 of the windshield 2, and reflects the image light toward the projection area 3. In other words, the reflective mirrors M2 reflect and magnify the image light and project it onto the projection area 3 through the opening 6 formed in the dashboard 5.

[0061] The reflective mirror M1 has a fixed installation angle. On the other hand, the reflective mirror M2 is equipped with a drive mechanism 16, and its installation angle can be variably adjusted via the drive mechanism 16. The drive mechanism 16 functions as a mirror drive unit and includes, for example, a motor, which rotates the reflective mirror M2. By adjusting the installation angle of the reflective mirror M2, it is possible to protect the display panel 15 by preventing sunlight from the outside from reaching it.

[0062] The image light projected onto the projection area 3 of the windshield 2 is reflected by this projection area 3 and enters the driver's eyes 7. As a result, the driver sees the image light projected onto the projection area 3 as a virtual image 8 beyond the transparent windshield 2, superimposed on the scenery outside the vehicle (roads, buildings, people, etc.). Furthermore, by rotating the reflective mirror M2 with the drive mechanism 16, the position of the virtual image 8 (position of the projection area 3) seen by the driver 7 can be adjusted vertically.

[0063] The information represented by the virtual image 8 includes various things, such as road signs, the current speed of the vehicle, and various information added to objects in the landscape, i.e., AR (Augmented Reality) information. If the virtual image 8 is AR information, it is displayed superimposed according to the position of the object. If the virtual image 8 is not AR, it is displayed independently at a predetermined position. Also, while the vehicle 11 is typically an automobile, it is not limited to this and may also be a railway vehicle or an aircraft.

[0064] Figures 7A and 7B show an example of video display by the in-vehicle video display device 10. As shown in Figures 7A and 7B, the driver 7 can view the video I (virtual image) based on the video light emitted by the in-vehicle video display device 10, which is reflected by the windshield 2 of the vehicle 11, through the windshield 2 of the vehicle 11. Here, as shown in Figure 7B, the display range PA in which the in-vehicle video display device 10 can display video is limited to a portion of the driver 7's field of view FA.

[0065] In order to widen the display range PA of the in-vehicle video display device 10, such as a head-up display or information projection light, it would be necessary to enlarge the in-vehicle video display device 10, which would make it difficult to mount the in-vehicle video display device 10 on the vehicle 11. As a result, the display range PA of the video displayed by the in-vehicle video display device 10 is limited. Here, the driver 7 cannot see the video displayed by the in-vehicle video display device 10 except when looking at a limited area in front of the vehicle 11. This is because the in-vehicle video display device 10 is fixed to the vehicle 11 and does not move in conjunction with the driver 7's viewpoint. On the other hand, the head-mounted display terminal 1 is, for example, a head-mounted display or AR glasses, and the positional relationship between the display 131 and the user is fixed, and the display 131 follows the user's head movement, making it possible to display video in the user's field of view. Therefore, by using the head-mounted display terminal 1, the problem of the limitation of the video display area of ​​the in-vehicle video display device 10 is solved, and a wide range of information is displayed to the user.

[0066] Figure 8 shows an example of information display in coordination with the in-vehicle video display device 10 on the head-mounted display terminal 1. Here, Figure 8 shows the view from the driver's seat window from the driver's perspective. In this example, the head-mounted display terminal 1 is an augmented reality (AR) glasses. The AR glasses may be worn on one eye or on both eyes. The user can see the external environment while driving through the AR glasses. In this example, only the image seen through the AR glasses on one eye is displayed.

[0067] Image I1 is an image reflected from the in-vehicle video display device 10 onto the projection area 3 of the windshield 2. Images I2 and I3 are images displayed on the head-mounted display terminal 1. The head-mounted display terminal 1 highlights and frames surrounding objects to be monitored, such as people or bicycles, within a predetermined range in front of the vehicle 11, based on information acquired by the rear camera 133 and the distance measuring sensor 153, in order to alert the driver. Alternatively, instead of acquiring information from the rear camera 133 and the distance measuring sensor 153, information acquired by the vehicle may be received via the communication I / F 160 and used.

[0068] Figure 9 is a flowchart showing an example of a coordinated display process with an in-vehicle video display device 10, which is performed on the head-mounted display terminal 1. This process is achieved when the processor 101 of the head-mounted display terminal 1 executes a program stored in the non-volatile memory 112.

[0069] When the head-mounted display terminal 1 starts processing (S1000), it performs a connection determination (S1) to determine whether it is connected to the in-vehicle video display device 10. The head-mounted display terminal 1 and the in-vehicle video display device 10 may be connected directly, or they may be connected via other devices such as the controller 100 of the vehicle 11. The connection method between the head-mounted display terminal 1 and the in-vehicle video display device 10 may be wired or wireless. An example of the connection determination (S1) process will be explained in detail later with reference to Figure 12.

[0070] If the head-mounted display terminal 1 determines that it is not connected to the in-vehicle video display device 10, it terminates processing without switching to the coordinated display mode and displays in normal display mode. The mode in which the display is performed in coordination with the in-vehicle video display device 10 is called the "coordinated display mode," and the mode in which the display is performed normally without coordination with the in-vehicle video display device 10 is called the "normal display mode."

[0071] When the head-mounted display terminal 1 determines that it is connected to the in-vehicle video display device 10, it switches to cooperative display mode. The head-mounted display terminal 1 detects display interference with the in-vehicle video display device 10 by aligning its coordinate system with the in-vehicle video display device 10 and adjusts the display area (S2). By calibrating the display space of the head-mounted display terminal 1 and the in-vehicle video display device 10, the head-mounted display terminal 1 can check whether the display images overlap based on their relative positions in the coordinate systems and display them without overlapping. It also becomes possible to transfer images between the head-mounted display terminal 1 and the in-vehicle video display device 10 within the user's field of view.

[0072] Although the in-vehicle video display device 10 is fixedly installed in the vehicle 11, the position of the head-mounted display terminal 1 is affected by the user's seating position and height, so calibration must be performed for each user. User-specific calibration information may be read from information already recorded in the storage device 110.

[0073] Furthermore, the head-mounted display terminal 1 may read and use past calibration results if the user is the same, but even if it is the same user, there is no guarantee that they will be in the same riding position each time, so it may be set to perform calibration each time. In addition, the head-mounted display terminal 1 may be set to perform calibration again when the out-camera 133 of the head-mounted display terminal 1 confirms that the image of the head-mounted display terminal 1 and the image of the in-vehicle video display device 10 overlap.

[0074] Furthermore, if the relative position of the coordinate systems of the head-mounted display terminal 1 and the in-vehicle video display device 10 cannot be correctly recognized, the displayed images from the head-mounted display terminal 1 and the in-vehicle video display device 10 may overlap, potentially preventing the information from being correctly recognized. In such cases, the display on the head-mounted display terminal 1 may be discontinued.

[0075] The process of adjusting the display area (S2) will be explained in detail later with reference to Figure 13.

[0076] In processing S3, the head-mounted display terminal 1 performs display adjustment processing with the in-vehicle video display device 10. If the head-mounted display terminal 1 and the in-vehicle video display device 10 do not perform coordinated display, their display contents may overlap, making it impossible to recognize them correctly or resulting in them being recognized as incorrect information. Therefore, when the display areas of the head-mounted display terminal 1 and the in-vehicle video display device 10 overlap, the head-mounted display terminal 1 prioritizes the display of one device and terminates part or all of the display of the other device. Which device's display to prioritize may be set in advance or may be set by the user.

[0077] Alternatively, the head-mounted display terminal 1 may set priorities for the display content shown on the head-mounted display terminal 1 and the in-vehicle video display device 10, and display only the content with the highest priority. The head-mounted display terminal 1 and the in-vehicle video display device 10 may be set to display different content, and only the content with the highest priority may be displayed. The priority of the display content may be set in advance or set by the user. Priority information regarding the set priorities is stored, for example, in the storage device 110.

[0078] Furthermore, the head-mounted display terminal 1 may terminate the display of part or all of its information based on the battery status of the head-mounted display terminal 1.

[0079] The process of display adjustment (S3) will be explained in detail later with reference to Figures 19A and 19B.

[0080] The head-mounted display terminal 1 displays video in the area set in S2 based on the priority set in S3 (S4). Then, the head-mounted display terminal 1 determines the display termination condition (S5). If the display termination condition is not met, the head-mounted display terminal 1 continues displaying in coordinated display mode. If the display termination condition is met, the head-mounted display terminal 1 terminates display in coordinated display mode, terminates connection with the in-vehicle video display device 10 (S6), and the process ends (S1001).

[0081] The process for determining the display termination condition (S5) will be explained in detail later with reference to Figure 27.

[0082] Figure 10 is a sequence diagram showing an example of communication between a head-mounted display terminal 1 and a vehicle 11 having an in-vehicle video display device 10.

[0083] First, the head-mounted display terminal 1 transmits a connection request to the vehicle 11 (S2001). The vehicle 11 determines whether or not it can connect with the head-mounted display terminal 1 and transmits the result to the head-mounted display terminal 1 (S3001).

[0084] The head-mounted display terminal 1 receives a connection status from the vehicle 11 and performs a connection determination (S2002) to determine whether the vehicle 11 has accepted the connection to the head-mounted display terminal 1. If the vehicle 11 does not accept the connection to the head-mounted display terminal 1 (NO in S2002), the process ends without transitioning to the cooperative display mode. In addition, if the head-mounted display terminal 1 does not receive a connection status from the vehicle 11 for a certain period of time, it also determines that the vehicle 11 has not accepted the connection to the head-mounted display terminal 1 and ends the process without transitioning to the cooperative display mode.

[0085] Figure 10 shows an example where the head-mounted display terminal 1 sends a connection request to the vehicle 11. However, the vehicle 11 may send a connection request to the head-mounted display terminal 1, and the head-mounted display terminal 1 may determine whether or not a connection with the vehicle 11 is possible and send that determination to the vehicle 11.

[0086] If the head-mounted display terminal 1 determines, based on the connection determination in S2002, that the vehicle 11 has accepted the connection (YES in S2002), it switches to cooperative display mode and starts adjusting the display area with the in-vehicle video display device 10 by aligning the coordinate systems with the in-vehicle video display device 10 (S2003).

[0087] Here, the head-mounted display terminal 1 requests the in-vehicle video display device 10 to display a calibration image, and the in-vehicle video display device 10 displays the calibration image in the projection area 3 of the windshield 2 (S3002). The calibration image is an image that shows the displayable range of the in-vehicle video display device 10.

[0088] The head-mounted display terminal 1 acquires a calibration image displayed by the in-vehicle video display device 10 from its out-camera 133, confirms the relative coordinates of the display area of ​​the in-vehicle video display device 10, and recognizes the position of the display area of ​​the in-vehicle video display device 10 within the display coordinate system of the head-mounted display terminal 1 (S2004). This enables the head-mounted display terminal 1 to display information while taking into account the display area of ​​the in-vehicle video display device 10. Alternatively, the head-mounted display terminal 1 can set a coordinate range corresponding to the display area of ​​the in-vehicle video display device 10 as a display prohibition area within its display space, thereby restricting the display to only the in-vehicle video display device 10 and preventing display interference.

[0089] The display area recognition in S2004 will be explained with reference to Figure 11. Figure 11 is a diagram showing an example of display area recognition. In Figure 11, the head-mounted display terminal 1 sets the display area 402 of the in-vehicle video display device 10 within the display area 401 of the display 131.

[0090] The head-mounted display terminal 1 recognizes the display area 402 of the in-vehicle video display device 10 from the calibration image and constructs the display area 402 of the in-vehicle video display device within the display area 401 of the head-mounted display terminal 1. This allows the head-mounted display terminal 1 to monitor and adjust the displayed image to ensure that the display area 402 of the in-vehicle video display device 10 and the image displayed on the head-mounted display terminal 1 do not overlap.

[0091] In this way, the head-mounted display terminal 1 can check whether the displayed images overlap by calibrating the display space between the head-mounted display terminal 1 and the in-vehicle video display device 10, and display them in a way that prevents overlap. It also becomes possible to transfer video between the head-mounted display terminal 1 and the in-vehicle video display device 10. For example, the head-mounted display terminal 1 can convert information acquired by the vehicle 11 into a display area for the head-mounted display terminal 1 and display it. It is also possible to display information acquired by the head-mounted display terminal 1 on the in-vehicle video display device 10.

[0092] Returning to Figure 10, once the head-mounted display terminal 1 has finished adjusting the display area, it notifies the vehicle 11 of the display area adjustment information (S2005). The vehicle 11 makes settings based on the received information and transmits the setting information to the head-mounted display terminal 1, thereby sharing the setting information between the head-mounted display terminal 1 and the vehicle 11 (S3003). The setting information here refers to information about the display content and setting conditions related to the display. The setting information may also be transmitted together with connection availability information.

[0093] Figure 10 shows an example where vehicle 11 transmits setting information to head-mounted display terminal 1. However, the head-mounted display terminal 1 may transmit setting information to vehicle 11, or both may transmit information to each other. Here, an example of sending and receiving setting information is described, but it is also possible to display the information without sending or receiving it.

[0094] Next, the head-mounted display terminal 1 and the vehicle 11 decide which display device to prioritize, the head-mounted display terminal 1 or the in-vehicle video display device 10, and send and receive information for coordinated display to each other to perform display adjustments (S2006, S3004). If the display of the in-vehicle video display device 10 is prioritized, the display of the head-mounted display terminal 1 is adjusted, and if the display of the head-mounted display terminal 1 is prioritized, the display of the in-vehicle video display device 10 is adjusted. For example, the head-mounted display terminal 1 transmits the content to be displayed on the head-mounted display terminal 1 to the in-vehicle video display device 10, and the in-vehicle video display device 10 transmits the content to be displayed on the in-vehicle video display device 10 to the head-mounted display terminal 1 to perform display adjustments.

[0095] In a coordinated display, the head-mounted display terminal 1 may pre-set priorities for its display content, and if the display areas of the head-mounted display terminal 1 and the in-vehicle video display device 10 overlap, it may display only the content with the higher priority. For example, the head-mounted display terminal 1 and the in-vehicle video display device 10 may be configured to prioritize the display of safety / warning information.

[0096] Furthermore, the head-mounted display terminal 1 and the in-vehicle video display device 10 may display different content. For example, information to be superimposed on surrounding objects may be displayed on the head-mounted display terminal 1, while other information (information not to be superimposed on surrounding objects) may be displayed on the in-vehicle video display device 10. Alternatively, safety / warning information may be displayed on the head-mounted display terminal 1, while other information may be displayed on the in-vehicle video display device 10.

[0097] After the display adjustment is complete, the vehicle 11 transmits information about the vehicle 11 (speed, navigation, etc.) (S3005) and information about people, bicycles, cars, etc. detected by various sensors 201 to 212 installed in the vehicle 11 (S3006) to the head-mounted display terminal 1, and starts a coordinated display based on this information (S3007).

[0098] The head-mounted display terminal 1 starts a coordinated display based on the information received from the vehicle 11 (S2007).

[0099] Figure 10 shows an example where the vehicle 11 transmits information to the head-mounted display terminal 1. However, the head-mounted display terminal 1 may transmit the acquired information to the vehicle 11 and display it on the in-vehicle video display device 10. Alternatively, the head-mounted display terminal 1 may perform display adjustments without exchanging information for coordinated display. In this case, a device such as an out-camera 133 may be used to check the display position and content of the in-vehicle video display device 10 and perform display adjustments.

[0100] The head-mounted display terminal 1 checks whether there has been a change in the display area of ​​the head-mounted display terminal 1 and the in-vehicle video display device 10 during coordinated display (S2008).

[0101] If there is a change in the display area of ​​the head-mounted display terminal 1 and the in-vehicle video display device 10 during coordinated display (YES in S2008), the head-mounted display terminal 1 returns to S2006 and adjusts the display according to the display priority.

[0102] For example, if the head-mounted display terminal 1 confirms in S2008 that the display images of the head-mounted display terminal 1 and the in-vehicle video display device 10 are overlapping, it returns to S2006 and terminates the display of either the head-mounted display terminal 1 or the in-vehicle video display device 10. Alternatively, the display position of either the overlapping display images of the head-mounted display terminal 1 or the in-vehicle video display device 10 may be moved, or both display positions may be moved to a position where they do not interfere with each other. For example, if the bicycle, which is the object being monitored, enters the display area of ​​the in-vehicle video display device 10, the head-mounted display terminal 1 may prioritize highlighting the object being monitored and terminate the display of the in-vehicle video display device 10, or change the display position of the in-vehicle video display device 10.

[0103] Furthermore, if the head-mounted display terminal 1 confirms in S2008 that the overlap between the display images of the head-mounted display terminal 1 and the in-vehicle video display device 10 has been resolved, it returns to S2006 and displays again the display image that had been stopped being displayed on the head-mounted display terminal 1 or the in-vehicle video display device 10. Alternatively, if the display position of the display image on the head-mounted display terminal 1 or the in-vehicle video display device 10 had been changed, it may be returned to its original display position.

[0104] When the display termination condition is met, the head-mounted display terminal 1 sends a termination request to the vehicle 11 (S2009), terminates the connection with the vehicle 11, and ends the coordinated display (S2010). Similarly, when the display termination condition is met, the vehicle 11 also sends a termination request to the head-mounted display terminal 1 (S3008) and ends the coordinated display (S3009). This termination request may also be sent from the vehicle 11 to the head-mounted display terminal 1. In this example, the head-mounted display terminal 1 is connected to the in-vehicle video display device 10, but if the head-mounted display terminal 1 is performing coordinated display independently, the head-mounted display terminal 1 may terminate the coordinated display on its own.

[0105] The termination request may be transmitted, for example, by the vehicle 11 when the operation ends, or by the head-mounted display terminal 1 when it is removed or when use ends.

[0106] Figure 12 shows an example of the connection determination process for S1 in Figure 9.

[0107] Example 1-1 is an example in which the coordinated display mode is automatically activated by connecting the head-mounted display terminal 1 to the in-vehicle video display device 10 or the vehicle 11. In Example 1-1, the head-mounted display terminal 1 determines whether one or more of the following conditions (1A), (1B), and (1C) are met. Condition (1A) is that the head-mounted display terminal 1 is connected to the in-vehicle video display device 10. Condition (1B) is that the head-mounted display terminal is connected to the vehicle 11. Condition (1C) is that the head-mounted display terminal 1 has activated the coordinated display mode. If the head-mounted display terminal 1 determines that one or more of the following conditions (1A), (1B), and (1C) are met, it can proceed to the next step S2.

[0108] Example 1-2 is an example in which the coordinated display mode is activated by connecting the head-mounted display terminal 1 to the in-vehicle video display device 10 or the vehicle 11, or by user operation. In Example 1-2, the head-mounted display terminal 1 determines whether one or more of the following conditions (1A), (1B), and (1D) are met. Condition (1A) is that the head-mounted display terminal 1 is connected to the in-vehicle video display device 10. Condition (1B) is that the head-mounted display terminal 1 is connected to the vehicle 11. Condition (1D) is that the head-mounted display terminal 1 has activated the coordinated display mode by user operation. If the head-mounted display terminal 1 determines that one or more of the following conditions (1A), (1B), and (1D) are met, it can proceed to the next step S2.

[0109] Example 1-3 is an example in which the coordinated display mode is not activated and the process ends because the head-mounted display terminal 1 is not connected to the in-vehicle video display device 10 or the vehicle 11. In Example 1-3, the head-mounted display terminal 1 determines whether one or more of the conditions (1E) and (1F) are met. Condition (1E) is that the head-mounted display terminal 1 is not connected to the in-vehicle video display device 10. Condition (1F) is that the head-mounted display terminal 1 is not connected to the vehicle 11. If the head-mounted display terminal 1 determines that one or more of the conditions (1E) and (1F) are met, it stops displaying the video and proceeds to the next step S1001. Although not described in Example 1-3, if the head-mounted display terminal 1 chooses not to activate the coordinated display mode through user operation, it may also stop displaying the video and proceed to the next step S1001.

[0110] Example 1-4 is an example of terminating without activating the coordinated display mode when the head-mounted display terminal 1 does not have a coordinated display mode. In Example 1-4, the head-mounted display terminal 1 determines whether condition (1G) is met. Condition (1G) is that the head-mounted display terminal 1 does not have a coordinated display mode. If the head-mounted display terminal 1 determines that condition (1G) is met, it stops displaying the video and proceeds to the next step S1001.

[0111] Example 1-5 is an example of how the head-mounted display terminal 1 terminates without activating the coordinated display mode when the coordinated display mode is set to not used. In Example 1-5, the head-mounted display terminal 1 determines whether condition (1H) is met. This example is when the settings allow for the selection of whether to use or not use the coordinated display mode. Condition (1H) is that the coordinated display mode of the head-mounted display terminal 1 is set to not used. If the head-mounted display terminal 1 determines that condition (1H) is met, it stops displaying the video and proceeds to the next step S1001.

[0112] Next, an example of display area control (S2) will be explained in detail with reference to Figure 13-18.

[0113] Figure 13 is a flowchart showing an example of the processing in display area adjustment (S2). When the cooperative display mode is activated and the process transitions from the previous step (S200) to display area adjustment (S2), for example, as shown in Figure 13, the head-mounted display terminal 1 performs relative coordinate confirmation (S21) and relative coordinate adjustment of the display area (S22). Then, the process transitions to the next step (S201). In display area adjustment (S2), the coordinate systems of the head-mounted display terminal 1 and the in-vehicle video display device 10 are aligned.

[0114] A specific example of the process in S21 will be explained using Figures 14 and 15.

[0115] Figure 14 is a table showing an example of the process for verifying the relative coordinates of the head-mounted display terminal 1. In S21, the relative position of the coordinate systems of the head-mounted display terminal 1 and the in-vehicle video display device 10 is calculated in order to perform coordinate system calibration. Furthermore, by combining multiple examples shown in Figure 14, the relative position of the coordinate systems may be calculated with even greater accuracy.

[0116] In Example 21-1, the head-mounted display terminal 1 checks the calibration image displayed by the in-vehicle video display device 10 and proceeds to the next step. The coordinate position of the in-vehicle video display device 10 is fixed relative to the vehicle 11. The coordinate position of the head-mounted display terminal 1 is fixed relative to the user's position. By capturing the calibration image for correction displayed by the in-vehicle video display device 10 with the rear camera 133, the head-mounted display terminal 1 can determine the relative position of the coordinate system between the head-mounted display terminal 1 and the in-vehicle video display device 10, based on the position and orientation of the head-mounted display terminal 1 within the vehicle 11.

[0117] In Example 21-2, the head-mounted display terminal 1 receives and confirms information regarding the display area of ​​the in-vehicle video display device 10, and proceeds to the next step. Based on the information regarding the display area of ​​the in-vehicle video display device 10 within the vehicle 11, the head-mounted display terminal 1 can determine the relative position of the coordinate system between the head-mounted display terminal 1 and the in-vehicle video display device 10.

[0118] In Example 21-3, the head-mounted display terminal 1 checks a specific location inside the vehicle and proceeds to the next step. The head-mounted display terminal 1 checks its relative position to the vehicle 11 by photographing a specific location, such as the four corners of the windshield 2, with the out-camera 133. Since the relative position between the vehicle 11 and the in-vehicle video display device 10 is uniquely determined, the head-mounted display terminal 1 can then grasp the relative position of the coordinate system between the head-mounted display terminal 1 and the in-vehicle video display device 10. Here, the four corners of the windshield 2 are given as specific locations inside the vehicle, but it could also be another structure fixed to the vehicle, such as the steering wheel or instrument panel. Alternatively, the head-mounted display terminal 1 may directly recognize the image displayed by the in-vehicle video display device 10 to grasp the relative position of the coordinate system between the head-mounted display terminal 1 and the in-vehicle video display device 10.

[0119] In Example 21-4, the head-mounted display terminal 1 receives and confirms the coordinate information of the head-mounted display terminal 1 within the vehicle 11, which has been identified by the vehicle 11, and proceeds to the next step. The vehicle 11 calculates the user's position within the vehicle 11 by photographing the user wearing the head-mounted display terminal 1 with the camera (in-vehicle camera) 215, and the head-mounted display terminal 1 obtains the user's position from the vehicle 11. As a result, the head-mounted display terminal 1 can understand the relative position of the coordinate system between the head-mounted display terminal 1 and the in-vehicle video display device 10.

[0120] Figures 15A and 15B show an example of display area adjustment based on the processing of Example 21-1 explained using Figure 14. Figures 15A and 15B show an example in which the display area of ​​the in-vehicle video display device 10 is set within the display area of ​​the head-mounted display terminal 1.

[0121] As shown in Figure 15A, the in-vehicle video display device 10 displays a calibration image 501 indicating the display area, and the head-mounted display terminal 1 captures that area, enabling display that takes into account the relative coordinate position within the head-mounted display terminal 1. In this example, the range of the display area is detected by displaying the entire display area in a single color.

[0122] As shown in Figure 15B, the calibration image 502 can also be displayed on the head-mounted display terminal 1 to prevent the image displayed on the head-mounted display terminal 1 from overlapping with the display area of ​​the in-vehicle video display device 10. In this example, the range of the display area is detected by displaying the image only at the four corners, which are the outermost edges of the entire display area. As another example, the range of the display area may be detected by displaying the image only on the outer frame of the display area.

[0123] These calibration images may be displayed only when cooperative mode is activated, or they may be displayed periodically while cooperative mode is active to correct relative coordinate positions. If the calibration images are displayed periodically, they may be displayed for a duration of, for example, less than 1 / 30th of a second. This allows for correction of relative coordinate positions without the user being aware of it.

[0124] A specific example of the process in S22 will be explained using Figures 16-18.

[0125] Figure 16 is a table showing an example of the process for adjusting the relative coordinates of the head-mounted display terminal 1. In S22, the coordinate system is corrected (aligned) by considering the difference in the relative position of the head-mounted display terminal 1 with respect to the vehicle 11 or the in-vehicle video display device 10. As a result, the head-mounted display terminal 1 can display the positions of objects such as people, cars, and obstacles detected on the vehicle 11 side on the display 131 of the head-mounted display terminal 1 in the direction of the detected coordinates, similar to the in-vehicle video display device 10.

[0126] In Example 22-1, the head-mounted display terminal 1 converts the information acquired from the vehicle 11 into the coordinate system of the head-mounted display terminal 1, and adjusts the difference in the relative position of the head-mounted display terminal 1 with respect to the vehicle 11 or the in-vehicle video display device 10 based on the converted information. After that, the head-mounted display terminal 1 performs video display.

[0127] In Example 22-2, the head-mounted display terminal 1 converts its coordinate system to the vehicle 11's coordinate system and adjusts the difference in the relative position of the head-mounted display terminal 1 with respect to the vehicle 11 or the in-vehicle video display device 10 based on the information acquired from the vehicle 11. Subsequently, the head-mounted display terminal 1 converts the vehicle 11's coordinate system to its own coordinate system and performs video display.

[0128] In Example 22-3, the head-mounted display terminal 1 converts its coordinate system to that of the in-vehicle video display device 10 and adjusts the difference in the relative position of the head-mounted display terminal 1 with respect to the vehicle 11 or the in-vehicle video display device 10 based on information acquired from the vehicle 11. Subsequently, the head-mounted display terminal 1 converts the coordinate system of the in-vehicle video display device 10 to that of the head-mounted display terminal 1 and performs video display.

[0129] Figure 17 is a table in a different format showing a specific example of the relative coordinate adjustment (step S22) in Figure 16. Figure 17 shows the relationship between the coordinate system of the information obtained by the head-mounted display terminal 1 from the vehicle 11, the coordinate system during the relative coordinate adjustment between the head-mounted display terminal 1 and the in-vehicle video display device 10, and the coordinate system when the information is finally displayed on the head-mounted display terminal 1.

[0130] In Example 1, the head-mounted display terminal 1 obtains coordinate information from the vehicle 11 based on the head-mounted display terminal 1, performs relative coordinate adjustment based on the coordinate system of the head-mounted display terminal 1, and displays the image on the display 131 in the coordinate system of the head-mounted display terminal 1.

[0131] In Example 2, the head-mounted display terminal 1 obtains coordinate information from the vehicle 11, converts the coordinate information from the vehicle 11 to the coordinate system of the head-mounted display terminal 1, performs relative coordinate adjustment, and then displays the image on the display 131 in the coordinate system of the head-mounted display terminal 1.

[0132] In Example 3, the head-mounted display terminal 1 obtains coordinate information from the vehicle 11 based on the in-vehicle video display device 10, converts the coordinate information based on the in-vehicle video display device 10 to the coordinate system based on the head-mounted display terminal 1, performs relative coordinate adjustment, and then displays the image on the display 131 in the coordinate system of the head-mounted display terminal 1.

[0133] In Example 4, the head-mounted display terminal 1 obtains coordinate information from the vehicle 11, performs relative coordinate adjustments in the vehicle 11-based coordinate system, converts it to the head-mounted display terminal 1-based coordinate system, and then displays the image on the display 131 in the coordinate system of the head-mounted display terminal 1.

[0134] In Example 5, the head-mounted display terminal 1 obtains coordinate information from the vehicle 11 based on the in-vehicle video display device 10, performs relative coordinate adjustments in the coordinate system based on the in-vehicle video display device 10, converts it to the coordinate system based on the head-mounted display terminal 1, and then displays the image on the display 131 in the coordinate system of the head-mounted display terminal 1.

[0135] In this embodiment, an example of adjusting the coordinate system on the head-mounted display terminal 1 has been described. However, the head-mounted display terminal 1 may receive coordinate system information adjusted by the vehicle 11 or the in-vehicle video display device 10 and perform video display. In that case, the head-mounted display terminal 1 provides relative position information to the vehicle 11 or the in-vehicle video display device 10, and the vehicle 11 or the in-vehicle video display device 10 adjusts the coordinate system information using the relative position information provided by the head-mounted display terminal 1. Alternatively, the vehicle 11 may detect the relative position of the head-mounted display terminal 1. For example, it is possible to detect the position and orientation of the head-mounted display terminal 1 worn by the user from information captured by the in-vehicle camera 215 of the vehicle 11.

[0136] Furthermore, if the coordinate system is adjusted using the vehicle 11 or the in-vehicle video display device 10, there will be a time lag between the coordinate system adjustment and the video display. During the coordinate system adjustment, the user may move their head, which could cause the relative position of the head-mounted display terminal 1 with respect to the vehicle 11 or the in-vehicle video display device 10 to change. For this reason, it is preferable to adjust the coordinate system using the head-mounted display terminal 1. Also, the coordinate system adjustment during the ride may be performed only once or multiple times.

[0137] Figure 18 is a diagram illustrating the coordinate system of the display of the head-mounted display terminal 1 when the user is riding in the vehicle 11. Figure 18 illustrates an example in which the display area 602 of the in-vehicle video display device 10 is set within the display area of ​​the head-mounted display terminal 1.

[0138] The coordinate space of the head-mounted display terminal 1 has a coordinate region 601 based on the user's head. On the other hand, the display area of ​​the in-vehicle video display device 10 is set to a coordinate system based on the vehicle 11 or a coordinate system based on the in-vehicle video display device 10. Sensors (such as an external camera 216 and a distance measuring sensor 207) are installed in various locations on the vehicle 11, and the information acquired by the sensors is corrected using information about the location where they are installed to inform the user of the location of the detected object. By converting the information detected by the vehicle 11 into a coordinate space based on the head-mounted display terminal 1 and displaying it, it becomes possible to correctly display the detected object on the head-mounted display terminal 1 regardless of the user's seating position or the direction of their face.

[0139] Next, an example of display adjustment (S3) will be explained in detail with reference to Figure 19-26.

[0140] Figures 19A and 19B are flowcharts showing examples of processing in display adjustment (S3). When processing transitions from the previous step (S300) to display adjustment (S3), for example, as shown in Figure 19A, the head-mounted display terminal 1 performs display adjustment (S31). Then, processing transitions to the next step (S301).

[0141] As shown in Figure 19B, the head-mounted display terminal 1 may perform adjustments with other display devices (S32). Also, the order in which S31 and S32 are performed does not matter.

[0142] In display adjustment (S3), after adjusting the display area in S2, adjustments are made to the display content between the head-mounted display terminal 1 and the in-vehicle video display device 10.

[0143] A specific example of the processing in S31 will be explained using Figures 20-26. [Example 1] Figure 20 is a table showing an example of the processing of display adjustment for the head-mounted display terminal 1 and the in-vehicle video display device 10 according to Example 1. In S31, the head-mounted display terminal 1 performs adjustments regarding the display content. If the display areas of the head-mounted display terminal 1 and the in-vehicle video display device 10 overlap, there is a risk that information will not be transmitted correctly, or that incorrect information will be recognized, contrary to the intent of the information. Therefore, if the display areas of the head-mounted display terminal 1 and the in-vehicle video display device 10 overlap, it is desirable to set the display settings to be exclusive so that the displayed images do not overlap.

[0144] In this embodiment, a priority is set for the head-mounted display terminal 1 and the in-vehicle video display device 10. If the display images from the head-mounted display terminal 1 and the in-vehicle video display device 10 overlap, the video display device with the lower priority performs the processing shown below.

[0145] Example 31-1 is an example in which the video from the in-vehicle video display device 10 is given priority display. In Example 31-1, if the video from the head-mounted display terminal 1 overlaps with the video area of ​​the in-vehicle video display device 10, the head-mounted display terminal 1 performs one or more of the following processes (1) to (4): (1) The head-mounted display terminal 1 hides a portion of the displayed video. (2) The head-mounted display terminal 1 hides all of the displayed video. (3) The head-mounted display terminal 1 moves the display position of a portion of the displayed video to a position where it does not overlap with the video from the in-vehicle video display device 10. (4) The head-mounted display terminal 1 moves the display position of all of the displayed video to a position where it does not overlap with the video from the in-vehicle video display device 10. As a result, the video from the in-vehicle video display device 10 is displayed preferentially, and correct information can be conveyed to the user without being obstructed by the video from the head-mounted display terminal 1.

[0146] Example 31-2 is an example in which the video from the head-mounted display terminal 1 is given priority display. In Example 31-2, if the video from the head-mounted display terminal 1 overlaps with the video area of ​​the in-vehicle video display device 10, the in-vehicle video display device 10 performs one or more of the following processes (5) to (8): (5) The in-vehicle video display device 10 hides a portion of the displayed video. (6) The in-vehicle video display device 10 hides all of the displayed video. (7) The in-vehicle video display device 10 moves the display position of a portion of the displayed video to a position where it does not overlap with the video from the head-mounted display terminal 1. (8) The in-vehicle video display device 10 moves the display position of all of the displayed video to a position where it does not overlap with the video from the head-mounted display terminal 1. As a result, the video from the head-mounted display terminal 1 is displayed preferentially, and correct information can be conveyed to the user without being obstructed by the video from the in-vehicle video display device 10.

[0147] Example 31-3 is an example of prioritizing display based on the priority of each video. In Example 31-3, when the video of the head-mounted display terminal 1 overlaps with the video area of ​​the in-vehicle video display device 10, the video display device with the lower priority among the head-mounted display terminal 1 and the in-vehicle video display device 10 performs one or more of the following processes (9) to (12): (9) The video display device with the lower priority hides part of the displayed video. (10) The video display device with the lower priority hides all of the displayed video. (11) The video display device with the lower priority moves part of the displayed video to a position where it does not overlap with the video of the video display device with the higher priority. (12) The video display device with the lower priority moves all of the displayed video to a position where it does not overlap with the video of the video display device with the higher priority. As a result, regardless of the display device, the video with the higher priority is displayed preferentially, and the correct information can be conveyed to the user without being obstructed by the video with the lower priority.

[0148] Figures 21A and 21B show an example of display adjustment based on the processing of Example 31-1 explained using Figure 20.

[0149] In Figure 21A, images are displayed in the user's field of view by the head-mounted display terminal 1 and the in-vehicle video display device 10. However, the displayed image 701 of the head-mounted display terminal 1 and the displayed image 702 of the in-vehicle video display device 10 overlap in the user's field of view, and a portion of the displayed image 702 of the in-vehicle video display device 10 is not recognizable.

[0150] Figure 21B shows the case where the display of the in-vehicle video display device 10 is set to take priority. As shown in Figure 21B, the head-mounted display terminal 1 performs a display adjustment in S31 to prioritize the display of the in-vehicle video display device 10, thereby hiding the display image 701 of the head-mounted display terminal 1. As a result, the head-mounted display terminal 1 can display images such that the display image 701 of the head-mounted display terminal 1 and the display image 702 of the in-vehicle video display device 10 do not overlap. In the example shown in the figure, the display image 701 of the head-mounted display terminal 1 is hidden, but the display position of the display image 701 may be moved, or the display format may be changed, such as by shrinking the display image 701 so that it does not overlap.

[0151] Figures 22A and 22B show an example of display adjustment based on the processing of Example 31-2 explained using Figure 20.

[0152] In Figure 22A, images are displayed in the user's field of view by the head-mounted display terminal 1 and the in-vehicle video display device 10. However, the displayed image 701 of the head-mounted display terminal 1 and the displayed image 702 of the in-vehicle video display device 10 overlap in the user's field of view, making it impossible to recognize the displayed image 701 of the head-mounted display terminal 1.

[0153] Figure 22B shows the case where the display of the head-mounted display terminal 1 is set to take priority. As shown in Figure 22B, by performing a display adjustment in S31 to prioritize the display of the head-mounted display terminal 1, the display image 702 of the in-vehicle video display device 10 is hidden. As a result, the head-mounted display terminal 1 can display images such that the display image 701 of the head-mounted display terminal 1 and the display image 702 of the in-vehicle video display device 10 do not overlap. In the example shown in the figure, a portion of the display image 702 of the in-vehicle video display device 10 is hidden, but the display position of the display image 702 may be partially moved, or a portion of the display image 702 may be reduced in size to prevent overlapping, thus changing the display format.

[0154] Furthermore, as shown in Example 31-3 of Figure 20, when priority display is performed based on the priority set for each video, for example, as shown in Figure 22B, safety-related information about the surroundings of the vehicle 11 sensed by the vehicle's sensors may be displayed preferentially. [Example 2] Figure 23 is a table showing an example of display adjustment processing for the head-mounted display terminal 1 and the in-vehicle video display device 10 according to Example 2.

[0155] In this embodiment, a priority is set for the display content of either the head-mounted display terminal 1 or the in-vehicle video display device 10, or both. When the display images of the head-mounted display terminal 1 and the in-vehicle video display device 10 overlap, the device for which a priority has been set for its display content will perform the following processing on the video display device displaying the lower priority content.

[0156] Example 31-4 is an example of prioritizing the display of safety-related information. In Example 31-4, when the image from the head-mounted display terminal 1 overlaps with the image area of ​​the in-vehicle video display device 10, the head-mounted display terminal 1 and the in-vehicle video display device 10 compare the image information and perform one or more of the following processes (1) to (4): (1) The head-mounted display terminal 1 and the in-vehicle video display device 10 prioritize the display of safety-related information. (2) The head-mounted display terminal 1 and the in-vehicle video display device 10 hide information other than safety-related information. (3) If there is an alternative display device, the head-mounted display terminal 1 and the in-vehicle video display device 10 hide information other than safety-related information and display it on the alternative display device. (4) If there is no alternative display device, the head-mounted display terminal 1 and the in-vehicle video display device 10 move the display position of information other than safety-related information.

[0157] Through the above process, users will be able to obtain the information necessary to ensure their own safety without it being compromised by other information.

[0158] Example 31-5 is an example of prioritizing the display of superimposed images of physical objects. In Example 31-5, when the image of the head-mounted display terminal 1 overlaps with the image area of ​​the in-vehicle video display device 10, the head-mounted display terminal 1 and the in-vehicle video display device 10 compare the image information and perform one or more of the following processes (5) to (6): (5) The head-mounted display terminal 1 and the in-vehicle video display device 10 prioritize the display of superimposed images of physical objects sensed by the vehicle 11. (6) The head-mounted display terminal 1 and the in-vehicle video display device 10 hide information other than superimposed images of physical objects sensed by the vehicle 11.

[0159] The above is an example where the superimposition of information sensed by the vehicle 11 onto real objects around the vehicle is prioritized. However, the superimposition of information sensed by the head-mounted display terminal 1 onto real objects around the vehicle may also be prioritized.

[0160] Example 31-6 is an example of prioritizing display based on the video category or video priority set by the user. In Example 31-6, when the video from the head-mounted display terminal 1 overlaps with the video area of ​​the in-vehicle video display device 10, the head-mounted display terminal 1 and the in-vehicle video display device 10 compare the video information and perform one or more of the following processes (7) to (10): (7) The head-mounted display terminal 1 and the in-vehicle video display device 10 prioritize displaying the video with higher priority based on the video priority set by the user. (8) The head-mounted display terminal 1 and the in-vehicle video display device 10 prioritize displaying the video from the video category with higher priority based on the video category priority set by the user. (9) When displaying multiple videos from the same video category, the head-mounted display terminal 1 and the in-vehicle video display device 10 prioritize displaying the video with higher priority within the same category. (10) When the head-mounted display terminal 1 and the in-vehicle video display device 10 display multiple videos of the same video category, the video of the display device with the higher priority will be displayed first.

[0161] First, videos from categories with higher priority may be displayed first. If videos overlap, the video with the highest priority within the same category may be displayed first. Alternatively, if videos of the same priority overlap, the video from the display device with the highest priority may be displayed first.

[0162] Furthermore, the priority settings for videos may be changed by the user, or the priority of some videos may be set so that it cannot be changed. For example, some videos may have their priority fixed to the highest level from the beginning, while others may be designated as not to be displayed and therefore their priority cannot be set.

[0163] Figures 24A and 24B show an example of video priority.

[0164] As shown in Figure 24A, the head-mounted display terminal 1 may store data in its storage device 110 in which display priority is associated with video categories. Priorities can be set, for example, from 1 to 5, and videos of video categories with lower priority numbers are displayed preferentially.

[0165] For example, videos 1 and 2 show vehicle control information acquired from vehicle 11, and are assigned a priority of 2. These videos allow the user to understand the vehicle's driving status and the operating status of the in-vehicle equipment.

[0166] Videos 3-5 contain safety-related information and are therefore given priority level 1 as they are necessary for maintaining safety during driving operations. These videos allow users to understand the surrounding environment of their vehicle and the conditions that affect the safety of their vehicle.

[0167] Videos 6-10 contain route information, entertainment information, message displays, and other information that is not necessarily required for driving operations, and are therefore assigned a priority of 3-5. These videos allow vehicle users to understand the vehicle's driving status and the operating status of onboard equipment.

[0168] Furthermore, the head-mounted display terminal 1 may store data in which the display priority is linked to the video information.

[0169] For example, Video 1 is vehicle speed information for vehicle 11 and is assigned priority 1. Video 2 is vehicle gear information for vehicle 11 and is assigned priority 2. Both Video 1 and Video 2 belong to the category of video vehicle control information with priority 2, but if Video 1 and Video 2 overlap, Video 1, with priority 1, will be displayed preferentially. The priority here may be set for each video category, or it may be set for each video within the same video category.

[0170] Furthermore, as shown in Figure 24B, the head-mounted display terminal 1 may store data in its storage device 110 in which the display priority is associated only with video information, rather than with video categories.

[0171] A specific example of the process in S32 will be explained using Figures 25 and 26. [Example 3] Figure 25 is a diagram showing an example of the display content of the head-mounted display terminal 1 and the in-vehicle video display device 10 according to Example 3.

[0172] In S32, the head-mounted display terminal 1 adjusts the display content using both the head-mounted display terminal 1 and the in-vehicle video display device 10. It is desirable that the head-mounted display terminal 1 and the in-vehicle video display device 10 display content that is more suitable according to their respective characteristics.

[0173] For example, the in-vehicle video display device 10 has a stable communication state because it is directly connected to the vehicle 11, and there is no concern about battery depletion because power is supplied from the vehicle 11. On the other hand, the display area of ​​the in-vehicle video display device 10 is fixed to the vehicle 11, and the display range is narrower than the sensing area of ​​the vehicle 11, making it difficult to continuously superimpose the AR object onto the detected target even when people, cars, or obstacles are detected in the surrounding area.

[0174] In contrast, the head-mounted display terminal 1 has a display area that is not fixed in space and can follow the direction the user is looking, making it possible to display images in various directions. On the other hand, the head-mounted display terminal 1 may experience unstable communication and concerns about battery depletion.

[0175] As shown in the table in Figure 25, the head-mounted display terminal 1 may store data in the storage device 110 that pre-sets the display device to display for each type of information. This prevents changes in display format, such as the movement of display information between display devices due to changes in display conditions. Changes in display format include, for example, differences in resolution and color tone between display devices. This is because the head-mounted display terminal 1 and the in-vehicle video display device 10 are not limited to the same display specifications, and therefore the display resolution, display frequency, display color gamut, display brightness, etc., may differ, and the same display may not be possible on both devices.

[0176] Example 2 is safety-related information, and since it is information necessary for maintaining safety related to driving operations, it is desirable to display it on a head-mounted display terminal 1 that can always display the information within the driver's field of view. By displaying it on the head-mounted display terminal 1, it becomes possible to display the object detected in the sensing area of ​​the vehicle 11 in the correct position. [Example 4] Figure 26 is a diagram showing an example of processing related to the battery of the head-mounted display terminal 1 according to Example 4.

[0177] The head-mounted display terminal 1 either has its own battery or is powered by another device such as a smartphone. While the in-vehicle video display device 10 is always powered by the vehicle 11, the power supply for the head-mounted display terminal 1 is left to the user's choice, and there is no guarantee that it will be powered or that the battery level will be sufficient when in use. Therefore, depending on the power supply status of the head-mounted display terminal 1, it may be difficult to continue displaying information. Thus, the head-mounted display terminal 1 needs to monitor its battery status and decide whether or not to display information based on the remaining battery level and any changes in the remaining battery level.

[0178] In Example 1, when the battery of the head-mounted display terminal 1 is being powered by a smartphone or the like, the head-mounted display terminal 1 continues to display in cooperative display mode. If the head-mounted display terminal 1 is not being powered, it may reduce power consumption by ending the display of some or all of the display content. In this case, the head-mounted display terminal 1 may be controlled to display the content that has been ended on another device such as the in-vehicle video display device 10. Alternatively, if the head-mounted display terminal 1 is not being powered, it may end cooperative display mode and end the display.

[0179] In Example 2, if the battery level of the head-mounted display terminal 1 is above a specified value, the head-mounted display terminal 1 continues to display in cooperative display mode.

[0180] In Example 3, if the battery level of the head-mounted display terminal 1 falls below a specified value, the head-mounted display terminal 1 exits the collaborative display mode and ends the display. This prevents the battery of another device, such as a smartphone, from being completely drained and affecting the operation of that device.

[0181] In Example 4, if the battery level of the head-mounted display terminal 1 is between a specified value of 1 and a specified value of 2, the head-mounted display terminal 1 will terminate the display of some of the content. In this case, the head-mounted display terminal 1 may transmit information indicating the content that has been terminated to another device, such as the in-vehicle video display device 10, via the communication I / F 160, and control the other device to display it.

[0182] Furthermore, if the battery level of the head-mounted display terminal 1 falls below the specified value of 1, the head-mounted display terminal 1 will exit the coordinated display mode and cease displaying. This makes it possible to gradually inform the user that the battery level is decreasing and that continued display is difficult.

[0183] In Example 5, if the head-mounted display terminal 1 is being powered by a smartphone or the like, and the discharge rate is faster than the charging rate, the head-mounted display terminal 1 will terminate the coordinated display mode and end the display. Alternatively, if the head-mounted display terminal 1 is being powered by a smartphone or the like, and the discharge rate is faster than the charging rate, the head-mounted display terminal 1 may terminate the display of some of the display content. In this case, the head-mounted display terminal 1 may be controlled to display the terminated display content on another device, such as the in-vehicle video display device 10.

[0184] Although Figure 26 illustrates an example of monitoring the battery status of the head-mounted display terminal 1, the battery level of a smartphone or other device connected to the head-mounted display terminal 1 may also be monitored. For example, if the battery level of the smartphone connected to the head-mounted display terminal 1 falls below a specified value, the head-mounted display terminal 1 may terminate the display of some of its contents.

[0185] Next, an example of the display termination condition determination (S5) will be explained in detail with reference to Figure 27. Figure 27 is a diagram showing an example of the display termination condition determination process in S5 of Figure 9.

[0186] In Example 5-1, when the head-mounted display terminal 1 changes mode from cooperative display mode to normal display mode by user operation, it terminates the display in cooperative display mode and terminates the connection with the in-vehicle video display device 10 (5A). Example 5-1 is an example in which the mode change is performed by the user's own operation (operation on the settings screen, button operation, voice operation, gesture operation, etc.). The user may change to normal mode when handing over driving to someone else or ending driving.

[0187] In Example 5-2, the head-mounted display terminal 1 terminates the display in cooperative display mode and terminates the connection with the in-vehicle video display device 10 when a mode change is performed from cooperative display mode to normal display mode via communication with the vehicle 11 or the in-vehicle video display device 10 (5B). Example 5-2 is an example in which the head-mounted display terminal 1 terminates cooperative display mode when it receives a signal indicating a mode change via communication with the vehicle 11 or the in-vehicle video display device 10. The head-mounted display terminal 1 may obtain information from the vehicle 11 when it is connected to the vehicle 11. For example, the head-mounted display terminal 1 may obtain information from OBD (On Board Diagnostics). Alternatively, the head-mounted display terminal 1 may terminate cooperative display mode when it detects that the connection with the vehicle 11 has been lost.

[0188] In Example 5-3, when the head-mounted display terminal 1 detects the end of driving, it terminates the display in cooperative display mode and terminates the connection with the in-vehicle video display device 10 (5C). Example 5-3 is an example in which the end of driving of the vehicle 11 is determined by input from the camera and / or sensors of the head-mounted display terminal 1, without user input or communication with the vehicle 11. Here, the head-mounted display terminal 1 may use the rear camera 133 to detect that the user is not holding the steering wheel or that the vehicle 11 has stopped moving based on the scenery in the real world. The head-mounted display terminal 1 may also detect that movement has stopped using an acceleration sensor 154, a positioning sensor 151, etc. The head-mounted display terminal 1 may also use the rear camera 133 to detect the user's movement from the driver's seat or the user's movement from inside the vehicle.

[0189] In Example 5-4, when the head-mounted display terminal 1 detects that it has been removed, it terminates the display in coordinated display mode and terminates the connection with the in-vehicle video display device 10 (5D). Example 5-4 is an example in which the termination of coordinated display mode is determined when the user detects that they have removed the head-mounted display terminal 1 from themselves.

[0190] In Example 5-5, the head-mounted display terminal 1 terminates the display in coordinated display mode and terminates the connection with the in-vehicle video display device 10 (5E) when it confirms that the head-mounted display terminal 1 is not in the driver's seat. Example 5-5 is an example in which the termination of coordinated display mode is determined when it is confirmed that the head-mounted display terminal 1 is not in the driver's seat based on input from the camera and / or sensors of the head-mounted display terminal 1. The head-mounted display terminal 1 may also confirm that the head-mounted display terminal 1 is not in the driver's seat using a positioning sensor 151 or the like. The head-mounted display terminal 1 may also detect the user's movement from the driver's seat, the user's movement from inside the vehicle, etc., using the rear camera 133.

[0191] In Example 5-6, the head-mounted display terminal 1 terminates the display in cooperative display mode and terminates the connection with the in-vehicle video display device 10 (5G) if the windshield 2 cannot be detected within the field of view of the head-mounted display terminal 1. Example 5-6 is an example in which the head-mounted display terminal 1 determines to terminate cooperative display mode when it is confirmed that the windshield 2 is not within its field of view based on input from its camera and / or sensors. The head-mounted display terminal 1 may also confirm that it is not in the driver's seat using a positioning sensor 151 or the like. The head-mounted display terminal 1 may also detect the user's movement from the driver's seat or the user's movement from inside the vehicle using the rear camera 133. When switching from manual driving to automatic driving, for example, the user may not be looking ahead to the vehicle, such as when the driver rotates the seat backward while seated in the driver's seat. In such cases, the windshield 2, which should be visible while driving, is not detected within the field of view, and since the user is not performing driving tasks, cooperative display mode may be terminated. When the user resumes driving tasks, cooperative display mode may be restored.

[0192] In Example 5-7, the head-mounted display terminal 1 terminates the display in cooperative display mode and ends its connection with the in-vehicle video display device 10 when it detects that the user is not holding the steering wheel (5H) or when it detects that the vehicle 11 has entered autonomous driving mode (5J). Example 5-7 is an example in which the head-mounted display terminal 1 determines to terminate cooperative display mode when it confirms that the user is not holding the steering wheel based on input from its camera and / or sensors. Another example in which the head-mounted display terminal 1 determines to terminate cooperative display mode when it detects that the vehicle 11 is driving in autonomous driving mode based on communication with the vehicle 11 or input from its external camera 133 and / or sensors.

[0193] In Example 5-8, the head-mounted display terminal 1 terminates the display in coordinated display mode and disconnects from the in-vehicle video display device 10 when it confirms that the vehicle 11's engine has stopped (5L), the vehicle 11 has entered parking mode (5M), or the user has exited the vehicle 11 (5N). Example 5-8 is an example in which the head-mounted display terminal 1 determines to terminate the coordinated display mode when it confirms that the vehicle 11's engine has stopped, the vehicle 11 has entered parking mode, or the user has exited the vehicle 11, based on input from the camera and / or sensors of the head-mounted display terminal 1, or information from the vehicle 11.

[0194] In Example 5-9, the head-mounted display terminal 1 terminates the display in coordinated display mode and terminates the connection with the in-vehicle video display device 10 when it detects a malfunction in the head-mounted display terminal 1 (5P). Example 5-9 is an example in which the head-mounted display terminal 1 determines to terminate the driving mode when it detects that it cannot display normally due to a malfunction. The head-mounted display terminal 1 may, for example, continue displaying on the other display if only one of the displays malfunctions. The head-mounted display terminal 1 may, for example, have a glasses-type structure and may continue displaying on the other display that is not malfunctioning if it detects a malfunction in either the left or right display. The head-mounted display terminal 1 may also terminate the driving mode only if both displays malfunction. In Example 5-9, the head-mounted display terminal 1 may prompt the user to stop using the head-mounted display terminal 1 by displaying a message on its screen indicating a display abnormality or by emitting an audio message.

[0195] In Example 5-10, when the head-mounted display terminal 1 receives an emergency signal from the vehicle 11, it terminates the display in coordinated display mode and terminates its connection with the in-vehicle video display device 10 (5Q). Example 5-10 is an example of determining the termination of driving mode when the vehicle 11 transmits an emergency signal. If an emergency signal is received, continuing the display on the head-mounted display terminal 1 may hinder the transmission of information via the emergency signal. Therefore, it is desirable to terminate the driving mode or terminate the display itself.

[0196] Figures 28A and 28B show another example of video display by the head-mounted display terminal 1.

[0197] Figure 28A shows an example where the head-mounted display terminal 1 displays an overlaid display 801 on a monitored object, such as a bicycle crossing a road, which is sensed by the out-camera 133 and the sensor group 150.

[0198] Figure 28B shows a user wearing the head-mounted display terminal 1 with their face turned to the left to focus on the left side of the road. In this case, a bicycle that should be focused on crossing the road from the right is not within the display area of ​​the head-mounted display terminal 1, making it impossible to create the superimposed display 801 shown in Figure 28A.

[0199] In such cases, the head-mounted display terminal 1 may display an arrow 802 indicating the direction in which the monitored object is located. The head-mounted display terminal 1 may also display an icon 803 indicating the presence of the monitored object around the display area in the direction of the monitored object. The head-mounted display terminal 1 may display both the arrow 802 and the icon 803, or either one. This display indirectly informs the user of the presence of a monitored object outside the display area.

[0200] In this embodiment, an example was described in which the processor 101 of the head-mounted display terminal 1 adjusts the display area with the in-vehicle video display device 10 and performs the process of displaying in coordinated display mode. However, the controller 100 of the in-vehicle video display device 10 may also adjust the display area with the head-mounted display terminal 1 and perform the process of displaying in coordinated display mode.

[0201] Although embodiments have been described above, the present invention is not limited to the embodiments described above, and includes various modifications and equivalent configurations within the spirit of the attached claims. For example, the embodiments described above are described in detail for the purpose of clearly illustrating the present invention, and the present invention is not necessarily limited to having all the configurations described. Also, for example, some of the configurations of the embodiments may be added, deleted, or replaced with other configurations.

[0202] The above embodiments include the following forms: (Note 1) A head-mounted display terminal comprising a processor, a display, and a communication interface. The processor communicates with an information display device via the communication interface and controls the display and at least one of the video displays of the display and the information display device so that the video displayed on the display and the video displayed by the information display device do not overlap. (Note 2) A head-mounted display terminal comprising a processor, a display, a communication interface, and a memory. The processor communicates with an in-vehicle device installed in a vehicle via the communication interface, stores priority information regarding the priority of video displays between the in-vehicle device and the processor in the memory, and controls the video display of the display based on the priority information stored in the memory when the video displayed on the display and the video displayed by the in-vehicle device overlap within the field of view of the user wearing the head-mounted display terminal. (Note 3) A method for displaying information on a head-mounted display terminal, which communicates with an in-vehicle device installed in a vehicle, stores priority information regarding the priority of video display between the head-mounted display terminal and the in-vehicle device, and controls the video display of the head-mounted display terminal based on the stored priority information when the video displayed on the head-mounted display terminal and the video displayed by the in-vehicle device overlap within the field of view of the user wearing the head-mounted display terminal. (Note 4) An information processing system comprising a head-mounted display terminal and an in-vehicle device installed in a vehicle. The information processing system communicates between the head-mounted display terminal and the in-vehicle device and controls at least one video display of the head-mounted display terminal and the in-vehicle device so that the video displayed on the head-mounted display terminal and the video displayed by the in-vehicle device do not overlap.

[0203] 1: Head-mounted display terminal 101: Processor 110: Storage device 120: Input I / F 130: Video input / output device 131: Display 140: Audio input / output device 150: Sensor group 160: Communication I / F

Claims

1. A head-mounted display terminal comprising a processor, a display, and a communication interface, wherein the processor communicates with an information display device via the communication interface and controls at least one image display of the display and the information display device so that the image displayed on the display and the image displayed by the information display device do not overlap.

2. A head-mounted display terminal comprising: a processor; a display; a communication interface; and a memory, wherein the processor communicates with an in-vehicle device installed in a vehicle via the communication interface; the memory stores priority information relating to the priority of video display between the in-vehicle device and the processor; and when the video displayed on the display and the video displayed by the in-vehicle device overlap within the field of view of the user wearing the head-mounted display terminal, the head-mounted display terminal controls the video display on the display based on the priority information stored in the memory.

3. A head-mounted display terminal according to claim 2, wherein the processor terminates part or all of the video display on the display when the video displayed on the display and the video displayed by the in-vehicle device overlap within the user's field of view, and the priority of the head-mounted display terminal is set lower than the priority of the in-vehicle device according to the priority information stored in the memory.

4. A head-mounted display terminal according to claim 2, wherein the processor moves part or all of the image displayed on the display when the image displayed on the display and the image displayed by the in-vehicle device overlap within the user's field of view, and the priority of the head-mounted display terminal is set lower than the priority of the in-vehicle device according to the priority information stored in the memory.

5. A head-mounted display terminal according to claim 2, further comprising an input interface used by the user for operation input, wherein the processor modifies the priority information stored in the memory based on the user's operation input via the input interface.

6. A head-mounted display terminal according to claim 2, wherein the priority information is information relating to the priority set for the display and the video content displayed on the in-vehicle device, respectively.

7. A head-mounted display terminal according to claim 2, wherein the processor generates an image to be displayed on the display based on information acquired from the in-vehicle equipment via the communication interface, and superimposes the generated image onto a real object within a predetermined range from the vehicle.

8. A head-mounted display terminal according to claim 2, wherein the processor acquires information of a monitored object located within a predetermined range from the vehicle from the in-vehicle equipment via the communication interface, and if the monitored object is located within the display area of ​​the display, the monitored object is highlighted and displayed, and if the monitored object is located outside the display area of ​​the display, a display indicating that the monitored object is located outside the display area is provided.

9. A head-mounted display terminal according to claim 2, wherein the processor aligns the coordinate systems of the display area of ​​the in-vehicle device and the display area of ​​the display, and adjusts so that the display area of ​​the in-vehicle device and the display area of ​​the display do not overlap.

10. A head-mounted display terminal according to claim 9, wherein the processor acquires a calibration image showing the display area of ​​the in-vehicle device displayed by the in-vehicle device, and performs coordinate system alignment based on the calibration image.

11. A head-mounted display terminal according to claim 9, wherein the processor receives information relating to the display area of ​​the in-vehicle device from the in-vehicle device and performs coordinate system alignment based on the information relating to the display area.

12. A head-mounted display terminal according to claim 2, wherein the processor has the processes of acquiring the display area of ​​the in-vehicle device and performing calibration of the display space from the position of the display area of ​​the in-vehicle device within the display space of the head-mounted display terminal.

13. A head-mounted display terminal according to claim 12, further comprising a camera capable of capturing images of the surroundings, wherein the processor performs the process of acquiring the display area of ​​the in-vehicle device by capturing a calibration image displayed by the in-vehicle device with the camera.

14. A head-mounted display terminal according to claim 12, wherein the processor has a process of setting the display area position of the in-vehicle device within the display space of the head-mounted display terminal as a display prohibition area by performing the calibration process, and the display prohibition area is fixed in a coordinate system based on the vehicle and does not follow the orientation of the head-mounted display terminal.

15. A head-mounted display terminal according to claim 2, further comprising a battery, wherein the processor detects the state of the battery, terminates part or all of the video display on the display if the battery is not supplying power or if the remaining battery level is below a specified value, and transmits information indicating the terminated display content to the in-vehicle device via the communication interface.

Images