Head-mounted display device and light blocking control method for head-mounted display device

Abstract

This head-mounted display device comprises a display that transmits external light, a light blocking filter that blocks the external light, a first motion sensor that detects motion of a user of the head-mounted display device and a second motion sensor of a type different from that of the first motion sensor, and a processor. While a display object is displayed on the display, the motion level of the user is assessed by using both a first sensor output outputted from the first motion sensor and a second sensor output outputted from the second motion sensor, and the light blocking filter is controlled so as to be in a light blocking state corresponding to the motion level.

Description

Head-mounted display device and light-shielding control method for head-mounted display device

[0001] The present invention relates to a head-mounted display device and a light-shielding control method for the head-mounted display device.

[0002] As a head-mounted display device used by wearing on the head, a transmissive head-mounted display (HMD: Head Mounted Display) capable of visually recognizing the outside world is known. Regarding this transmissive HMD, Patent Document 1 discloses that "it includes a walking detection unit that detects the walking state of the user, and the control unit adjusts the transmittance of the area of the display unit based on the walking state of the user detected by the walking detection unit (extracted from the specification)".

[0003] Japanese Unexamined Patent Application Publication No. 2022-144057

[0004] According to Patent Document 1, it is possible to make it easier for a user wearing an HMD to confirm a display object and check the surroundings during walking. However, no consideration is given to the compatibility of confirming the display object and checking the surroundings during the transition movement from a lying or sitting position to a standing position as a preparatory movement for the walking movement, or during the movement of turning the face in the called direction without movement. In the future, if the HMD becomes more popular and the wearing scenarios of the HMD cover a wide range, fine control of the HMD according to various actions and action levels will be required, and the invention described in Patent Document 1 that only assumes the walking state cannot meet that requirement.

[0005] The present invention has been made in view of the above circumstances, and an object thereof is to provide a technique for ensuring the visibility of the surrounding environment while maintaining the visibility of the display object according to various actions performed by the user while wearing the head-mounted display device.

[0006] To solve the above problems, the present invention has the configuration described in the claims. For example, a head-mounted display device comprising a display that transmits ambient light, a light-shielding filter that blocks the ambient light, a first motion sensor and a second motion sensor of a different type from the first motion sensor that detect the movement of the user of the head-mounted display device, and a processor, wherein the processor determines the user's movement level by using in combination the first sensor output output by the first motion sensor and the second sensor output output by the second motion sensor while displaying a display object on the display, and controls the light-shielding state of the light-shielding filter in accordance with the movement level.

[0007] According to the present invention, it is possible to provide a technology that ensures the visibility of the surrounding environment while maintaining the visibility of the displayed object, in response to various actions performed by the user while wearing a head-mounted display device. Other objectives, configurations, and effects will be revealed in the following embodiments.

[0008] This is a schematic diagram of the HMD's appearance. This is a hardware configuration diagram of the HMD. This is a functional block diagram of the control unit. This is an explanatory diagram showing an example of pixel alignment processing between the display object of the display and the shading pattern of the shading filter. This is a diagram illustrating human movement. This is a diagram showing an example of an operation level definition table. This is a diagram showing the relationship between shading intensity and shading range corresponding to the operation level. This is a diagram showing the relationship between shading intensity and shading range corresponding to ambient light illuminance. This is a diagram showing example shading control 1 corresponding to the operation level. This is a diagram showing example shading control 2 corresponding to the operation level. This is a flowchart of the basic control flow of the HMD. This is a flowchart of the shading control (shading intensity control) flow corresponding to the operation level and ambient light illuminance. This is a flowchart of the shading control (shading range control) flow corresponding to the operation level and ambient light illuminance. This is a flowchart of the shading control flow corresponding to the operation level and the size (occupancy rate) of the display object. This is a flowchart of the shading control (shading intensity) flow corresponding to the operation level and gaze detection.

[0009] Embodiments of the present invention will be described below with reference to the drawings. Throughout the drawings, the same components and processes are denoted by the same reference numerals, and redundant explanations are omitted.

[0010] In this embodiment, an HMD is used as the head-mounted display device, but smart glasses, head-mounted projectors, VR helmets, head-mounted microdisplays, AR or VR compatible visors (AR: Augmented Reality, VR: Virtual Reality), etc. can also be used as head-mounted display devices.

[0011] Figure 1 is a schematic diagram of the external appearance of the HMD1 according to this embodiment.

[0012] As shown in Figure 1, the HMD 1 is equipped with an out-camera 110, an illuminance sensor 114, and a distance sensor 118 on the outer front of the frame 101. The HMD 1 is also equipped with a display 125, a light-shielding filter 150, a right-eye gaze sensor 112, and a left-eye gaze sensor 113 on the inner front of the frame 101. In this embodiment, the side facing the user wearing the HMD 1 is referred to as the inner side, and the side not facing the user is referred to as the outer side. Furthermore, the HMD 1 is equipped with a motion sensor group 140 on the left side of the frame 101 and a control unit 500 that controls the operation of the HMD 1 on the right side. The control unit 500 is composed of a processor 160 and a memory 170 (see Figure 2).

[0013] The HMD1 communicates with the input controller 400 via wireless or wired communication and operates by receiving operation input signals. The control unit 500 may also be configured to have its functions integrated into the input controller 400.

[0014] Figure 2 is a hardware configuration diagram of HMD1.

[0015] As shown in Figure 2, the HMD1 includes an out-camera 110, an in-camera 111, a right-eye gaze sensor 112, a left-eye gaze sensor 113, an illuminance sensor 114, a motion sensor group 140, a microphone 121, a speaker 122, a timer 123, an operation input interface (I / F) 124, a display 125, a network communication device 135 and an antenna 136, a light-shielding filter 150, a processor 160, and a memory 170. These components are connected to each other via a bus 199.

[0016] The rear camera 110 captures the view of the area in front of the HMD1, and acquires a camera image by converting light entering from the lens into an electrical signal using an image sensor.

[0017] The in-camera 111 captures the user's face, particularly their eyes, on the HMD 1. The right gaze sensor 112 and the left gaze sensor 113 detect the movement and direction of the right and left eyes, respectively, and output gaze information indicating the user's gaze direction. The process for detecting eye movement can utilize well-known techniques commonly used as eye-tracking processing. For example, a method using corneal reflection involves irradiating the face with an infrared LED (Light Emitting Diode) and capturing the image with an infrared camera. The position of the reflected light on the cornea (corneal reflection) created by the infrared LED irradiation is used as a reference point, and the movement of the eye and gaze direction are detected based on the position of the pupil relative to the position of the corneal reflection.

[0018] Timer 123 measures time, such as the time elapsed after detection of a stop in operation, or the time elapsed while an operation level is maintained. It also measures the dwell time of the user's gaze as detected by the right gaze sensor 112 and the left gaze sensor 113. Triggers for starting measurement include, for example, detecting a new operation level or detecting a new movement of the gaze. To avoid taking measurements every time there is even a slight movement, the trigger for starting measurement may be set to occur only when there is a gaze movement exceeding a predetermined amount.

[0019] An input controller 400 is connected to the operation input I / F 124. The input controller 400 is a device that provides operation instructions to the HMD 1, and consists of an operating component such as a keyboard or touch panel of a smartphone, PC, or tablet, and a device with communication functionality. In the example in Figure 1, the input controller 400 is connected by wire, but it may also be connected to the HMD 1 via a short-range wireless communication line to send and receive information. In the case of wireless communication, a wireless communication device is required. The input controller 400 may also be built into the HMD 1. In that case, for example, the HMD 1 may be equipped with an operating component such as a keyboard or touch panel as the input controller 400. Alternatively, for example, a selection of operation instructions may be displayed on the display 125, and operation instructions may be input according to the user's gaze or hand gestures.

[0020] The network communication device 135 is connected to the server 200 via the external network 300 and can send and receive data with the server 200. The server 200 may generate a display object and send the display object to the HMD1 via the external network 300. Alternatively, the HMD1 may be configured to generate and display the display object itself. The network communication device 135 is, for example, a mobile network, Wi-Fi (registered trademark), Bluetooth (registered trademark), lrDA (Infrared Data Association), Zigbee (registered trademark), HorneRF (Home Radio Frequency, registered trademark), or wireless LAN (IEEE 802.11a, IEEE 802.11b, IEEE 802.11g), LTE (Long Term Evolution), 4G (4th Generation Mobile Communication System), 5G (5th Generation Mobile Communication). It consists of a communication interface that conforms to the communication standards of the System.

[0021] The display 125 is a transmissive display. The display 125 is positioned in front of one or both eyes of the user wearing the HMD 1. External light that passes through the display 125 enters the user's eyes. The processor 160 also displays display objects on the display 125, so the display light also enters the user's eyes. As a result, the user sees an external view in which the displayed objects are superimposed on the real world.

[0022] A light-shielding filter 150 is positioned outside the display 125. The light-shielding filter 150 is a polarizing filter that can switch between a state in which ambient light is incident on a part of the light-shielding filter 150 and a state in which ambient light is blocked.

[0023] The processor 160 is configured, for example, using a CPU. The processor 160 loads and executes programs 180 stored in memory 170, specifically the operating system 181 and application programs 182, and reads information data 190 as needed for use in the execution process of the processor 160. As a result, the processor 160 functions as a data acquisition unit 161, a data processing unit 162, a display control unit 163, and a light-shielding control unit 164. The processor 160 displays display objects on the display 125 and partially controls the light-shielding filter 150 as needed.

[0024] The memory 170 is composed of, for example, ROM, RAM, etc., and stores various programs 180 and information data 190 handled by the processor, etc. The memory 170 also stores temporary information such as images acquired by the rear camera 110 and detection information.

[0025] The information data 190 includes light-shielding filter setting information 191, static state information 192, and operating level information 193. The light-shielding filter setting information 191 indicates parameters set to control the light-shielding filter, such as light-shielding intensity, light-shielding range, light-shielding position, and light-shielding change rate.

[0026] The motion sensor group 140 includes multiple sensors for detecting the user's momentum. In this embodiment, examples of motion sensors include an acceleration sensor 115, a gyro sensor 116, a geomagnetic sensor 117, a distance sensor 118, a GPS receiver 119, and a positioning sensor 120. In Figure 1, the distance sensor 118 is shown separately from the motion sensor group 140, but since the distance sensor 118 can measure the relative distance between the HMD 1 and an object in the outside world, it is also possible to calculate the movement of the HMD 1 from the change in that relative distance. Therefore, the distance sensor 118 can also be used as a motion sensor that constitutes the motion sensor group 140. In this embodiment, the first motion sensor and the second motion sensor for detecting the user's movement amount are configured by one or more combinations of multiple sensors of different types, such as the acceleration sensor 115, a gyro sensor 116, a geomagnetic sensor 117, a distance sensor 118, a positioning sensor (GPS receiver 119 and positioning sensor 120), and an out-camera 110 (corresponding to an outside-view camera). In this embodiment, a first motion sensor and a second motion sensor are provided, but the system is not limited to two motion sensors. Three or more motion sensors may be provided.

[0027] The motion sensor group 140 is a system that detects the orientation and rotational speed of the mounted head, and is classified into "3DoF" and "6DoF" depending on the movement and direction detected by the sensors. DoF is an abbreviation for "Degree of Freedom," which means "the degree of freedom that the sensor can detect." In the case of 3DoF, it means that there are three axes that can be detected: the X axis, Y axis, and Z axis. By detecting the up / down, left / right, and tilt axes of the head to which the HMD1 is attached, it is possible to recognize which direction the user is looking in: up / down, left / right, diagonally down / diagonally up. 6DoF is a system in which "the sensor has six degrees of freedom that can be detected," and in addition to head movement, it also detects "the wearer's own movement in the X, Y, and Z axis directions," thereby recognizing the user's "movement." Specifically, it becomes possible to recognize movement in the "forward / backward," "left / right," and "up / down" directions.

[0028] As a 3DoF sensor, sensors are used to detect head rotation and tilt. Specifically, a 3-axis accelerometer 115 measures acceleration to detect head tilt and movement, a gyroscope 116 measures angular velocity to detect rotational motion, and a geomagnetic sensor 117 uses the Earth's magnetic field to detect direction.

[0029] Furthermore, the 6DoF sensor uses sensors to detect not only 3DoF functionality but also positional movement (forward / backward, left / right, up / down). For example, the rear camera 110 may scan the surrounding environment and track the position, or the distance measuring sensor 118 may detect the distance to an object. The distance measuring sensor 118 may be a ToF sensor (LiDAR) using infrared light, or the rear camera 110 may be configured as a stereo camera capable of measuring distance. User movement, stillness, and lying down can be detected by utilizing the 6DoF sensor output.

[0030] Furthermore, the processor 160 may acquire heart rate and breathing patterns obtained from vital sensors, such as heart rate sensors and respiratory sensors, sensing the user of the HMD1. These vital sensor outputs may then be combined with the 6DoF sensor outputs to determine whether the user is stationary or not.

[0031] Figure 3 is a functional block diagram of the control unit 500. The data acquisition unit 161 acquires detection data from the illuminance sensor 114, motion sensor group 140, rear camera 100, front camera 111, right gaze sensor 112, left gaze sensor 113, and timer 123, and records it in the memory 170 as terminal information.

[0032] The data processing unit 162 reads terminal information from the memory 170 and performs processes such as determining the operating level. As an example of the process for determining the operating level, if the user of the HMD1 is on a train or bus, even if the user's movement is detected by the acceleration sensor 115, the system may use the rear camera 110 to photograph the user's feet in order to determine that the user is stationary. This makes it possible to determine that the user is stationary even if they are on a moving object. Alternatively, the system may use the GPS receiver 119 to detect the speed of movement of the location information and determine that the user is on board.

[0033] The display control unit 163 displays display objects on the display 125 and adjusts the brightness of the display 125.

[0034] The light-shielding control unit 164 adjusts the transmittance of the light-shielding filter 150. In doing so, the light-shielding control unit 164 can change the transmittance for each sub-region of the light-shielding filter 150. Furthermore, the light-shielding control unit 164 can also change the size of the sub-region. In order to shield only the area around a display object displayed on the display 125 using the light-shielding filter 150, according to its shape and size, it is necessary to associate the pixel positions of the display 125 with the pixel positions of the light-shielding filter 150. A known method may be used for the pixel alignment process between the display object on the display 125 and the light-shielding pattern of the light-shielding filter 150. For example, by associating the pixel positions of the display with the pixel positions of the light-shielding filter in the factory default settings, it is possible to change the light-shielding pattern of the light-shielding filter 150 in accordance with the display object. In the pixel alignment process, a grid pattern may be used, or a rectangle indicating the display area or a sample of the display object may be used as an adjustment pattern. Alternatively, if the size and display area of ​​the display object are fixed, a light-shielding pattern corresponding to that size and area can be stored in advance, and the light-shielding filter pattern can be programmed to change according to the operating level and then controlled.

[0035] Figure 4 is an explanatory diagram showing an example of pixel alignment processing between the display object of the display 125 and the light-shielding pattern of the light-shielding filter 150.

[0036] The data processing unit 162 causes the display control unit 163 to display the first grid pattern 125a on the display 125, as shown in Figure 4, and outputs the pixel positions p1(x1, y1), p2(x2, y2), ..., pn(xn, yn) (where n is the number of vertices in the first grid pattern) of each vertex to the light shielding control unit 164. The light shielding control unit 164 displays a second grid pattern 150a, which has the same shape as the first grid pattern 125a, on the light shielding filter 150. Initially, the first grid pattern 125a and the second grid pattern 150a are misaligned, as shown in the upper part of Figure 4.

[0037] Therefore, the data processing unit 162 shifts the display position of the second grid pattern 150a by the amount of arrow A so that the first grid pattern 125a and the second grid pattern 150a overlap, and adjusts them so that they coincide. As a result, data is formed in which the first grid pattern 125a and the second grid pattern 150a overlap, as shown in the middle of Figure 4. At this time, the data processing unit 162 outputs the pixel positions p1(s1, t1), p2(s2, t2), ..., pn(sn, tn) (where n is the number of vertices of the second grid pattern) of each vertex of the second grid pattern 150a in the light-shielding filter 150 to the light-shielding control unit 164, and the light-shielding control unit 164 holds these pixel positions.

[0038] The light-shielding control unit 164 generates position correspondence data 150b that associates each of the pixel positions p1(x1, y1), p2(x2, y2), ..., pn(xn, yn) at each vertex of the display 125 with each of the pixel positions p1(s1, t1), p2(s2, t2), ..., pn(sn, tn) at each vertex of the light-shielding filter 150, and stores it in the memory 170. Here, an example using a grid pattern for pixel alignment processing is shown, but a rectangle indicating the display area or a sample of a display object may be used as the adjustment pattern. The pixel alignment processing shown in Figure 4 is just one example, and the pixel alignment processing between the light-shielding filter 150 and the display 125 is not limited to this.

[0039] Figure 5 is a diagram illustrating human movement.

[0040] Basic human movements include lying down, turning over, sitting up, sitting up, standing up, standing, and walking. Therefore, as a preliminary step before setting the movement level, the types of static states and the movements that start from each static state are classified. In this embodiment, corresponding to this movement classification, the light-shielding state, such as the transmittance of the light-shielding filter 150 and the size of the light-shielding range, and the display state, such as the brightness of the display object displayed on the display 125, are controlled to maintain the visibility of the display object while ensuring the visibility of the surrounding environment. Hereinafter, this type of control will be referred to as visibility control. For example, the initial basic setting may be "viewing in a static state ⇒ visibility control is activated when a movement state is reached," and "viewing in a movement state ⇒ visibility control is further activated when the movement level changes" may be an advanced version of this basic setting. Note that visibility control is not limited to controlling both the light-shielding state and the display state; for example, it may be light-shielding control only.

[0041] Figure 6 shows an example of an motion level definition table (corresponding to motion level definition information). In Figure 6, Va represents the detected value of acceleration, Vg represents the detected value of head orientation and tilt, and Vp represents the detected value of distance traveled. TH represents the sensor detection threshold for determining the motion level; for example, THa1 represents the first threshold (TH1) in the acceleration sensor. In this example, a second threshold (TH2) that is larger than the first threshold (TH1) is set, and the determination is made using two threshold levels.

[0042] The motion level definition table in Figure 6 (corresponding to motion level definition information) defines four motion levels, A, B, C, and D, by combining the sensor outputs of multiple motion sensors. The motion levels are in the order of increasing motion: A, B, C, and D. Motion level A represents the highest motion level, such as starting to run. Motion level B represents the second highest motion level, such as starting to walk. Motion level C represents the third highest motion level, such as standing up from a seated position. Motion level D represents the lowest motion level, such as turning the head (face) or sitting up. In this example, four categories are used, but this is not the only option; the number of categories can be increased or decreased by changing the combination of sensors used or the number of thresholds.

[0043] Also, when an operation level has been detected previously, the visual recognition control may be changed according to a combination with the newly detected operation level.

[0044] Due to the control associated with a sudden change in the operation level, it is conceivable that the light-shielding state or the display state of the display object suddenly changes, which may affect the user. For example, if the light-shielding intensity suddenly changes from a low state to a high state, it suddenly becomes difficult to see the surrounding environment, and there is a risk that the user may feel that it is difficult to check the surroundings. Therefore, when the previously detected operation level is greater than the newly detected operation level, it is desirable to keep the light-shielding intensity corresponding to the previously detected operation level and switch after a predetermined time. For example, if operation level A has been detected previously and operation level C is newly detected, the light-shielding intensity corresponding to operation level A is maintained for a predetermined time and then switched to the light-shielding intensity corresponding to operation level C. Also, instead of switching from operation level A to operation level C, it may be changed step by step to the light-shielding intensity corresponding to operation level A, operation level B, and operation level C. Note that a low light-shielding intensity means a high transmittance of the light-shielding filter, and a high light-shielding intensity means a low transmittance of the light-shielding filter.

[0045] On the other hand, when switching the light-shielding intensity from a high state to a low state, rather than maintaining the light-shielding intensity of the previously detected operation level for a predetermined time and then changing it, or changing it step by step from the light-shielding intensity of the previously detected operation level, it is desirable to switch to the light-shielding intensity corresponding to the newly detected operation level as quickly as possible. However, it is not limited to this. Depending on the user, when the light-shielding intensity suddenly becomes low, it may become dazzling and actually more difficult to see. Therefore, according to the user's settings, it may be switched after a predetermined time or switched step by step.

[0046] In addition, when controlling using only the operation level definition table shown in FIG. 6, even if the user is riding on a train or the like and moving at high speed, the state is determined to be operation level A. However, it may not be appropriate to perform control assuming human movement. Therefore, in addition to control using the operation level definition table, it is desirable to set the light shielding state and the display state of display objects corresponding to specific situations so that the user can select them. For example, set modes of light shielding intensity and luminance of display objects appropriate for each situation, such as for automobiles, motorcycles, and train rides. By allowing the user to select a mode, regardless of the sensor output, the corresponding light shielding intensity and luminance may be fixed.

[0047] Also, even if the sensor output of the acceleration sensor 115 is below the threshold value, if the moving distance Vp calculated based on the change amount of the position data output by the GPS receiver 119 satisfies THp2 ≤ Vp, it may be determined that the vehicle is in a high-speed moving state. In the high-speed moving state, basically, priority is given to surrounding confirmation and the light shielding degree is reduced. However, if it is determined from the position data acquired by the GPS receiver 119 that the vehicle is moving on the route where the train passes, it is determined that the user is on the train, and without user mode selection, the light shielding intensity and luminance in the train ride mode may be changed. By pre-holding the threshold values for this determination (for example, if Va < THa2 and THp2 ≤ Vp, there is a possibility of a train) and the position data, automatic discrimination of the user state during high-speed movement can be performed.

[0048] Note that the initial stationary state is first set to any one of the stationary states (1) to (3) in FIG. 5. In this embodiment, since it is characterized in that control is performed in response to the start of operation, the setting of the visual recognition control in each stationary state is optional. For example, the visual recognition control in the stationary state may be divided or may be unified without division.

[0049] Figure 7A shows operation level-shading control information, which associates the relationship between shading intensity and shading range corresponding to the user's operation level. Figure 7B shows ambient light illuminance-shading control information, which associates the relationship between shading intensity and shading range corresponding to ambient light illuminance. This information is used when shading control is performed as a visibility control. Figures 7A and 7B do not mean that shading intensity and shading range change in pairs, but rather that each represents how control is performed in relation to the operation level and the magnitude of ambient light illuminance. The operation level-shading control information and ambient light illuminance-shading control information are stored in memory 170 as one of the shading filter setting information 191.

[0050] HMD1 may perform shading control based only on the operating level-shading control information, or only on the ambient light illuminance-shading control information, or it may perform shading control according to the operating level while shading control corresponding to ambient light illuminance is in operation. When shading control is performed using both the operating level and ambient light illuminance, shading control based on the operating level shall take precedence. Furthermore, control based on ambient light illuminance does not necessarily have to be ON. However, if it is desirable to maintain fine visibility and surrounding awareness in both the stationary and operating states, it is desirable to perform shading control using both the operating level and ambient light illuminance. In this case, the control processing of shading control based on the operating level and shading control based on ambient light shall be independent and not linked.

[0051] Figure 8 shows an example of light shielding control 1 corresponding to the operating level. This example is characterized by changing the transmittance of the light shielding filter depending on the operating level.

[0052] In the stationary state (2), the display object 600 is displayed in the center of the display area of ​​the display 125, and the light-shielding filter 150 is controlled so that the light-shielding area 601 surrounding the display object 600 has the lowest transmittance, i.e., the highest degree of light shielding. The light-shielding area 601 covers, in principle, the entire area and part of the display object. This can be arbitrarily changed by setting an area definition according to the operation level.

[0053] In this state, if the processor 160 of the HMD1 determines that the operating level has transitioned to operating level D based on the sensor output of the motion sensor, light shielding control corresponding to operating level D is performed. As a result, the transmittance of the light-shielding area 601 increases to the transmittance of the light-shielding area 602, and the user becomes able to see the outside world.

[0054] Furthermore, when the processor 160 determines, based on the sensor output of the motion sensor, that the operation level has transitioned from D to C, light shielding control corresponding to operation level C is performed. As a result, the transmittance of the light shielding range 603 increases further.

[0055] Furthermore, the rate at which the transmittance changes may be adjusted according to the relative value of the motion, for example, by relatively slowing down the rate at which the transmittance changes when the motion level is relatively low. Also, when the motion level changes, the transmittance may not be changed immediately, but after a predetermined time has elapsed. In addition, the transmittance may be changed in steps. For example, the processor 160 may maintain the light-shielding range 601 immediately after determining that the motion level is D, and then gradually increase the transmittance to that of the light-shielding range 602. By changing the transmittance in steps, discomfort caused by changes in visibility can be minimized. Since there are individual differences in the time required for the user's eyes to adapt to changes in light and dark, it is desirable that the time for changing the transmittance be set by the user via the operation input I / F 124.

[0056] Furthermore, the rate at which the transmittance changes may be adjusted according to the change in the activity level. For example, if it is determined that there has been a transition between multiple activity levels, such as from activity level D to activity level B, the transmittance may be changed in steps, and the rate of the stepwise change may be controlled to be faster than in the case of a single-level transition. This makes it possible to reduce the time it takes to adjust the transmittance to suit the current activity level while avoiding the discomfort of visibility caused by abrupt changes in transmittance.

[0057] Figure 9 shows an example of light shielding control 2 corresponding to the operating level. This example is characterized by changing the size of the light shielding area depending on the operating level.

[0058] In the stationary state (3), the display object 610 is displayed in the center of the display area of ​​the display 125, and the light-shielding filter 150 is controlled so that the light-shielding area 611 surrounding the display object 610 has the lowest transmittance, i.e., the highest degree of light shielding. The light-shielding area 611 generally covers the entire area and part of the display object. This can be arbitrarily changed by setting an area definition according to the operating level.

[0059] In this state, if the processor 160 determines that the operating level has transitioned to operating level B based on the sensor output of the operating sensor, light shielding control corresponding to operating level B is performed. That is, the area of ​​the light shielding range 612 is set to be smaller than that of the light shielding range 611. As a result, the area further outside the light shielding range 612 becomes the lowest level of light shielding that can be set on the HMD1, such as 100% transmittance, making it easier for the user to see the outside world.

[0060] Furthermore, when the processor 160 determines that the operation has transitioned from operation level B to operation level A based on the sensor output of the operation sensor, light shielding control corresponding to operation level A is performed. That is, the area of ​​the light shielding range 613 is set to be smaller than that of the light shielding range 612. As a result, the area around the display object 610 becomes the lowest level of light shielding, and the outside world becomes more widely visible. At this time, in order to prioritize checking the surroundings while ensuring the visibility of the display object, it is preferable that the light shielding range 613 be shaped to follow the display object and slightly smaller than the display object. In order to ensure the visibility of the display object, it is desirable that the area of ​​the light shielding range 613 be 80% or more of the display object area. However, if checking the surroundings is the highest priority, this is not limited to this, and the area of ​​the light shielding range 613 may be made almost disappear.

[0061] In this embodiment, the light-shielding areas 611, 612, and 613 are set to rectangles corresponding to the shape of the display object, but are not limited to this. The light-shielding areas 611, 612, and 613 can be any shape as long as the light-shielding area is set to overlap the back side of the display object in order to ensure the visibility of the display object, for example, they may be circular or elliptical.

[0062] Furthermore, if the surroundings can be checked, the light-shielding area may change in a gradient manner according to the operating level. "Gradient manner" means that the transmittance at the boundary between the light-shielding area and the area outside the light-shielding area in the display area of ​​the display 125 is continuously changed so that it becomes the transmittance of the area outside the light-shielding area. The transmittance may be changed in a gradient manner whether the light-shielding area is expanded or contracted. Moreover, during the process of changing the operating level, the size of the light-shielding area may be changed while the transmittance at the boundary between the light-shielding area and the area outside the light-shielding area is changed in a gradient manner. This eliminates areas in the display 125 where the transmittance changes abruptly, and minimizes the discomfort caused by changes in visibility.

[0063] Alternatively, the position of the display object can be set in advance, and the corresponding light-shielding area can be controlled by pattern control.

[0064] Furthermore, the rate at which the light-shielding area changes may be adjusted according to the relative value of the momentum, for example, by relatively slowing down the rate at which the light-shielding area is reduced when the momentum of the operating level is relatively low. In addition, when the operating level changes, the size of the light-shielding area may not be changed immediately, but rather after a predetermined time has elapsed, to change to the light-shielding area corresponding to the new operating level. Alternatively, the size of the light-shielding area may be changed in stages. For example, immediately after the processor 160 determines that the operating level is B, it maintains the light-shielding area 611 and gradually reduces it to the light-shielding area 612. By changing the size of the light-shielding area in stages, the discomfort caused by changes in visibility is minimized.

[0065] Furthermore, the rate at which the light-shielding range changes may be modified according to the change in the activity level. For example, if it is determined that there has been a transition between multiple activity levels, such as from activity level C to activity level A, the size of the light-shielding range may be changed in steps, and the rate of the stepwise change may be controlled to be faster than in the case of a single-level transition. This makes it possible to reduce the time it takes to adjust the light-shielding range to suit the current activity level while avoiding discomfort in visibility caused by abrupt changes in the light-shielding range. Since there are individual differences in the time it takes for the user's eyes to adapt to changes, it is desirable that the time it takes to change the size of the light-shielding range be set by the user via the operation input I / F 124.

[0066] Figure 10 is a flowchart showing the basic control flow of the HMD1. In the basic control flow, visual control is performed when the operating level changes from "viewing in a stationary state" to "entering an operating state". This flow starts, for example, when the execution of an application program that displays a display object is instructed. However, it is not limited to this, and this flow may also be started when the main power of the HMD1 is turned on. Alternatively, this flow may be set to start at a predetermined time or time interval based on schedule data stored in memory 170.

[0067] The display control unit 163 displays a display object on the display 125 (S101). Multiple motion sensor groups 140 detect sensor outputs (S102), and the data acquisition unit 161 temporarily stores these sensor outputs in the memory 170. The data processing unit 162 determines the current posture based on the multiple types of sensor outputs temporarily stored in the memory 170 (S103). In the basic control processing, a stationary state is detected as the user's current posture.

[0068] The data processing unit 162 compares the operating level-light shielding control information (Figure 7A) with the current posture determined in step S103, determines the light shielding control content (including control content for at least one of the light shielding intensity and light shielding range) (S104), and outputs the light shielding control content from the data processing unit 162 to the light shielding control unit 164. The light shielding control unit 164 controls the light shielding filter 150 according to the light shielding control content determined in step S104 (S105). In the basic control flow of Figure 10, it is assumed that the light shielding intensity corresponding to the stationary state is set as the operating condition immediately after the start of the initial setup. The light shielding intensity may be kept the same regardless of the type of stationary state.

[0069] When the user starts moving and the motion sensor group 140 senses further and detects a new sensor output (S106), the data processing unit 162 determines whether the user has started moving based on the new sensor output (S107). When the data processing unit 162 detects that the user has started moving (change) (S107: YES), it refers to the movement level definition table (Figure 5) and determines the movement level after the change (S108).

[0070] The data processing unit 162 refers to the operating level-light shielding control information (Figure 7A) and determines the light shielding control content that matches the changed operating level (S110). Here, if the light shielding control content is determined by considering both the operating level and the ambient light illuminance, in step S110, the light shielding control content of the light shielding filter 500 corresponding to the ambient light illuminance detected by the illuminance sensor 114 is determined by referring to the ambient light illuminance-light shielding control information (Figure 7B).

[0071] In step S111, the light-shielding control unit 164 performs light-shielding control (including at least one of adjusting the transmittance and adjusting the size of the light-shielding range) according to the operating level, and the display control unit 163 performs light emission control (brightness adjustment) of the display object. In light emission control, if the operating level is relatively high, the light emission of the display object is reduced. This improves the visibility of the surrounding environment. On the other hand, if the operating level is relatively low, the light emission of the display object is increased. When the operating level is relatively low, it is often a situation where the user can easily check the surrounding environment, so the light emission is controlled to be stronger in order to prioritize the visibility of the display object. After performing the control in step S111, the process returns to step S106, and the data processing unit 162 repeats the process.

[0072] Returning to step S107, if the data processing unit 162 does not detect the start (change) of user operation (S107: NO), the data processing unit 162 determines whether a state without operation start (change) has continued for a certain period of time (S109). If it is not possible to detect a period of maintaining a predetermined certain level of operation (S109: NO), it can be inferred that further changes in the level of operation will occur, that is, the user's operation is not stable, so the process returns to step S106 and is repeated.

[0073] On the other hand, if there is no operation (variation) for a certain period of time (S109: YES), the basic control flow is terminated while maintaining the visibility control at that operation level. Alternatively, the light-shielding state, such as the transmittance and the size of the light-shielding area when stationary, and the display state, such as the brightness of the display object, may be stored in memory 170 in advance, and the stored state may be restored after the basic control flow is terminated.

[0074] As mentioned above, sudden changes in the operating level may cause the shading state and the display state of the display object to switch abruptly, potentially affecting the user. Therefore, in step S111, the shading rate of the shading filter and the brightness of the display object may be controlled to change in stages. Also, frequent changes in the shading state and the display state of the display object may cause flickering and actually reduce visibility. Therefore, after executing the control in step S111, the control state may be maintained until a predetermined period has elapsed, and the shading rate, etc., may not be changed even if the user's operation start (change) is detected. Even after the basic control flow is completed, the shading state and the display state of the display object may be controlled to return to the stored state after a predetermined time has elapsed.

[0075] According to the basic control flow, when the user's activity level changes from a stationary state to an active state, light shielding control is performed according to the activity level. This allows for prioritizing the visibility of the surrounding environment over the visibility of the HMD1's display object in situations with high activity levels, such as sudden movements, and prioritizing the visibility of the display object while ensuring visibility of the surrounding environment in situations with low activity levels, such as slow movements.

[0076] Furthermore, in addition to light-blocking control, the luminescence of the display object can be changed to suppress the decrease in visibility of the display object that occurs with light-blocking control.

[0077] Furthermore, when adjusting the transmittance according to the operating level as a light shielding control, specifically, in step S110, the data processing unit 162 determines the light shielding intensity as a light shielding control content according to the operating level determined in step S108, and determines the transmittance of the light shielding filter. Then, in step S111, the light shielding control unit 164 controls the light shielding filter 150 according to the transmittance determined by the data processing unit 162. An example of changing the transmittance is as described above with reference to Figure 8.

[0078] Furthermore, when adjusting the shading range according to the operating level as part of the shading control, specifically, in step S110, the data processing unit 162 detects the area on the display 125 where the display object is displayed, and determines one or more of the position, size, and shape of the shading range as shading control content according to the operating level determined in step S108. Then, in step S111, the shading control unit 164 controls the shading filter 150 according to the position, size, and shape of the shading range determined by the data processing unit 162. An example of changing the shading range is shown in the above explanation with reference to Figure 9.

[0079] Furthermore, in step S111, when controlling the emission of light (brightness adjustment) of the display object, the display control unit 163 controls the emission intensity to decrease as the operating level increases, for example. This allows for prioritizing the visibility of the surrounding environment. On the other hand, when the operating level is low, it is easier to check the surroundings, so the emission intensity is controlled to increase as the operating level decreases. Note that in step S111 of Figure 10, the light shielding control and emission control are described as one step, but it is not necessary to perform both controls at the same time. For example, when the operating level changes in the direction of increasing, the transmittance may be increased first to prioritize checking the surroundings, and then the emission intensity may be increased with a slight delay.

[0080] Furthermore, depending on the operating level and other conditions, it is also possible to perform only one of the light-blocking control or light-emitting control, such as changing only the light-blocking rate. However, by controlling both light-blocking control and light-emitting control, such as increasing the transmittance of the light-blocking filter and decreasing the light-emitting intensity of the display object when the operating level is relatively high, and decreasing the transmittance and increasing the light-emitting intensity when the operating level is relatively low, it is possible to improve the visibility of the display object while ensuring the visibility of the surrounding environment.

[0081] Figure 11 is a flowchart showing the light-shielding control (light-shielding intensity control) flow corresponding to the operating level and ambient light intensity.

[0082] Similar to the basic control flow in Figure 10, after steps S101 to S105 are executed, the illuminance sensor 114 detects the ambient light illuminance (S120), and the data processing unit 162 determines the light shielding control content (light shielding intensity A) corresponding to the ambient light illuminance (S121). The light shielding control unit 164 controls the transmittance of the light shielding filter 150 to achieve the light shielding intensity determined by the data processing unit 162 (S122).

[0083] Next, steps S106 to S108 are executed, and once the operating level after the change is determined, in the example shown in Figure 11, the light-shielding intensity is controlled as visual control by steps S123 to S127. Specifically, the data processing unit 162 determines the light-shielding intensity B according to the operating level (S123). The data processing unit 162 compares the light-shielding intensity A determined in step S121 with the light-shielding intensity B (S124). If the light-shielding intensity A is greater than the light-shielding intensity B (S124: YES), the data processing unit 162 changes the setting of the light-shielding intensity B as the light-shielding control content (S125). The light-shielding control unit 164 controls the transmittance of the light-shielding filter 150 so that it corresponds to the transmittance of the light-shielding intensity B (S126).

[0084] On the other hand, if the light-shielding intensity A is less than or equal to the light-shielding intensity B (S124: NO), the transmittance of the light-shielding filter 150 is not changed, and the setting of light-shielding intensity A is maintained (S127). After steps S126 and S127, the process returns to step S106 and is repeated until the control flow is terminated in step S109. According to this example, visibility control can be performed by considering both the operating level and the ambient light intensity.

[0085] Figure 12 is a flowchart showing the shading control (shading range control) flow corresponding to the operating level and ambient light intensity.

[0086] Similar to the control flow in Figure 11, after steps S101 to S105 are executed, the illuminance sensor 114 detects the ambient light intensity (S120). Subsequently, the data processing unit 162 detects the area of ​​the display object (S130) and determines the shading control content (one or more of the position, size, and shape of the shading range A) according to the ambient light intensity (S131). The shading control unit 164 controls the shading filter 150 of the shading range A according to the shading control content determined in step S131 (S132).

[0087] Subsequently, steps S106 to S108 are executed, and once the operating level after the change is determined, in the example shown in Figure 12, the light-shielding range is controlled as visual control by steps S140 to S144. Specifically, the data processing unit 162 determines the light-shielding range B according to the operating level (S140), and compares the light-shielding range A determined in step S131 according to the ambient light intensity with the light-shielding range B determined in step S140 according to the operating level (S141).

[0088] If the data processing unit 162 determines that the light-shielding range A is larger (S141: YES), it changes the light-shielding control content to the setting of light-shielding range B (S142). The light-shielding control unit 164 controls the light-shielding filter 150 to change the position, size, and shape of the light-shielding range according to the setting of light-shielding range B (S143).

[0089] On the other hand, if the data processing unit 162 determines that the light-shielding range A is less than or equal to the light-shielding range B (S141: NO), it does not change the light-shielding range of the light-shielding filter and maintains the setting of light-shielding range A (S144). After steps S143 and S144, the process returns to step S106 and is repeated until the control flow ends in step S109.

[0090] In this example, by controlling the shading range while considering both the operating level and ambient light intensity, visibility control can be implemented to prioritize surrounding confirmation.

[0091] In step S126 in Figure 11 and steps S132 and S142 in Figure 12, when controlling the light shielding rate and light shielding range, the control may be performed to change them in stages, similar to the basic control flow, or the control state may be maintained after the control is executed until a predetermined period has elapsed, without changing the light shielding rate, etc. Examples of changing the transmittance and light shielding range are as described above using Figures 8 and 9.

[0092] Note that while Figures 11 and 12 only show shading intensity control and shading range control, respectively, as shading control, the system is not limited to these. For example, steps S121 to S127 of Figure 11 may be added to the control flow of Figure 12 to perform both shading intensity control and shading range control. Furthermore, when performing both shading intensity control and shading range control, one of the controls may be performed according to a newly detected operating level, rather than based on ambient light, similar to the control in the basic control flow. Also, while Figures 11 and 12 only show shading control as visibility control, the system is not limited to this. Similar to the basic control flow, in addition to shading control, display control of display objects such as brightness adjustment may also be performed.

[0093] Figure 13 is a flowchart showing the light-shielding control flow corresponding to the operating level and the size (occupancy rate) of the display object. In this example, either the light-shielding intensity or the light-shielding range is changed according to the size of the display object.

[0094] Similar to the basic control flow in Figure 10, steps S101 to S108 are executed to determine the operation level after the change. Then, the data processing unit 162 detects the area of ​​the display object (S150) and calculates the occupancy rate of the display object within the effective display area of ​​the display 125 (S151). If multiple display objects are displayed, the occupancy rate may be calculated based on their total display range, or it may be calculated based on the size of the largest display object.

[0095] If the occupancy rate is greater than a preset threshold (S152: YES), the data processing unit 162 determines the light-shielding intensity according to the operating level (S153). The light-shielding control unit 164 controls the light-shielding filter 150 so that the transmittance corresponds to the determined light-shielding intensity (S154).

[0096] If the occupancy rate is below a threshold (S152: NO), the data processing unit 162 determines one or more of the position, size, and shape of the light-shielding range according to the operating level (S155). The light-shielding control unit 164 controls the light-shielding filter 150 according to the determined light-shielding range (S156).

[0097] In this example, if the display object is displayed quite large (occupies a high percentage) within the effective display area of ​​the display 125, and it is determined that it may affect surrounding visibility depending on the operation level (size and speed of operation), the shading control according to the operation level is limited to changing only the shading intensity (transmittance). When the display object is large, if the shading range is adjusted to match the display object, the shading range becomes large. Therefore, by changing either the shading intensity or the shading range according to the size of the display object, it is possible to prevent a decrease in visibility of the outside world. However, this is not the only option; the shading intensity may be controlled according to the operation level regardless of the occupancy rate of the display object, and only the shading range may be switched on or off depending on the occupancy rate.

[0098] Figure 14 is a flowchart showing the light-shielding control (light-shielding intensity) flow corresponding to the activity level and gaze detection. In this example, when the user's gaze is detected and it is determined that the user is gazing at the effective display area of ​​the display 125 for a certain period of time, in addition to the control according to the activity level, control according to the area being gazed at is performed.

[0099] Similar to the basic control flow in Figure 10, steps S101 to S108 are executed to determine the operating level after the change. Then, the data processing unit 162 determines the light-shielding intensity according to the operating level (S160), and the light-shielding control unit 164 controls the light-shielding filter 150 so that the transmittance corresponds to the determined light-shielding intensity (S161). The right gaze sensor 112 and the left gaze sensor 113 then detect the user's gaze (S162), and the data processing unit 162 determines whether the user is gazing at the area where the display object is displayed (hereinafter referred to as the display object area) for a certain period of time (S163). If the data processing unit 162 determines that the user is gazing at the display object area for a certain period of time (S163: YES), the light-shielding control unit 164 sets a new light-shielding intensity that is stronger than the light-shielding intensity determined in step S160, and controls the light-shielding filter 150 to lower its light-shielding rate (S164). This allows for prioritizing the visibility of the display object.

[0100] On the other hand, if the data processing unit 162 determines that the user is not gazing at the display object area for a certain period of time (S163: NO), it determines whether the user is gazing at an area other than the display object (hereinafter referred to as the background area), such as the background, for a certain period of time (S165). If the data processing unit 162 determines that the user is gazing at the background area for a certain period of time (S165: YES), the light-shielding control unit 164 sets a new light-shielding intensity that is lower than the light-shielding intensity determined in step S160, and controls the light-shielding filter 150 to lower its light-shielding intensity (S166). This allows for prioritizing the visibility of the surrounding environment, such as the background. If the data processing unit 162 determines that the user is not gazing at the background area for a certain period of time (S165: NO), the system returns to step S106 without changing the transmittance, while maintaining the light-shielding intensity determined in step S160.

[0101] Furthermore, the constant time in steps S163 and S165 is not limited to a fixed time. Instead, gaze time information indicating the gaze time corresponding to the operating level or light-shielding intensity may be stored in the memory 170, and the threshold time used to determine whether the user is gazing may be changed according to the operating level determined in step S108 or the light-shielding intensity determined in step S160. Alternatively, different times may be used as the constant time in steps S163 and S165.

[0102] Furthermore, in the example shown in Figure 14, the shading intensity is controlled as shading control corresponding to the operating level and gaze detection, but the shading range may be controlled instead of the shading intensity. In this case, after determining the operating level after the change in step S108, instead of steps S160 to S161, the data processing unit 162 detects the display object area and determines one or more of the position, size, and shape of the shading range according to the operating level and the display object area, and the shading control unit 164 controls the shading filter 150 according to the shading range determined by the data processing unit 162. After that, steps S162 to S163 are executed, and if the data processing unit 162 determines that the user is gazing at the display object area for a certain period of time (S163: YES), instead of step S164, the shading control unit 164 sets a new shading intensity that is larger than the shading range determined before step S162 and controls the shading filter 150. This makes it possible to prioritize the visibility of the display object.

[0103] On the other hand, if the data processing unit 162 determines that the user is not gazing at the display object area for a certain period of time (S163: NO), but is gazing at the background area for a certain period of time (S165: YES), the light-shielding control unit 164 sets a new light-shielding intensity that is smaller than the light-shielding range determined before step S162, and controls the light-shielding filter 150. This allows for prioritizing the visibility of the surrounding environment.

[0104] While embodiments of the present invention have been described above, it goes without saying that the configurations for realizing the technology of the present invention are not limited to the above embodiments, and various modifications are conceivable. For example, the embodiments described above are described in detail for the purpose of explaining the present invention in an easy-to-understand manner, and are not necessarily limited to those comprising all the described configurations. Furthermore, it is possible to replace a part of the configuration of one embodiment with the configuration of another embodiment, and it is also possible to add the configuration of another embodiment to the configuration of one embodiment. All of these fall within the scope of the present invention. In addition, the numbers and messages that appear in the text and figures are merely examples, and using different ones will not impair the effects of the present invention.

[0105] Furthermore, the programs described in each processing example may be independent programs, or multiple programs may constitute a single application program. The order in which each processing step is performed may also be changed. Additionally, some processing steps may be executed in parallel. For example, in the flows shown in Figures 11 and 12, the sensor output detection for attitude determination in step S102 and the ambient light illuminance detection in step S104 may be performed in parallel, and the shading control content may be determined in steps S121 and S131 based on this detection data.

[0106] The functions of the present invention described above may be implemented in hardware, in whole or in part, by designing them, for example, using integrated circuits, general-purpose processors, or application-specific processors. A processor includes transistors and other circuits and is considered a circuit or processing circuit. Alternatively, the functions may be implemented in software by a microprocessor unit, CPU, etc., interpreting and executing an operating program that realizes each function. Furthermore, the scope of software implementation is not limited, and hardware and software may be used in combination. In addition, some or all of each function may be implemented by a server. The server only needs to be able to perform functions in cooperation with other components via communication, and its form is not limited to, for example, a local server, cloud server, edge server, or network service. Information such as programs, tables, and files that realize each function may be stored in memory, a recording device such as a hard disk or SSD (Solid State Drive), or a recording medium such as an IC card, SD card, or DVD, or it may be stored in a device on a communication network.

[0107] Furthermore, the control lines and information lines shown in the diagram are those deemed necessary for explanation and do not necessarily represent all control lines and information lines on the product. In reality, it can be assumed that almost all components are interconnected.

[0108] The above embodiment includes the following invention: (Note 1) A head-mounted display device comprising: a display that transmits ambient light; a light-shielding filter that blocks the ambient light; a first motion sensor and a second motion sensor of a different type from the first motion sensor that detect the movement of the user of the head-mounted display device; and a processor, wherein the processor determines the user's movement level by using in combination the first sensor output output by the first motion sensor and the second sensor output output by the second motion sensor while a display object is displayed on the display, and controls the light-shielding state of the light-shielding filter in accordance with the movement level.

[0109] (Note 2) A method for controlling light shielding of a head-mounted display device, comprising: a step in which the processor of the head-mounted display device determines the user's movement level by using in combination a first sensor output output by a first motion sensor that detects the user's movement and a second sensor output output by a second motion sensor that detects the user's movement while a display object is displayed on a display that transmits ambient light; and a step in which the light shielding state of a light shielding filter that shields ambient light corresponds to the movement level.

[0110] 1: HMD, 101: Frame, 110: Rear camera, 111: Front camera, 112: Right gaze sensor, 113: Left gaze sensor, 114: Illuminance sensor, 115: Accelerometer, 116: Gyroscope, 117: Geomagnetic sensor, 118: Distance sensor, 119: GPS receiver, 120: Positioning sensor, 121: Microphone, 122: Speaker, 123: Timer, 124: Operation input I / F, 125: Display, 125a: First grid pattern, 135: Network communicator, 136: Antenna, 140: Motion sensor group, 150: Light-shielding filter, 150a: Second grid pattern, 150b: Position-corresponding data, 160: Pro 161: Data acquisition unit, 162: Data processing unit, 163: Display control unit, 164: Light shielding control unit, 170: Memory, 180: Program, 181: Operating system, 182: Application program, 190: Information data, 191: Light shielding filter setting information, 192: Static state information, 193: Operation level information, 199: Bus, 200: Server, 300: External network, 400: Input controller, 500: Control unit, 600: Display object, 601: Light shielding range, 602: Light shielding range, 603: Light shielding range, 610: Display object, 611: Light shielding range, 612: Light shielding range, 613: Light shielding range

Claims

1. A head-mounted display device comprising: a display that transmits ambient light; a light-shielding filter that blocks the ambient light; a first motion sensor and a second motion sensor of a different type from the first motion sensor that detect the movement of the user of the head-mounted display device; and a processor, wherein the processor determines the user's movement level by using in combination the first sensor output output by the first motion sensor and the second sensor output output by the second motion sensor while a display object is displayed on the display, and controls the light-shielding state of the light-shielding filter in accordance with the movement level.

2. A head-mounted display device according to claim 1, wherein the processor refers to operation level definition information that defines a plurality of operation levels with different amounts of user movement, determines the user's operation level based on the value of the first sensor output and the value of the second sensor output, and controls the light-shielding state according to the determined operation level.

3. A head-mounted display device according to claim 2, wherein the light-shielding state is the light-shielding intensity of the light-shielding filter, and the processor determines that the user's activity level is at a relatively high operating level, in which case it relatively lowers the light-shielding intensity to increase the visibility of the ambient light, and determines that the user's activity level is at a relatively low operating level, in which case it relatively raises the light-shielding intensity to reduce the visibility of the ambient light.

4. A head-mounted display device according to claim 1, wherein the processor controls the light-shielding state by changing the transmittance of the light-shielding filter.

5. A head-mounted display device according to claim 4, wherein the processor changes the rate of change of the transmittance according to the relative value of the momentum of the operating level.

6. A head-mounted display device according to claim 5, wherein the processor relatively slows down the rate of change of the transmittance when the amount of motion of the operating level is relatively low.

7. A head-mounted display device according to claim 1, wherein the processor controls the light-shielding state by changing the size of the light-shielding range that is shielded by the light-shielding filter in the display area of ​​the display.

8. A head-mounted display device according to claim 7, wherein the processor relatively slows down the speed at which the light-shielding range is reduced when the amount of motion of the operating level is relatively low.

9. A head-mounted display device according to claim 7, wherein the processor expands or shrinks the light-shielding range by continuously changing the transmittance of the boundary of the light-shielding range to the transmittance of the peripheral area of ​​the light-shielding range in the display area of ​​the display.

10. A head-mounted display device according to claim 1, further comprising an illuminance sensor for detecting ambient light intensity, wherein the processor controls the shading state by the shading filter by using in combination the user's operating level determined based on the value of the first sensor output and the value of the second sensor output, and the ambient light intensity.

11. A head-mounted display device according to claim 1, further comprising a gaze sensor for detecting the gaze of a user, wherein the light-shielding state is the light-shielding intensity of the light-shielding filter, and the processor, based on the gaze information of the user detected by the gaze sensor, relatively increases the light-shielding intensity when it determines that the user is viewing a display object displayed on the display, and relatively decreases the light-shielding intensity when it determines that the user is viewing a background area in the display area of ​​the display where the display object is not displayed.

12. A head-mounted display device according to claim 1, wherein the light-shielding state is the light-shielding intensity of the light-shielding filter, and the processor calculates the amount of change in the operating level determined based on the first sensor output and the second sensor output detected this time, relative to the operating level determined based on the first sensor output and the second sensor output detected last time, and controls the rate of change of the light-shielding intensity according to the amount of change in the operating level.

13. A head-mounted display device according to claim 12, wherein the processor changes the light-shielding intensity in steps when the amount of change in the operating level corresponds to a change of multiple levels, and makes the rate of change of the light-shielding intensity faster than the rate of change when the amount of change in the operating level corresponds to a change of one level.

14. A head-mounted display device according to claim 1, wherein the processor controls the light-shielding state of the light-shielding filter in accordance with the operating level and adjusts the brightness of the display object.

15. A method for controlling light shielding of a head-mounted display device, comprising: a step of determining the user's movement level by using in combination a first sensor output output by a first motion sensor that detects the user's movement and a second sensor output output by a second motion sensor that detects the user's movement, while the processor of the head-mounted display device is displaying a display object on a display that transmits ambient light; and a step of controlling the light shielding state of a light shielding filter that shields ambient light in accordance with the movement level.

Images