Electronic apparatus, determination method, and program

Abstract

This electronic apparatus has a control unit. The control unit detects a change in a position of an eyeball in an image captured by an imaging unit. The control unit calculates a VOR gain on the basis of vibration of a moving body detected by a vibration sensor and the change in the position of the eyeball. The control unit determines the degree of concentration by a person with the eyeball on the basis of the VOR gain.

Description

Electronic device, determination method, and program Cross-Reference to Related Application

[0001] The present application claims priority from Japanese Patent Application No. 2025-050724 filed in Japan on March 25, 2025, the entire disclosure of which prior application is incorporated herein by reference.

[0002] The present disclosure relates to an electronic device, a determination method, and a program.

[0003] Safe driving of a moving body requires concentration of the driver. Therefore, studies have been made to observe the driver's concentration, and when the concentration decreases, issue a warning to the driver or provide driving support. It has been proposed to observe the concentration based on the vestibulo-ocular reflex (see Patent Document 1).

[0004] Japanese Unexamined Patent Publication No. 2019-195376

[0005] An electronic device according to a first aspect comprises a control unit that measures a change in position of an eyeball in an image captured by an imaging unit, calculates a VOR gain based on vibration of a moving body detected by a vibration sensor and the change in position of the eyeball, and determines a concentration level of a holder of the eyeball based on the VOR gain.

[0006] A determination method according to a second aspect, which is performed by a computer, comprises: detecting vibration of a moving body; capturing an image of a front surface of a head of a driver seated in a driver's seat in front of the driver's seat of the moving body to generate an image; measuring a change in position of an eyeball in the image; calculating a VOR gain based on the vibration of the moving body and the change in position of the eyeball; and determining a concentration level of a holder of the eyeball based on the VOR gain.

[0007] A program according to a third aspect causes a computer to: measure a change in position of an eyeball in an image acquired from an imaging unit; calculate a VOR gain based on vibration of a moving body detected by a vibration sensor and the change in position of the eyeball; and determine a concentration level of a holder of the eyeball based on the VOR gain.

[0008] This is a block diagram illustrating the schematic configuration of an electronic device according to one embodiment. This is a layout diagram showing the electronic device of Figure 1 mounted in a specific orientation within a mobile body. This is a diagram showing the relationship between pupil movement and the eyeball for calculating the angular velocity of eye movement. This is a diagram showing the relationship between the rotation of the mobile body and the rotation of the head for calculating the angular velocity of head movement. This is a flowchart illustrating the calculation process performed by the electronic device of Figure 1 in one embodiment.

[0009] Hereinafter, embodiments of electronic devices to which this disclosure is applied will be described with reference to the drawings. The following description will also serve as a description of the determination method and program to which this disclosure is applied.

[0010] As shown in Figure 1, the electronic device 10 according to one embodiment is configured to include a control unit 11. The electronic device 10 may further include an imaging unit 12, a vibration sensor 13, an output unit (warning unit) 14, an input unit 15, and a storage unit 16.

[0011] The electronic device 10 may be a portable information terminal device such as a smartphone, tablet, notebook PC, or mobile device. The electronic device 10 may be mountable on a mobile body. In configurations where the electronic device 10 has an imaging unit 12 or a vibration sensor 13, it may be mountable on a mobile body so as to be fixed in a specific posture. The electronic device 10 may be detachably mounted on a mobile body by a holder provided on the mobile body. The electronic device 10 may be a car navigation system, TV, car audio equipment, etc.

[0012] Mobile entities may include, for example, vehicles, ships, and aircraft. Vehicles may include, for example, automobiles, industrial vehicles, railway vehicles, passenger vehicles, and fixed-wing aircraft that travel on runways. Automobiles may include, for example, passenger cars, trucks, buses, motorcycles, and trolleybuses. Industrial vehicles may include, for example, industrial vehicles for agriculture and construction. Industrial vehicles may include, for example, forklifts and golf carts. Industrial vehicles for agriculture may include, for example, tractors, cultivators, transplanters, binders, combines, and lawnmowers. Industrial vehicles for construction may include, for example, bulldozers, scrapers, excavators, cranes, dump trucks, and road rollers. Vehicles may include those that are powered by human effort. The classification of vehicles is not limited to the examples given above. For example, automobiles may include industrial vehicles that can travel on roads. The same vehicle may be included in multiple classifications. Ships may include, for example, jet skis, boats, and tankers. Aircraft may include, for example, fixed-wing aircraft, rotary-wing aircraft, etc.

[0013] The imaging unit 12 may generate an image as a signal by imaging. As shown in Figure 2, the imaging unit 12 can image the front of the head of a passenger ps seated in a seat st of the mobile body 17 while the electronic device 10 is mounted in a specific posture within the mobile body 17. The passenger ps may be the driver of the mobile body 17. The imaging unit 12 is, for example, a camera capable of imaging at a speed of 30 fps.

[0014] The vibration sensor 13 may detect vibrations. The vibration sensor 13 is fixed within the electronic device 10 and may be mounted so as to be fixed to the moving body 17 as described above, thereby detecting vibrations of the moving body 17. The vibration sensor 13 may be composed of, for example, a three-axis accelerometer or a three-axis gyroscope. The vibration sensor 13 may detect acceleration or angular velocity along each axis.

[0015] The output unit 14 may be controlled to issue a warning based on the concentration level, which will be described later. The output unit 14 may output the warning via video, audio, etc. The output unit 14 may be, for example, a display that outputs the warning as video, or a speaker that outputs the warning as audio. The display may be, for example, an LCD or an organic EL display. The output unit 14 may further output information to notify the user.

[0016] The input unit 15 may include at least one input interface for detecting user input. The input interface may be, for example, a physical key, a capacitive key, a pointing device, a touchscreen integrated with the display of the output unit 14, or a microphone.

[0017] The storage unit 16 may include semiconductor memory, magnetic memory, or optical memory. Examples of semiconductor memory include RAM (Random Access Memory) and ROM (Read Only Memory). Examples of RAM include SRAM (Static Random Access Memory) and DRAM (Dynamic Random Access Memory). Examples of ROM include EEPROM (Electrically Erasable Programmable Read Only Memory). The storage unit 16 may function as a main memory, auxiliary memory, or cache memory. The storage unit 16 may store data used in the operation of the electronic device 10 and data obtained by the operation of the electronic device 10. The memory unit 16 stores, for example, system programs, application programs, and embedded software.

[0018] The control unit 11 may be configured to include at least one processor, at least one dedicated circuit, or a combination thereof. The processor may be a general-purpose processor such as a CPU (Central Processing Unit) or a GPU (Graphics Processing Unit), or a dedicated processor specialized for a specific process. The dedicated circuit may be, for example, an FPGA (Field-Programmable Gate Array) or an ASIC (Application Specific Integrated Circuit). The control unit 11 may execute processes related to the operation of the electronic device 10 while controlling each part of the electronic device 10.

[0019] The control unit 11 measures the change in the position of the eyeball in the image being captured at the time of imaging. More specifically, the control unit 11 may measure the change in position using the pupil as the position of the eyeball. The change in the position of the eyeball is the change in the position of the eyeball between images captured at different times. Images captured at different times may be images captured consecutively in time. The control unit 11 may detect pixels in the image that contain an image of the eyeball as the position of the eyeball. If multiple pixels contain an image of the eyeball, the center of the multiple pixels containing the image of the eyeball may be identified as the position of the eyeball.

[0020] The control unit 11 calculates the VOR (vestibulo-ocular reflex) gain based on the vibration of the moving body 17 detected by the vibration sensor 13 and the change in the position of the eyeball in the image. The VOR gain is the ratio of the angular velocity of eye movement to the angular velocity of eye movement.

[0021] The control unit 11 may calculate the eye movement angular velocity ω1 based on the movement of the eyeball. As shown in Figure 3, the eye movement angular velocity ω1 is calculated using the first distance d1 in the height direction of the pupil pl in the entire eye images eye1 and eye2 of an arbitrary image and the next frame, and the second distance d2 from the rotation center c1 of a typical human eyeball to the surface of the eyeball, where ω1 = tan －１ It may be calculated approximately using (d1 / (d2×Δt)). The height direction may be the vertical direction. Δt is the image acquisition interval between any image and the next frame.

[0022] Furthermore, the control unit 11 may calculate the ideal eye movement angular velocity based on the vibration of the mobile body 17. The ideal eye movement angular velocity ω2 is the ideal value of the eye movement angular velocity and is the angular velocity in the opposite direction to the movement of the head. As shown in Figure 4, the head of the passenger ps on the mobile body 17 rotates in the opposite direction to the vibration of the mobile body 17 around the base of the neck bn within the mobile body 17. Based on this phenomenon, the control unit 11 may recognize the vibration in the opposite direction to the vibration of the mobile body 17 detected by the vibration sensor 13 as the movement of the head. Therefore, the control unit 11 may calculate the ideal eye movement angular velocity ω2 as the angular velocity of the rotation of the mobile body 17 around the left-right axis, based on the vibration of the mobile body 17 detected by the vibration sensor 13.

[0023] The relationship between the rotation of the occupant ps's head in the mobile body 17 and the vibration of the mobile body 17 within the mobile body 17 may be a relationship other than that shown in Figure 4. The relationship between the rotation of the occupant ps's head in the mobile body 17 and the vibration of the mobile body 17 within the mobile body 17 may be various depending on the occupant's body shape, the magnitude of the vibration of the mobile body, the magnitude of the acceleration of the vibration of the mobile body, and other factors. When the mobile body 17 vibrates downward, the occupant ps's head may move forward or backward. When the mobile body 17 vibrates to the right, the occupant ps's head may move to the left or to the right. When the mobile body 17 vibrates to the left, the occupant ps's head may move to the left or to the right. When the mobile body 17 vibrates forward, the occupant ps's head may move forward or backward. Furthermore, when the occupant's head moves, it may rotate around the neck, chin, back of the head, shoulders, top of the head, or other points such as the temples.

[0024] The control unit 11 may calculate the VOR gain by dividing the calculated eye movement angular velocity ω1 by the calculated ideal eye movement angular velocity ω2.

[0025] The control unit 11 determines the level of concentration of the eyeball holder, or in other words, the passenger ps, based on the calculated VOR gain. The control unit 11 may control the output unit 14 to issue a warning based on the level of concentration. For example, the control unit 11 may determine that a warning should be issued if the VOR gain continuously falls below a threshold value, such as 0.5, for a reference period of time, such as one minute.

[0026] Next, the determination process performed by the electronic device 10 in this embodiment will be explained using the flowchart in Figure 5. The determination process starts each time an image frame is acquired.

[0027] In step S100, the control unit 11 detects the movement of the pupil pl based on the newly acquired image. After detection, the process proceeds to step S101.

[0028] In step S101, the control unit 11 calculates the angular velocity of rotation of the moving body 17 about the left-right axis based on the vibration of the moving body 17 detected by the vibration sensor 13. After calculation, the process proceeds to step S102.

[0029] In step S102, the control unit 11 calculates the VOR gain based on the pupil pl movement detected in step S100 and the angular velocity calculated in step S101. After calculation, the process proceeds to step S103.

[0030] In step S103, the control unit 11 determines whether the VOR gain calculated in step S103 is less than a determination threshold. If it is not less than the determination threshold, the determination process ends. If it is less than the determination threshold, the process proceeds to step S104.

[0031] In step S104, the control unit 11 determines whether the reference time has elapsed while the VOR gain is below the determination threshold. If the reference time has not elapsed, the determination process ends. If the reference time has elapsed, the process proceeds to step S105.

[0032] In step S105, the control unit 11 controls the output unit 14 to output a warning indicating that the concentration level has decreased. After the output, the determination process ends.

[0033] The electronic device 10 of this embodiment, configured as described above, includes a control unit 11 that measures the change in the position of the eyeball in the image captured by the imaging unit 12, calculates a VOR gain based on the vibration of the moving body 17 detected by the vibration sensor 13 and the change in the position of the eyeball, and determines the degree of concentration of the eyeball holder based on the VOR gain. With this configuration, the electronic device 10 determines the degree of concentration based on the detection results of the imaging unit 12 and the vibration sensor 13, which can be mounted on the moving body 17, so that the degree of concentration based on the vestibulo-ocular reflex can be determined using a general-purpose device such as a smartphone.

[0034] Furthermore, the electronic device 10 of this embodiment further comprises at least one of the imaging unit 12 and the vibration sensor 13. With this configuration, the electronic device 10 can determine the degree of concentration using at least one of the imaging unit 12 and the vibration sensor 13 that a general-purpose electronic device 10 may have, without communicating with at least one of the imaging unit 12 and the vibration sensor 13 fixed to the mobile body 17. Therefore, the electronic device 10 can determine the degree of concentration even in a mobile body 17 that does not have at least one of the imaging unit 12 and the vibration sensor 13 mounted on it.

[0035] Furthermore, the electronic device 10 of this embodiment is further equipped with an output unit (warning unit) 14 that issues a warning based on the level of concentration. With this configuration, the electronic device 10 can notify the passenger ps when the level of concentration is low.

[0036] In one embodiment, (1) the electronic device includes a control unit that measures the change in the position of the eyeball in the image captured by the imaging unit, calculates a VOR gain based on the vibration of the moving body detected by the vibration sensor and the change in the position of the eyeball, and determines the degree of concentration of the person holding the eyeball based on the VOR gain.

[0037] (2) The electronic device described in (1) further comprises at least one of the imaging unit and the vibration sensor.

[0038] (3) In the electronic device described in (1) or (2) above, the vibration sensor can be mounted on the moving body in a manner that enables it to detect vibrations of the moving body.

[0039] (4) In the electronic device according to any one of (1) to (3) above, the imaging unit is mountable on the moving body so as to be capable of imaging the front of the head of a occupant seated in a seat of the moving body.

[0040] (5) The electronic device according to any one of (1) to (4) above is a smartphone, a tablet terminal, a notebook PC, or a mobile terminal that can be mounted on the moving body.

[0041] (6) The electronic device according to any one of (1) to (5) above further includes a warning unit that issues a warning based on the concentration level.

[0042] In one embodiment, (7) the determination method is a determination method executed by a computer, comprising: detecting vibration of a moving body; imaging the front of the head of a driver seated in a seat in front of the seat of the moving body to generate an image; measuring a change in position of an eyeball in the image; calculating a VOR gain based on the vibration of the moving body and the change in position of the eyeball; and determining the concentration level of the holder of the eyeball based on the VOR gain.

[0043] In one embodiment, (8) the program causes a computer to: measure a change in position of an eyeball in an image acquired from an imaging unit; calculate a VOR gain based on vibration of a moving body detected by a vibration sensor and the change in position of the eyeball; and determine the concentration level of the holder of the eyeball based on the VOR gain.

[0044] The embodiments of the electronic device 10 have been described above. As embodiments of the present disclosure, in addition to a method or a program for implementing the apparatus, embodiments can also be implemented as a storage medium having the program recorded thereon (for example, an optical disk, a magneto-optical disk, a CD-ROM, a CD-R, a CD-RW, a magnetic tape, a hard disk, a memory card, or the like).

[0045] Furthermore, the implementation form of the program is not limited to application programs such as object code compiled by a compiler and program code executed by an interpreter, and may be in the form of a program module or the like incorporated into an operating system. Further, the program may or may not be configured such that all processing is performed only by the CPU on the control board. The program may be configured such that part or all thereof is executed by another processing unit mounted on an expansion board or expansion unit attached to the board as necessary.

[0046] The diagrams for explaining the embodiments according to the present disclosure are schematic. The dimensional ratios and the like in the drawings do not necessarily match those in reality.

[0047] Although the embodiments according to the present disclosure have been described based on various drawings and examples, it should be noted that those skilled in the art can make various variations or modifications based on the present disclosure. Therefore, it should be noted that these variations or modifications are included in the scope of the present disclosure. For example, functions and the like included in each component or the like can be rearranged so as not to be logically inconsistent, and a plurality of components or the like can be combined into one or divided.

[0048] For example, in the present embodiment, the imaging unit 12 and the vibration sensor 13 are configured to be provided in the electronic device 10, but they may not be provided therein. For example, a configuration may be adopted in which the electronic device 10 acquires, as information, an image and the vibration of the moving body 17 through communication from each of the imaging unit and the vibration sensor mounted on the moving body 17. In this configuration, the imaging unit mounted on the moving body 17 is fixed so as to be capable of imaging the front of the head of a driver seated in the driver's seat of the moving body. Also, in this configuration, the vibration sensor is fixed to the moving body 17 so as to be capable of detecting vibration of the moving body 17.

[0049] All of the constituent elements described in this disclosure, and / or all of the disclosed methods or steps of processing, can be combined in any combination except for any combination in which these features are mutually exclusive. Furthermore, each of the features described in this disclosure can be replaced by an alternative feature that works for the same, equivalent, or similar purposes, unless expressly disregarded. Thus, unless expressly disregarded, each of the disclosed features is merely an example of a comprehensive set of identical or equivalent features.

[0050] Furthermore, the embodiments relating to this disclosure are not limited to any specific configuration of the embodiments described above. The embodiments relating to this disclosure can be extended to all novel features or combinations thereof described herein, or all novel methods or processing steps or combinations thereof described herein.

[0051] In this disclosure, the designations "First," "Second," etc., are identifiers used to distinguish the configurations. Configurations distinguished by the designations "First," "Second," etc., in this disclosure may have their numbers swapped. For example, the first interval may swap the identifiers "First" and "Second" with the second interval. The swapping of identifiers occurs simultaneously. The configurations remain distinguishable even after the swapping of identifiers. Identifiers may be deleted. Configurations from which identifiers have been deleted are distinguished by codes. The designations "First," "Second," etc., in this disclosure should not be used alone to interpret the order of the configurations or to justify the existence of smaller numbered identifiers.

[0052] 10 Electronic equipment 11 Control unit 12 Imaging unit 13 Vibration sensor 14 Output unit 15 Input unit 16 Memory unit bn Base of neck ps Passenger st Seat

Claims

1. An electronic device comprising: an imaging unit that measures the change in the position of the eyeball in the image it captures; a vibration sensor that calculates a VOR gain based on the vibration of a moving object and the change in the position of the eyeball; and a control unit that determines the degree of concentration of the person holding the eyeball based on the VOR gain.

2. An electronic device according to claim 1, further comprising at least one of the imaging unit and the vibration sensor.

3. The electronic device according to claim 1 or 2, wherein the vibration sensor is an electronic device that can be mounted on the moving body in a manner that enables it to detect vibrations of the moving body.

4. An electronic device according to any one of claims 1 to 3, wherein the imaging unit is an electronic device that can be mounted on the mobile body so as to be able to image the front of the head of a passenger seated in the seat of the mobile body.

5. An electronic device according to any one of claims 1 to 4, wherein the electronic device is a smartphone, tablet, notebook PC, or mobile terminal that can be mounted on the mobile body.

6. An electronic device according to any one of claims 1 to 5, further comprising a warning unit that issues a warning based on the concentration level.

7. A determination method performed by a computer, comprising: detecting vibrations of a moving body; capturing an image of the front of the head of a driver seated in the seat in front of the seat of the moving body and generating an image; measuring changes in the position of the eyeballs in the image; calculating a VOR gain based on the vibrations of the moving body and the changes in the position of the eyeballs; and determining the degree of concentration of the eyeball holder based on the VOR gain.

8. A program that causes a computer to measure the change in the position of the eyeball in an image acquired from an imaging unit, calculate a VOR gain based on the vibration of a moving object detected by a vibration sensor and the change in the position of the eyeball, and determine the degree of concentration of the person holding the eyeball based on the VOR gain.

Images