Display device and electronic apparatus

Abstract

The present invention is a display device capable of calculating an orientation with high accuracy even when there are a plurality of causes of errors.　The display device comprises: a display unit that displays image data on the basis of an imaging result obtained by an imaging unit; and a display control unit that controls the display unit. The display control unit includes: an estimation unit (27) that estimates the orientation or an orientation change amount of the imaging unit on the basis of information from a detection unit that detects the orientation or an orientation change of the imaging unit; and an identification unit (37) that identifies the cause of an error occurring in the orientation or orientation change amount estimated by the estimation unit, on the basis of the difference between the orientation or orientation change amount of the imaging unit, which are detected on the basis of the imaging result obtained by the imaging unit, and the orientation or orientation change amount estimated by the estimation unit.

Description

Display devices and electronic devices

[0001] This disclosure relates to display devices and electronic equipment.

[0002] Traditionally, smart glasses and other devices with Augmented Reality (AR) functionality have utilized odometry technology. Odometry technology allows, for example, the user's viewpoint changes to be reflected in digital content.

[0003] Odometry techniques include, for example, visual odometry, which analyzes images captured by cameras to determine the orientation of the imaging device, and inertial odometry, which uses an inertial measurement unit (IMU) to measure rotational speed, etc. However, visual odometry is known to consume a lot of power, and inertial odometry is known to have large errors due to integral calculations (e.g., gyrodrift).

[0004] Furthermore, visual inertial odometry (VIO), which combines visual odometry and inertial odometry, has been proposed. For example, a method has been proposed that can achieve low power consumption and high accuracy odometry by correcting high-rate IMU data with low-rate image data (see, for example, Patent Document 1).

[0005] Japanese Patent Publication No. 2024-075553

[0006] However, the method described in Patent Document 1 can only correct errors in the IMU data. Correction of other errors that affect the attitude calculation of the imaging device is not disclosed.

[0007] Therefore, this disclosure provides a display device that can perform posture calculations with high accuracy even when there are multiple sources of error.

[0008] To solve the above problems, the present disclosure provides a display device comprising: a display unit that displays image data based on imaging results from an imaging unit; and a display control unit that controls the display unit, wherein the display control unit includes: an estimation unit that estimates the posture or amount of posture change of the imaging unit based on information from a detection unit that detects the posture or change in posture of the imaging unit; and an identification unit that identifies the cause of an error in the posture or amount of posture change estimated by the estimation unit based on the difference between the posture or amount of posture change of the imaging unit detected based on imaging results from the imaging unit and the posture or amount of posture change estimated by the estimation unit.

[0009] The aforementioned identifying unit may identify the cause of multiple errors that occurred at different timings.

[0010] The identifying unit may, in addition to the difference, identify the cause of the multiple errors based on at least one of the gyro sensor drift error or the geomagnetic disturbance error.

[0011] The system may further include a correction unit that corrects the plurality of errors based on at least one of the drift error or the geomagnetic disturbance error.

[0012] The correction unit may correct the attitude or attitude change amount obtained from the gyro sensor if the plurality of errors include the drift error of the gyro sensor, or correct the gain for the attitude obtained from the geomagnetic sensor if the plurality of errors do not include the drift error of the gyro sensor.

[0013] The identifying unit may identify the cause of the error in the posture or posture change amount estimated by the estimation unit, and the cause of the error in the posture or posture change amount detected based on the image data.

[0014] The identification unit may identify the cause of the error based on a correction parameter that corrects the posture or posture change amount detected based on the image data and the posture or posture change amount estimated by the estimation unit, and a correction formula that calculates the correction amount of the correction parameter.

[0015] The error in the attitude or attitude change amount estimated by the estimation unit includes at least one of a mechanical position error of the gyro sensor, a mechanical position error of the geomagnetic sensor, a drift error of the gyro sensor, an error in geomagnetic disturbance, a sampling time error of the gyro sensor, or a sampling time error of the geomagnetic sensor. The error detected based on the image data may include at least one of a mechanical position error of the imaging unit, a lens distortion error of the imaging unit, or a sampling time error of the imaging unit.

[0016] Further, according to the present disclosure, a display unit that displays image data based on an imaging result by an imaging unit, an acquisition unit that acquires the attitude or attitude change amount of the imaging unit detected based on the imaging result by the imaging unit when a predetermined condition is satisfied, a correction unit that corrects information from a detection unit that detects the attitude or attitude change of the imaging unit based on the attitude or attitude change amount acquired by the acquisition unit, and a determination unit that determines whether or not the predetermined condition is satisfied are provided. A display device is provided.

[0017] The determination unit may determine whether or not the predetermined condition is satisfied based on any one of the position, attitude, or captured image data of the imaging unit.

[0018] A position specifying unit that specifies the current position of the imaging unit, a movement determination unit that determines whether or not the current position specified by the position specifying unit has moved by a predetermined threshold or more, and an imaging condition acquisition unit that acquires imaging conditions corresponding to the current position specified by the position specifying unit when it is determined that the movement is by the predetermined threshold or more are provided. The determination unit may determine whether or not the predetermined condition is satisfied based on the imaging conditions acquired by the imaging condition acquisition unit.

[0019] The imaging conditions include information regarding the direction in which the imaging unit should perform imaging. The predetermined condition may include that the imaging unit has reached an attitude that enables imaging in the direction included in the imaging conditions.

[0020] The predetermined condition may include that the amount of change from the attitude or attitude change amount most recently detected by the detection unit is a predetermined threshold or more.

[0021] The acquisition unit acquires information from an arithmetic unit that calculates the posture or the amount of change in posture of the imaging unit based on the imaging result of the imaging unit, and the predetermined condition may include that the imaging unit can image a subject suitable for the arithmetic unit to calculate the posture or the amount of change in posture.

[0022] An extraction unit that extracts a subject by segmentation from image data including one or more subjects is provided, and the predetermined condition may include that the extraction unit can extract a subject suitable for the arithmetic unit to acquire the posture or the amount of change in posture from the image data including the one or more subjects.

[0023] The predetermined condition may include at least one of the remaining battery level, the heat generation condition, or the calculation margin.

[0024] The specifying unit specifies the cause of the error based on the posture or the amount of change in posture of the imaging unit detected based on the image data captured by the imaging unit, and the display unit may display the image data captured by the imaging unit.

[0025] The specifying unit specifies the cause of the error based on the posture or the amount of change in posture of the imaging unit detected based on the first image data captured by the imaging unit, and the display unit may display second image data whose display position is adjusted based on the posture or the amount of change in posture of the imaging unit detected based on the first image data.

[0026] Further, according to the present disclosure, an electronic device is provided that includes a display device having a display unit, a control device, an imaging unit that performs imaging for displaying image data on the display unit, a detection unit that detects the posture or the change in posture of the imaging unit, an estimation unit that estimates the amount of change in the posture or the posture of the imaging unit based on the information from the detection unit, and a specifying unit that specifies the cause of an error in the amount of change in the posture or the posture estimated by the estimation unit based on the difference between the amount of change in the posture or the posture of the imaging unit detected based on the image data and the amount of change in the posture or the posture estimated by the estimation unit.

[0027] The display device has a display control unit that controls the display unit, and the control unit or the display control unit may perform processing on at least one of the estimation unit and the identification unit.

[0028] A block diagram showing a first configuration example of an electronic device according to the first embodiment of this disclosure. A diagram illustrating VPS processing. A diagram illustrating local VPS processing. A diagram illustrating AR image data generated by the application unit. A block diagram showing a second configuration example of an electronic device according to the first embodiment of this disclosure. A block diagram showing a detailed configuration of the display control unit, etc., according to the first embodiment of this disclosure. A block diagram showing a more detailed configuration of the VPS processing unit and error correction unit of Figure 6. A flowchart showing the operation of the local VPS timing generation unit. A flowchart showing a first example of the operation of the display control unit or application unit. A flowchart showing a second example of the operation of the display control unit or application unit. A flowchart showing a method for setting the Jacobian matrix and attitude calculation formula. A first diagram illustrating the timing at which VPS processing is effective. A second diagram illustrating the timing at which VPS processing is effective. A block diagram showing a detailed configuration of the display control unit, etc., according to the second embodiment of this disclosure. A diagram showing an imaging condition table that can be acquired by the imaging condition acquisition unit. A flowchart showing the operation of the imaging condition acquisition unit. A flowchart showing a first example of the operation of the VPS timing generation unit. A diagram illustrating segmentation processing. A flowchart showing a second example of the operation of the VPS timing generation unit. Circuit diagram of a pixel circuit relating to the first specific example. Circuit diagram of a pixel circuit relating to the second specific example. Circuit diagram of a pixel circuit relating to the third specific example. Circuit diagram of a pixel circuit relating to the fourth specific example. Circuit diagram of a pixel circuit relating to the fifth specific example. Circuit diagram of a pixel circuit relating to the sixth specific example. Circuit diagram of a pixel circuit relating to the seventh specific example. Circuit diagram of a pixel circuit relating to the eighth specific example. Circuit diagram of a pixel circuit relating to the ninth specific example. Diagram showing an example of the appearance of the first head-mounted display. Diagram showing an example of the appearance of the second head-mounted display. Front view of a digital still camera. Rear view of a digital still camera. Diagram showing an example of the appearance of a smartphone. Diagram showing an example of the interior of a vehicle viewed from the rear of the vehicle. Diagram showing an example of the interior of a vehicle viewed from the left rear of the vehicle.

[0029] Embodiments of the display device and electronic device will be described below with reference to the drawings. While the main components of the display device and electronic device will be described below, there may be components and functions not shown or described. The following description does not exclude any components or functions not shown or described.

[0030] (First Embodiment) Figure 1 is a block diagram showing a first configuration example of an electronic device having a display device 10 according to the first embodiment of the present disclosure. The electronic device 1 in Figure 1 comprises a display device 10, an imaging unit (Camera) 2, an application unit (AP, or control device) 3, a communication unit (communication unit) 4, an inertial measuring unit (IMU) 5, a geomagnetic sensor 6, and a GPS receiver 7.

[0031] Electronic device 1 may be, for example, AR glasses, smart glasses, a smartphone, or an HMD (Head Mounted Display). Alternatively, electronic device 1 may be a PC (Personal Computer), a PDA (Personal Digital Assistant), or a tablet device.

[0032] The display device 10 displays at least one of the image data captured by the imaging unit 2 (first image data) or the AR image data generated by the application unit 3 (second image data). The display device 10 comprises a display unit 11 and a display control unit (Driver) 12.

[0033] The display unit 11 has, for example, a liquid crystal display. The display unit 11 has a plurality of display elements. The plurality of display elements are arranged in a two-dimensional array on the liquid crystal display, for example.

[0034] The display unit 11 may have a configuration that includes a lens portion that transmits at least a portion of the incident light. In this case, the display unit 11 displays either or both of the image data captured by the imaging unit 2 or the AR image data on the lens portion. This allows the display unit 11 to display the AR image data by superimposing it on the real-world scenery projected through the lens.

[0035] The display control unit 12 includes, for example, an integrated circuit (IC) or a large-scale integrated circuit (Large Scale Integration). The display control unit 12 drives the display unit 11 based on the video signal input from the application unit 3. Specifically, the display control unit 12 applies voltage to a plurality of display elements within the display unit 11 to display image data on the display unit 11.

[0036] The imaging unit 2 is, for example, a world-facing camera. The imaging unit 2 supplies the captured image data (camera image) to the application unit 3. The imaging unit 2 may supply the image data to the application unit 3 at a fixed interval (for example, 30 fps).

[0037] The imaging unit 2 can be fitted with, for example, a frame-type vision sensor such as a CMOS (Complementary Metal Oxide Semiconductor) image sensor. Alternatively, the imaging unit 2 may be fitted with an EVS (Event-based Vision Sensor). Or, the imaging unit 2 may have a configuration in which CMOS pixels and EVS pixels are mixed. An example of applying an EVS to the imaging unit 2 will be described later.

[0038] The application unit (image generation unit) 3 generates a video signal based on the image data captured by the imaging unit 2. The application unit 3 also generates AR image data, including AR content (third image data), based on the above image data, and can display it on the display device 10.

[0039] AR image data is, for example, image data obtained by overlaying AR content onto image data captured by the imaging unit 2.

[0040] Alternatively, the AR image data is image data in which AR content is positioned to align with a real-world subject captured by the imaging unit 2. For example, the AR image data may be image data in which AR content is positioned to align with a real-world object reflected through AR glasses or the like.

[0041] The AR content may be generated by the application unit 3, or it may be acquired by the application unit 3 from a recording unit within the electronic device 1 or from an external device. The AR content may also be supplied to the application unit 3 via a network or the like.

[0042] AR includes VR (Virtual Reality), MR (Mixed Reality), and XR (Extended Reality).

[0043] The application unit 3 acquires VPS (Visual Positioning System or Visual Positioning Service) information from the communication unit 4, sensor information from the inertial measurement device 5 and geomagnetic sensor 6, and location information from the GPS receiver 7. Based on at least a portion of the sensor information, VPS information, or location information, the application unit 3 determines whether or not to generate AR image data, and determines the type of AR content, the size of the AR content, the position or angle at which the AR content will be superimposed, etc. (hereinafter also referred to as AR content superposition processing).

[0044] Furthermore, the application unit 3 controls the operation of the imaging unit 2, communication unit 4, inertial measuring device 5, geomagnetic sensor 6, and GPS receiver 7 as needed.

[0045] The communication unit 4 communicates with an external VPS server (processing unit) 13 of the electronic device 1 and performs VPS processing. The VPS server 13 is, for example, a cloud server. The VPS server 13 stores, for example, point cloud data of feature points extracted from multiple images.

[0046] Figure 2 is a diagram illustrating the VPS processing. Figure 2 shows the imaging unit 2 and the image data img captured by the imaging unit 2. In the example in Figure 2, the image data img contains images of multiple buildings. Point cloud data Pts of feature points can be extracted from the image data img.

[0047] In VPS processing, the position and orientation of the imaging unit 2 can be estimated by comparing the extracted point cloud data Pts with the point cloud data in the VPS server 13.

[0048] In Figure 1, the communication unit 4 acquires image data captured by the imaging unit 2 via the application unit 3. The communication unit 4 transmits the acquired image data to the VPS server 13 and requests VPS processing. In addition to transmitting the image data, the communication unit 4 may also transmit to the VPS server 13 position information acquired from the GPS receiver 7, and position and attitude information estimated from the sensor information of the inertial measurement device 5 and the geomagnetic sensor 6.

[0049] The VPS server 13 performs VPS processing as shown in Figure 2 to estimate the position and orientation of the imaging unit 2. The VPS server 13 transmits the estimated position and orientation of the imaging unit 2 as VPS information to the communication unit 4. The communication unit 4 inputs the received VPS information to the application unit 3. The VPS information includes, for example, information such as latitude, longitude, altitude, azimuth angle, elevation angle, and roll angle.

[0050] The VPS processing may be performed inside the electronic device 1, but is not limited to the above. For example, the communication unit 4 may acquire at least a portion of the point cloud data held by the VPS server 13 as VPS information. The communication unit 4 may transmit location information, etc., acquired from the GPS receiver 7 to the VPS server 13 and cause the VPS server 13 to search for point cloud data of buildings, etc., located near the imaging unit 2. The application unit 3 may perform VPS processing as shown in Figure 2 based on the point cloud data acquired from the VPS server 13 and the image data captured by the imaging unit 2.

[0051] In addition to the VPS processing shown in Figure 2, the application unit 3 may estimate the amount of change in the orientation (or at least one of the position) of the imaging unit 2 from multiple image data captured by the imaging unit 2. Since this estimation process does not require communication with the VPS server 13, it is also referred to as local VPS processing in this specification.

[0052] Figure 3 illustrates local VPS processing. Figure 3 shows two image data files, img_a and img_b, captured by the imaging unit 2. Image data img_a is image data captured at imaging time t. Image data img_b is image data captured at imaging time t+Δt, a small time interval Δt after imaging time t. Image data img_a and img_b have an overlapping range A where the imaging ranges overlap.

[0053] From the image data img_a and img_b, the point cloud data Pts_a and Pts_b of feature points can be extracted, respectively. Furthermore, from their respective overlapping ranges A, the corresponding feature point data Ps can be extracted.

[0054] The position of feature point data Ps differs between image data img_a and image data img_b. By comparing the position of feature point data Ps in image data img_a with the position of feature point data Ps in image data img_b, the amount of change in the orientation (or position) of the imaging unit 2 can be estimated.

[0055] Image data used for VPS processing (or local VPS processing) is, for example, color (RGB) image data. However, it is not limited to this, and monochrome image data may also be used. Alternatively, depth images generated by distance measurement or the like may be used.

[0056] When EVS is applied to the imaging unit 2, the imaging unit 2 can capture event image data that includes information about events such as brightness changes. Event image data can detect changes in the position of the subject, etc., at high speed, high accuracy, and low power consumption compared to normal image data captured by a CMOS image sensor. The application unit 3 may use the event image data to estimate the amount of attitude change instead of local VPS processing.

[0057] Let's return to Figure 1 and continue the explanation. The inertial measurement device 5 includes, for example, an angle or angular velocity sensor (gyro sensor), an acceleration sensor, and a temperature sensor. The inertial measurement device 5 outputs the attitude (and position) of the imaging unit 2, or the amount of change in attitude (and position), as sensor information. Note that the electronic device 1 may have a configuration that simply includes a gyro sensor instead of the inertial measurement device 5.

[0058] The geomagnetic sensor 6 outputs the orientation of the imaging unit 2 as sensor information. The GPS receiver 7 outputs the position of the imaging unit 2. Note that either the geomagnetic sensor 6 or the GPS receiver 7, or both, may be omitted from the electronic device 1.

[0059] The inertial measuring device 5, the geomagnetic sensor 6, and the GPS receiver 7 may, more precisely, output the attitude or change in attitude of the entire electronic device 1 (hereinafter also simply referred to as attitude, etc.), or the position or change in position (hereinafter also simply referred to as position, etc.). In this specification, the attitude, etc. (or position, etc.) output by the inertial measuring device 5, the geomagnetic sensor 6, and the GPS receiver 7 may be referred to as the attitude, etc. (or position, etc.) of the imaging unit 2.

[0060] In the electronic device 1, it is desirable to use a combination of the output values ​​from the VPS processing, the inertial measurement device 5, the geomagnetic sensor 6, and the GPS receiver 7 as appropriate. For example, errors may occur in the inertial measurement device 5 due to integral calculations (e.g., gyrodrift). Therefore, it is desirable to appropriately correct the output value of the inertial measurement device 5 based on the output values ​​of other sensors or estimated values ​​obtained by VPS processing.

[0061] The geomagnetic sensor 6 and GPS receiver 7 may experience errors (for example, errors due to geomagnetic disturbances) when there are multiple buildings near the imaging unit 2, or when the imaging unit 2 is located inside a building. In such situations, it is desirable to substitute or correct the output values ​​of the geomagnetic sensor 6 and GPS receiver 7 with the output values ​​of other sensors or estimated values ​​obtained by VPS processing.

[0062] Furthermore, VPS processing consumes more power than the inertial measurement device 5, etc. Therefore, it is desirable that VPS processing be performed at a lower rate (lower frequency) than the sensor information acquisition process from the inertial measurement device 5, etc. After VPS processing has been performed once, the electronic device 1 can perform AR content alignment based on sensor information from the inertial measurement device 5 and the geomagnetic sensor 6, etc., until the next VPS processing is performed.

[0063] The components of the electronic device 1 shown in Figure 1 do not necessarily have to be built into the electronic device 1. For example, some or all of the imaging unit 2, application unit 3, communication unit 4, inertial measuring device 5, geomagnetic sensor 6, and GPS receiver 7 may be located in an external device connected to the electronic device 1 via USB or the like.

[0064] Furthermore, the electronic device 1 may estimate the attitude (and position) of the imaging unit 2 by SfM (Structure from Motion) or SLAM (Simultainius Localization and Mapping) instead of the inertial measurement device 5, geomagnetic sensor 6, GPS receiver 7, and local VPS processing. For example, the application unit 3 may perform SfM or SLAM based on a plurality of image data captured by the imaging unit 2. SfM and SLAM may be used in combination with the inertial measurement device 5, etc. In the low-latency AR system described later, SfM and SLAM may be performed by the display control unit 12. In addition to the above-mentioned SfM and SLAM, the electronic device 1 may estimate the attitude (and position) of the imaging unit 2 by other methods (for example, LiDER: Light Detection and Ranging, or movement detection using a MIMO antenna, etc.).

[0065] The application unit 3 can overlay AR content onto image data based on VPS processing, the attitude of the imaging unit 2 output by the inertial measurement device 5, and the geomagnetic sensor 6.

[0066] Figure 4 illustrates the AR image data generated by the application unit 3. Figure 4 shows the AR image data displayed on the electronic device 1 (a smartphone in the example shown in Figure 4). The AR image data includes AR content obj_a and image data obj_b of a real-world subject captured by the imaging unit 2.

[0067] In Figure 4, the AR content obj_a and the image data obj_b are aligned. Specifically, in the example in Figure 4, the AR content obj_a is positioned so as to be in contact with the interface of the image data obj_b.

[0068] Here, if the orientation of the electronic device 1 (i.e., the imaging unit 2) changes, the angle of the subject relative to the imaging unit 2 changes, causing a misalignment between the AR content obj_a and the image data obj_b. In response to this, the application unit 3 rotates the AR content obj_a in accordance with the change in the orientation of the imaging unit 2. As a result, even if the orientation of the imaging unit 2 changes, the electronic device 1 can align the AR content obj_a and the image data obj_b.

[0069] In addition to the example in Figure 4, the electronic device 1 may align the AR content obj_a and the image data obj_b such that either the AR content obj_a or the image data obj_b overlaps with at least a portion of the other.

[0070] In addition to the above, the electronic device 1 may also be configured to display AR content and have glasses or the like that allow the real scenery to pass through and be displayed in the user's field of view (for example, AR glasses). In this case, the electronic device 1 may perform alignment between the AR content and the real scenery seen through the glasses or the like, rather than using image data captured by the imaging unit 2.

[0071] Figure 5 is a block diagram showing a second configuration example of an electronic device having a display device 10 according to the first embodiment of the present disclosure. The electronic device 1a shown in Figure 5 differs from the electronic device 1 in Figure 1 in that the display control unit 12 receives sensor information and position information from the inertial measuring device 5, geomagnetic sensor 6, and GPS receiver 7.

[0072] The display control unit 12 directly detects changes in the orientation of the imaging unit 2 from the inertial measurement device 5, etc. The display control unit 12 extracts a portion of the video signal acquired from the application unit 3 (for example, AR content) and performs correction according to the changes in orientation. As a result, changes in AR image data due to changes in the orientation of the imaging unit 2 can be reflected in the display unit 11 with lower latency than the electronic device 1 in Figure 1. In this specification, the AR system of the electronic device 1a in Figure 5 is also referred to as a low-latency AR system.

[0073] The display control unit 12 may request the application unit 3 to update the video signal (i.e., redraw the image data) if the position or orientation of the imaging unit 2 changes significantly. At this time, the application unit 3 may obtain the orientation of the imaging unit 2 from the display control unit 12. Alternatively, at least a portion of the inertial measuring device 5, geomagnetic sensor 6, and GPS receiver 7 may input sensor information and position information to both the application unit 3 and the display control unit 12.

[0074] In the electronic device 1a shown in Figure 5, the display control unit 12 outputs a control signal to the imaging unit 2, controlling the imaging timing of the imaging unit 2, etc. In addition, the electronic device 1a may be configured in the same way as the electronic device 1 in Figure 1, with the application unit 3 controlling the imaging unit 2.

[0075] At least a portion of the VPS processing or local VPS processing of the application unit 3 may be executed by the display control unit 12. Alternatively, the electronic device 1a may be configured without the application unit 3.

[0076] Figure 6 is a block diagram showing the detailed configuration of the application unit 3 and the display control unit 12 according to the first embodiment of the present disclosure. The application unit 3 includes an image acquisition unit 21, a VPS information acquisition unit (acquisition unit) 22, and a VPS processing unit 23. The display control unit 12 includes a sensor information acquisition unit (first detection unit) 24, a sensor information acquisition unit (second detection unit) 25, a position information acquisition unit (position identification unit) 26, a posture estimation unit (estimation unit) 27, an error correction unit 28, an integrated filter unit 29, and a posture information utilization unit 30.

[0077] The configuration of the components shown in Figure 6 can be arbitrarily changed to either the application unit 3 or the display control unit 12. That is, at least a part of the image acquisition unit 21, the VPS information acquisition unit 22, or the VPS processing unit 23 may be located in the display control unit 12. Also, at least a part of the sensor information acquisition units 24 and 25, the position information acquisition unit 26, the attitude estimation unit 27, the error correction unit 28, the integrated filter unit 29, or the attitude information utilization unit 30 may be located in the application unit 3.

[0078] The application unit 3 and display control unit 12 in Figure 6 can be applied, for example, to the electronic device 1a in Figure 5. Alternatively, the application unit 3 and display control unit 12 in Figure 6 may be applied to the electronic device 1 in Figure 1.

[0079] The image acquisition unit 21 acquires image data from the imaging unit 2. The VPS information acquisition unit 22 acquires VPS information via the communication unit 4 based on the image data acquired by the image acquisition unit 21. The VPS processing unit 23 performs processing to cause the VPS information acquisition unit 22 to acquire VPS information, or performs local VPS processing, etc.

[0080] The VPS processing unit 23 can estimate the orientation of the imaging unit 2 through VPS processing. Furthermore, the VPS processing unit 23 can estimate the amount of orientation change of the imaging unit 2 through local VPS processing.

[0081] The sensor information acquisition unit 24 acquires sensor information from the inertial measurement device 5. The sensor information acquisition unit 25 acquires sensor information from the geomagnetic sensor 6. The position information acquisition unit 26 acquires position information from the GPS receiver 7. The sensor information acquisition unit 25 and the position information acquisition unit 26 may be omitted.

[0082] The attitude estimation unit 27 estimates the attitude of the imaging unit 2 based on the sensor information acquired by the sensor information acquisition units 24 and 25. The attitude estimation unit 27 corrects the attitude acquired from the inertial measurement device 5 with the attitude acquired by the geomagnetic sensor 6.

[0083] For example, the attitude estimation unit 27 multiplies the attitude acquired by the geomagnetic sensor 6 by a predetermined correction gain (hereinafter also called the geomagnetic correction gain) and outputs a correction value. The attitude estimation unit 27 estimates the attitude of the imaging unit 2 by adding the output correction value to the attitude acquired from the inertial measurement device 5.

[0084] The error correction unit 28 detects the difference (hereinafter also referred to as the error) between the attitude estimated by the attitude estimation unit 27 and the attitude estimated by the VPS processing unit 23. The error correction unit 28 also identifies at least one of the error causes generated by the attitude estimation unit 27 or the VPS processing unit 23, and outputs a correction amount for the identified error cause.

[0085] The causes of errors in the attitude estimation unit 27 include, for example, integral errors of the inertial measuring device 5 (e.g., gyro drift errors), errors of the geomagnetic sensor 6 due to geomagnetic disturbances, and mechanical positional errors of the inertial measuring device 5 and the geomagnetic sensor 6 (e.g., errors caused by misalignment of mounting position and direction). Specifically, the mechanical positional errors of the inertial measuring device 5 include mechanical positional errors of the gyro sensor and acceleration sensor. The causes of errors in the VPS processing unit 23 include, for example, mechanical positional errors of the imaging unit 2 and errors due to lens distortion of the imaging unit 2.

[0086] The error correction unit 28 may identify errors such as mechanical position errors of the inertial measuring device 5, geomagnetic sensor 6, and imaging unit 2, including errors caused by individual differences during manufacturing, positional misalignment due to retrofitting of sensors, and errors caused by deformation of the electronic device 1 (for example, the temple portion of the AR glasses with the sensor attached widens). The error correction unit 28 may also identify other mechanical position errors.

[0087] Furthermore, if a synchronization error occurs in any of the inertial measuring device 5, geomagnetic sensor 6, or VPS processing unit 23, the error correction unit 28 may correct the sampling time error of any of the inertial measuring device 5, geomagnetic sensor 6, or VPS processing unit 23.

[0088] The integrated filter unit 29 calculates the orientation of the imaging unit 2 based on the orientation estimated by the orientation estimation unit 27 and the orientation estimated by the VPS processing unit 23. The integrated filter unit 29 includes, for example, a Kalman filter.

[0089] The posture information utilization unit 30 performs AR content overlay processing, etc., based on the posture calculated from the integrated filter unit 29. The posture information utilization unit 30 may also use location information obtained from the location information acquisition unit 26 or the VPS processing unit 23, etc., for AR content overlay processing.

[0090] Figure 7 is a block diagram showing a more detailed configuration of the VPS processing unit 23 and the error correction unit 28. The VPS processing unit 23 includes a VPS processing main unit 31, a VPS timing generation unit 32, a local VPS processing unit (attitude change amount estimation unit) 33, and a local VPS timing generation unit 34. The error correction unit 28 includes a difference detection unit 35, an error history holding unit (first holding unit) 36, an error cause identification unit (identification unit) 37, and a correction amount generation unit (correction unit) 38. The error correction unit 28 may also have a correction formula holding unit 39.

[0091] The VPS processing main unit 31 transmits image data and requests VPS processing to the VPS server 13, for example, via the VPS information acquisition unit 22. The VPS processing main unit 31 may also perform VPS processing based on point cloud data acquired from the VPS server 13.

[0092] The VPS timing generation unit 32 instructs the VPS processing main unit 31 on the timing to request VPS processing from the VPS server 13 or to perform VPS processing (hereinafter collectively referred to simply as VPS processing).

[0093] The VPS timing generation unit 32 may instruct the VPS processing main unit 31 to perform VPS processing at regular intervals (for example, every tens of seconds). Alternatively, the VPS timing generation unit 32 may instruct VPS processing at a timing in which it is determined that VPS processing is effective based on the orientation of the imaging unit 2 calculated by the integrated filter unit 29. The method for determining whether or not VPS processing is effective will be described later.

[0094] The local VPS processing unit 33 performs local VPS processing based on multiple image data acquired from the imaging unit 2. The local VPS timing generation unit 34 instructs the local VPS processing unit 33 on the timing for performing local VPS processing. Note that the local VPS processing unit 33 and the local VPS timing generation unit 34 may be omitted.

[0095] The local VPS processing unit 33 may receive the posture estimated by the posture estimation unit 27 (for example, the amount of posture change). The local VPS processing unit 33 may also modify the feature point matching based on the input amount of posture change.

[0096] Figure 8 is a flowchart illustrating the operation of the local VPS timing generation unit 34. First, the attitude estimation unit 27 estimates the attitude of the imaging unit 2 based on sensor information acquired from the inertial measurement device 5 (and the geomagnetic sensor 6) (step S1).

[0097] Next, the local VPS timing generation unit 34 determines whether the overlap range of the field of view with the image data captured in the previous local VPS processing (or VPS processing) is below a threshold (step S2). In step S2, for example, it determines whether the area of ​​overlap range A in Figure 3 is below a threshold.

[0098] In step S2, the determination can be made based on, for example, the orientation of the imaging unit 2 at the time of the previous local VPS processing, the current orientation of the imaging unit 2, the amount of change in the orientation of the imaging unit 2 from the time of the previous local VPS processing to the present, or the rate of change in the orientation of the imaging unit 2.

[0099] Alternatively, the local VPS timing generation unit 34 may acquire the field of view (FOV) from the imaging unit 2, etc. The local VPS timing generation unit 34 may compare the field of view of the imaging unit 2 at the timing of the previous local VPS processing with the current field of view of the imaging unit 2 and make the determination in step S2.

[0100] In step S2, if the overlap range is determined to be below a threshold, the imaging unit 2 and the local VPS processing unit 33 are instructed to perform local VPS processing (step S3). In step S3, first, the imaging unit 2 captures image data. The local VPS processing unit 33 also performs local VPS processing based on the image data captured in the previous local VPS processing and the currently captured image data to estimate the amount of change in the posture of the imaging unit 2.

[0101] After step S3, the local VPS timing generation unit 34 waits for a predetermined time before performing the process in step S1. Also, if in step S2 it is determined that the overlap range is above the threshold, the local VPS timing generation unit 34 waits for a predetermined time before performing the process in step S1.

[0102] As shown in Figure 3, local VPS processing requires an overlap range between the two image data. Therefore, in the process shown in Figure 8, local VPS processing is performed each time the overlap range falls below a threshold. This allows local VPS processing to be executed while an overlap range exists, thus enabling effective local VPS processing. Furthermore, it reduces the frequency of power-hungry local VPS processing.

[0103] Furthermore, if it is determined in step S2 that there is no overlap range with the image data captured earlier due to a sudden change in the field of view, or if the local VPS has not been executed in the past, the imaging unit 2 may capture new image data. Alternatively, after a predetermined time has elapsed from the imaging timing, it may be determined in step S2 whether the overlap range with the newly captured image data falls below a threshold.

[0104] Returning to Figure 7, the explanation continues. The difference detection unit 35 detects the difference between the VPS main processing unit 31 or the local VPS processing unit 33 and the attitude estimation unit 27. The error history holding unit 36 ​​holds the difference detected by the difference detection unit 35 as an error history. Note that the error history holding unit 36 ​​does not need to hold the detected difference as an error history if it is below a predetermined threshold.

[0105] The error history storage unit 36 ​​stores, for example, multiple (e.g., 100) error histories within a predetermined period. This allows the error correction unit 28 to determine the trend of error fluctuations over time.

[0106] The error cause identification unit 37 identifies one or more error causes based on multiple error histories held by the error history holding unit 36. In other words, the error cause identification unit 37 can identify the causes of multiple errors that occurred at multiple different timings. The error cause identification unit 37 may also identify the error cause based on a correction formula held by the correction formula holding unit 39.

[0107] The correction formula holding unit 39 holds, for example, a Jacobian matrix for determining the degree of influence of each of multiple correction parameters on the error. In this case, the error cause identification unit 37 can identify one or more correction parameters (i.e., error causes) that need correction based on the Jacobian matrix. The error correction unit 28 can also change the identified correction parameters to values ​​that produce less error.

[0108] The correction formula holding unit 39 may hold correction formulas entered before the shipment of the electronic device 1 or display device 10. Alternatively, the correction formula holding unit 39 may hold correction formulas obtained from a cloud server or the like, or correction formulas entered by a user or the like.

[0109] The correction amount generation unit 38 generates a correction amount for each of the one or more error causes identified by the error cause identification unit 37. Alternatively, the error correction unit 28 may omit the correction amount generation unit 38 and have the error cause identification unit 37 generate the correction amount.

[0110] The VPS processing unit 23 and attitude estimation unit 27 in Figure 7, for example, handle multiple correction parameters β1, β2...β n The posture calculation formula f(β1, β2..., β) includes n The posture calculation formula f is retained. The posture calculation formula f can be input, for example, before shipment, similar to the correction formula in the correction formula retention unit 39. The integrated filter unit 29 may also retain at least a part of the posture calculation formula f.

[0111] Correction parameters β1 to β nIt may include a geomagnetic correction gain, a sensor correction value, a weighting coefficient for attitude calculation, a correction value for a mechanical position error of the inertial measurement device 5, the geomagnetic sensor 6, and the imaging unit 2, a correction value for lens distortion of the imaging unit 2, a correction value for sampling error, or a Kalman gain, etc. The geomagnetic correction gain, the sensor correction value, the weighting coefficient for attitude calculation, and the correction value for the mechanical position / direction deviation of the sensor are held, for example, by the attitude estimation unit 27. The correction values for the mechanical position error and the lens distortion of the imaging unit 2 are held, for example, by the VPS processing unit 23. The Kalman gain and the correction value for sampling error are held, for example, by the integrated filter unit 29.

[0112] As an example, the following formula (1) can be used for the attitude calculation formula f in the VPS processing unit 23. The right side of the following formula (1) corresponds to the attitude calculation formula f. r0, p0, y0: The attitude (roll angle, pitch angle, and yaw angle) estimated by the VPS processing unit 23. r vps , p vps , y vps : The attitude obtained by VPS processing or local VPS processing. β c11 ~β c33 , β s1 ~β s3 : The lens distortion correction coefficient of the imaging unit 2 (for example, affine transformation). β r_pos , β p_pos , β y_pos : The mounting position correction coefficient of the imaging unit 2.

[0113] Among the above parameters, β c11 ~β c33 , β s1 ~β s3 , β r_pos , β p_pos , and β y_pos can be treated as correction parameters.

[0114] A Madgwick Filter may be used in the attitude estimation unit 27. The correction parameters β1~β n may include the drift correction value of the Madgwick Filter and the correction gain of the geomagnetic sensor 6, etc.

[0115] Note that the correction parameters β1 to β n Furthermore, the posture calculation formula f may be modified by redesigning the parameters from equation (1) above and applying the corrected parameters and posture calculation formula. Parameter redesign will be described later.

[0116] The VPS processing unit 23, attitude estimation unit 27, and integrated filter unit 29 adjust the correction parameters β1 to β based on the correction values ​​generated by the error cause identification unit 37. n The input value may be corrected.

[0117] The integrated filter unit 29 can calculate the attitude of the imaging unit 2 from the attitude estimated by the VPS processing unit 23 and the attitude estimation unit 27 using the attitude calculation formula f.

[0118] Figure 9 is a flowchart showing a first example of the operation of the display control unit 12 or the application unit 3. In the operation shown in Figure 9, the error cause identification unit 37 can determine whether the error is caused by geomagnetic disturbance or by gyrodrift. The following describes an example in which the display control unit 12 performs the process shown in Figure 9.

[0119] First, the display control unit 12 identifies the reference orientation and reference position of the imaging unit 2 (step S11). Specifically, the display control unit 12 acquires position information from the GPS receiver 7, and the VPS processing main unit 31 performs VPS processing to acquire VPS information. Based on the acquired position information and VPS information, the display control unit 12 identifies the orientation and position of the imaging unit 2 in the reference coordinate system (i.e., the reference orientation and reference position).

[0120] Next, the attitude estimation unit 27 estimates the attitude of the imaging unit 2 based on the reference attitude and reference position identified in step S11, and sensor information acquired from the inertial measurement device 5 (and the geomagnetic sensor 6) (step S12).

[0121] The attitude estimation unit 27 can estimate the amount of attitude change from the reference attitude (i.e., relative attitude) based on sensor information acquired from the inertial measurement device 5 (and the geomagnetic sensor 6). Furthermore, the attitude estimation unit 27 can estimate the current attitude of the imaging unit 2 in the reference coordinate system based on the reference attitude and the relative attitude.

[0122] Next, the VPS timing generation unit 32 or the local VPS timing generation unit 34 determines whether or not it is time to perform VPS processing or local VPS processing (hereinafter also simply referred to as VPS processing, etc.) (step S13). If it is determined in step S13 that it is not time to perform VPS processing, etc., then in step S12 the orientation of the imaging unit 2 is estimated periodically.

[0123] If it is determined in step S13 that it is time to perform VPS processing, the VPS processing main unit 31 or the local VPS processing unit 33 performs VPS processing (step S14). In step S14, the imaging unit 2 captures image data and either transmits the captured image data to the VPS server 13 or performs local VPS processing based on the captured image data.

[0124] The difference detection unit 35 detects the difference between the posture etc. estimated in step S12 and the posture etc. estimated in step S14. The difference detection unit 35 also determines whether the detected difference (i.e., error) is greater than or equal to a threshold (step S15).

[0125] If an error exceeding a threshold is detected in step S15, the detected difference is stored as an error history in the error history storage unit 36. Subsequently, the error cause identification unit 37 determines from the multiple error histories stored in the error history storage unit 36 ​​whether the occurring error is a gyro drift error or not (step S16).

[0126] The absolute value of the gyrodrift error tends to increase approximately linearly with time. On the other hand, errors caused by geomagnetic disturbances tend to fluctuate randomly depending on the environment surrounding the imaging unit 2. Therefore, in step S16, it is possible to determine whether or not an error is due to gyrodrift based on whether or not the error is increasing approximately linearly from multiple error histories.

[0127] If, in step S16, the error is not determined to be due to gyrodrift, that is, if it is determined to be an error due to geomagnetic disturbance, the correction amount generation unit 38 and the attitude estimation unit 27 perform geomagnetic disturbance correction (step S17). The correction amount generation unit 38 instructs the attitude estimation unit 27 to reduce the geomagnetic correction gain. The correction amount generation unit 38 generates the amount of reduction in the geomagnetic correction gain.

[0128] The correction amount generation unit 38 may generate a reduction amount for the geomagnetic correction gain according to the maximum, minimum, average, median, or variance of a plurality of error histories. Alternatively, the correction amount generation unit 38 may weight each of the plurality of error histories according to the retained timing (for example, giving greater weight to error histories with newer retained timings), or it may extract a portion of the new error histories and generate a reduction amount for the geomagnetic correction gain. Alternatively, the correction amount generation unit 38 may generate a predetermined reduction amount, or it may generate a reduction amount based on sensor information acquired from the geomagnetic sensor 6. Alternatively, the reduction amount for the geomagnetic correction gain may be adjusted according to the current geomagnetic correction gain. For example, if the geomagnetic correction gain has already decreased from its initial value, the reduction amount for the geomagnetic correction gain may be reduced.

[0129] If the error is determined to be due to gyrodrift in step S16, the correction amount generation unit 38 and the attitude estimation unit 27 perform gyro correction (step S18). The correction amount generation unit 38 generates a correction amount corresponding to the difference detected by the difference detection unit 35, for example. The correction amount generation unit 38 may also adjust the correction amount according to the geomagnetic correction gain. The attitude estimation unit 27 corrects the attitude and other data acquired from the inertial measurement device 5 with the generated correction amount.

[0130] In addition, in steps S17 and S18, the integrated filter unit 29 may perform the correction instead of the attitude estimation unit 27.

[0131] In step S18, if the geomagnetic correction gain has decreased in the past due to processing in step S17 or other means, the geomagnetic correction gain may be increased. In step S18, the geomagnetic correction gain may be initialized all at once, or it may be increased in small increments to approach the initial value.

[0132] After steps S17 and S18, the process of step S12 is repeated. In the processing after steps S17 and S18, the attitude estimated in the VPS processing of step S14 may be identified as the reference attitude.

[0133] If no error exceeding the threshold is detected in step S15, the error correction unit 28 does not need to perform correction (step S19). Also, the error history holding unit 36 ​​does not need to hold the difference detected by the difference detection unit 35. Furthermore, in step S19, the geomagnetic correction gain may be increased, similar to step S18.

[0134] If the display control unit 12 performs the processing shown in Figure 9, the correction formula holding unit 39 may be omitted.

[0135] In addition to the method described in Figure 9, for example, a threshold that increases based on the difference detected by the difference detection unit 35 may be set, and it may be determined whether the next difference detected by the difference detection unit 35 exceeds that threshold. If the difference detected by the difference detection unit 35 exceeds the above threshold multiple times, it may be determined that the error is due to gyro drift. In this method, the error history holding unit 36 ​​may be omitted.

[0136] Figure 10 is a flowchart showing a second example of the operation of the display control unit 12 or the application unit 3. In the operation shown in Figure 10, the error cause identification unit 37 can identify various error causes, not limited to errors due to geomagnetic disturbances and errors due to gyrodrift.

[0137] In Figure 10, first, the processes in steps S11 to S14 are performed, similar to those in Figure 9. Next, the difference detection unit 35 detects the difference between the posture etc. estimated by the posture estimation unit 27 in step S12 and the posture etc. estimated by the VPS processing unit 23 in step S14. The error history holding unit 36 ​​holds the posture etc. estimated by the posture estimation unit 27, the posture etc. estimated by the VPS processing unit 23, and the detected difference as an error history (step S21).

[0138] Before step S21, it is also possible to determine whether the detected difference is greater than or equal to a threshold, similar to step S15 in Figure 9. If the difference is less than the threshold, the error history holding unit 36 ​​does not need to hold the difference.

[0139] Next, the error cause identification unit 37 uses the Jacobian matrix and error history to determine the correction parameters β1 to β of the attitude calculation formula f. n The contribution of is calculated (step S22). The Jacobian matrix can be obtained from the correction formula holding unit 39. Subsequently, the correction amount generation unit 38, VPS processing unit 23, attitude estimation unit 27, and integrated filter unit 29 correct the correction parameters (step S23).

[0140] In step S22, for example, the contribution can be calculated using the Jacobian matrix and the error history. Furthermore, steps S22 and S23 allow one step of the correction parameter optimization process using the nonlinear least squares method with the steepest descent method to be executed. The details of the processes in steps S22 and S23 are described below.

[0141] In step S14, the difference in posture, etc., is, for example, the difference in roll e. r , pitch difference e p , and the difference e of yaw y This can be obtained. Also, in step S14, the roll x estimated by the attitude estimation unit 27 is obtained. r , pitch x p , and yaw x y The VPS processing unit 23 estimated the role y r , pitch y p , and y y This can be obtained.

[0142] The values ​​estimated by the attitude estimation unit 27 and the VPS processing unit 23 are not limited to roll, pitch, and yaw. For example, the attitude estimation unit 27 and the VPS processing unit 23 may estimate quaternions. The difference detection unit 35 can obtain the difference in quaternions. Furthermore, quaternions may also be applied to the attitude calculation formula f, etc. Roll, pitch, and yaw may be arbitrarily converted to other values ​​other than quaternions.

[0143] Among multiple differences, any difference e corresponds to the estimated pose x, y, and correction parameters β1 to β. n Using this, e = e(x, y, β1, β2..., β n It can be expressed as follows. Hereafter, in this specification, this equation will also be called the model function. The difference e will be called the dependent variable, and the attitude x and y will be called the independent variables, and the correction parameters β1 to β n These are called fitting parameters.

[0144] The correction formula holding unit 39, for example, the difference e r , e p , and e y Correction parameters β1, β2...β n The Jacobian matrix J containing the components obtained by the partial derivative with respect to can be preserved. The Jacobian matrix J is expressed as shown in equation (2).

[0145] Multiple error histories consist of m data points (where m is, for example, three times the number of error histories) (x1, y1, e1), (x2, y2, e2), ..., (x m , y m , e m ) has.

[0146] The error correction unit 28 adjusts the correction parameters β1, β2...β so that the sum of squared residuals R shown in equation (3) below is minimized. n Correct the following (hereinafter collectively referred to as β). Alternatively, instead of the sum of squared residuals R in equation (3), the correction parameter β may be adjusted so that the standardized sum of squared residuals is minimized.

[0147] The minimum sum of squared residuals R can be estimated using the Jacobian matrix J and the steepest descent method. Furthermore, equation (3) includes the current correction parameters β1 to β n From the sum of squared residuals R calculated by applying this, the current correction parameters β1 to β n The contribution (coefficient of determination) can be calculated.

[0148] The error cause identification unit 37 can identify one or more error causes based on their contribution. For example, if the contribution of the geomagnetic correction gain among the correction parameters is large, it can be determined that the error cause includes an error due to geomagnetic disturbance. Alternatively, if the contribution of the correction value to the mechanical position error or sampling error of the imaging unit 2 is large, it can be determined that the error cause includes a mechanical position error or sampling error.

[0149] In step S23, the correction amount generation unit 38 can generate a correction amount for each correction parameter based on the contribution of the correction parameter. The VPS processing unit 23, the attitude estimation unit 27, and the integrated filter unit 29 generate a correction amount based on the generated correction amount for each correction parameter β1 to β n Correct it.

[0150] As described above, the operation shown in Figure 10 allows for correction of mechanical positional errors or sampling errors in the imaging unit 2. Correcting these errors is particularly useful, for example, when the imaging unit 2 is externally mounted and prone to misalignment, or when the inertial measuring device 5 or the like is externally mounted and synchronization with the application unit 3 or the like is difficult.

[0151] Figure 11 is a flowchart showing the method for setting the Jacobian matrix J and the posture calculation formula f. First, the correction parameters β1 to β n The determined correction parameters β1 to β n Based on this, the posture calculation formula f and the Jacobian matrix J are created (steps S32 and S33). Subsequently, the Jacobian matrix J is analyzed to obtain the correction parameters β1 to β n It is determined whether or not the result includes multiple parameters with a high degree of linear dependence (hereinafter also referred to as a parameter set) (step S34).

[0152] If it is determined that a set of parameters with high linear dependence is included, parameter redesign (parameter shrinking) is performed (step S35). A set of parameters with high linear dependence can be combined into a single parameter. Based on the redesigned correction parameter, the attitude calculation formula f and the Jacobian matrix J are created in steps S32 and S33. After that, the determination in step S34 is performed again.

[0153] If it is determined in step S34 that the parameter set does not contain a high degree of linear dependence, the created attitude calculation formula f and Jacobian matrix J are input to the electronic device 1 (step S36).

[0154] In addition to the methods described in Figures 10 and 11, the error correction unit 28 may also identify the cause of the error and generate a correction amount using machine learning. For example, the error history holding unit 36, the error cause identification unit 37, and the correction amount generation unit 38 may have a configuration that includes an RNN (Recurrent Neural Network) model. The difference detected by the difference detection unit 35 may be input to the RNN model, and the error cause and correction amount may be output.

[0155] As described above, the electronic device 1 according to the first embodiment of this disclosure includes a posture estimation unit 27 that estimates the posture of the imaging unit 2 based on sensor information, a VPS processing unit 23 that estimates the posture based on VPS processing, and an error cause identification unit 37 that identifies the cause of the difference between the estimated value of the posture estimation unit 27 and the estimated value of the VPS processing unit 23.

[0156] In the process shown in Figure 9, the error cause identification unit 37 can determine whether the cause of the error in the attitude estimation unit 27 is gyro drift or geomagnetic disturbance. In the process shown in Figure 10, the error cause identification unit 37 can identify multiple error causes. Furthermore, in the process shown in Figure 10, it is possible to identify not only the error cause in the attitude estimation unit 27 but also the error cause in the VPS processing unit 23.

[0157] As described above, the error cause identification unit 37 can identify one or more error causes. This enables the error correction unit 28 to perform appropriate correction (dynamic calibration) for the identified error causes. In other words, the electronic device 1 can perform attitude calculations with high accuracy even when there are multiple error causes.

[0158] This enables the display device 10 to perform highly accurate alignment of AR content. Furthermore, when the error cause identification unit 37 is located in the display control unit 12, the alignment of AR content can be performed at a faster speed than when the error cause is identified in the application unit 3.

[0159] (Second Embodiment) VPS processing includes imaging processing and communication processing with a server, etc., and consumes a large amount of power. In addition, if the feature points extracted from the imaged subject and the point cloud data held in the server cannot be sufficiently matched, the accuracy of VPS processing will decrease. For this reason, it is desirable that VPS processing be performed at an effective timing, that is, at a timing when high-precision VPS processing is expected. The electronic device 1 according to the second embodiment of this disclosure has a function to perform VPS processing at an effective timing.

[0160] Figures 12A and 12B illustrate the timing at which VPS processing is effective. Figures 12A and 12B show image data captured by the imaging unit 2, respectively. Multiple types of feature points p1, p2, and p3 can be extracted from the image data. Feature points p1, p2, and p3 are most effective in this order (for example, they have a large difference in brightness from the surroundings).

[0161] The image data in Figures 12A and 12B have subjects obj_c and obj_d that contain many effective feature points p1 and p2. Subject obj_c is, for example, a building photographed outdoors. Subject obj_d is, for example, a door located indoors.

[0162] As shown in Figures 12A and 12B, effective VPS processing is possible when an image can be captured of a subject containing many effective feature points.

[0163] Figure 13 is a block diagram showing the detailed configuration of the application unit 3a and the display control unit 12a according to the second embodiment of the present disclosure. The application unit 3a includes an image acquisition unit 21, a VPS information acquisition unit 22, and a VPS processing unit 31a. The display control unit 12a includes sensor information acquisition units 24 and 25, a position information acquisition unit 26, a posture estimation unit 27, an integrated filter unit 29, a posture information utilization unit 30, and a VPS timing generation unit (determination unit) 32a. The arrangement of the above components between the application unit 3a and the display control unit 12a can be arbitrarily changed.

[0164] In the application unit 3a and display control unit 12a of Figure 13, the error correction unit 28, local VPS processing unit 33, and local VPS timing generation unit 34 of Figure 7 can be omitted.

[0165] Furthermore, the display control unit 12a (or application unit 3a) in Figure 13 further includes an imaging condition acquisition unit 41, an imaging condition holding unit (second holding unit) 42, a movement amount determination unit (movement determination unit) 43, a segmentation unit (extraction unit) 44, an equipment status acquisition unit 45, and a correction unit 46. Note that the imaging condition acquisition unit 41, the imaging condition holding unit 42, the segmentation unit 44, and the equipment status acquisition unit 45 may be omitted.

[0166] The imaging condition acquisition unit 41 acquires imaging condition data (for example, the imaging condition table described later) from a predetermined server (for example, a VPS server 13) via the communication unit 4. The imaging condition holding unit 42 holds the imaging condition data acquired by the imaging condition acquisition unit 41. The movement amount determination unit 43 determines the movement amount of the imaging unit 2 from the information of the position information acquisition unit 26 and inputs it to the imaging condition acquisition unit 41.

[0167] The segmentation unit 44 performs segmentation processing based on the image data acquired by the image acquisition unit 21.

[0168] The device status acquisition unit 45 acquires device status information such as battery level, thermal margin, or processing power of the electronic device 1 or the display control unit 12a. In the following, an example of the device status acquisition unit 45 acquiring the device status of the electronic device 1 will be described.

[0169] The correction unit 46 corrects the posture information estimated by the posture estimation unit 27 or the sensor information acquired by the sensor information acquisition units 24 and 25, based on the posture information of the imaging unit 2 acquired by the VPS processing unit 31a. Alternatively, the correction unit 46 may be omitted, and the integrated filter unit 29 may perform the above correction.

[0170] The VPS processing unit 31a in Figure 13 performs VPS processing as shown in Figure 2 (or Figure 3) when predetermined conditions are met. The VPS timing generation unit 32a in Figure 13 determines whether the above predetermined conditions are met based on the imaging condition data held by the imaging condition holding unit 42, the results of the segmentation processing, or the equipment status acquired by the equipment status acquisition unit 45. If the above predetermined conditions are met, the VPS timing generation unit 32a can instruct the VPS processing unit 31a to perform VPS processing.

[0171] Figure 14 is a diagram showing the imaging condition table that can be acquired by the imaging condition acquisition unit 41. In this specification, each row from the second row onward in the table in Figure 14 is also referred to as a record. That is, Figure 14 shows three records. Each record contains data on roll angle, pitch angle, yaw angle, VPS effectiveness, nighttime luminescence, and variability.

[0172] The imaging conditions table in Figure 14 contains information on one or more subjects located near the imaging unit 2 from which feature points can be extracted. These one or more subjects may include, for example, buildings or groups of buildings. Natural objects such as mountains may also be included.

[0173] The VPS server 13 may have point cloud data corresponding to each subject. Alternatively, the VPS server 13 may generate an imaging condition table based on the point cloud data used for VPS processing.

[0174] The imaging conditions table, for example, contains information for one subject per record. For example, each record contains information about the range of orientations in which the subject can be imaged in the reference coordinate system, such as roll angle, pitch angle, and yaw angle. Each record contains information about the roll angle, pitch angle, and yaw angle in the reference coordinate system.

[0175] For example, if the orientation of the imaging unit 2 is within the range of roll angle, pitch angle, and yaw angle within the record, the imaging unit 2 can image the subject. As a result, the VPS processing unit 31a can perform high-precision VPS processing based on the image data of the subject and the point cloud data corresponding to the subject in the VPS server 13.

[0176] The communication unit 4 can obtain an imaging condition table based on location information acquired from, for example, the GPS receiver 7. Specifically, the communication unit 4 transmits location information to a server (for example, a VPS server 13) that has an imaging condition table. The server that has acquired the location information extracts information on subjects located within a predetermined range from the position of the imaging unit 2, for example, and transmits it to the communication unit 4 as an imaging condition table. The imaging condition table may also include location information of the subjects.

[0177] Furthermore, a server with an imaging condition table may change the priority of subjects to be extracted based on information such as VPS effectiveness, nighttime luminescence, and variability. VPS effectiveness is information indicating the accuracy of VPS processing using image data captured from subjects. The higher the VPS effectiveness, the higher the priority the server can give to extracting subjects.

[0178] Nighttime luminescence is information indicating the brightness (luminescence) of a subject at night. If a subject emits light, for example, due to internal or external lighting at night, the nighttime luminescence will be high. When the server transmits the imaging condition table to the communication unit 4 at night, it may prioritize extracting information on subjects with high nighttime luminescence.

[0179] Furthermore, when imaging subjects with high luminescence at night, the imaging unit 2 may shorten the exposure period for imaging to prevent image blur and overexposure of image data, thereby improving the accuracy of feature point extraction. Similarly, when imaging subjects with low luminescence at night, the exposure period for imaging may be lengthened.

[0180] Changeability indicates the possibility that the shape or other characteristics of a subject may change over time, or that the subject may disappear altogether. For example, mountains have low changeability, while simple structures such as prefabricated huts have high changeability. The server can extract subjects with higher priority the lower their changeability.

[0181] Furthermore, the server may change the priority of subject extraction based on the timestamp and other factors of the subject's information, as well as its likelihood of change. For example, if the timestamp is old and highly likely to change, the priority of subject extraction may be lowered. The imaging condition table transmitted to the communication unit 4 may include the timestamp and other factors of the subject's information.

[0182] Furthermore, information regarding VPS effectiveness, nighttime luminescence, and variability may be omitted from the imaging conditions table. In addition, the imaging conditions table may include information indicating the imaging timing (for example, information indicating imaging within X seconds).

[0183] Each record in the imaging conditions table may be stored on a database server or the like. Alternatively, file data containing at least a portion of the imaging conditions table may be stored on a file server or the like.

[0184] Figure 15 is a flowchart showing the operation of the imaging condition acquisition unit 41. First, the imaging condition acquisition unit 41 and the location information acquisition unit 26 acquire location information from the GPS receiver 7. Based on the location information, the imaging condition acquisition unit 41 and the communication unit 4 acquire imaging condition data (for example, the imaging condition table in Figure 14) from the VPS server 13 or the like (step S41).

[0185] Next, after a predetermined period has elapsed, the imaging condition acquisition unit 41 and the position information acquisition unit 26 acquire position information from the GPS receiver 7 (step S42). The movement amount determination unit 43 determines whether the change in the position of the imaging unit 2 since the last time imaging condition data was acquired is greater than or equal to a threshold (step S43).

[0186] If the change in the position of the imaging unit 2 is greater than or equal to a threshold, the imaging condition acquisition unit 41 and the communication unit 4 acquire imaging condition data based on the position information acquired in step S42 (step S44). After a predetermined period of time has elapsed, the process in step S42 is performed again.

[0187] If, in step S43, the change in the position of the imaging unit 2 is less than a threshold, the imaging condition acquisition unit 41 performs the process of step S42 again after a predetermined period of time has elapsed.

[0188] Figure 16 is a flowchart showing a first example of the operation of the VPS timing generation unit 32a. First, the attitude estimation unit 27 estimates the attitude based on sensor information (step S51). Next, the VPS timing generation unit 32a determines whether the estimated attitude matches the imaging conditions (step S52).

[0189] Specifically, in step S52, it is determined whether the attitude estimated in step S51 falls within the range of roll angle, pitch angle, and yaw angle for each record in the imaging condition table. Alternatively, in step S52, it may be determined whether the attitude estimated by the integrated filter unit 29 matches the imaging conditions.

[0190] If it is determined that the imaging conditions are met, the VPS timing generation unit 32a determines whether a certain period of time has elapsed since the previous VPS processing (step S53). If a certain period of time has elapsed since the previous VPS processing, the VPS timing generation unit 32a and the equipment status acquisition unit 45 determine whether the equipment status of the electronic device 1 is good or not (step S54).

[0191] For example, in step S54, it is determined whether there is sufficient battery charge remaining for the electronic device 1, whether there is sufficient thermal capacity for the electronic device 1 (i.e., whether the electronic device 1 is not overheating), and whether there is sufficient computing power remaining for the electronic device 1. The VPS timing generation unit 32a and the device status acquisition unit 45 may determine that the device status of the electronic device 1 is good if these conditions are met. Step S54 may be omitted.

[0192] If the electronic device 1 is in good condition, the VPS timing generation unit 32a instructs the VPS processing unit 31a to perform VPS processing (step S55). As a result, the imaging unit 2 captures image data and the VPS processing unit 31a performs VPS processing. The correction unit 46 corrects the attitude etc. estimated by the attitude estimation unit 27 based on the attitude etc. estimated by the VPS processing.

[0193] If it is determined in step S54 that the device status of electronic device 1 is not good, it is determined whether a long time has passed since the last VPS and whether the device is in a VPS-ready state (step S56).

[0194] In other words, if VPS processing has not been performed for a long period of time, even if the battery level, thermal margin, and computing power of the electronic device 1 are insufficient, if there is enough leeway in the battery level, etc. to allow VPS processing to be performed, the VPS processing in step S55 will be executed. Step S56 may be omitted.

[0195] After step S55, the process in step S51 is performed after a certain period of time has elapsed. Also, if it is determined in step S52 that the imaging conditions are not met, if it is determined in step S53 that a certain period of time has not elapsed since the last VPS, or if it is determined in step S56 that a long period of time has not elapsed since the last VPS, or that the electronic device 1 is not in a state where it can perform a VPS, the process in step S51 is performed after a certain period of time has elapsed.

[0196] In step S55 of Figure 16, as described above, the imaging unit 2 can image a subject for which point cloud data exists within the VPS server 13. This enables the VPS processing unit 31a to perform high-precision VPS processing. Note that if the VPS timing generation unit 32a operates as shown in Figure 16, the segmentation unit 44 in Figure 13 may be omitted.

[0197] Electronic device 1 can execute the processes in Figure 15 and Figure 16 in parallel. Electronic device 1 may execute the processes in steps S42 and S43 of Figure 15 at a frequency of, for example, once per second. Also, electronic device 1 may execute the process in Figure 16 at a frequency of, for example, once every 0.001 seconds.

[0198] In steps S54 and S56 of Figure 16, a decision may be made on whether or not to perform VPS processing based on the communication environment in addition to the equipment status. For example, if the communication environment is poor, a decision may be made not to perform VPS processing. Similarly, in step S43 of Figure 15, a decision may be made on whether or not to acquire imaging condition data based on the communication environment.

[0199] In addition to the operation shown in Figure 16, the display control unit 12a or the application unit 3a may guide the user to enable effective VPS processing. For example, the display device 10 may display AR content such as an arrow to guide the user to change the orientation of the imaging unit 2 to an orientation (hereinafter also referred to as the imaging direction) within the range of roll angle, pitch angle, or yaw angle specified in the imaging condition table. The AR content may be a character that moves in the imaging direction. The user may also be guided by sound information, etc. Alternatively, the display device 10 may superimpose the subject specified in each record of the imaging condition table (for example, a building, a mountain, etc.) and guide the user so that the subject is largely within the field of view. Alternatively, the orientation of the imaging unit 2 may be changed to the imaging direction by an arbitrary drive unit, etc. In addition to the above, for example, the electronic device 1 may determine an orientation or direction in which it can perform effective VPS based on point cloud data recorded in the VPS server 13 and guide the user by AR content or sound information, etc. The electronic device 1 may store orientations or directions in which effective VPS processing has been performed in the past and guide the user to those orientations or directions.

[0200] Furthermore, the communication unit 4 may acquire the imaging condition table and also acquire point cloud data corresponding to each record in the imaging condition table from the VPS server 13. In other words, after acquiring the imaging condition table, VPS processing may be performed without using the communication unit 4.

[0201] Furthermore, the electronic device 1 may acquire imaging condition data by means other than the communication unit 4 transmitting location information. For example, an edge computer or the like that has information on nearby buildings, etc., may detect the electronic device 1 located nearby and transmit an imaging condition table. The edge computer or the like may also transmit point cloud data, etc., along with the imaging condition table.

[0202] Figure 17 is a diagram illustrating the segmentation (semantic segmentation) process. Figure 17 shows image data captured by the imaging unit 2. The segmentation unit 44 can classify one or more subjects contained in the image data into multiple classes.

[0203] In segmentation processing, subjects can be classified into classes such as Sky, Building, Pole, Road, Pavement, Tree, Sign Symbol, Fence, Car, Pedestrian, Bicyclist, or Mountain. In Figure 17, multiple subjects are distinguished by different hatches for each class.

[0204] Among the various types of classes, it is desirable for VPS processing to extract feature points from subjects of classes that do not move and whose shape does not change (for example, buildings or mountains).

[0205] The segmentation unit 44 classifies the subjects in the image data into, for example, (1) buildings, (2) mountains, and (3) others. The VPS timing generation unit 32a can determine whether or not to perform VPS processing based on whether or not the image data contains subjects that are classified as, for example, (1) buildings or (2) mountains.

[0206] The class classified by the segmentation unit 44, and information on which class to perform VPS processing on, may be obtained from a cloud server or the like.

[0207] Figure 18 is a flowchart showing a second example of the operation of the VPS timing generation unit 32a. First, the VPS timing generation unit 32a estimates the attitude in the process of step S51, similar to Figure 17. Next, the VPS timing generation unit 32a determines whether the difference (attitude change) between the attitude estimated in the previous segmentation and the attitude estimated in step S51 is greater than or equal to a threshold (step S61).

[0208] If it is determined that the change in posture is above a threshold, the VPS timing generation unit 32a determines whether the electronic device 1 is in a state where it can perform VPS processing (step S62). In step S62, it is possible to determine whether the device is in a state where it can perform VPS processing based on the determinations in steps S53, S54, and S56 of Figure 17.

[0209] If it is determined that VPS processing is possible, the imaging unit 2 captures image data, and the segmentation unit 44 performs segmentation processing on the image data (step S63). The VPS timing generation unit 32a also determines whether or not there are subjects for which VPS processing is effective (step S64). In step S64, for example, it is determined whether or not there are subjects that fall under the categories of (1) buildings or (2) mountains.

[0210] If it is determined that there is a subject for which VPS processing is effective, the VPS processing unit 31a performs VPS processing based on the image data captured in step S63. The correction unit 46 corrects the posture etc. estimated by the posture estimation unit 27 based on the posture etc. estimated by the VPS processing (step S65).

[0211] After step S65, the process in step S51 is performed after a certain period of time has elapsed. Also, if the posture change is below the threshold in step S61, if it is determined in step S62 that the VPS processing is not possible, or if it is determined in step S64 that there are no subjects for which VPS processing is effective, the process in step S51 is performed after a certain period of time has elapsed.

[0212] In the operation shown in Figure 18, the VPS processing unit 31a can perform high-precision VPS processing based on image data that includes one or more subjects for which VPS processing is effective. Note that when the VPS timing generation unit 32a performs the operation shown in Figure 18, the imaging condition acquisition unit 41 and the imaging condition holding unit 42 shown in Figure 13 may be omitted.

[0213] Furthermore, the VPS timing generation unit 32a may perform an operation that combines the operation in Figure 16 and the operation in Figure 18. For example, in step S55 of Figure 16, the processing in steps S63 to S65 of Figure 18 may be performed.

[0214] In addition to the above, the VPS timing generation unit 32a may adjust the interval of VPS processing (hereinafter also referred to as the VPS rate). For example, the VPS rate may be adjusted by the gyrodrift or integration rate of the inertial measurement device 5. The VPS rate may also be increased when there is a large difference between the attitude obtained from the inertial measurement device 5 and the attitude obtained from the geomagnetic sensor 6. The VPS rate may also be increased when there is a large change in ambient temperature and the accuracy of the inertial measurement device 5 decreases. The VPS rate may also be increased when the accuracy of the GPS receiver 7 and geomagnetic sensor 6 decreases, such as when there are multiple buildings near the imaging unit 2 or when the imaging unit 2 is located inside a building. In this case, the VPS timing generation unit 32a may predict the degree of decrease in accuracy of the GPS receiver 7 and geomagnetic sensor 6 based on the rebar utilization rate of the building obtained from a cloud server or the like, or it may determine the effectiveness of VPS based on the presence or absence of point cloud data of subjects inside or around the building and adjust the VPS rate. Furthermore, the VPS timing generation unit 32a may adjust the VPS rate based on the equipment status (battery level, thermal margin, or computing power) acquired by the equipment status acquisition unit 45, or the surrounding communication environment, etc.

[0215] Furthermore, if it is possible to determine the timing for imaging structures or natural objects that can be effectively processed using VPS based on position information obtained from the GPS receiver 7, sensor information from the inertial measurement device 5, etc., and information on surrounding structures or natural objects obtained from the cloud server, etc., VPS processing may be performed preferentially. Also, if the attitude change of the imaging unit 2 is large and the accuracy of feature point extraction decreases, the VPS rate may be reduced. If the cessation of attitude change of the imaging unit 2 or a decrease in the amount of change is detected, VPS processing may be performed preferentially.

[0216] Furthermore, the electronic device 1 according to the second embodiment of this disclosure may have a configuration that includes the local VPS processing unit 33 shown in Figure 7. In this case, the electronic device 1 may decide whether to perform local VPS processing or VPS processing based on power consumption or expected processing delay.

[0217] Furthermore, the electronic device 1 may be configured to dynamically adjust the calculation accuracy (processing level) of VPS processing (or local VPS processing). The electronic device 1 may adjust the calculation accuracy of VPS processing, etc., based on the VPS rate, the equipment status acquired by the equipment status acquisition unit 45, the surrounding communication environment, the effectiveness of the VPS, the accuracy of the inertial measuring device 5, etc., power consumption, calculation amount, etc. Similarly, the electronic device 1 may adjust the resolution of the image data used for VPS processing, etc.

[0218] Furthermore, the electronic device 1 may predict what subject to be captured by the imaging unit 2 is based on information obtained from a cloud server or the like, calculate an appropriate shutter speed for VPS processing, and have the imaging unit 2 capture the image. For example, when capturing images at night, the exposure time may be reduced according to the amount of light from illuminated signs, etc., in order to speed up the imaging process and suppress image blur.

[0219] Thus, the electronic device 1 according to the second embodiment of this disclosure can perform VPS processing at an effective timing. Specifically, the VPS timing generation unit 32a can determine the timing at which VPS processing should be performed based on imaging condition data acquired from the VPS server 13 or the like. Alternatively, the VPS timing generation unit 32a can determine the timing at which VPS processing should be performed based on segmentation processing. Furthermore, the VPS timing generation unit 32a can determine whether or not to perform VPS processing based on the device status or the like.

[0220] As described above, the electronic device 1 can avoid performing VPS processing at times when VPS is not effective (i.e., when accuracy cannot be obtained). This allows the electronic device 1 to reduce the power consumption of VPS processing. The power consumption reduction effect is particularly large when depth images generated by distance measurement, etc., are used for VPS processing. The electronic device 1 according to the second embodiment of this disclosure can reduce battery capacity and can be miniaturized.

[0221] Electronic device 1 can also prioritize VPS processing if the accuracy of the inertial measuring device 5 or the like is degraded. This makes it possible to effectively correct the attitude and other values ​​estimated from the inertial measuring device 5 or the like, thereby improving the accuracy of attitude estimation and the accuracy of AR content alignment.

[0222] Electronic device 1 can also adjust the VPS rate or the calculation accuracy of VPS processing based on the device status, etc. Furthermore, electronic device 1 can guide the user to a position where effective VPS processing can be performed by displaying AR content, etc.

[0223] The processing and configuration of the electronic device 1 according to the second embodiment of this disclosure can also be applied to the electronic device 1 according to the first embodiment. For example, one or both of the VPS timing generation unit 32 or the local VPS timing generation unit 34 in Figure 7 may perform the operation shown in Figure 16 or Figure 18.

[0224] (Example of Pixel Configuration) The following describes pixels that can be applied to the display elements in the display unit 11 in Figure 1 or Figure 5.

[0225] In the following, regarding the components, only the essential parts of this disclosure are shown, along with other circuits and other appropriate components necessary for display. However, the pixels of the display unit 11 also include other components not shown as necessary for displaying images, etc.

[0226] The following examples indicate whether each transistor is n-type or p-type, but these are merely examples and are not restrictive; the polarity of the transistor is not particularly important as long as it functions correctly.

[0227] Furthermore, the light-emitting element in the following is, for example, an LED (Light Emitting Diode). LEDs include OLEDs used in micro-LED displays and OELs (Organic Electro Luminescence) used in organic EL displays. The light-emitting element may be configured such that its cathode is connected to ground voltage and it emits light due to the current flowing from the anode. Alternatively, as another example, the pixel may change its light emission intensity by controlling the liquid crystal.

[0228] Figure 19 shows an example configuration of a pixel PIX. The pixel PIX includes a capacitor C01, transistors MN02 to MN03, and a light-emitting element EL. Transistors MN02 to MN03 are N-type MOSFETs (Metal Oxide Semiconductor Field Effect Transistors). The gate of transistor MN02 is connected to the control line WSL, the other of its source and drain is connected to the signal line SGL, and one of its source and drain is connected to the gate of transistor MN03 and one end of capacitor C01. One end of capacitor C01 is connected to one of the source and drain of transistor MN02 and the gate of transistor MN03, and the other end is connected to one of the source and drain of transistor MN03 and the anode of the light-emitting element EL. The gate of transistor MN03 is connected to one of the source and drain of transistor MN02 and one end of capacitor C01, the other source and drain is connected to the power line VCCP, and one of the source and drain is connected to the other end of capacitor C01 and the anode of light-emitting element EL. The anode of light-emitting element EL is connected to one of the source and drain of transistor MN03 and the other end of capacitor C01, and its cathode is connected to the power line Vcath. The voltage of the power line VCCP is switched as appropriate between a first voltage and a second voltage lower than the first voltage.

[0229] In this configuration, when transistor MN02 is turned on in a pixel PIX, the voltage across capacitor C01 is set based on the pixel signal supplied from signal line SGL. During the period when the voltage of power line VCCP is the first voltage, transistor MN03 supplies a current to the light-emitting element EL corresponding to the voltage across capacitor C01. The light-emitting element EL emits light based on the current supplied from transistor MN03. In this way, the pixel PIX emits light with a brightness corresponding to the pixel signal. During the period when the voltage of power line VCCP is the second voltage, the light-emitting element EL is extinguished.

[0230] Figure 20 shows another example of a pixel PIX configuration. This pixel PIX includes capacitors C11 and C12, transistors MP12 to MP15, and light-emitting element EL. Transistors MP12 to MP15 are P-type MOSFETs. The gate of transistor MP12 is connected to the control line WSL, one of its source and drain is connected to the signal line SGL, and the other of its source and drain is connected to the gate of transistor MP14 and the other end of capacitor C12. One end of capacitor C11 is connected to the power line VCCP, and the other end is connected to one end of capacitor C12, the other of the source and drain of transistor MP13, and one of the source and drain of transistor MP14. One end of capacitor C12 is connected to the other end of capacitor C11, the other of the source and drain of transistor MP13, and one of the source and drain of transistor MP14, and the other end is connected to the other of the source and drain of transistor MP12 and the gate of transistor MP14. The gate of transistor MP13 is connected to the control line DSL, one of its source and drain is connected to the power line VCCP, and the other of its source and drain is connected to one of the source and drain of transistor MP14, the other end of capacitor C11, and one end of capacitor C12. The gate of transistor MP14 is connected to the other of the source and drain of transistor MP12 and the other end of capacitor C12, one of its source and drain is connected to the other of the source and drain of transistor MP13, the other end of capacitor C11, and one end of capacitor C12, and the other of its source and drain is connected to the anode of the light-emitting element EL and one of the source and drain of transistor MP15. The gate of transistor MP15 is connected to the control line AZSL, one of its source and drain is connected to the other of the source and drain of transistor MP14 and the anode of the light-emitting element EL, and the other of its source and drain is connected to the power line VSS.

[0231] In this configuration, in a pixel PIX, when transistor MP12 is turned on, the voltage across capacitor C12 is set based on the pixel signal supplied from signal line SGL. Transistor MP13 is turned on and off based on the signal from control line DSL. Transistor MP14 supplies a current to the light-emitting element EL corresponding to the voltage across capacitor C12 while transistor MP13 is on. The light-emitting element EL emits light based on the current supplied from transistor MP14. In this way, the pixel PIX emits light with brightness corresponding to the pixel signal. Transistor MP15 is turned on and off based on the signal from control line AZSL. While transistor MP15 is on, the voltage at the anode of the light-emitting element EL is initialized by being set to the voltage of power line VSS.

[0232] Furthermore, transistors MP12 to MP15 may be transistors made of low-temperature polycrystalline silicon (LTPS). Also, at least one of transistors MP12 and MP15 may be a transistor made of oxide semiconductor.

[0233] Figure 21 shows another example of a pixel PIX configuration. This pixel PIX includes a capacitor C21, transistors MN22 to MN25, and a light-emitting element EL. Transistors MN22 to MN25 are N-type MOSFETs. The gate of transistor MN22 is connected to the control line WSL, the other of its source and drain is connected to the signal line SGL, and one of its source and drain is connected to the gate of transistor MN24 and one end of capacitor C21. One end of capacitor C21 is connected to one of the source and drain of transistor MN22 and the gate of transistor MN24, and the other end is connected to one of the source and drain of transistor MN24, the other of the source and drain of transistor MN25, and the anode of the light-emitting element EL. The gate of transistor MN23 is connected to the control line DSL, the other of its source and drain is connected to the power line VCCP, and one of its source and drain is connected to the other of the source and drain of transistor MN24. The gate of transistor MN24 is connected to one of the source and drain of transistor MN22 and one end of capacitor C21, the other source and drain is connected to one of the source and drain of transistor MN23, and one source and drain is connected to the other end of capacitor C21, the other source and drain of transistor MN25 and the anode of light-emitting element EL. The gate of transistor MN25 is connected to the control line AZSL, the other source and drain is connected to one of the source and drain of transistor MN24, the other end of capacitor C21 and the anode of light-emitting element EL, and one source and drain is connected to the power line VSS.

[0234] In this configuration, in a pixel PIX, when transistor MN22 is turned on, the voltage across capacitor C21 is set based on the pixel signal supplied from signal line SGL. Transistor MN23 is turned on and off based on the signal from control line DSL. Transistor MN24 supplies a current to the light-emitting element EL corresponding to the voltage across capacitor C21 while transistor MN23 is on. The light-emitting element EL emits light based on the current supplied from transistor MN24. In this way, the pixel PIX emits light with brightness corresponding to the pixel signal. Transistor MN25 is turned on and off based on the signal from control line AZSL. While transistor MN25 is on, the voltage at the anode of the light-emitting element EL is initialized by being set to the voltage of power line VSS.

[0235] Transistors MN22 to MN25 may be transistors made of low-temperature polycrystalline silicon (LTPS). Furthermore, at least one of transistors MN22 and MN25 may be a transistor made of oxide semiconductor.

[0236] Figure 22 shows another example of a pixel PIX configuration. This pixel PIX includes a capacitor C31, transistors MP32 to MP36, and a light-emitting element EL. Transistors MP32 to MP36 are P-type MOSFETs. The gate of transistor MP32 is connected to the control line WSL, one of its source and drain is connected to the signal line SGL, and the other of its source and drain is connected to the gate of transistor MP33, the other of its source and drain of transistor MP34, and the other end of capacitor C31. One end of capacitor C31 is connected to the power line VCCP, and the other end is connected to the other of its source and drain of transistor MP32, the gate of transistor MP33, and the other of its source and drain of transistor MP34. The gate of transistor MP34 is connected to control line AZSL1, one of its source and drain is connected to the other of the source and drain of transistor MP33, and one of the source and drain of transistor MP35, and the other of its source and drain is connected to the other of the source and drain of transistor MP32, the gate of transistor MP33, and the other end of capacitor C31. The gate of transistor MP35 is connected to control line DSL, one of its source and drain is connected to the other of the source and drain of transistor MP33, and one of the source and drain of transistor MP34, and the other of its source and drain is connected to one of the source and drain of transistor MP36 and the anode of the light-emitting element EL. The gate of transistor MP36 is connected to control line AZSL2, one of its source and drain is connected to the other of the source and drain of transistor MP35, and the anode of the light-emitting element EL, and the other of its source and drain is connected to power line VSS.

[0237] In this configuration, in a pixel PIX, when transistor MP32 is turned ON, the voltage across capacitor C31 is set based on the pixel signal supplied from signal line SGL. Transistor MP35 is turned ON or OFF based on the signal from control line DSL. Transistor MP33 supplies a current to the light-emitting element EL corresponding to the voltage across capacitor C31 while transistor MP35 is ON. The light-emitting element EL emits light based on the current supplied from transistor MP33. In this way, the pixel PIX emits light with a brightness corresponding to the pixel signal. Transistor MP34 is turned ON or OFF based on the signal from control line AZSL1. While transistor MP34 is ON, the drain and gate of transistor MP33 are connected to each other. Transistor MP36 is turned ON or OFF based on the signal from control line AZSL2. While transistor MP36 is ON, the voltage of the anode of the light-emitting element EL is initialized by being set to the voltage of power line VSS.

[0238] Transistors MP32 to MP36 may be transistors made of low-temperature polycrystalline silicon (LTPS). Furthermore, at least one of transistors MP32, MP34, and MP36 may be a transistor made of oxide semiconductor.

[0239] Figure 23 shows another example of the pixel PIX configuration. One end of capacitor C48 is connected to the signal line SGL1, and the other end is connected to the power line VSS. One end of capacitor C49 is connected to the signal line SGL1, and the other end is connected to the signal line SGL2. Transistor MP49 is a P-type MOSFET, with its gate connected to the control line WSL2, one of its source and drain connected to the signal line SGL1, and the other of its source and drain connected to the signal line SGL2.

[0240] Each pixel PIX includes a capacitor C41, transistors MP42 to MP46, and a light-emitting element EL. Transistors MP42 to MP46 are P-type MOSFETs. The gate of transistor MP42 is connected to the control line WSL1, one of its source and drain is connected to the signal line SGL2, and the other of its source and drain is connected to the gate of transistor MP43 and the other end of capacitor C41. One end of capacitor C41 is connected to the power line VCCP, and the other end is connected to the other of its source and drain of transistor MP42 and the gate of transistor MP43. The gate of transistor MP43 is connected to the other of its source and drain of transistor MP42 and the other end of capacitor C41, one of its source and drain is connected to the power line VCCP, and the other of its source and drain is connected to one of its source and drain of transistors MP44 and MP45. The gate of transistor MP44 is connected to control line AZSL1, one of its source and drain is connected to the other of the source and drain of transistor MP43, and one of the source and drain of transistor MP45, and the other of its source and drain is connected to signal line SGL2. The gate of transistor MP45 is connected to control line DSL, one of its source and drain is connected to the other of the source and drain of transistor MP43, and one of the source and drain of transistor MP44, and the other of its source and drain is connected to one of the source and drain of transistor MP46 and the anode of the light-emitting element EL. The gate of transistor MP46 is connected to control line AZSL2, one of its source and drain is connected to the other of the source and drain of transistor MP45 and the anode of the light-emitting element EL, and the other of its source and drain is connected to power line VSS.

[0241] In this configuration, in a pixel PIX, when transistor MP42 is turned ON, the voltage across capacitor C41 is set based on the pixel signal supplied to signal line SGL1. Transistor MP45 is turned ON or OFF based on the signal on control line DSL. Transistor MP43 supplies a current to the light-emitting element EL corresponding to the voltage across capacitor C41 while transistor MP45 is ON. The light-emitting element EL emits light based on the current supplied by transistor MP43. In this way, the pixel PIX emits light with a brightness corresponding to the pixel signal. Transistor MP44 is turned ON or OFF based on the signal on control line AZSL1. While transistor MP44 is ON, the drain of transistor MP43 and signal line SGL2 are connected to each other. Transistor MP46 is turned ON or OFF based on the signal on control line AZSL2. While transistor MP46 is ON, the voltage of the anode of the light-emitting element EL is initialized by setting it to the voltage of power line VSS.

[0242] Furthermore, transistors MP42 to MP46 and MP49 may be transistors using low-temperature polycrystalline silicon (LTPS). Also, at least one of transistors MP42, MP46, and MP49 may be a transistor using an oxide semiconductor.

[0243] Figure 24 shows another example of a pixel PIX configuration. Multiple pixels PIX are arranged in a matrix in the display area Da, and the display area Da is located between the first control unit Ct1 and the second control unit Ct2.

[0244] The first control unit Ct1 includes transmission gates TG45 and TG46, transistors MP56 and MP57, and capacitor C61. Transistors MP56 and MP57 are P-type MOSFETs. A pixel signal is supplied to one end of transmission gate TG45, and the other end of transmission gate TG45 is connected to signal line SGL1. One end of transmission gate TG46 is connected to signal line SGL2, and the other end of transmission gate TG46 is connected to power line Vorst. One end of capacitor C61 is connected to signal line SGL1, and the other end is connected to power line VSS1. The gate of transistor MP56 is connected to control line INIL, one of its source and drain is connected to power line Vini, and the other of its source and drain is connected to signal line SGL2. The gate of transistor MP57 is connected to control line ELL, one of its source and drain is connected to power line Vel, and the other of its source and drain is connected to signal line SGL2.

[0245] The second control unit Ct2 includes a transmission gate TG72, a transistor MP73, and a capacitor C82. The transistor MP73 is a P-type MOSFET. One end of the transmission gate TG72 is connected to the signal line SGL1, and the other end is connected to the other of the source and drain of the transistor MP73 and to one end of the capacitor C82. The gate of the transistor MP73 is connected to the control line REFL, one of the source and drain is connected to the power line Vref, and the other of the source and drain is connected to the other end of the transmission gate TG72 and one end of the capacitor C82. One end of the capacitor C82 is connected to the other end of the transmission gate TG72 and the other of the source and drain of the transistor MP73, and the other end is connected to the signal line SGL2.

[0246] Each pixel PIX includes a capacitor C132, transistors MP121 to MP125, and a light-emitting element EL. Transistors MP121 to MP125 are P-type MOSFETs. The gate of transistor MP122 is connected to the control line WSL, one of its source and drain is connected to the signal line SGL2, and the other of its source and drain is connected to the gate of transistor MP121 and the other end of capacitor C132. One end of capacitor C132 is connected to the power line Vel, and the other end is connected to the other of its source and drain of transistor MP122 and the gate of transistor MP121. The gate of transistor MP121 is connected to the other of its source and drain of transistor MP122 and the other end of capacitor C132, one of its source and drain is connected to the power line Vel, and the other of its source and drain is connected to one of its source and drain of transistors MP123 and MP124. The gate of transistor MP123 is connected to the control line AZSL, one of its source and drain is connected to the other of the source and drain of transistor MP121, and one of the source and drain of transistor MP124, and the other of its source and drain is connected to the signal line SGL2. The gate of transistor MP124 is connected to the control line DSL, one of its source and drain is connected to the other of the source and drain of transistor MP121, and one of the source and drain of transistor MP123, and the other of its source and drain is connected to one of the source and drain of transistor MP125 and the anode of the light-emitting element 130. The gate of transistor MP125 is connected to the control line AZSL, the other of its source and drain is connected to the power line Vorst, and one of its source and drain is connected to the other of the source and drain of transistor MP124 and the anode of the light-emitting element 130.

[0247] In this configuration, in a pixel PIX, when transistor MP122 is turned ON, the voltage across capacitor C132 is set based on the pixel signal supplied to one end of transmission gate TG45. Transistor MP124 is turned ON or OFF based on the signal on control line DSL. Transistor MP121 supplies a current to the light-emitting element EL corresponding to the voltage across capacitor C132 during the period when transistor MP124 is ON. The light-emitting element EL emits light based on the current supplied from transistor MP121. In this way, the pixel PIX emits light with brightness corresponding to the pixel signal. Transistors MP123 and MP125 are turned ON or OFF based on the signal on control line AZSL. During the period when transistor MP123 is ON, the other of the source and drain of transistor MP121 and one of the source and drain of transistor MP124 are connected to signal line SGL2. During the period when transistor MP125 is ON, the voltage of the anode of the light-emitting element EL is initialized by setting it to the voltage of power line Vorst. Furthermore, transistor MP56 is switched on and off based on the signal of control line INIL, transistor MP57 is switched on and off based on the signal of control line ELL, and transistor MP73 is switched on and off based on the signal of control line REFL. When transistor MP56 is turned on, signal line SGL2 is set to the voltage of power line Vini, and when transistor MP57 is turned on, signal line SGL2 is set to the voltage of power line Vel. When transistor MP73 is turned on, one end of capacitor C82 is initialized by being set to the voltage of power line Vref.

[0248] Furthermore, transistors MP121 to MP125, MP56, and MP57 may be transistors made of low-temperature polycrystalline silicon (LTPS). Also, at least one of transistors MP122 and MP125 may be a transistor made of oxide semiconductor.

[0249] Figure 25 shows another example of a pixel PIX configuration. This pixel PIX includes a capacitor C51, transistors MP52 to MP60, and a light-emitting element EL. Transistors MP52 to MP60 are P-type MOSFETs. The gate of transistor MP52 is connected to the control line WSL, one of its source and drain is connected to the signal line SGL, and the other of its source and drain is connected to the other of its source and drain of transistor MP53 and one of its source and drain of transistor MP54. The gate of transistor MP53 is connected to the control line DSL, one of its source and drain is connected to the power line VCCP, and the other of its source and drain is connected to the other of its source and drain of transistor MP52 and one of its source and drain of transistor MP54. The gate of transistor MP54 is connected to one of the source and drain of transistor MP55, the other of the source and drain of transistor MP57, and the other end of capacitor C51. One of the source and drain is connected to the other of the source and drain of transistors MP52 and MP53, and the other of the source and drain is connected to one of the source and drain of transistors MP58 and MP59. One end of capacitor C51 is connected to the power line VCCP, and the other end is connected to the gate of transistor MP54, one of the source and drain of transistor MP55, and the other of the source and drain of transistor MP57. Capacitor C51 may include two capacitors connected in parallel with each other. The gate of transistor MP55 is connected to the control line AZSL1, one of the source and drain is connected to the gate of transistor MP54, the other of the source and drain of transistor MP57, and the other end of capacitor C51, and the other of the source and drain is connected to one of the source and drain of transistor MP56. The gate of transistor MP56 is connected to the control line AZSL1, one of its source and drain is connected to the other of its source and drain, and the other of its source and drain is connected to the power line VSS.The gate of transistor MP57 is connected to the control line WSL, the other of its source and drain is connected to the gate of transistor MP54, one of the source and drain of transistor MP55, and the other end of capacitor C51, and one of its source and drain is connected to the other of its source and drain of transistor MP58. The gate of transistor MP58 is connected to the control line WSL, the other of its source and drain is connected to one of the source and drain of transistor MP57, the other of its source and drain is connected to the other of its source and drain of transistor MP54, and one of its source and drain is connected to one of its source and drain of transistor MP59. The gate of transistor MP59 is connected to the control line DSL, the other of its source and drain is connected to the other of its source and drain of transistor MP54, and one of its source and drain is connected to one of its source and drain of transistor MP58, and the other of its source and drain is connected to one of its source and drain of transistor MP60, and the anode of the light-emitting element EL. The gate of transistor MP60 is connected to the control line AZSL2, one of its source and drain is connected to the other of its source and drain of transistor MP59, and to the anode of the light-emitting element EL, while the other of its source and drain is connected to the power line VSS.

[0250] In this configuration, in the pixel PIX, the voltage across capacitor C51 is set based on the pixel signal supplied from signal line SGL when transistors MP52, MP54, MP58, and MP57 are turned ON. Transistors MP53 and MP59 are turned ON and OFF based on the signal from control line DSL. Transistor MP54 supplies a current to the light-emitting element EL corresponding to the voltage across capacitor C51 while transistors MP53 and MP59 are ON. The light-emitting element EL emits light based on the current supplied from transistor MP54. In this way, the pixel PIX emits light with brightness corresponding to the pixel signal. Transistors MP55 and MP56 are turned ON and OFF based on the signal from control line AZSL1. While transistors MP55 and MP56 are ON, the gate voltage of transistor MP54 is initialized by being set to the voltage of power line VSS. Transistor MP60 is turned ON and OFF based on the signal from control line AZSL2. During the period when transistor MP60 is ON, the voltage of the anode of the light-emitting element EL is initialized by setting it to the voltage of the power line VSS.

[0251] Transistors MP52 to MP60 may be transistors using low-temperature polycrystalline silicon (LTPS). Also, at least one of transistors MP55 to MP58 and MP60 may be a transistor using an oxide semiconductor.

[0252] Figure 26 shows another example of a pixel PIX configuration. The signals of control line WSNL and control line WSPL are inverted signals of each other.

[0253] Each pixel PIX includes capacitors C61 and C62, transistors MN63, MP64, MN65-MN67, and light-emitting element EL. Transistors MN63, MN65-MN67 are N-type MOSFETs, and transistor MP64 is a P-type MOSFET. The gate of transistor MN63 is connected to the control line WSNL, and the other of its source and drain is connected to the signal line SGL and one of the source and drain of transistor MP64. The other of its source and drain is connected to the other of the source and drain of transistor MP64, one end of capacitors C61 and C62, and the gate of transistor MN65. The gate of transistor MP64 is connected to the control line WSPL, and the other of its source and drain is connected to the signal line SGL and the other of the source and drain of transistor MN63. The other of its source and drain is connected to one of the source and drain of transistor MN63, one end of capacitors C61 and C62, and the gate of transistor MN65. Capacitor C61 is constructed using, for example, a MOM (Metal Oxide Metal) capacitor, with one end connected to one of the source and drain of transistor MN63, the other of the source and drain of transistor MP64, one end of capacitor C62, and the gate of transistor MN65, and the other end connected to the power line VSS2. Capacitor C61 may also be constructed using, for example, a MOS capacitor or a MIM (Metal Insulator Metal) capacitor. Capacitor C62 is constructed using, for example, a MOS capacitor, with one end connected to one of the source and drain of transistor MN63, the other of the source and drain of transistor MP64, one end of capacitor C61, and the gate of transistor MN65, and the other end connected to the power line VSS2. Capacitor C62 may also be constructed using, for example, a MOM capacitor or a MIM capacitor. The other end of capacitor C62 may also be connected to the power line VSS3 (not shown).The gate of transistor MN65 is connected to one of the source and drain of transistor MN63, the other of the source and drain of transistor MP64, and one end of capacitors C61 and C62. The other of the source and drain is connected to the power line VCCP, and the other of the source and drain is connected to the other of the source and drain of transistors MN66 and MN67. The gate of transistor MN66 is connected to the control line AZL, and the other of the source and drain is connected to one of the source and drain of transistor MN65 and the other of the source and drain of transistor MN67, and the other of the source and drain is connected to the power line VSS1. The gate of transistor MN67 is connected to the control line DSL, and the other of the source and drain is connected to one of the source and drain of transistor MN65 and the other of the source and drain of transistor MN66, and the other of the source and drain is connected to the anode of the light-emitting element EL. Alternatively, transistor MN67 and control line DSL may be omitted, and one of the source and drain of transistor MN65 may be connected to the other of the source and drain of transistor MN66, and to the anode of the light-emitting element EL.

[0254] In this configuration, at least one of transistors MN63 and MP64 is turned on in the pixel PIX, setting the voltage across capacitors C61 and C62 based on the pixel signal supplied from signal line SGL. Transistor MN67 is turned on and off based on the signal from control line DSL. Transistor MN65 supplies a current to the light-emitting element EL corresponding to the voltage across capacitors C61 and C62 during the period when transistor MN67 is on. The light-emitting element EL emits light based on the current supplied from transistor MP65. In this way, the pixel PIX emits light with a brightness corresponding to the pixel signal. Transistor MN66 may be turned on and off based on the signal from control line AZL. Transistor MN66 may also function as a resistive element having a resistance value corresponding to the signal from control line AZL. In this case, transistors MN65 and MN66 constitute a so-called source follower circuit.

[0255] Furthermore, transistors MN63, MP64, and MN65-MN67 may be transistors using low-temperature polycrystalline silicon (LTPS). Also, at least one of transistors MN63, MP64, and MN66 may be a transistor using an oxide semiconductor.

[0256] Figure 27 shows another example of a pixel PIX configuration. This pixel PIX includes a capacitor C71, transistors MN72 to MN77, and a light-emitting element EL. Transistors MN72 to MN77 are N-type MOSFETs. The gate of transistor MN72 is connected to the control line WSL, the other of its source and drain is connected to the signal line SGL, and one of its source and drain is connected to one of the source and drain of transistor MN74 and the other of the source and drain of transistor MN75. One end of capacitor C71 is connected to the gate of transistor MN74 and one of the source and drain of transistor MN76, and the other end is connected to the other of the source and drain of transistor MN77, one of the source and drain of transistor MN75, and the anode of the light-emitting element EL. The gate of transistor MN73 is connected to control line DSL1, the other of its source and drain is connected to power line VCCP, and one of its source and drain is connected to the other of its source and drain of transistor MN74 and the other of its source and drain of transistor MN76. The gate of transistor MN74 is connected to one of its source and drain of transistor MN76 and one end of capacitor C71, the other of its source and drain is connected to one of its source and drain of transistor MN73 and the other of its source and drain of transistor MN76, and one of its source and drain is connected to one of its source and drain of transistor MN72 and the other of its source and drain of transistor MN75. The gate of transistor MN75 is connected to control line DSL2, the other of its source and drain is connected to one of the source and drain of transistor MN72 and one of the source and drain of transistor MN74, and one of its source and drain is connected to the other end of capacitor C71, the other of the source and drain of transistor MN77 and the anode of light-emitting element EL.The gate of transistor MN76 is connected to the control line AZSL, and the other of its source and drain is connected to one of the source and drain of transistor MN73 and the other of the source and drain of transistor MN74, with one of its source and drain connected to the gate of transistor MN74 and one end of capacitor C71. The gate of transistor MN77 is connected to the control line AZSL, and the other of its source and drain is connected to the other end of capacitor C71, one of the source and drain of transistor MN75 and the anode of light-emitting element EL, with one of its source and drain connected to the power line VSS.

[0257] In this configuration, in the pixel PIX, the voltage across capacitor C71 is set based on the pixel signal supplied from signal line SGL when transistors MN72, MN74, and MN76 are turned on. Transistor MN73 is turned on and off based on the signal from control line DSL1, and transistor MN75 is turned on and off based on the signal from control line DSL2. Transistor MN74 supplies a current to the light-emitting element EL corresponding to the voltage across capacitor C71 while transistors MN73 and MN75 are turned on. The light-emitting element EL emits light based on the current supplied from transistor MN74. In this way, the pixel PIX emits light with brightness corresponding to the pixel signal. Transistor MN77 is turned on and off based on the signal from control line AZSL. While transistor MN77 is turned on, the voltage of the anode of the light-emitting element EL is initialized by being set to the voltage of power line VSS.

[0258] Transistors MN72 to MN77 may be transistors made of low-temperature polycrystalline silicon (LTPS). Transistor MN76 may be a transistor made of oxide semiconductor.

[0259] (Application Examples) The following describes application examples of the electronic device 1 and display device 10 described in the above embodiment and modified examples.

[0260] (Application Example 1) Figure 28 shows an example of the appearance of a head-mounted display 110. The head-mounted display 110 has, for example, a glasses-shaped display unit 111 and ear hooks 112 on both sides for attachment to the user's head. The technology according to the above embodiment can be applied to such a head-mounted display 110.

[0261] (Application Example 2) Figure 29 shows an example of the appearance of another head-mounted display 120. The head-mounted display 120 is a transmissive head-mounted display having a main body 121, an arm 122, and a lens barrel 123. This head-mounted display 120 is attached to eyeglasses 128. The main body 121 has a control board and a display unit for controlling the operation of the head-mounted display 120. This display unit emits image light of the displayed image. The arm 122 connects the main body 121 and the lens barrel 123 and supports the lens barrel 123. The lens barrel 123 projects the image light supplied from the main body 121 via the arm 122 towards the user's eyes through the lenses 129 of the eyeglasses 128. The technology according to the above embodiment can be applied to such a head-mounted display 120.

[0262] This head-mounted display 120 is a so-called light guide plate type head-mounted display, but is not limited to this; for example, it may be a so-called birdbath type head-mounted display. This birdbath type head-mounted display includes, for example, a beam splitter and a partially transparent mirror. The beam splitter outputs light encoded with image information toward the mirror, and the mirror reflects the light toward the user's eyes. Both the beam splitter and the partially transparent mirror are partially transparent. This allows light from the surrounding environment to reach the user's eyes.

[0263] (Application Example 3) Figures 30A and 30B show an example of the external appearance of a digital still camera 130, with Figure 30A showing a front view and Figure 30B showing a rear view. This digital still camera 130 is a single-lens reflex type camera with interchangeable lenses and has a camera body 131, an imaging lens unit 132, a grip 133, a monitor 134, and an electronic viewfinder 135. The imaging lens unit 312 is an interchangeable lens unit and is located near the center of the front of the camera body 311. The grip 133 is located on the left side of the front of the camera body 311, and the photographer holds this grip 133. The monitor 134 is located to the left of the center of the rear of the camera body 131. The electronic viewfinder 135 is located on the rear of the camera body 131, above the monitor 134. The photographer can look through the electronic viewfinder 135 to see the light image of the subject guided by the shooting lens unit 132 and determine the composition. The technology according to the above embodiment can be applied to the electronic viewfinder 135.

[0264] (Application Example 4) Figure 31 shows an example of the appearance of a smartphone 150. The smartphone 150 has a display unit 151 that displays various information and an operation unit 152 that includes buttons and the like for receiving user input. The technology according to the above embodiment can be applied to this display unit 151.

[0265] (Application Example 5) Figures 32A and 32B show an example of a vehicle configuration to which the technology of this disclosure is applied. Figure 32A shows an example of the interior of the vehicle as seen from the rear of the vehicle 200, and Figure 32B shows an example of the interior of the vehicle as seen from the left rear of the vehicle 200.

[0266] The vehicles in Figures 32A and 32B include a center display 201, a console display 202, a head-up display 203, a digital rearview mirror 204, a steering wheel display 205, and a rear entertainment display 106.

[0267] The center display 201 is positioned on the dashboard 261, facing the driver's seat 262 and the passenger seat 263. Figure 32A shows an example of a horizontally elongated center display 201 extending from the driver's seat 262 to the passenger seat 263, but the screen size and placement of the center display 201 are not limited to this. The center display 201 can display information detected by various sensors. As a specific example, the center display 201 can display images captured by an image sensor, distance images to obstacles in front of and to the side of the vehicle measured by a ToF sensor, and the body temperature of occupants detected by an infrared sensor. The center display 201 can be used to display at least one of the following: safety-related information, operation-related information, life logs, health-related information, authentication / identification-related information, and entertainment-related information.

[0268] Safety-related information includes data based on sensor detection results, such as drowsiness detection, distraction detection, detection of mischief by passengers, seatbelt usage status, and detection of unattended occupants. Operation-related information includes information on occupant gestures detected using sensors. Gestures may include operation of various in-vehicle equipment, such as air conditioning, navigation systems, AV (Audio Visual) systems, and lighting systems. Lifelogs include the lifelogs of all occupants. For example, lifelogs include records of each occupant's actions. By acquiring and saving lifelogs, it is possible to check the condition of occupants in the event of an accident. Health-related information includes the occupant's body temperature detected using temperature sensors, and information on the occupant's health status inferred from the detected body temperature. Alternatively, information on the occupant's health status may be inferred based on the occupant's face captured by an image sensor. Furthermore, information on the occupant's health status may be inferred based on the occupant's responses obtained by conversing with the occupant using automated voice. Authentication / identification-related information includes information such as keyless entry functions that use sensors for facial recognition and functions that automatically adjust seat height and position based on facial recognition. Entertainment-related information includes information on AV equipment operation by occupants detected by sensors, and information on content to be displayed that is appropriate for occupants detected and recognized by sensors.

[0269] The console display 202 can be used, for example, to display life log information. The console display 202 is located near the shift lever 265 in the center console 264 between the driver's seat 262 and the passenger seat 263. The console display 202 can also display information detected by various sensors. In addition, the console display 202 may display images of the area around the vehicle captured by an image sensor, or it may display distance images to obstacles around the vehicle.

[0270] The head-up display 203 is virtually displayed behind the windshield 266 in front of the driver's seat 262. The head-up display 203 can be used to display, for example, at least one of safety-related information, operation-related information, life logs, health-related information, authentication / identification-related information, and entertainment-related information. Because the head-up display 203 is often virtually positioned in front of the driver's seat 262, it is suitable for displaying information directly related to vehicle operation, such as vehicle speed, fuel level, and battery level.

[0271] The digital rearview mirror 204 can not only display the area behind the vehicle, but also show the condition of the rear-seat passengers. Therefore, it can be used, for example, to display life log information of rear-seat passengers.

[0272] The steering wheel display 205 is positioned near the center of the vehicle's steering wheel 267. The steering wheel display 205 can be used to display at least one of the following: safety-related information, operation-related information, life log, health-related information, authentication / identification-related information, and entertainment-related information. In particular, because the steering wheel display 205 is located near the driver's hands, it is suitable for displaying life log information such as the driver's body temperature, or information related to the operation of AV equipment, air conditioning equipment, etc.

[0273] The rear entertainment display 206 is mounted on the back of the driver's seat 262 and the passenger seat 263 and is intended for viewing by rear-seat passengers. The rear entertainment display 206 can be used to display at least one of the following: safety-related information, operation-related information, life logs, health-related information, authentication / identification-related information, and entertainment-related information. In particular, because the rear entertainment display 206 is in front of the rear-seat passengers, it displays information relevant to the rear-seat passengers. For example, the rear entertainment display 206 may display information related to the operation of AV equipment or air conditioning equipment, or it may display the results of temperature sensor measurements of the rear-seat passengers' body temperature, etc.

[0274] The technologies described in the above embodiments can be applied to these center displays 201, console displays 202, head-up displays 203, digital rear mirrors 204, steering wheel displays 205, and rear entertainment displays 206.

[0275] Furthermore, this technology can take the following configurations: (1) A display device comprising: a display unit that displays image data based on the imaging results of an imaging unit; and a display control unit that controls the display unit, wherein the display control unit comprises: an estimation unit that estimates the posture or amount of posture change of the imaging unit based on information from a detection unit that detects the posture or change of posture of the imaging unit; and a specification unit that identifies the cause of an error in the posture or amount of posture change estimated by the estimation unit based on the difference between the posture or amount of posture change of the imaging unit detected based on the imaging results of the imaging unit and the posture or amount of posture change estimated by the estimation unit; (2) The display device according to (1), wherein the specification unit identifies the cause of a plurality of errors that occurred at multiple different timings; (3) The display device according to (2), wherein the specification unit identifies the cause of the plurality of errors based on at least one of the drift error of the gyro sensor or the error of geomagnetic disturbance in addition to the difference; (4) The display device according to (3), further comprising: a correction unit that corrects the plurality of errors based on at least one of the drift error or the error of geomagnetic disturbance. (5) The display device according to (4), wherein the correction unit corrects the attitude or attitude change amount obtained from the gyro sensor if the plurality of errors include the drift error of the gyro sensor, and corrects the gain for the attitude obtained from the geomagnetic sensor if the plurality of errors do not include the drift error of the gyro sensor. (6) The display device according to (2), wherein the identification unit identifies the cause of the error in the attitude or attitude change amount estimated by the estimation unit and the cause of the error in the attitude or attitude change amount detected based on the image data. (7) The display device according to (6), wherein the identification unit identifies the cause of the error based on a correction parameter that corrects the attitude or attitude change amount detected based on the image data and the attitude or attitude change amount estimated by the estimation unit, and a correction formula that calculates the correction amount of the correction parameter.(8) The display device according to (6) or (7), wherein the error in the attitude or attitude change amount estimated by the estimation unit includes at least one of the mechanical position error of the gyro sensor, the mechanical position error of the geomagnetic sensor, the drift error of the gyro sensor, the geomagnetic disturbance error, the sampling time error of the gyro sensor, or the sampling time error of the geomagnetic sensor, and the error detected based on the image data includes at least one of the mechanical position error of the imaging unit, the lens distortion error of the imaging unit, or the sampling time error of the imaging unit. (9) A display device comprising: a display unit that displays image data based on the imaging result of the imaging unit; an acquisition unit that acquires the attitude or attitude change amount of the imaging unit detected based on the imaging result of the imaging unit when predetermined conditions are met; a correction unit that corrects information from a detection unit that detects the attitude or attitude change of the imaging unit based on the attitude or attitude change amount acquired by the acquisition unit; and a determination unit that determines whether or not the predetermined conditions are met. (10) The display device according to (9), wherein the determination unit determines whether or not the predetermined conditions are met based on the position, orientation, or image data of the imaging unit. (11) The display device according to (9) or (10), comprising: a position determination unit that identifies the current position of the imaging unit; a movement determination unit that determines whether or not the current position identified by the position determination unit has moved by a predetermined threshold; and an imaging condition acquisition unit that, when it is determined that the current position has moved by a predetermined threshold, acquires imaging conditions corresponding to the current position identified by the position determination unit, wherein the determination unit determines whether or not the predetermined conditions are met based on the imaging conditions acquired by the imaging condition acquisition unit. (12) The display device according to (11), wherein the imaging conditions include information relating to the direction in which the imaging unit should image, and the predetermined conditions include the imaging unit being in an orientation that allows imaging in the direction included in the imaging conditions. (13) The display device according to (9) or (10), wherein the predetermined conditions include the change from the orientation or change in orientation most recently detected by the detection unit being greater than or equal to a predetermined threshold.(14) The display device according to (13), wherein the acquisition unit acquires information from a calculation unit that calculates the posture or posture change amount of the imaging unit based on the imaging results of the imaging unit, and the predetermined condition includes that the imaging unit can image a subject suitable for the calculation unit to calculate the posture or posture change amount. (15) The display device according to (14), comprising an extraction unit that extracts a subject from image data including one or more subjects by segmentation, and the predetermined condition includes that the extraction unit can extract a subject from image data including one or more subjects that is suitable for the calculation unit to acquire the posture or posture change amount. (16) The display device according to any one of (9) to (15), wherein the predetermined condition includes at least one of the following conditions: battery level, heat generation conditions, or calculation capacity. (17) The display device according to any one of (1) to (8), wherein the identification unit identifies the cause of the error based on the posture or posture change amount of the imaging unit detected based on the image data captured by the imaging unit, and the display unit displays the image data captured by the imaging unit. (18) The display device according to any one of (1) to (8), wherein the identification unit identifies the cause of the error based on the posture or amount of posture change of the imaging unit detected based on the first image data captured by the imaging unit, and the display unit displays second image data whose display position has been adjusted based on the posture or amount of posture change of the imaging unit detected based on the first image data. (19) The display device according to any one of (9) to (16), wherein the acquisition unit acquires the posture or amount of posture change of the imaging unit detected based on the image data captured by the imaging unit, and the display unit displays image data captured by the imaging unit. (20) The display device according to any one of (9) to (16), wherein the acquisition unit acquires the posture or amount of posture change of the imaging unit detected based on the first image data captured by the imaging unit, and the display unit displays second image data whose display position has been adjusted based on the posture or amount of posture change of the imaging unit detected based on the first image data.(21) A display device according to any one of (1) to (8), (17), or (18), comprising: a communication unit that communicates with a communication device; and an acquisition unit that acquires the attitude or attitude change amount of the imaging unit detected based on image data of a predetermined subject captured by the imaging unit and point cloud data of the subject held by the communication device. (22) A display device according to any one of (9) to (16), (19), or (20), comprising: a communication unit that communicates with a communication device, wherein the acquisition unit acquires the attitude or attitude change amount of the imaging unit detected based on image data of a predetermined subject captured by the imaging unit and point cloud data of the subject held by the communication device. (23) A display device according to (21) or (22), wherein the acquisition unit acquires the attitude or attitude change amount of the imaging unit detected by VPS. (24) The display device according to any one of (1) to (22), further comprising a posture change amount estimation unit that estimates the amount of posture change of the imaging unit based on a third image data captured by the imaging unit and a fourth image data captured by the imaging unit at a time different from the third image data. (25) The display device according to (24), wherein the posture change amount estimation unit estimates the amount of posture change of the imaging unit based on a first feature point extracted from the third image data and a second feature point extracted from the fourth image data that corresponds to the first feature point. (26) The display device according to any one of (1) to (25), further comprising an image generation unit that generates a sixth image data by superimposing a fifth image data on a first image data captured by the imaging unit based on the posture or amount of posture change detected based on the first image data. (27) Electronic device comprising: a display device having a display unit; and a control device, the control device comprising: an imaging unit that takes images for displaying image data on the display unit; a detection unit that detects the posture or posture change of the imaging unit; an estimation unit that estimates the posture or amount of posture change of the imaging unit based on information from the detection unit; and an identification unit that identifies the cause of an error in the posture or amount of posture change estimated by the estimation unit based on the difference between the posture or amount of posture change of the imaging unit detected based on the image data and the posture or amount of posture change estimated by the estimation unit.(28) The electronic device according to (27), wherein the display device has a display control unit that controls the display unit, and the control unit or the display control unit performs processing of at least one of the estimation unit and the identification unit. (29) A communication device comprising: an information holding unit that holds information of an object in which point cloud data is stored in a storage unit; and a communication unit that transmits information regarding the direction in which the imaging unit should image based on the information of the object. (30) The communication device according to (29), wherein the communication unit transmits information regarding the direction in which the imaging unit should image to an electronic device having an imaging unit and an acquisition unit that acquires the orientation of the imaging unit based on the point cloud data and image data captured by the imaging unit. (31) The display device according to (7), wherein the correction formula includes a Jacobian matrix having a partial derivative with respect to the correction parameter. (32) The display device according to (13), wherein the determination unit determines whether or not to instruct the imaging unit to image when the change in orientation or the amount of change from the orientation most recently detected by the detection unit is greater than or equal to a predetermined threshold.

[0276] The aspects of this disclosure are not limited to the individual embodiments described above, but include various modifications that a person skilled in the art could conceive, and the effects of this disclosure are not limited to those described above. In other words, various additions, modifications, and partial deletions are possible, as long as they do not depart from the conceptual idea and spirit of this disclosure derived from the claims and their equivalents.

[0277] 1, 1a Electronic equipment, 2 Imaging unit, 3, 3a Application unit, 4 Communication unit, 5 Inertial measurement device, 6 Geomagnetic sensor, 7 GPS receiver, 10 Display device, 11 Display unit, 12, 12a Display control unit, 13 VPS server, 21 Image acquisition unit, 22 VPS information acquisition unit, 23 VPS processing unit, 24, 25 Sensor information acquisition unit, 26 Position information acquisition unit, 27 Attitude estimation unit, 28 Error correction unit, 29 Integrated filter unit, 30 Attitude information utilization unit, 31 Main VPS processing unit, 31a VPS processing unit, 32, 32a VPS timing generation unit, 33 Local VPS processing unit, 34 Local VPS timing generation unit, 35 Difference detection unit, 36 Error history retention unit, 37 Error cause identification unit, 38 Correction amount generation unit, 39 Correction formula retention unit, 41 Imaging condition acquisition unit, 42 Image condition holding unit, 43 Movement amount determination unit, 44 Segmentation unit, 45 Equipment status acquisition unit, 46 Correction unit

Claims

1. A display device comprising: a display unit that displays image data based on imaging results from an imaging unit; and a display control unit that controls the display unit, wherein the display control unit includes: an estimation unit that estimates the posture or amount of posture change of the imaging unit based on information from a detection unit that detects the posture or change in posture of the imaging unit; and an identification unit that identifies the cause of an error in the posture or amount of posture change estimated by the estimation unit based on the difference between the posture or amount of posture change of the imaging unit detected based on imaging results from the imaging unit and the posture or amount of posture change estimated by the estimation unit.

2. The display device according to claim 1, wherein the identifying unit identifies the cause of multiple errors that occurred at multiple different timings.

3. The display device according to claim 2, wherein the identifying unit identifies the cause of the plurality of errors based on at least one of the drift error of the gyro sensor or the error of geomagnetic disturbance, in addition to the difference.

4. The display device according to claim 3, further comprising a correction unit for correcting the plurality of errors based on at least one of the drift error or the geomagnetic disturbance error.

5. The display device according to claim 4, wherein the correction unit corrects the attitude or attitude change amount obtained from the gyro sensor if the plurality of errors include the drift error of the gyro sensor, and corrects the gain for attitude obtained from the geomagnetic sensor if the plurality of errors do not include the drift error of the gyro sensor.

6. The display device according to claim 2, wherein the identifying unit identifies the cause of the error in the posture or posture change amount estimated by the estimation unit and the cause of the error in the posture or posture change amount detected based on the image data.

7. The display device according to claim 6, wherein the identifying unit identifies the cause of the error based on a correction parameter for correcting the posture or posture change amount detected based on the image data and the posture or posture change amount estimated by the estimation unit, and a correction formula for calculating the correction amount of the correction parameter.

8. The display device according to claim 6, wherein the error in the attitude or attitude change amount estimated by the estimation unit includes at least one of the mechanical position error of the gyro sensor, the mechanical position error of the geomagnetic sensor, the drift error of the gyro sensor, the geomagnetic disturbance error, the sampling time error of the gyro sensor, or the sampling time error of the geomagnetic sensor, and the error detected based on the image data includes at least one of the mechanical position error of the imaging unit, the lens distortion error of the imaging unit, or the sampling time error of the imaging unit.

9. A display device comprising: a display unit that displays image data based on the imaging results of an imaging unit; an acquisition unit that acquires the posture or posture change amount of the imaging unit detected based on the imaging results of the imaging unit when predetermined conditions are met; a correction unit that corrects information from a detection unit that detects the posture or posture change of the imaging unit based on the posture or posture change amount acquired by the acquisition unit; and a determination unit that determines whether or not the predetermined conditions are met.

10. The display device according to claim 9, wherein the determination unit determines whether or not the predetermined conditions are met based on the position, orientation, or image data of the imaging unit.

11. The display device according to claim 9, comprising: a position identification unit that identifies the current position of the imaging unit; a movement determination unit that determines whether the current position identified by the position identification unit has moved by more than a predetermined threshold; and an imaging condition acquisition unit that, when it is determined that the current position has moved by more than a predetermined threshold, acquires imaging conditions corresponding to the current position identified by the position identification unit, wherein the determination unit determines whether the predetermined conditions are met based on the imaging conditions acquired by the imaging condition acquisition unit.

12. The display device according to claim 11, wherein the imaging conditions include information relating to the direction in which the imaging unit should capture images, and the predetermined conditions include the imaging unit being in a position that enables it to capture images in the direction included in the imaging conditions.

13. The display device according to claim 9, wherein the predetermined condition includes that the amount of change from the posture or posture change amount most recently detected by the detection unit is greater than or equal to a predetermined threshold.

14. The display device according to claim 13, wherein the acquisition unit acquires information from a calculation unit that calculates the posture or posture change amount of the imaging unit based on the imaging results of the imaging unit, and the predetermined condition includes that the imaging unit can image a subject suitable for the calculation unit to calculate the posture or posture change amount.

15. The display device according to claim 14, comprising an extraction unit for extracting subjects from image data including one or more subjects by segmentation, wherein the predetermined condition includes that the extraction unit is capable of extracting subjects from the image data including one or more subjects that are suitable for the calculation unit to obtain a posture or a change in posture amount.

16. The display device according to claim 9, wherein the predetermined conditions include at least one of the following conditions: battery level, heat generation conditions, or computing power.

17. The display device according to claim 1, wherein the identifying unit identifies the cause of the error based on the posture or change in posture of the imaging unit detected based on the image data captured by the imaging unit, and the display unit displays the image data captured by the imaging unit.

18. The display device according to claim 1, wherein the identifying unit identifies the cause of the error based on the posture or change in posture of the imaging unit detected based on the first image data captured by the imaging unit, and the display unit displays a second image data whose display position has been adjusted based on the posture or change in posture of the imaging unit detected based on the first image data.

19. Electronic device comprising: a display device having a display unit; a control device; an imaging unit that performs imaging for displaying image data on the display unit; a detection unit that detects the posture or posture change of the imaging unit; an estimation unit that estimates the posture or amount of posture change of the imaging unit based on information from the detection unit; and an identification unit that identifies the cause of an error in the posture or amount of posture change estimated by the estimation unit based on the difference between the posture or amount of posture change of the imaging unit detected based on the image data and the posture or amount of posture change estimated by the estimation unit.

20. The electronic device according to claim 19, wherein the display device has a display control unit that controls the display unit, and the control device or the display control unit performs processing of at least one of the estimation unit and the identification unit.

Images