Image control device, mobile body, display system, and image control method

Abstract

The present invention improves visibility of an image based on a display image. This image control device comprises: a calculation unit that calculates a first distortion amount in a first region, which can be visually recognized when a visual recognition unit is viewed from the front, of an image based on a display image and a second distortion amount in a second region, which cannot be visually recognized when the visual recognition unit is viewed from the front, of the image based on the display image; and a first correction unit that executes, on the basis of the first distortion amount and the second distortion amount, distortion correction processing on the image based on the display image.

Description

Image control device, moving body, display system, and image control method

[0001] The present disclosure relates to an image control device and the like.

[0002] Conventionally, for example, the display device described in Patent Document 1 is known.

[0003] Japanese Patent Laid-Open No. 2022-63533

[0004] An image control device according to one aspect of the present disclosure is an image control device that controls a display device, wherein the display device includes: a display panel that displays a display image; an optical system capable of forming an image based on the display image at a position different from that of the display panel; and a housing that accommodates the display panel therein and has a visual recognition portion, and the image control device includes: a calculation unit that calculates a first distortion amount in a first region of the image that is visible when the visual recognition portion is viewed from the front, and a second distortion amount in a second region of the image that is not visible when the visual recognition portion is viewed from the front; and a first correction unit that executes distortion correction processing on the image based on the first distortion amount and the second distortion amount.

[0005] An image control method according to one aspect of the present disclosure is an image control method for controlling a display device, wherein the display device includes: a display panel that displays a display image; an optical system capable of forming an image based on the display image at a position different from that of the display panel; and a housing that accommodates the display panel therein and has a visual recognition portion, and the image control method includes: a calculation step of calculating a first distortion amount in a first region of the image that is visible when the visual recognition portion is viewed from the front, and a second distortion amount in a second region of the image that is not visible when the visual recognition portion is viewed from the front; and a correction step of executing distortion correction processing on the image based on the first distortion amount and the second distortion amount.

[0006] This is a diagram illustrating the configuration of the display device according to this disclosure. This is a cross-sectional view showing an example of the specific configuration of the display device according to this disclosure. This is a cross-sectional view showing an example of the main component configuration of the display device according to this disclosure. This is a diagram illustrating an example of the application of an imaging device equipped with the display device according to this disclosure. This is a diagram illustrating an example of the interior of a mobile vehicle to which an imaging device equipped with the display device according to this disclosure is applied. This is a schematic diagram illustrating how a virtual image appears from the front of the display device according to this disclosure. This is a schematic diagram illustrating how a virtual image appears from the front of the display device according to this disclosure. This is a schematic diagram illustrating the projection of a virtual image in the display device according to this disclosure. This is a block diagram illustrating an example of an image control system according to this disclosure. This is a schematic diagram illustrating an example of a method for generating a composite image. This is a schematic diagram illustrating the outline of the processing flow from when the control unit according to this disclosure performs distortion correction processing on a composite image obtained from a display image and obtains a corrected image. This is a schematic diagram illustrating the outline of the determination process that determines the correspondence between coordinates on the input image and coordinates on the output image. This is a flowchart illustrating an example of processing by the control unit according to this disclosure. This is a schematic diagram showing a state in which a second region is divided and imaged along the direction from the outer edge of the first region to the outer edge of the virtual image, and a schematic diagram showing an example of a composite image generated based on the imaging result. This is a schematic diagram showing the state in which the second region is divided and imaged along the above direction, and a schematic diagram showing an example of a composite image generated based on the said imaging results.

[0007] There is a desire to improve the visibility of images based on displayed images. According to one aspect of this disclosure, the visibility of images based on displayed images can be improved.

[0008] [Basic Configuration of Display Device] Figure 1 is a schematic diagram showing the configuration of the display device 1 according to the present disclosure. Figure 2 is a cross-sectional view showing an example of a specific configuration of the display device 1. Figure 3 is a cross-sectional view showing an example of the main components of the display device 1. As shown in Figures 1, 2 and 3, the display device 1 may include a display panel 2, a housing 36, a viewing window 38, and an optical system 3.

[0009] The display device 1 may direct a portion of the display light emitted from the display panel 2 into the eyes of the user 22, allowing the user 22 to view it as an image, picture, or aerial image. The display device 1 can allow the user 22 to view the display on the display panel 2 at a position different from the position of the display panel 2, using the display light emitted from the display panel 2. In one embodiment of this disclosure, the display device 1 may allow the user 22 to view the display light as a virtual image V. The virtual image V may be formed on the side of the display device 1 that is further away from the user 22. The virtual image V may be an upright virtual image that is an enlarged version of the display image displayed on the display panel 2. The virtual image V may be formed inside the housing 36 or outside the housing 36. The virtual image V may be formed on the side of the user 22 that is further away from the display panel 2 or closer to the display panel 2. The virtual image V may be formed on the side of the user 22 that is further away from the viewing window 38 or closer to the viewing window 38.

[0010] The display device 1 in one embodiment of this disclosure may be a non-attachable device to the user 22. That is, it may not be attached to the user 22 but may be fixed to the environment. The display device 1 may be fixed to, for example, a wall, column, or ceiling. The display device 1 may also be fixed to the interior of a vehicle. The display device 1 may be attached to the user 22. When attached to the user 22, the display device 1 may have a mounting part (not shown) so that the opening 37 is fixed at the position of the user 22's eyes.

[0011] The display panel 2 has a display surface 2a, and a display image may be displayed on the display surface 2a. In other words, the display panel 2 may emit display light of the display image from the display surface 2a. The display panel 2 may be configured to emit linearly polarized display light. The following description will focus on, but is not limited to, the case in which the display panel 2 emits S-wave polarized display light.

[0012] The display panel 2 may be a liquid crystal panel. The liquid crystal panel may have a known liquid crystal panel configuration. Known liquid crystal panels may be, for example, IPS (In-Plane Switching), FFS (Fringe Field Switching), VA (Vertical Alignment), ECB (Electrically Controlled Birefringence), and the like.

[0013] The display device 1 may include an irradiator 4 that illuminates the display panel 2 in a planar manner. The irradiator 4 may also be called a backlight. The irradiator 4 may be an edge-lit backlight or a direct-lit backlight. An edge-lit backlight has one or more light sources arranged on the outer periphery of the display panel 2, and the light emitted from the light sources may be guided by a light guide plate to the entire back surface of the display panel 2 and uniformly dispersed. A direct-lit backlight has multiple light sources arranged on the back side of the display panel 2, and the display panel 2 may be illuminated by light emitted from the multiple light sources. The light sources of the irradiator 4 may be cold cathode fluorescent lamps, halogen lamps, or xenon lamps, or they may be light-emitting diodes (LEDs), organic light-emitting diodes (OLEDs), semiconductor lasers (LDs), etc. If the light source of the irradiator 4 is an LD with excellent monochromaticity, the design of the optical system 3, in particular the design of optical components whose optical properties are wavelength-dependent, may be made easier.

[0014] The display panel 2 is not limited to a liquid crystal panel (transmissive display panel). The display panel 2 may be a self-emissive display panel that includes self-emissive elements such as a light-emitting diode (LED), an organic light-emitting diode (OLED), or a semiconductor laser (LD).

[0015] The housing 36 may be a housing in which the display panel 2 is located inside. The housing 36 may have an opening 37 in which a part of the wall surface is cut out. The housing 36 may have a viewing area that allows the inside of the housing 36 to be seen from the outside of the housing 36. The opening 37 may function as a viewing area.

[0016] The housing 36 may have a member that makes the inside of the housing 36, which is located in the opening 37, visible from the outside of the housing 36. The viewing window 38 may be arranged to at least partially block the opening 37. The viewing window 38 may be a light-transmitting plate. The viewing window 38 may transmit light emitted from the optical system 3. The viewing window 38 may be formed of light-transmitting glass or resin or the like. The member that makes the inside of the housing 36, which is located in the opening 37, visible from the outside of the housing 36 (for example, the viewing window 38) may function as a viewing part.

[0017] The viewing window 38 may be located in front of the display panel 2 in the housing 36 when the direction in which the display panel 2 displays the display image is considered to be forward. The direction in which the display panel 2 displays the display image may be, for example, the direction from the display panel 2 toward the user 22 when the display device 1 is in use.

[0018] The optical system 3 may project the display light emitted from the display panel 2 as an image based on the display image into the user's field of view 22. The image based on the display image into the user's field of view 22 may be formed at a position different from the display panel. For example, the optical system 3 may project the display light emitted from the display panel 2 as a virtual image V into the user's field of view 22. In other words, the virtual image V may be an example of an image based on the display image displayed by the display panel 2, which is formed by the optical system 3. The virtual image V may be formed at a position different from the display panel 2. The optical system 3 may be configured to include a first phase difference plate 5, a semi-transparent mirror 6, a second phase difference plate 7, and a reflective polarizing plate 8, as shown in Figures 2 and 3. The optical system 3 may be located inside the housing 36. If the display device 1 has a viewing window 38, the optical system 3 may be positioned so as to be surrounded by the housing 36 and the viewing window 38. The first phase difference plate 5, the semi-transparent mirror 6, the second phase difference plate 7, and the reflective polarizing plate 8 may be arranged in this order in the direction of emission of display light from the display panel 2 (positive direction in the Z-axis direction).

[0019] Here, the optical system 3 may be capable of forming a virtual image V in a way that makes it visible through the viewing window 38. In other words, the virtual image V may be visible by looking through the viewing window 38. To put it another way, the virtual image V cannot be seen without the viewing window 38. However, the display device 1 does not have to have a viewing window 38. If the display device 1 does not have a viewing window 38, the optical system 3 only needs to be capable of forming a virtual image V in a way that makes it visible through the aperture 37.

[0020] The first phase difference plate 5 may be located on the side of the display surface 2a of the display panel 2. The second phase difference plate 7 may be located away from the first phase difference plate 5 in the direction of emission of display light from the display panel 2. The first phase difference plate 5 and the second phase difference plate 7 may be quarter-wave plates. The first phase difference plate 5 and the second phase difference plate 7 may give a phase difference of 1 / 4 wavelength to the polarization plane (polarization plane in the direction of electric field oscillation) of the incident light. This makes it possible to reflect a portion of the display light emitted from the display panel 2 with the reflective polarizer 8 and have it incident on the semi-transparent mirror 6. The positional relationship between the first phase difference plate 5 and the second phase difference plate 7 may be defined such that when the first phase difference plate 5 and the second phase difference plate 7 are viewed along the Z-axis direction, the lagging axis of the second phase difference plate 7 is perpendicular to the lagging axis of the first phase difference plate 5. The positional relationship between the first phase difference plate 5 and the second phase difference plate 7 may be defined such that, when viewed along the Z-axis direction, the lagging axis of the second phase difference plate 7 is parallel to the lagging axis of the first phase difference plate 5.

[0021] The first phase difference plate 5 and the second phase difference plate 7 only need to be able to provide the necessary phase difference to the light transmitted through them so that the light transmitted through them is reflected by the reflective polarizer plate 8. That is, for example, when the polarization obtained by transmitting through the first phase difference plate 5 and the second phase difference plate 7 is taken as the second polarization, the first phase difference plate 5 and the second phase difference plate 7 may be other wavelength plates or combinations thereof, rather than quarter-wave plates, as long as the second polarization is obtained. In this disclosure, the case where the first phase difference plate 5 and the second phase difference plate 7 are quarter-wave plates will be explained as an example. Furthermore, the first phase difference plate and the second phase difference plate may be film-like members.

[0022] Furthermore, the second phase difference plate 7 only needs to be able to provide the necessary phase difference to the light that has passed through the second phase difference plate 7 so that the light that has been reflected by the reflective polarizer 8 and passed through the second phase difference plate 7 passes through the reflective polarizer 8 again when it reaches the reflective polarizer 8. In other words, for example, if the polarization obtained after being reflected by the reflective polarizer 8 and passed through the second phase difference plate 7 is taken as the first polarization, the second phase difference plate 7 may be a wave plate other than a quarter wave plate, as long as the first polarization can be obtained.

[0023] The first phase difference plate 5 may be integrated with the display panel 2. "Integration" may mean that the two members are arranged in contact with each other, or that the two members are joined to each other by an optically transparent adhesive such as OCA (Optically Clear Adhesive). However, the first phase difference plate 5 may be positioned away from the display surface 2a in the direction of emission of display light from the display panel 2.

[0024] The semi-transparent mirror 6 may be positioned between the first phase difference plate 5 and the second phase difference plate 7. The semi-transparent mirror 6 may transmit a portion of the incident light (for example, approximately 50%) and reflect the remainder (for example, approximately 50%). The transmittance and reflectance of light incident on the semi-transparent mirror 6 are not limited to 50%. The semi-transparent mirror 6 may have a function to collect or focus light. Specifically, the semi-transparent mirror 6 may have a function to collect or focus light that has been incident on and reflected by the semi-transparent mirror 6. The semi-transparent mirror 6 may reflect a portion of the display light reflected by the reflective polarizer 8 and direct it into the eyes of the user 22. This makes it possible for the user 22 to see the virtual image V. The semi-transparent mirror 6 may be a concave mirror having a concave reflective surface 6a, as shown in Figure 3. The reflective surface 6a of the semi-transparent mirror 6 may be positioned on the side of the second phase difference plate 7. The semi-transparent mirror 6 may include a spherical, aspherical, or free-form shape in at least a portion of its reflective surface 6a. The semi-transparent mirror 6 may focus or concentrate light more effectively than other components of the optical system 3. In other words, the semi-transparent mirror 6 may have a larger degree of focusing, convergence, or an index expressed as the reciprocal of the focal length than other components of the optical system 3. The reflective surface 6a of the semi-transparent mirror 6 may have a greater curvature than other components of the optical system 3. The optical system 3 may have only the semi-transparent mirror 6 as a component with a focusing or converging function. Furthermore, the semi-transparent mirror 6 may be composed of a holographic optical element (HOE), or its surface shape may have a Fresnel shape.

[0025] The semi-transparent mirror 6 is composed of, for example, a substrate and a semi-transparent reflective layer located on the surface of the substrate. The substrate may have a transmittance of 100% or close to 100% for light in the visible light band. The substrate may be made of, for example, a resin material, a glass material, etc. The resin material may be, for example, an acrylic resin, a polycarbonate resin, etc. The semi-transparent reflective layer may be a thin metal film. The thin metal film may be made of, for example, a metal material such as aluminum or chromium. The semi-transparent reflective layer is not limited to a thin metal film, and may be, for example, a dielectric multilayer film, etc. The semi-transparent mirror 6 may be configured to reflect light with the semi-transparent reflective layer. The semi-transparent reflective layer may be formed on the surface of the substrate located on the side of the second phase difference plate 7.

[0026] The reflective polarizer 8 may be located on the side of the second phase difference plate 7 opposite to the side of the semi-transparent mirror 6. In other words, the reflective polarizer 8 may be located downstream of the second phase difference plate 7 in the direction of emission of display light from the display panel 2. The reflective polarizer 8 may transmit a portion of the incident light and reflect the remainder. In this embodiment, the reflective polarizer 8 may be configured to reflect polarized light having a polarization axis parallel to the polarization axis of the display light (also called S-wave polarized light or second polarized light) and transmit polarized light having a polarization axis perpendicular to the polarization axis of the display light (also called P-wave polarized light or first polarized light). In this case, the positional relationship between the first phase difference plate 5 and the second phase difference plate 7 may be defined such that, when the first phase difference plate 5 and the second phase difference plate 7 are viewed along the Z-axis direction, the lagging axis of the second phase difference plate 7 is perpendicular to the lagging axis of the first phase difference plate 5. Furthermore, for example, the reflective polarizer 8 may be configured to reflect polarized light having a polarization axis perpendicular to the polarization axis of the display light (also called P-wave polarized light or second polarized light) and transmit polarized light having a polarization axis parallel to the polarization axis of the display light (also called S-wave polarized light or first polarized light). In this case, the positional relationship between the first phase difference plate 5 and the second phase difference plate 7 may be defined such that when the first phase difference plate 5 and the second phase difference plate 7 are viewed along the Z-axis direction, the lagging axis of the second phase difference plate 7 and the lagging axis of the first phase difference plate 5 are parallel. This makes it possible for the user 22 to view the virtual image V. The reflective polarizer 8 may be integrated with the second phase difference plate 7.

[0027] The reflective polarizer 8 may have the function of diverging the light that is incident on the semi-transparent mirror 6 and reflected. The reflective polarizer 8 may have the function of focusing or converging the light that is incident on the semi-transparent mirror 6 and reflected. The reflective polarizer 8 may be flat, or it may have a concave shape located on the display panel 2 side, or it may have a convex shape located on the display panel 2 side. Furthermore, the reflective polarizer 8 may be composed of a holographic optical element (HOE), or its surface shape may have a Fresnel shape.

[0028] The reflective polarizer 8 may be a wire grid polarizer comprising, for example, a substrate and a plurality of metal nanowires (also called a metal nanowire grid) located on the surface of the substrate. The substrate may have a transmittance of 100% or nearly 100% for light in the visible light band. The substrate may be made of, for example, a resin material, a glass material, etc. The metal nanowires may be made of, for example, a metal material such as aluminum, chromium, or titanium oxide. The metal nanowires may be arranged along one direction. The reflective polarizer 8 can transmit light components vibrating in a direction perpendicular to the grid and can reflect light components vibrating in a direction parallel to the grid.

[0029] The display device 1 may include a controller 50. The controller 50 may be connected to each component of the display device 1 and control each component. The controller 50 may control the irradiator 4. The controller 50 may control the display image displayed on the display panel 2 and the irradiator 4. The controller 50 may control the irradiator 4 based on the display image displayed on the display panel 2. The controller 50 may be configured to include one or more processors. The processors may include a general-purpose processor configured to load a specific program and execute a specific function, and a dedicated processor specialized for a specific process. The processors may include a PLD (Programmable Logic Device). The controller 50 may be either a SoC (System-on-a-Chip) or a SiP (System In a Package) in which one or more processors cooperate. The controller 50 includes a storage unit, which may store various information or programs for operating each component of the display device 1. The storage unit may be composed of, for example, a semiconductor memory. The memory unit may function as the work memory of the controller 50.

[0030] The optical function of the optical system 3 will now be described. The display panel 2 may emit display light that is S-wave polarized (first linearly polarized light L1). The display light of the first linearly polarized light L1 emitted from the display panel 2 may pass through the first phase difference plate 5 and be converted into light of the first circularly polarized light C1. A portion of the first circularly polarized light C1 that has passed through the first phase difference plate 5 (for example, approximately 50%) may pass through the semi-transparent mirror 6. The first circularly polarized light C1 that has passed through the semi-transparent mirror 6 may pass through the second phase difference plate 7 and be converted into light of the second linearly polarized light L2, whose polarization direction is parallel to the first linearly polarized light L1 (i.e., S-wave polarized light). The light of the second linearly polarized light L2 may be incident on the reflective polarizer 8. As described above, the reflective polarizer 8 may reflect S-wave polarized light and transmit P-wave polarized light. The light of the second linearly polarized light L2 incident on the reflective polarizer 8 may be reflected by the reflective polarizer 8 and converted into light of the third linearly polarized light L3. The light of the third linearly polarized light L3 may pass through the second phase difference plate 7 and be converted into the light of the second circularly polarized light C2. A portion of the light of the second circularly polarized light C2 that has passed through the second phase difference plate 7 (for example, about 50%) may be reflected by the semitransparent mirror 6 and converted into the light of the third circularly polarized light C3. The light of the third circularly polarized light C3 may pass through the second phase difference plate 7 and be converted into the light of the fourth linearly polarized light L4 whose polarization direction is perpendicular to the first linearly polarized light L1 (i.e., it is P-wave polarized). The light of the fourth linearly polarized light L4 may pass through the reflective polarizer 8 and be emitted to the outside. The amount of light (luminance) emitted from the display device 1 may be, for example, about 25% of the amount of light (luminance) of the display light emitted from the display panel 2.

[0031] The first phase difference plate 5, the semi-transparent mirror 6, the second phase difference plate 7, and the reflective polarizing plate 8 may be held by a holding member (not shown) to maintain their relative positions. Air may be interposed between the first phase difference plate 5 and the second phase difference plate 7 (i.e., between the first phase difference plate 5 and the semi-transparent mirror 6, and between the semi-transparent mirror 6 and the second phase difference plate 7). The display device 1 may be configured without a member made of a resin material such as polymer between the first phase difference plate 5 and the second phase difference plate 7. This reduces the risk of deformation of the semi-transparent mirror 6 when the resin material is cured during the manufacturing process of the display device 1, and positional misalignment between the semi-transparent mirror 6 and the first phase difference plate 5 and the second phase difference plate 7. Furthermore, resin materials such as polymers have material-specific retardation, which can reduce the risk of changing the polarization state of light transmitted through the resin material. As a result, a decrease in display quality can be reduced.

[0032] Since the optical system 3 is an on-axis type optical system in which the optical axis of the incident light and the optical axis of the emitted light substantially coincide, the space occupied by the optical system 3 can be reduced, and as a result, the display device 1 can be miniaturized. In addition, because the optical system 3 is on-axis, distortion and brightness unevenness of the virtual image V seen by the user 22 can be reduced, and the design of the optical system 3 is simplified.

[0033] In the display device 1, the optical path length of the light emitted from the display panel 2, passing through the semi-transparent mirror 6, reflected by the reflective polarizer 8, and returning to the semi-transparent mirror 6 may be smaller than the focal length of the semi-transparent mirror 6. In this case, a virtual image V can be made visible to the user 22. In the display device 1, the optical path length of the light emitted from the display panel 2, passing through the semi-transparent mirror 6, reflected by the reflective polarizer 8, and returning to the semi-transparent mirror 6 may be larger than the focal length of the semi-transparent mirror 6. In this case, a real image can be made visible to the user 22.

[0034] In Figure 3, for the sake of illustration, the optical path of light incident on the reflective polarizer 8 and the optical path of light reflected by the reflective polarizer 8 are shown shifted in the height direction (Y-axis direction). Similarly, the optical path of light incident on the semi-transparent mirror 6 and the optical path of light reflected by the semi-transparent mirror 6 are shown shifted in the height direction (Y-axis direction). However, in reality, the display light emitted from the display panel 2 may propagate substantially along a single axis.

[0035] Furthermore, the configuration of the display device 1 shown in Figures 2 and 3 may be just one example of a device that projects a virtual image V or a real image as an image based on the display image I. That is, the display device 1 only needs to have an optical system capable of projecting a virtual image V or a real image as an image based on the display image I. Such an optical system may, for example, comprise a first semi-transparent mirror, a first phase difference plate, a second semi-transparent mirror, a second phase difference plate, and a polarizing plate. Alternatively, for example, the optical system may comprise a first semi-transparent mirror, a first phase difference plate, a second semi-transparent mirror, a second phase difference plate, and a third semi-transparent mirror. These components may be arranged in this order in the direction of emission of display light from the display panel 2.

[0036] Next, an imaging device according to one embodiment of the present disclosure will be described. The imaging device 100 of this embodiment may include a display device 1. The imaging device 100 may cause the user 22 to view the display light emitted from the display panel 2 as a virtual image V. Since the imaging device 100 includes the display device 1, a compact imaging device can be realized, and the user 22 can view a virtual image V with improved display quality. The imaging device 100 may also cause the user 22 to view the display light emitted from the display panel 2 as a real image.

[0037] Figure 4 shows an example of the application of the imaging device 100. The imaging device 100 may be mounted on a vehicle 23, as shown in Figure 4. The vehicle 23 may be an example of a mobile body on which the imaging device 100 is mounted. However, the mobile body on which the imaging device 100 is mounted is not limited to a vehicle 23. The mobile body may be an aircraft or a ship, etc. Figure 4 shows the case where the vehicle 23 is a passenger car, but the vehicle 23 is not limited to a passenger car and may be an automobile such as a truck, bus, or trolleybus, or a motorcycle. The position of the display device 1 is arbitrary inside the vehicle 23. The display device 1 may be located on the dashboard (instrument panel), inside the dashboard, on the ceiling of the passenger compartment, on the A-pillar, etc. The imaging device 100 may share some of its components with other devices and parts provided by the vehicle 23.

[0038] As an example of this disclosure, the imaging device 100 may constitute a display system comprising a display device 1 and a camera 102 that captures images of the scenery around the vehicle 23, as shown in Figure 4. Here, the scenery around the vehicle 23 may be at least one of the front, rear, sides, above, and below the vehicle 23. The camera 102 may include, for example, a CCD (Charge Coupled Device) image sensor or a CMOS (Complementary Metal Oxide Semiconductor) image sensor. The display device 1 and the camera 102 may be connected by wired communication and / or wireless communication. In the vehicle 23, the display device 1 and the camera 102 may be connected via a vehicle network such as a CAN (Control Area Network).

[0039] The display device 1 may be configured to display at least a portion of the captured image captured by the camera 102 on the display panel 2. In this case, the imaging device 100 can allow the user 22 (driver of the vehicle 23) to view the scenery behind the vehicle 23 as a virtual image V (hereinafter also referred to as virtual image V1) formed on a side farther from the imaging device 100. As a result, the user 22 can view the scenery behind the vehicle 23 without significantly changing the viewing distance (point of gaze) while driving the vehicle 23, making it easier to see the virtual image V1 and improving driving safety. Furthermore, since the imaging device 100 is a small imaging device, even if it is placed in the driver's cab of the vehicle 23, it does not occupy a large volume in the driver's cab and is less likely to interfere with driving. The display device 1, which is mounted on the vehicle 23 and configured to allow the user 22 to view the scenery behind the vehicle 23 as a virtual image V1, is also called a digital rearview mirror.

[0040] Figure 5 shows an example of the interior of a vehicle 23 to which the imaging device 100 is applied. The imaging device 100 may also be applied to a digital side mirror. In this case, as shown in Figure 5, the imaging device 100 may include a display device 1 located on the left A-pillar of the vehicle 23 (hereinafter also referred to as the left-side display device 1L) and a camera 102 that captures the left rear of the vehicle 23 (hereinafter also referred to as the left-side camera 102L). Alternatively, the imaging device 100 may include a display device 1 located on the right A-pillar of the vehicle 23 (hereinafter also referred to as the right-side display device 1R) and a camera 102 that captures the right rear of the vehicle 23 (hereinafter also referred to as the right-side camera 102R). The left-side display device 1L may allow the user 22 to view the image of the left rear of the vehicle 23 captured by the left-side camera 102L as a virtual image V (hereinafter also referred to as the virtual image V2). The right-side display device 1R may allow the user 22 to view a virtual image V (hereinafter also referred to as virtual image V3) of the right rear of the vehicle 23 captured by the right-side camera 102R. The image may be a moving image (also referred to as a video) or a still image. The left-side camera 102L may be positioned in the same position as the left-side door mirror, and the right-side camera 102R may be positioned in the same position as the right-side door mirror.

[0041] The image forming apparatus 100 may be configured such that the distances between the eyes (or eye box) of a user 22 and each of the virtual images V2 and V3 substantially match each other. In this case, the user 22 can check the conditions of the left rear and right rear of the vehicle 23 without greatly changing the gaze distance (the distance between the eyes of the user 22 and the gaze point gazed by the user 22). Therefore, driving safety can be improved. The eye box may mean a region in real space where the eyes of the user 22 are assumed to be located.

[0042] The image forming apparatus 100 may be configured such that the distances between the eyes (or eye box) of a user 22 and each of the virtual images V1 to V3 substantially match each other. In this case, the user 22 can check the conditions directly behind, left rear and right rear of the vehicle 23 without greatly changing the gaze distance. Therefore, driving safety can be improved.

[0043] The image forming apparatus 100 may be applied to a cluster 29 in a dashboard of a vehicle 23 (see FIG. 5). In this case, the display apparatus 1 may allow the user 22 to visually recognize an image showing information related to driving such as vehicle speed, engine rotation speed, remaining fuel amount, etc., as a virtual image V (hereinafter also referred to as virtual image V4).

[0044] The image forming apparatus 100 may be applied to a CID (Center Information Display) 30 (see FIG. 5). In this case, the display apparatus 1 is arranged in a center cluster of the vehicle 23, and may allow the user 22 to visually recognize an image showing information related to navigation, in-vehicle environment (for example, settings of an air conditioner, an audio device, etc.), etc., as a virtual image V (hereinafter also referred to as virtual image V5).

[0045] The image forming apparatus 100 may be configured such that the distances between the eyes (or eye box) of a user 22 and each of the virtual images V4 and V5 substantially match each other. In this case, the user 22 can check information related to driving of the vehicle 23 and information related to navigation, in-vehicle environment, etc., without greatly changing the gaze distance. Therefore, driving safety can be improved.

[0046] The image forming apparatus 100 may be configured such that the distances between the eyes (or eye box) of a user 22 and each of the virtual images V1 to V5 are substantially the same. In this case, the user 22 can check the area immediately behind the vehicle 23, the left rear area and the right rear area without greatly changing the gaze distance, and can also check information related to driving the vehicle 23, navigation, the in-vehicle environment, and the like. Therefore, driving safety can be improved.

[0047] The image forming apparatus 100 may be applied to a PID (Passenger Information Display) 31 (see FIG. 5). In this case, the display device 1 is arranged near the passenger seat on the dashboard, and may allow a fellow passenger to visually recognize, as a virtual image V, video of entertainment content and video showing information related to an audio device, an air conditioner, and the like.

[0048] The image forming apparatus 100 may be applied to an RSE (Rear Seat Entertainment) system 32 (see FIG. 4). In this case, the display device 1 is arranged on the back surface of a front seat, and may allow a fellow passenger sitting on a rear seat of the vehicle 23 to visually recognize, as a virtual image V, video of entertainment content and video showing information related to an audio device, an air conditioner, and the like.

[0049] (Peeping area in virtual image) FIGS. 6 and 7 are schematic diagrams for explaining how the virtual image V is viewed from the front side of the display device 1. FIG. 6 is a schematic diagram when the display device 1 is viewed from above (a schematic diagram when viewed from the positive side toward the negative side along the Y-axis). FIG. 7 is a schematic diagram when the display device 1 is viewed from the side (a schematic diagram when viewed from the negative side toward the positive side along the X-axis). In FIGS. 6 and 7, only the display panel 2, the semi-transparent mirror 6, the opening 37, and the viewing window 38 are illustrated as the configuration of the display device 1.

[0050] As shown in Figures 6 and 7, the display device 1 can form areas that appear in the field of view of the left and right eyes when the user's head moves, that is, viewing areas that can be peered into with either the left or right eye. Specifically, the display device 1 may generate a virtual image V having a first area R1 that is visible when the aperture 37 is viewed from the front, and a second area R2 (viewing area) that is not visible when the aperture 37 is viewed from the front.

[0051] The size (dimensions) of the second region R2 can be controlled, for example, by controlling the size of the display surface 2a, the magnification factor of the virtual image V, etc. Alternatively, the size of the second region R2 can be controlled by controlling the image display area (the area where the image is actually displayed) on the display surface 2a. Increasing the image display area enlarges the second region R2. Decreasing the image display area reduces the second region R2.

[0052] Furthermore, for example, within the user's 22 field of view, the virtual image V may be larger than the aperture 37. The size of the second region R2 can also be controlled by the size of the aperture 37. That is, by appropriately designing the size of the aperture 37, the second region R2 can be formed and its size can be controlled. When the size of the second region R2 is controlled by the aperture 37, the size of the semi-transparent mirror 6 only needs to be large enough to project the image of the entire display surface 2a into the user's 22 field of view, which simplifies the design of the optical system 3.

[0053] Here, the projection of the virtual image V in the display device 1 will be explained using Figure 8. In Figure 8, the illuminator 4 and optical components that do not contribute to the projection distance (virtual image distance) and magnification of the virtual image V (first phase difference plate 5 and second phase difference plate 7) are omitted. The distance between the display panel 2 and the semi-transparent mirror 6 may be considered as "0". In the following explanation, the focal length of the concave mirror semi-transparent mirror 6 may be denoted as f, the distance between the display panel 2 and the reflective polarizer 8 may be denoted as a1, and the distance between the semi-transparent mirror 6 and the reflective polarizer 8 may be denoted as a2.

[0054] The display device 1 may be configured to project a virtual image V by magnifying the virtual image Q of the display surface 2a formed by the reflective polarizing plate 8 using a concave mirror, which is a semi-transparent mirror 6. As shown in Figure 8, the virtual image Q is located on the opposite side of the concave mirror, which is a semi-transparent mirror 6, from the reflective polarizing plate 8, and the distance from the reflective polarizing plate 8 may be a1. The virtual image Q may be an image of the display surface 2a magnified to 1x.

[0055] The virtual image distance b and virtual image magnification m of the virtual image V may be expressed by the following equations (1) and (2), respectively. The virtual image distance b is the distance between the virtual image V and the semitransparent mirror 6, and the virtual image magnification m may be the magnification ratio of the virtual image V relative to the display surface 2a.

[0056] b = 1 / (1 / (a1+a2)-1 / f) ... (1) m = b / (a1+a2) = f / (f-(a1+a2)) ... (2) In other words, the virtual image distance b (the position where the virtual image V is formed) can be controlled by appropriately changing the focal length (f) of the semitransparent mirror 6, the distance (a1) between the display panel 2 and the reflective polarizer 8, and / or the distance (a2) between the semitransparent mirror 6 and the reflective polarizer 8. Furthermore, the virtual image magnification m (the size of the virtual image V) can be controlled by appropriately changing f, a1, and a2.

[0057] In this embodiment, for example, as in Figure 8, the virtual image V may be located behind the display panel 2 at a distance of b = 1 / (1 / (a1+a2)-1 / f). Also in this embodiment, for example, as in Figure 8, the magnification of the virtual image V is f / (f-(a1+a2)), and since a1+a2>0, the magnification of the virtual image V may be greater than 1. In other words, the virtual image V may be larger than the display image I.

[0058] As shown in Figures 6 and 7, the visible area of ​​the virtual image V differs depending on the viewing position of the virtual image V. The size of the virtual image V visible through the aperture 37 may be determined by the end of the semi-transparent mirror 6. In this embodiment, the viewing position by the user 22 (the imaging position of the virtual image V by the imaging device 120, described later) may be a planar PL at a position away from the aperture 37.

[0059] As shown by reference numeral 1001 in Figure 6, when the aperture 37 is viewed from the front, the user 22 can perceive the entire first region R1 of the virtual image V as the first image Vi1. That is, when the aperture 37 is viewed along the normal NL passing through the center of the aperture 37 (the normal passing through the center of the plane containing the aperture 37) (when viewed from the "center" position in the figure), the user 22 can perceive the first image Vi1.

[0060] As shown by reference numeral 1002 in Figure 6, when the aperture 37 is viewed from the front and the aperture 37 is viewed from the left of the normal NL, the user 22 can see the second image Vi2, which includes the entire right-hand region R22 of the virtual image V, which is part of the second region R2. There may be a position on the straight line connecting the right end of the virtual image V and the right end of the semi-transparent mirror 6 where the right end of the virtual image V can be seen. In this embodiment, the position where the right end of the virtual image V can be seen ("left end" in the figure) may be the intersection of the straight line and the plane PL.

[0061] As shown by reference numeral 1003 in Figure 6, when the aperture 37 is viewed from the front and the aperture 37 is viewed from the right of the normal NL, the user 22 can see the third image Vi3, which includes the entire left region R23 of the virtual image V, which is part of the second region R2. There may be a position on the straight line connecting the left end of the virtual image V and the left end of the semitransparent mirror 6 where the left end of the virtual image V can be seen. In this embodiment, the position where the left end of the virtual image V can be seen ("right end" in the figure) may be the intersection of the straight line and the plane PL.

[0062] As shown by reference numeral 1011 in Figure 7, when the aperture 37 is viewed from below the normal NL, the user 22 can see the fourth image Vi4, which includes the entire upper region R24 of the virtual image V, which is part of the second region R2. There may be a position on the straight line connecting the upper end of the virtual image V and the upper end of the semi-transparent mirror 6 where the upper end of the virtual image V can be viewed. In this embodiment, the position where the upper end of the virtual image V can be viewed (referred to as the "lower end" in the figure) may be the intersection of this straight line and the plane PL.

[0063] As shown by reference numeral 1012 in Figure 7, when the aperture 37 is viewed from above the normal NL, the user 22 can see the fifth image Vi5, which includes the entire lower region R25 of the virtual image V, which is part of the second region R2. There may be a position on the straight line connecting the lower end of the virtual image V and the lower end of the semi-transparent mirror 6 where the lower end of the virtual image V can be viewed. In this embodiment, the position where the lower end of the virtual image V can be viewed ("upper end" in the figure) may be the intersection of the straight line and the plane PL.

[0064] Thus, consider the case where the virtual image V is viewed from a position on a straight line connecting an arbitrary point of the virtual image V (including the edge of the virtual image V) and the edge of the semi-transparent mirror 6 closest to that arbitrary point, in a plane that includes the normal vector NL. In this case, the user 22 can view an image through the aperture 37 that includes the arbitrary point of the virtual image V at its edge and has approximately the same size as the first image Vi1, as an image that includes the second region R2.

[0065] [Embodiment 1] Next, an example of an image control device 130 that controls the display device 1 described above will be explained. Before explaining the image control device 130, first, an example of an image control system 110 will be explained.

[0066] [Image Control System] Figure 9 is a block diagram showing an example of an image control system 110 according to the present disclosure. The image control system 110 according to the present disclosure may be a system for controlling a display device 1. As shown in Figure 9, the image control system 110 may include, for example, a display device 1, an imaging device 120, and an image control device 130.

[0067] As described above, the display device 1 may include a display panel 2 for displaying a display image I, an optical system 3 capable of forming an image based on the display image I at a position different from the display panel 2, and a housing 36 in which the display panel 2 is located. The housing 36 may have an aperture 37. In the following description, the case in which the image based on the display image I is a virtual image V will be used as an example, but as described above, the image may be a real image.

[0068] The imaging device 120 may be a device that images at least a portion of the virtual image V formed by the display device 1 through the aperture 37. The imaging device 120 may image the virtual image V from any position on the planar PL shown in Figures 6 and 7. By imaging the virtual image V from the arbitrary position, the imaging device 120 can capture an image that can be viewed by the user 22 from the arbitrary position.

[0069] The imaging device 120 may, for example, acquire a first image Vi1 as an image of the first region R1 of the virtual image V captured through the aperture 37 from the "center" position in Figure 6 (a position where the aperture 37 is viewed from the front). The imaging device 120 may also acquire an image including the second region R2 of the virtual image V captured through the aperture 37 from a position other than the front of the aperture 37. The imaging device 120 may, for example, acquire at least one of the following images as an image including the second region R2 captured in a planar PL: ・A second image Vi2 captured through the aperture 37 from the "left end" position in Figure 6. ・A third image Vi3 captured through the aperture 37 from the "right end" position in Figure 6. ・A fourth image Vi4 captured through the aperture 37 from the "bottom end" position in Figure 7. ・A fifth image Vi5 captured through the aperture 37 from the "top end" position in Figure 7. - An image including the second region R2, captured through the aperture 37 from a direction other than the upward, downward, left, and right directions mentioned above.

[0070] As a result, even if a second region R2 that is not visible exists, the entire virtual image V based on the displayed image I can be subjected to distortion correction processing by the first correction unit 144, which will be described later.

[0071] The position at which the imaging device 120 captures the virtual image V may be predetermined. As described above, the size and position of the virtual image V formed by the display device 1 may be determined, for example, by the structure of the display device 1. Therefore, the range of the virtual image V that can be obtained when imaging from a planar PL can also be predetermined. Range data showing the relationship between the imaging position and the range of the virtual image V captured from that imaging position may be stored in the storage unit 150.

[0072] The image control device 130 may be a device that controls the display device 1. The image control device 130 may perform distortion correction of the displayed image I or virtual image V. The image control device 130 may include, for example, a control unit 140 and a storage unit 150. The control unit 140 may comprehensively control each component of the image control device 130. The storage unit 150 may store data and programs necessary for control by the control unit 140. Details of the control unit 140 will be described later.

[0073] [Details of the Image Display Device] Next, the details of the image control device 130 will be described using Figure 9. As shown in Figure 9, the control unit 140 may include, for example, an acquisition unit 141, an image generation unit 142, a calculation unit 143, a first correction unit 144, and a second correction unit 145.

[0074] The acquisition unit 141 may acquire an image (captured image) that includes at least a part of the virtual image V captured by the imaging device 120 and formed by the display device 1. The acquisition unit 141 may acquire a first image Vi1 captured through the aperture 37 from a position where the aperture 37 is viewed from the front, and an image capturing the second region R2 of the virtual image V. The image capturing the second region R2 may be an image that includes the second region R2 as described above, such as the second image Vi2 to the fifth image Vi5.

[0075] The image generation unit 142 may generate a composite image by combining the first region R1 and the second region R2 of the virtual image V. Specifically, the image generation unit 142 may generate a composite image using the first image Vi1, which is the image of the first region R1 acquired by the acquisition unit 141, and images including the second region R2 acquired by the acquisition unit 141 (for example, the second image Vi2 to the fifth image Vi5). The composite image may be an image corresponding to the entire virtual image V.

[0076] Figure 10 is a schematic diagram illustrating an example of a method for generating a composite image VS. Reference numeral 1021 in Figure 10 describes an example of a composite image VS generated using a first image Vi1 and a fourth image Vi4. As described above, the first image Vi1 may be an image of the first region R1 captured from the "center" position in Figure 6. The fourth image Vi4 is an image captured from the "bottom end" position in Figure 7 and may be an image that includes the entire upper region R24 as an image of the second region R2.

[0077] The image generation unit 142 may generate a composite image VS by superimposing a common region in the first image Vi1 and the fourth image Vi4. The image generation unit 142 may identify the common region by, for example, referring to the range data described above. Through this synthesis, the image generation unit 142 can generate an image of the virtual image V that includes the first region R1 and the upper region R24.

[0078] The image generation unit 142 may perform the superposition of the first image Vi1 with the second image Vi2, the third image Vi3, and the fifth image Vi5 in the same way as the superposition of the first image Vi1 with the fourth image Vi4. When the first image Vi1 and the second image Vi2 are superimposed, the image generation unit 142 may obtain an image of the virtual image V that includes the first region R1 and the right-side region R22 (an image composed of the first region R1 and the right-side region R22 in reference numeral 1022 of Figure 10). When the first image Vi1 and the third image Vi3 are superimposed, the image generation unit 142 may obtain an image that includes the first region R1 and the left-side region R23, and when the first image Vi1 and the fifth image Vi5 are superimposed, the image generation unit 142 may obtain an image that includes the first region R1 and the lower region R25.

[0079] Furthermore, the second region R2 may contain corner regions R26 to R29 that are not included in the right region R22, the left region R23, the upper region R24, and the lower region R25. The imaging device 120 may acquire images including corner region R26, images including corner region R27, images including corner region R28, and images including corner region R29 by appropriately changing the imaging position. The image generation unit 142 may then generate four images including the first region R1 and each of the corner regions R26 to R29 by superimposing the first image Vi1 and each of the images including corner regions R26 to R29.

[0080] In this way, the control unit 140 acquires the first image Vi1 and also acquires multiple images that include any edge of the virtual image V, thereby acquiring an image that includes the entire virtual image V. Therefore, as shown by reference numeral 1022 in Figure 10, the control unit 140 can acquire a virtual image V as an captured image, which is composed of a first region R1 and a second region R2 (regions R22 to R29) that is a viewing region.

[0081] Figure 11 is a schematic diagram showing the general flow of processing from when the control unit 140 performs distortion correction processing on the composite image VS obtained from the display image I to when it obtains a corrected image VC. The display image I that is the source of processing for the first correction unit 144, which will be described later, may be an image that displays the dot pattern shown in Figure 11. The display image I is an image that allows the coordinate positions on the image to be identified in the processing of the calculation unit 143 and the first correction unit 144.

[0082] As shown in Figure 11, when a display image I is formed as a virtual image V, distortion may occur in the virtual image V. Figure 11 shows an example where distortion occurs in the virtual image V, which includes a second region R2 that is not visible when viewed from the front. When the imaging device 120 captures the virtual image V from the "center" position shown in Figure 6, it may acquire an image of the first region R1 (an image including each dot indicated by a ● in the virtual image V). The first correction unit 144 may process the image captured by the imaging device 120. Therefore, the first correction unit 144 does not need to be able to process the entire virtual image V, including the second region R2 (each dot indicated by a ○ in the virtual image V), by acquiring only an image of the first region R1. Consequently, the first correction unit 144 does not need to be able to perform distortion correction processing on the entire virtual image V.

[0083] As described above, since the imaging device 120 captures the virtual image V from multiple directions, the acquisition unit 141 may acquire not only the image of the first region R1 but also an image including the second region R2. Therefore, the image generation unit 142 may use the image of the first region R1 (first image Vi1) and the images including the second region R2 (second image Vi2 to fifth image Vi5, etc.) to generate a composite image VS that includes the first region R1 (each dot indicated by a ● mark) and the second region R2 (each dot indicated by hatching). This makes it possible for the first correction unit 144 to perform distortion correction processing on the entire virtual image V.

[0084] The calculation unit 143 may calculate a first distortion amount in the first region R1 of the virtual image V and a second distortion amount in the second region R2 of the virtual image V. In this embodiment, the calculation unit 143 may calculate the first distortion amount and the second distortion amount by calculating the distortion amount of the composite image VS generated by the image generation unit 142.

[0085] Known techniques may be used as the method for calculating the amount of distortion. As described above, the magnification ratio of the virtual image V relative to the display image I can be calculated. For this reason, the calculation unit 143 may calculate the amount of distortion at each coordinate position as the distance between each coordinate position on the virtual image V when the display image I is magnified without distortion and each coordinate position on the actually formed virtual image V that corresponds to each coordinate position. As described above, in this embodiment, a display image I with a dot pattern may be used for the distortion correction processing by the first correction unit 144. For this reason, the calculation unit 143 may calculate the above distance, i.e., the amount of distortion, for each dot pattern. The first correspondence between each coordinate position on the virtual image V when the display image I is magnified without distortion and each coordinate position on the display image I may be determined from the magnification ratio.

[0086] The first correction unit 144 may perform distortion correction processing on the virtual image V based on the first distortion amount and the second distortion amount calculated by the calculation unit 143. In this embodiment, the first correction unit 144 may perform distortion correction processing on the virtual image V based on the distortion amount of the composite image VS calculated by the calculation unit 143.

[0087] The first correction unit 144 can employ known methods as distortion correction processing. As an example of a determination process that determines the correspondence between coordinate positions on the input image and coordinate positions on the output image, which is performed in the distortion correction processing, for example, remap, a function of Python, may be used.

[0088] Figure 12 is a schematic diagram illustrating the outline of the determination process for determining the correspondence between coordinate positions on the input image and coordinate positions on the output image. As shown by reference numeral 1031 in Figure 12, consider the case where the coordinate positions on the output image are identified for each of the four points on the input image. In this case, according to the above determination process, the coordinates within the range enclosed by these four points can also be interpolated using a predetermined method to determine the correspondence between coordinate positions between the input and output images. One example of a predetermined method is linear interpolation. This interpolation may be performed based on the amount of distortion calculated by the calculation unit 143.

[0089] As shown by reference numeral 1032 in Figure 12, consider the case where the display image I is the input image and the output image is the virtual image V. In this case as well, it is possible to identify the correspondence between each dot pattern of the virtual image V when the display image I is enlarged without distortion, and each dot pattern of the composite image VS (the entire virtual image V). Furthermore, it is possible to identify the positional relationship of the coordinates within the area enclosed by the four dot patterns between the virtual image V and the composite image VS.

[0090] Therefore, the calculation unit 143 may identify a second correspondence between the coordinate positions on the display image I and the coordinate positions on the composite image VS as a result of calculating the amount of distortion of the composite image VS. Then, as shown by reference numeral 1033 in Figure 12, the first correction unit 144 may set the virtual image V (distortion-free virtual image) that would occur if the display image I were enlarged without distortion as the output image. Based on the positions of each dot pattern on the distortion-free virtual image and the identified second correspondence, the first correction unit 144 may generate a corrected image by correcting the virtual image V (composite image VS) based on the display image as the input image.

[0091] The first correction unit 144 may convert each coordinate position on the corrected image to each coordinate position on the display image I when the corrected image is resized to the size of the display image I. The first correction unit 144 may then determine a distortion correction pattern 151 that shows the correspondence between each coordinate position and each coordinate position on the distortion-free display image I (the original input image shown as reference numeral 1032 in Figure 12). That is, the distortion correction pattern 151 may indicate the amount of distortion correction in the display image I. The first correction unit 144 may store the determined distortion correction pattern 151 in the storage unit 150.

[0092] As shown in Figure 11, the composite image VS may be an image that shows the entire virtual image V, including the image of the first region R1 and the image of the second region R2. Therefore, the first correction unit 144 may generate a corrected image VC based on the dot patterns when the composite image VS is converted into a distortion-free virtual image V and the second correspondence relationship described above. As a result, the first correction unit 144 may generate a corrected image VC in which the direction of distortion is opposite to that of the composite image VS (a corrected image VC in which the distortion in the composite image VS is canceled out).

[0093] The second correction unit 145 may perform distortion correction processing on the display image I based on the results of the distortion correction processing on the virtual image V performed by the first correction unit 144. This display image I may be the image displayed on the display panel 2 after the distortion correction pattern 151 has been generated. Therefore, this display image I may be different from the display image I used by the first correction unit 144 to generate the distortion correction pattern 151 (the image displaying the dot pattern in the above example).

[0094] The second correction unit 145 may refer to the distortion correction pattern 151 stored in the storage unit 150 by the first correction unit 144 and perform distortion correction processing on the display image I to be displayed on the display panel 2. The second correction unit 145 may then output the display image I after distortion correction processing to the controller 50 of the display device 1. As a result, the controller 50 can display the display image I with overall reduced distortion on the display panel 2.

[0095] As described above, the distortion correction pattern 151 may be generated based on a distortion-free virtual image set by the first correction unit 144. Therefore, by performing distortion correction processing on the display image I using the distortion correction pattern 151, the display device 1 can form a virtual image V with overall reduced distortion. Consequently, no matter from which direction the user 22 views the virtual image V through the aperture 37, they can view a virtual image V with reduced distortion.

[0096] <Image Control Method> Figure 13 is a flowchart showing an example of processing by the control unit 140 (image control method). Reference numeral 1041 in Figure 13 is a flowchart showing an example of the distortion correction pattern 151 generation process, and reference numeral 1042 is a flowchart showing an example of the display process using the distortion correction pattern 151.

[0097] (Distortion Correction Pattern Generation Process) First, the display device 1 may display a display image I for generating a distortion correction pattern 151 on the display panel 2. As described above, this display image I may be an image that displays a dot pattern. The display device 1 may form a virtual image V based on this display image I. Next, the imaging device 120 may capture a part of the virtual image V formed by the display device 1 through the aperture 37 from a predetermined position on the planar PL. Specifically, the imaging device 120 may capture a first image Vi1, which is an image of the first region R1, and an image that includes the second region R2. As an image that includes the second region R2, the imaging device 120 may capture four images, for example, the second image Vi2 to the fifth image Vi5, and the corner regions R26 to R29, respectively. As shown by reference numeral 1041, the acquisition unit 141 may acquire the first image Vi1 captured by the imaging device 120 and an image including the second region R2 (S1).

[0098] Next, the image generation unit 142 may generate a composite image VS by combining the first region R1 and the second region R2 using the first image Vi1 and the image including the second region R2 (S2). Next, the calculation unit 143 may calculate the amount of distortion of the composite image VS based on the coordinate positions on the display image I and the coordinate positions on the composite image VS (S3). Step S3 may be an example of a calculation step in which the calculation unit 143 calculates the amount of distortion of the virtual image V based on the first amount of distortion in the first region R1 of the virtual image V and the second amount of distortion in the second region R2 of the virtual image V.

[0099] Next, the first correction unit 144 may perform distortion correction processing on the composite image VS based on the amount of distortion of the composite image VS calculated by the calculation unit 143 (S4). Step S4 may be an example of a correction step in which the first correction unit 144 performs distortion correction processing on the virtual image V based on the first amount of distortion and the second amount of distortion calculated by the calculation unit 143. The first correction unit 144 may then store the result of the distortion correction processing on the virtual image V as a distortion correction pattern 151 in the storage unit 150 (S5).

[0100] For example, the calculation unit 143 may identify a second correspondence between coordinate positions on the display image I and coordinate positions on the composite image VS as a result of calculating the amount of distortion of the composite image VS. The first correction unit 144 may generate a corrected image VC by correcting the distorted virtual image V based on the display image I, based on this second correspondence and a set distortion-free virtual image. The first correction unit 144 may then generate a distortion correction pattern 151 as a result of distortion correction processing on the composite image VS, based on this corrected image VC and the display image I, and store it in the storage unit 150.

[0101] (Display processing using distortion correction pattern) As shown in reference numeral 1042, the acquisition unit 141 may acquire a new display image I (S11). The second correction unit 145 may perform distortion correction processing on the new display image I acquired by the acquisition unit 141 based on the result of the distortion correction processing performed by the first correction unit 144 (S12). In this embodiment, the second correction unit 145 may perform distortion correction processing on the new display image I by referring to the distortion correction pattern 151 generated by the first correction unit 144. After that, the controller 50 of the display device 1 may display the display image I after distortion correction processing by the second correction unit 145 on the display panel 2 (S13). As a result, a virtual image V based on the display image I after distortion correction processing may be formed.

[0102] <Modification> In the above, the image control device 130 is implemented as a separate device from the display device 1, but it is not limited to this, and the image control device 130 may be provided in the display device 1. In this case, the image control device 130 may be implemented by the controller 50. The image control device 130 may also be provided in the vehicle 23. The display device 1 equipped with the image control device 130 may also be provided in the vehicle 23.

[0103] Furthermore, not all functions of the image control device 130 are necessarily implemented by the controller 50; some functions of the image control device 130 may be implemented by the controller 50. In this case, for example, the second correction unit 145 may be implemented as a function of the controller 50. In this case, the distortion correction pattern 151 generated by the first correction unit 144 may be stored in the display device 1. The controller 50 may perform distortion correction processing on the display image I to be displayed and display the display image I after distortion correction processing on the display panel 2.

[0104] Furthermore, the above description illustrates an example in which the control unit 140 generates a composite image VS by combining an image of the first region R1 and an image including the second region R2, and performs distortion correction processing on the composite image VS based on the amount of distortion of the composite image VS. However, the control unit 140 does not have to generate a composite image VS. In this case, the control unit 140 may calculate a first amount of distortion in the first region R1 and a second amount of distortion in the second region R2, and perform distortion correction processing on the virtual image V, which is the source of the distortion calculation, based on the first and second amounts of distortion. In this case, the control unit 140 may specify, for example, the following correspondence as the second correspondence. The control unit 140 may specify, for example, the correspondence between the coordinate position on the display image I and the coordinate position on the first region R1 of the virtual image V based on the display image I, and the correspondence between the coordinate position on the display image I and the coordinate position on the second region R2 of the virtual image V.

[0105] <Effects> Generally, when a virtual image V is generated based on a displayed image I, distortion may occur in the virtual image V. Therefore, in order to improve the visibility of the virtual image V, it is desirable to reduce the distortion that occurs in the virtual image V.

[0106] The virtual image V formed by the display device 1 may include a first region R1 that is visible when the aperture 37 is viewed from the front, and a second region R2 that is not visible when the aperture 37 is viewed from the front. In other words, in this embodiment, only a portion of the virtual image V may be visible through the aperture 37. Therefore, if distortion correction processing is performed on a portion of the virtual image V that is visible from one direction through the aperture 37, distortion correction processing may not be performed on the other regions. Consequently, if the region visible in the virtual image V differs depending on the direction in which the virtual image V is viewed, the user 22 may see a region where distortion correction processing has not been performed.

[0107] When the imaging device 120 performs distortion correction processing on the virtual image V it has captured, the range of the virtual image V captured through the aperture 37 may be only a portion of the virtual image V. Therefore, as described above, a situation may occur where distortion correction processing is not performed on areas other than the portion that is captured.

[0108] It is also conceivable to perform distortion correction processing on the region of the virtual image V that the user 22 can see by performing eye tracking and monitoring the user's gaze. However, imaging equipment for eye tracking is expensive, and controlling the distortion correction processing based on eye tracking can be complex.

[0109] In this embodiment, the control unit 140 may perform distortion correction processing on the virtual image V based on a first distortion amount in a first region R1 of the virtual image V that is visible when the aperture 37 is viewed from the front, and a second distortion amount in a second region R2 of the virtual image V that is not visible when the aperture 37 is viewed from the front. Therefore, the control unit 140 can perform distortion correction processing on the entire virtual image V generated based on the display image I. Accordingly, the control unit 140 can perform distortion correction processing on the entire new display image I by performing distortion correction processing on the new display image I that is to be displayed after the distortion correction processing, based on the result of the distortion correction processing.

[0110] The display device 1 may form a virtual image V including a first region R1 and a second region R2 based on this new display image I. Therefore, if the visible region in the virtual image V differs depending on the direction from which the virtual image V is viewed, the visible region in the virtual image V may be the region from which distortion correction processing has been performed, regardless of the direction from which the virtual image V is viewed. Therefore, in situations where the virtual image V can be viewed from any direction, the visibility of the virtual image V can be improved without performing eye tracking.

[0111] Alternatively, the image generation unit 142 may generate a composite image VS by combining the first region R1 and the second region R2, and the calculation unit 143 may calculate the amount of distortion of the composite image VS. In this case, the control unit 140 can perform a distortion correction process on the entire composite image VS as a distortion correction process based on the first amount of distortion and the second amount of distortion. Therefore, a distortion correction process that is performed on the entire virtual image V as a single process can be adopted as the distortion correction process.

[0112] Furthermore, the image generation unit 142 may generate a composite image VS using an image of a first region R1 captured through the aperture 37 from a position that views the aperture 37 from the front, and an image including a second region R2 captured through the aperture 37 from a position other than the front of the aperture 37.

[0113] Even if the range of the virtual image V captured from the aperture 37 is only a portion of the virtual image V, the control unit 140 can acquire an image including the first region R1 and the second region R2 by capturing images from a position facing the aperture 37 and from positions other than facing the aperture 37. Therefore, a composite image VS including the entire virtual image V can be generated. Accordingly, the control unit 140 can perform distortion correction processing on the entire virtual image V based on the imaging results of the virtual image V from multiple directions.

[0114] [Embodiment 2] Another embodiment of the present disclosure is described below. For convenience of explanation, components having the same function as those described in the above embodiments are denoted by the same reference numerals, and their descriptions are not repeated.

[0115] In Embodiment 1, the imaging device 120 may capture an image including the entire second region R2 by capturing an image including the edges of the virtual image V as an image including the second region R2. Specifically, the imaging device 120 may capture the entire right region R22, the entire left region R23, the entire upper region R24, the entire lower region R25, and the entire corner regions R26 to R29 as an image including the second region R2 (see reference numeral 1022 in Figure 10). In this embodiment, the imaging device 120 may capture an image including each region obtained by dividing the second region R2 along the direction from the outer edge of the first region R1 to the outer edge of the virtual image V. The control unit 140 may generate an image including the entire second region R2 based on the images including each captured region.

[0116] Figures 14 to 16 are schematic diagrams showing the state in which the second region R2 is divided and imaged along the direction from the outer edge E1 of the first region R1 to the outer edge E2 of the virtual image V, and schematic diagrams showing an example of a composite image generated based on the said imaging results. In Figures 14 to 16, the case in which the upper region R24 is divided and imaged will be explained as an example.

[0117] Reference numeral 1051 in Figure 14 and 1061 in Figure 15 are schematic diagrams showing the imaging range of the virtual image V when the imaging device 120 images the virtual image V from a first position PO1 and a second position PO2, which are located between the "center" position and the "lower end" position, respectively. Reference numeral 1071 in Figure 16 is a schematic diagram showing the imaging range of the virtual image V when the imaging device 120 images the virtual image V from the "lower end" position.

[0118] The second position PO2 may be closer to the "bottom end" position than the first position PO1. In this example, the first position PO1 and the second position PO2 may be positions that divide the "center" position and the "bottom end" position into three parts.

[0119] As shown by reference numeral 1051 in Figure 14, the imaging device 120 may acquire a first segmented image Vi41 that includes a portion of the upper region R24 adjacent to the first region R1 by capturing a virtual image V from the first position PO1 through the aperture 37.

[0120] As shown by reference numeral 1052 in Figure 14, the image generation unit 142 may generate a first composite image VS1 by superimposing a region common to the first image Vi1 and the first divided image Vi41. The first composite image VS1 may include the entirety of the above-mentioned partial region. The above-mentioned partial region may be the first divided region R41, which is one of a plurality of regions obtained by dividing the upper region R24 along the above-mentioned direction.

[0121] As shown by reference numeral 1061 in Figure 15, the imaging device 120 may acquire the second segmented image Vi42 by imaging a virtual image V from the second position PO2 through the aperture 37. The second segmented image Vi42 may be an image that includes another part of the upper region R24 adjacent to the first region R1, and that includes another part of the first segmented region R41.

[0122] As shown by reference numeral 1062 in Figure 15, the image generation unit 142 may generate a second composite image VS2 by superimposing the first composite image VS1 and the second divided image Vi42 such that common areas in the first image Vi1 and the second divided image Vi42 overlap. The image generation unit 142 may generate a second composite image VS2 that includes a first region R1, a first divided region R41 adjacent to the first region R1, and a second divided region R42 which is part of another partial region and adjacent to the first divided region R41. The second divided region R42 may be one of a plurality of regions obtained by dividing the upper region R24 along the above direction.

[0123] As shown by reference numeral 1071 in Figure 16, the imaging device 120 may acquire a fourth image Vi4 that includes the entire upper region R24 by imaging a virtual image V through the aperture 37 from the "lower end" position.

[0124] As shown by reference numeral 1072 in Figure 16, the image generation unit 142 may generate a third composite image VS3 by superimposing the second composite image VS2 and the fourth image Vi4 such that common regions in the first image Vi1 and the fourth image Vi4 overlap. The image generation unit 142 may generate a third composite image VS3 that includes a first region R1, a first divided region R41, a second divided region R42, and a third divided region R43 of the upper region R24 that does not overlap with the first divided region R41 and the second divided region R42. The third divided region R43 may be one of a plurality of regions obtained by dividing the upper region R24 along the above direction. In other words, the image generation unit 142 may generate a third composite image VS3 as the composite image VS, which includes a first divided region R41, a second divided region R42, and a third divided region R43 obtained by dividing the upper region R24 along the above direction.

[0125] Thus, the image generation unit 142 may generate a composite image in which different images are embedded in each of the multiple regions divided along the direction by superimposing images captured along the direction for the upper region R24. In the above description, an example was given in which the image generation unit 142 embeds different images in each of the multiple regions divided in the upper region R24, but the image generation unit 142 may also generate a composite image in which different images are embedded in each of the multiple regions divided along the direction for other regions of the second region R2.

[0126] The number of regions divided in the above direction is not particularly limited. Furthermore, the size of each region in the above direction is not particularly limited. Therefore, in the above example, the imaging position is not limited to the first position PO1 and the second position PO2; it can be any of several positions between the "center" position and the "bottom end" position, and the distances between each imaging position do not need to be equal. However, if the width of the multiple regions in the above direction is too small, it may become difficult to calculate the amount of distortion in those regions (i.e., calculate the amount of distortion correction). Therefore, the width of the multiple regions may be set to a size that allows for the calculation of the amount of distortion. The width of the multiple regions may be set to a range (number of pixels) greater than or equal to the minimum value assumed to be the amount of distortion.

[0127] As described above, the image generation unit 142 may generate a composite image using a plurality of regions obtained by dividing the second region R2 along the above direction as the second region R2. In this embodiment, the image generation unit 142 may generate a composite image using the image of the first region R1 and a plurality of images of the second region R2, each of which includes a plurality of regions.

[0128] The calculation unit 143 may calculate the second distortion amount in the second region R2 by calculating the amount of distortion for each of the multiple regions obtained by dividing the second region R2 along the above direction. As described in Embodiment 1, the calculation unit 143 may calculate the distance between each coordinate position of the virtual image V when the display image I is enlarged without distortion and each coordinate position of the virtual image V that is actually imaged and corresponds to each coordinate position, as the amount of distortion at each coordinate position. The coordinate positions of the virtual image V that is actually imaged may consist of the coordinate position in the first image Vi1 and the coordinate positions in the images corresponding to the multiple regions obtained by imaging the virtual image V from each imaging position. The calculation unit 143 may then identify the correspondence between the coordinate position on the display image I and the coordinate position on the composite image generated by superimposing images that include each of the multiple regions as described above (corresponding to the second correspondence).

[0129] Generally, the amount of distortion tends to increase as the distance from the center of the virtual image V increases. The control unit 140 can perform distortion correction processing that takes this tendency into account by calculating the amount of distortion for each of the multiple regions into which the second region R2 is divided. As a result, the control unit 140 can correct the distortion of the virtual image V with greater accuracy. Consequently, the visibility of the virtual image V can be further improved.

[0130] Specifically, when generating a composite image using different images captured for each region, the edge regions of each image captured from various directions, where the amount of distortion tends to be large, can be used as the image for the second region R2. Therefore, the control unit 140 can generate a composite image that more accurately reflects the amount of distortion when the virtual image V is viewed from various directions, and can perform distortion correction processing using this composite image. Consequently, even when the user 22 views the virtual image V from various directions, the visibility of the virtual image V can be further improved.

[0131] Considering these effects, it can be said that the larger the second region R2 in the virtual image V, the more effective it is to perform distortion correction processing on the virtual image V based on the amount of distortion for each of the multiple regions into which the second region R2 is divided (based on a composite image created by combining the images of each of the multiple regions).

[0132] [Embodiment 3] For example, although the above describes a display device equipped with a display panel, it is not limited to this. For example, a device without a display panel may be equipped with an optical system.

[0133] For example, the housing of a display device may include a display panel mounting section. The display panel mounting section may be capable of mounting a display panel. The display panel mounting section may be located on a part of the wall surface of the housing, or it may be located inside the housing. In this case, the display panel may be located inside the housing. Alternatively, the display panel mounting section may be located outside the housing. That is, the display panel may be located outside the housing. In this case, the housing may have an opening in which a part of the wall surface is cut out. The display panel mounting section may be positioned relative to the housing so that the display light emitted from the display panel installed in the display panel mounting section is guided into the inside of the housing through the opening. The display panel mounting section may be connected to the outer wall of the housing, or it may be connected to the outer wall of the housing so as to close at least a part of the opening. A light-transmitting member may be placed in the opening, and this member may be, for example, glass or resin. For example, Figures 1 and 2 show a display device in which a display panel is installed in the display panel mounting section, but there may also be a device in which a display panel is not installed in the display panel mounting section. In this case, the device may be a display panel housing having a housing with a viewing section, an optical system, and a display panel mounting section on which a display panel can be installed. The display panel housing may also realize the configuration of the display device of each embodiment described above. That is, the position of the display panel mounting section of the display panel housing may be defined such that when a display panel is installed in the display panel mounting section, it will have the configuration of each embodiment described above. Furthermore, the housing of the display panel housing may have an opening through which the display panel can be inserted. In this case, the display panel housing may have the same configuration as the display device, except that the housing has an opening and the display panel can be inserted from the outside.

[0134] [Example of implementation by software] The functions of the image control device 130 (hereinafter referred to as "device") can be realized by a program that causes a computer to function as the device, and by a program that causes a computer to function as each control block of the device (particularly each part included in the control unit 140).

[0135] In this case, the device may include a computer having at least one control device (e.g., a processor) and at least one storage device (e.g., memory) as hardware for executing the program. By executing the program using this control device and storage device, each of the functions described in each of the embodiments above may be realized.

[0136] The above program may be recorded on one or more computer-readable recording media, not temporary ones. These recording media may or may not be provided by the above device. In the latter case, the program may be supplied to the above device via any wired or wireless transmission medium.

[0137] Furthermore, some or all of the functions of each of the above control blocks can also be implemented by logic circuits. For example, an integrated circuit in which logic circuits functioning as each of the above control blocks are formed may also be included in the scope of this disclosure. In addition, it is also possible to implement the functions of each of the above control blocks by, for example, a quantum computer.

[0138] Furthermore, each process described in the above embodiments may be performed by AI (Artificial Intelligence). In this case, the AI ​​may operate on the control device described above, or it may operate on another device (for example, an edge computer or a cloud server).

[0139] [Summary] An image control device according to Embodiment 1 of the present disclosure is an image control device for controlling a display device, the display device comprising: a display panel for displaying a display image; an optical system capable of forming an image based on the display image at a position different from the display panel; and a housing in which the display panel is located and which has a viewing section, the image control device comprising: a calculation unit for calculating a first distortion amount in a first region of the image that is visible when the viewing section is viewed from the front; and a second distortion amount in a second region of the image that is not visible when the viewing section is viewed from the front; and a first correction unit for performing distortion correction processing on the image based on the first distortion amount and the second distortion amount.

[0140] In the image control device according to embodiment 2 of the present disclosure, in embodiment 1, the calculation unit calculates the amount of distortion for each of the plurality of regions obtained by dividing the second region along the direction from the outer edge of the first region to the outer edge of the image, thereby calculating the amount of distortion.

[0141] An image control device according to embodiment 3 of the present disclosure includes an image generation unit that generates a composite image by combining the first region and the second region in embodiment 1 or 2, and the calculation unit calculates a first distortion amount and a second distortion amount by calculating the distortion amount of the composite image.

[0142] In the image control device according to embodiment 4 of the present disclosure, in embodiment 3, the image generation unit generates the composite image using an image of the first region captured through the viewing unit from a position that is facing the viewing unit, and an image including the second region captured through the viewing unit from a position other than facing the viewing unit.

[0143] An image control device according to aspect 5 of the present disclosure includes an image generation unit that generates a composite image by combining the first region and the second region in aspect 2, wherein the image generation unit generates the composite image using the plurality of regions as the second region.

[0144] The image control device according to embodiment 6 of the present disclosure generates the composite image using an image of the first region captured through the viewing unit from a position that views the viewing unit from the front, and a plurality of images including each of the plurality of regions, which include the second region captured through the viewing unit from a position other than the front of the viewing unit.

[0145] The image control device according to embodiment 7 of the present disclosure, in embodiment 4 or 6, the image including the second region includes, in a plane at a position away from the viewing unit, at least one of the following: (1) an image captured via the viewing unit from above a normal line passing through the center position of the viewing unit; (2) an image captured via the viewing unit from below a normal line; (3) an image captured via the viewing unit from to the left of a normal line when the viewing unit is viewed from the front; (4) an image captured via the viewing unit from to the right of a normal line when the viewing unit is viewed from the front; and (5) an image captured via the viewing unit from a direction other than the upward, downward, left, and right directions.

[0146] The image control device according to embodiment 8 of the present disclosure includes, in any of embodiments 1 to 7, a second correction unit that performs distortion correction processing on the displayed image based on the result of the distortion correction processing on the image performed by the first correction unit.

[0147] The display device according to aspect 9 of this disclosure comprises an image control device as described in any of aspects 1 to 8.

[0148] The mobile body according to aspect 10 of the present disclosure comprises an image control device as described in any of aspects 1 to 8.

[0149] A display system according to aspect 11 of this disclosure comprises a display device as described in aspect 9 and a camera capable of communicating with the display device, wherein the display panel displays an image captured by the camera.

[0150] The mobile body according to aspect 12 of this disclosure comprises the display system described in aspect 13.

[0151] An image control device according to embodiment 13 of the present disclosure is an image control device for controlling a display panel housing device, the display panel housing device comprising: a display panel installation unit on which a display panel for displaying a display image can be installed; an optical system capable of forming an image based on the display image at a position different from the display panel; and a housing having a viewing unit, the image control device comprising: a calculation unit that calculates a first distortion amount in a first region of the image that is visible when the viewing unit is viewed from the front; and a second distortion amount in a second region of the image that is not visible when the viewing unit is viewed from the front; and a first correction unit that performs distortion correction processing on the image based on the first distortion amount and the second distortion amount.

[0152] An image control method according to aspect 14 of the present disclosure is an image control method for controlling a display device, the display device comprising: a display panel for displaying a display image; an optical system capable of forming an image based on the display image at a position different from the display panel; and a housing in which the display panel is located and which has a viewing section, the image control method comprising: a calculation step of calculating a first distortion amount in a first region of the image that is visible when the viewing section is viewed from the front, and a second distortion amount in a second region of the image that is not visible when the viewing section is viewed from the front; and a correction step of performing a distortion correction process on the image based on the first distortion amount and the second distortion amount.

[0153] Each aspect of the present disclosure may be implemented by a computer, in which case the image control program for the image control device that enables the computer to implement the image control device by operating the computer as each part (software element) of the image control device, and the computer-readable recording medium on which the program is recorded, also fall within the scope of the present disclosure.

[0154] [Additional Notes] The inventions described in this disclosure have been explained based on the drawings and embodiments. However, the inventions described in this disclosure are not limited to the embodiments described above. That is, the inventions described in this disclosure can be modified in various ways within the scope shown in this disclosure, and embodiments obtained by appropriately combining the technical means disclosed in different embodiments are also included in the technical scope of the inventions described in this disclosure. In other words, it should be noted that it is easy for those skilled in the art to make various modifications or alterations based on this disclosure. Furthermore, it should be noted that these modifications or alterations are included in the scope of this disclosure.

[0155] 1 Display device 2 Display panel 3 Optical system 23 Vehicle 37 Aperture 36 Housing 130 Image control device 142 Image generation unit 143 Calculation unit 144 First correction unit 145 Second correction unit I Display image R1 First region R2 Second region R41 First divided region (region) R42 Second divided region (region) R43 Third divided region (region) V Virtual image (image based on the display image) Vi1 First image (image of the first region) Vi2-Vi5 Second to fifth images (image including the second region) VS Composite image

Claims

1. An image control device for controlling a display device, wherein the display device comprises: a display panel for displaying a display image; an optical system capable of forming an image based on the display image at a position different from the display panel; and a housing in which the display panel is located and which has a viewing section, and the image control device comprises: a calculation unit for calculating a first distortion amount in a first region of the image that is visible when the viewing section is viewed from the front, and a second distortion amount in a second region of the image that is not visible when the viewing section is viewed from the front; and a first correction unit for performing distortion correction processing on the image based on the first distortion amount and the second distortion amount.

2. The image control device according to claim 1, wherein the calculation unit calculates the second distortion amount by calculating the distortion amount for each of a plurality of regions obtained by dividing the second region along the direction from the outer edge of the first region to the outer edge of the image.

3. The image control device according to claim 1 or 2, comprising an image generation unit that generates a composite image by combining the first region and the second region, wherein the calculation unit calculates the first distortion amount and the second distortion amount by calculating the distortion amount of the composite image.

4. The image control device according to claim 3, wherein the image generation unit generates the composite image using an image of the first region captured through the viewing unit from a position that views the viewing unit from the front, and an image including the second region captured through the viewing unit from a position other than the front of the viewing unit.

5. The image control device according to claim 2, comprising an image generation unit that generates a composite image by combining the first region and the second region, wherein the image generation unit generates the composite image using the plurality of regions as the second region.

6. The image control device according to claim 5, which generates the composite image using an image of the first region captured through the viewing unit from a position in which the viewing unit is viewed from the front, and a plurality of images including each of the plurality of regions as an image including the second region captured through the viewing unit from a position other than in front of the viewing unit.

7. The image control device according to claim 4 or 6, wherein the image including the second region includes, in a plane at a position separated from the viewing unit, at least one of: (1) an image captured via the viewing unit from above a normal line passing through the center position of the viewing unit; (2) an image captured via the viewing unit from below a normal line; (3) an image captured via the viewing unit from to the left of a normal line when the viewing unit is viewed from the front; (4) an image captured via the viewing unit from to the right of a normal line when the viewing unit is viewed from the front; and (5) an image captured via the viewing unit from a direction other than the upward, downward, left, and right directions.

8. An image control device according to any one of claims 1 to 7, further comprising a second correction unit that performs distortion correction processing on the displayed image based on the result of distortion correction processing on the image performed by the first correction unit.

9. A display device comprising an image control device according to any one of claims 1 to 8.

10. A mobile body comprising an image control device according to any one of claims 1 to 8.

11. A display system comprising: a display device according to claim 9; and a camera capable of communicating with the display device, wherein the display panel displays an image captured by the camera.

12. A mobile body comprising the display system described in claim 11.

13. An image control device for controlling a display panel housing, wherein the display panel housing comprises: a display panel installation section on which a display panel for displaying a display image can be installed; an optical system capable of forming an image based on the display image at a position different from the display panel; and a housing having a viewing section, and the image control device comprises: a calculation unit that calculates a first distortion amount in a first region of the image that is visible when the viewing section is viewed from the front, and a second distortion amount in a second region of the image that is not visible when the viewing section is viewed from the front; and a first correction unit that performs distortion correction processing on the image based on the first distortion amount and the second distortion amount.

14. An image control method for controlling a display device, wherein the display device comprises: a display panel for displaying a display image; an optical system capable of forming an image based on the display image at a position different from the display panel; and a housing in which the display panel is located and which has a viewing section, and the image control method includes: a calculation step of calculating a first distortion amount in a first region of the image that is visible when the viewing section is viewed from the front, and a second distortion amount in a second region of the image that is not visible when the viewing section is viewed from the front; and a correction step of performing a distortion correction process on the image based on the first distortion amount and the second distortion amount.

Images