Imaging device, image processing method, and program

Abstract

This imaging device comprises: a first imaging unit that comprises an optical system and an imaging element; a sensor unit that is capable of sensing a range that includes an imaging range of the first imaging unit and is wider than the imaging range; and an image processing unit that generates a composite image obtained by combining a first image obtained by the first imaging unit with a second image obtained by capturing an image of the outside of the imaging range of the first image by the sensor unit, and performs distortion correction on the composite image.

Description

Imaging Device, Image Processing Method, Program 【0001】 This technology relates to an imaging device, an image processing method, and a program, and more particularly to an image processing technology including image distortion correction. 【0002】 Performing distortion correction on a captured image in an imaging device is well-known. Patent Document 1 below discloses a configuration provided with correction means for performing distortion aberration correction on image information in an imaging device. 【0003】 Japanese Patent Application Laid-Open No. 2011-97131 【0004】 By the way, for example, when shooting with a wide-angle lens, there are cases where the area on the wide-angle side is desired to be captured without distortion depending on the subject or shooting intention, and cases where perspective (Perspective: sense of depth) is applied to enhance the visual effect for shooting. 【0005】 Changing the amount of perspective effect can be achieved by post-shooting image editing, by changing the lens during shooting, or by using a conversion lens, but these are time-consuming. Also, some cameras can switch on / off distortion correction or switch between barrel distortion, pincushion distortion, etc. in the camera settings, but there is no device that can finely adjust the distortion correction amount as intended by the user. Depending on the correction intensity, there may also be cases where the image looks like a circular fisheye, that is, there are black gaps where there is no image in the periphery. 【0006】 Therefore, this technology proposes an image processing method that appropriately utilizes distortion correction as an image effect while making black gaps less likely to occur. 【0007】The imaging device according to this technology comprises a first imaging unit equipped with an optical system and an image sensor; a sensor unit that includes the imaging range of the first imaging unit and is capable of sensing a wider range than the imaging range; and an image processing unit that generates a composite image by combining a first image obtained by the first imaging unit with an image outside the imaging range of the first image obtained by the sensor unit, and performs distortion correction on the composite image. By combining a first image obtained by the first imaging unit with an image outside the imaging range that is outside the field of view of the first image obtained by the first imaging unit, using a second image obtained by the sensor unit, a composite image with a wider field of view than the first image is generated. Then, distortion correction is performed on the composite image. 【0008】 This is an explanatory diagram of a smartphone, which is an example of an imaging device according to an embodiment of this technology. This is a block diagram of the configuration of the imaging device according to the embodiment. This is a block diagram of the main part related to distortion correction of the imaging device according to the first embodiment. This is an explanatory diagram of the synthesis process and distortion correction according to the embodiment. This is an explanatory diagram of the synthesis process and preview display according to the embodiment. This is an explanatory diagram of the preview display according to the embodiment. This is a block diagram of the main part related to distortion correction of the imaging device according to the second embodiment. This is a block diagram of the main part related to distortion correction of the imaging device according to the third embodiment. This is a block diagram of the main part related to distortion correction of the imaging device according to the fourth embodiment. This is a flowchart of the image processing according to the embodiment. This is a flowchart of the synthesis process according to the embodiment. This is a flowchart of the details of image synthesis according to the embodiment. This is an explanatory diagram of the calculation area according to the embodiment. This is an explanatory diagram of edge detection according to the embodiment. This is an explanatory diagram of area detection according to the embodiment. This is an explanatory diagram of image extrapolation according to the embodiment. This is a flowchart of the processing for video according to the embodiment. This is an explanatory diagram of the calculation area based on optical flow according to the embodiment. 【0009】 The embodiments will be described below in the following order: <1. Configuration of the imaging device> <2. Image processing of the first embodiment> <3. Image processing of the second embodiment> <4. Image processing of the third embodiment> <5. Image processing of the fourth embodiment> <6. Processing examples> <7. Summary and modified examples> 【0010】In this disclosure, "distortion correction" refers to both correction that eliminates distortion present in captured images and correction that actively adds distortion to images. Furthermore, "image" refers to both still images and videos. 【0011】 <1. Configuration of the Imaging Device> In this embodiment, the explanation will mainly be given using the example of launching a camera application on a smartphone to take a picture. Therefore, the imaging device 1 in this embodiment is a smartphone, which is a general-purpose information processing device equipped with a lens and an image sensor for imaging. 【0012】 Figure 1 shows the external appearance of a smartphone, which is an example of an imaging device 1. The smartphone's casing is, for example, a flat, roughly rectangular parallelepiped shape, and a display panel 2 is formed on the first surface 100 of the casing, on which various images are displayed by the application program that is launched. 【0013】 The second surface 200 is the back of the smartphone, where the lens unit 5 is located. The lens unit 5 includes, for example, three lenses 6. These may be an ultra-wide lens 6U, a wide lens 6W, and a telephoto lens 6T. The lens unit 5 also has a Time of Flight (ToF) sensor 9 located between the wide lens 6W and the telephoto lens 6T. As shown in the figure, the light-emitting and light-receiving parts of the ToF sensor 9 are exposed. Adjacent to the lens unit 5 are a flash unit 8 and a photometering unit 7 for measuring ambient light. 【0014】 A shutter button 3 is provided on the side of the casing. In addition to the shooting button displayed on the camera application on the display panel 2, the user can also take still images using the shutter button 3. 【0015】Figure 2 shows an example configuration of an imaging device 1, which is, for example, a smartphone. The imaging device 1 comprises a control unit 30, imaging units 31A, 31B, 31C, a display unit 32, an operation unit 33, a photometric processing unit 34, a distance sensor unit 35, a storage unit 36, and a memory unit 37. Note that the configuration in Figure 2 is merely an example showing the essential parts, and it may also include components not shown, and it is not necessary to include all of the components shown. 【0016】 The imaging unit 31A includes an optical system corresponding to the wide-angle lens 6W, an image sensor for receiving incident light, and a signal processing circuit for the photoelectrically converted signal from the image sensor. The imaging unit 31B includes an optical system corresponding to the ultra-wide-angle lens 6U, an image sensor for receiving incident light, and a signal processing circuit for the photoelectrically converted signal from the image sensor. The imaging unit 31C includes an optical system corresponding to the telephoto lens 6T, an image sensor for receiving incident light, and a signal processing circuit for the photoelectrically converted signal from the image sensor. 【0017】 The optical systems of the imaging units 31A, 31B, and 31C include a zoom lens, a focus lens, an aperture mechanism, and a drive mechanism for these optical elements. This optical system guides light from the subject and focuses it onto the image sensor. In this embodiment, an example of processing using imaging unit 31A, which performs wide-angle imaging, and imaging unit 31B, which performs ultra-wide-angle imaging, will be described in detail later. 【0018】 The image sensors in the imaging units 31A, 31B, and 31C are configured as, for example, CCD (Charge Coupled Device) type, CMOS (Complementary Metal Oxide Semiconductor) type, etc. The image sensors perform processes such as CDS (Correlated Double Sampling) processing and AGC (Automatic Gain Control) processing on the electrical signal obtained by photoelectric conversion of the received light, and further perform A / D (Analog / Digital) conversion processing. The captured image signal as digital data is then output to the control unit 30. 【0019】The display unit 32 consists of the display panel 2 and its display driving circuit, and displays various information to the user. The display panel 2 is made up of a liquid crystal display (LCD), an electroluminescent (EL) panel, or the like. 【0020】 The display unit 32 performs various displays on the display screen based on instructions from the control unit 30. For example, the display unit 32 displays a preview image taken during shooting, as well as various icons and controls as part of the user interface. In other words, it performs a GUI (Graphical User Interface) display on the screen. 【0021】 The operation unit 33 comprehensively represents the input devices for the user to perform various operations. Specifically, these include physical controls such as the shutter button 3, and a touch panel provided on the display panel 2 that detects operations on controls (buttons, icons, sliders, etc.) displayed on the display panel 2. The operation unit 33 detects user operations, and signals corresponding to the input operations are sent to the control unit 30. 【0022】 The photometric processing unit 34 has the photometric unit 7 and its processing circuit described above. The photometric processing unit 34 transmits the photometric information to the control unit 30. 【0023】 The distance measuring sensor unit 35 comprises a light-emitting unit, a light-receiving unit, and a calculation control unit that controls the light emission and performs calculations based on the received light signal, and performs processing to calculate the distance (depth) to the subject. The calculated distance information is supplied to the control unit 30. The ability to calculate the distance to the subject also means that the shape of the subject can be determined. 【0024】The storage unit 36 ​​performs processing to record image files (content files) such as still image data and video data, as well as attribute information of image files, thumbnail images, etc., on a recording medium such as non-volatile memory. The actual form of the storage unit 36 ​​can be diverse. For example, the storage unit 36 ​​may be a circuit unit that performs recording and playback on flash memory built into the imaging device 1, or it may be a card recording and playback circuit that performs recording and playback access on a memory card (for example, portable flash memory) that can be attached to and removed from the imaging device 1. In addition, the storage unit 36 ​​may be implemented as an HDD (Hard Disk Drive) or SSD (Solid State Drive) when built into the imaging device 1. 【0025】 The control unit 30 is comprised of a microcomputer (arithmetic processing unit) equipped with a CPU (Central Processing Unit). The memory unit 37 stores information used by the control unit 30 for processing. This memory unit 37 includes, for example, ROM (Read Only Memory), RAM (Random Access Memory), flash memory, etc. The memory unit 37 may be a memory area built into the microcomputer chip that constitutes the control unit 30, or it may be comprised of a separate memory chip. 【0026】 The RAM in the memory unit 37 is used as a workspace for the CPU of the control unit 30 during various data processing operations, and is used for the temporary storage of data and programs. The ROM and flash memory (non-volatile memory) in the memory unit 37 are used to store the OS (Operating System) for the CPU to control each part, application programs for various operations, and various setting information. In this embodiment, parallax information based on the lens placement is also stored in the ROM and flash memory. For example, the distance between the cameras is stored as parallax information for the fixedly positioned ultra-wide lens 6U and wide lens 6W. 【0027】The control unit 30 controls the entire imaging device 1 by executing programs stored in the ROM or flash memory of the memory unit 37. For example, the control unit 30 controls the zoom, focus, and aperture adjustment of the optical system of the imaging unit 31, controls the shutter speed of the image sensor, processes signals for captured image data, performs imaging and recording operations in response to user operations, and plays back recorded image files. 【0028】 The figure shows the image processing unit 30a as a processing function in the control unit 30 according to this embodiment. The image processing unit 30a performs processing on the captured image data that is captured and input by the imaging units 31A, 31B, and 31C, such as RAW correction processing, Y / C development processing, codec processing, and file generation processing. The image data output after processing by these image processing units 30a is stored in the storage medium by the storage unit 36. In particular, the image processing unit 30a performs processing such as image data synthesis processing and distortion correction processing of the image data obtained from imaging by the imaging units 31A and 31B, as well as UI (User Interface) processing related thereto. 【0029】 <2. Image Processing of the First Embodiment> The image processing by the imaging device 1 described above will now be explained. First, the image processing related to distortion correction as the first embodiment will be explained in Figure 3. Figure 3 shows the imaging units 31A and 31B and the image processing unit 30a. The image processing unit 30a is shown as a processing function of the control unit 30 realized by software, and therefore the processing of the image processing unit 30a described below is the processing executed by the control unit 30 based on the software program. 【0030】 The imaging unit 31A includes an optical system 10A, an image sensor 22A, an output processing unit 23A, a lens setting unit 24A, and an imager setting unit 25A. The imaging unit 31B also includes an optical system 10B, an image sensor 22B, an output processing unit 23B, a lens setting unit 24B, and an imager setting unit 25B. 【0031】Optical system 10A includes a condensing lens 11A, a zoom lens 12A, a focusing lens 13A, a lens 14A, and optical mechanisms (such as an aperture mechanism and a filter) not shown. Optical system 10B includes a condensing lens 11B, a lens 12B, a focusing lens 13B, and optical mechanisms (not shown). 【0032】 The lens setting unit 24A is bidirectionally connected to the zoom lens 12A and the focus lens 13A. The lens setting unit 24A supplies drive instructions to the zoom lens 12A and the focus lens 13A. These instructions may include target position, speed, and step pattern. The lens setting unit 24A also receives information from the zoom lens 12A and the focus lens 13A regarding their drive status (whether they are driving or stopped) and current position. 【0033】 The lens setting unit 24B is bidirectionally connected to the focus lens 13B. The lens setting unit 24B supplies drive commands to the focus lens 13B and receives information such as the drive status and current position from the focus lens 13B. 【0034】 These lens setting units 24A and 24B perform the necessary settings and drive the lens in response to the user's zoom and focus operations. They can also drive the lens for autofocus control. 【0035】 The imager setting unit 25A sets the shutter speed and other parameters for the image sensor 22A. The imager setting unit 25B sets the shutter speed and other parameters for the image sensor 22B. 【0036】 In the imaging unit 31A, the optical system 10A guides the subject light to the image sensor 22A, which then captures the image, i.e., converts it to an electrical signal. The electrical signal from the image sensor 22A undergoes gain processing, A / D conversion processing, etc., in the output processing unit 23A, and is sent to the image processing unit 30a as a first image (image data). 【0037】In the imaging unit 31B, the optical system 10B guides the subject light to the image sensor 22B, which then captures the image, i.e., converts it to an electrical signal. The electrical signal from the image sensor 22B is processed by the output processing unit 23B, which performs gain processing, A / D conversion, etc., and is then sent to the image processing unit 30a as a second image (image data). 【0038】 In the following explanation, the imaging unit 31A will be referred to as the "A imaging system," and the imaging unit 31B as the "B imaging system." The first image obtained by the A imaging system will be referred to as the "A image," and the second image obtained by the B imaging system will be referred to as the "B image." The B image is a wider-angle image than the A image. 【0039】 The above images A and B are input to the image processing unit 30a. The image processing unit 30a then combines image B with image A to generate a composite image with a wider field of view than image A alone. In other words, it is a process that connects the surrounding pixel information obtained from the wider-angle image B to the area around image A. As a result, even if the user performs distortion correction as an image effect on the composite image, it generates a perspective effect image in which blackouts do not occur (or occur with minimal blackouts). Figure 3 shows the image processing unit 30a with a processing block for this process. 【0040】 In the image processing unit 30a, distortion correction is first performed on image A by the A distortion correction unit 51A, and then on image B by the B distortion correction unit 51B. This is because the distortion characteristics of the A imaging system and the B imaging system are different, so the process corrects images A and B to a state with less distortion before image synthesis. Figure 4 schematically shows images A 90 and B 91 with distortion characteristics, and images A 90 and B 91 after distortion correction. Because the focal lengths of the A imaging system and the B imaging system are different, meaning the lens configurations are different, images A 90 and B 91 are distorted in different ways. Generally, wider angles are more prone to distortion. Distortion correction is performed on each image after correcting these differences in distortion, as this makes subsequent synthesis processing easier. In addition to distortion correction, it is also advisable to correct coma aberration, chromatic aberration, lens shading, etc., at this time. 【0041】The B image 91 corrected for distortion by the B distortion correction unit 51B in FIG. 3 is processed by the B image processing unit 53 to be in a state suitable for synthesis with the A image 90. Since the image sensors and lens configurations are different between the A imaging system and the B imaging system, the A image 90 and the B image 91 deviate in parts other than the angle of view. For example, color reproduction, gradation, gamma characteristics, black level, etc. Therefore, in the B image processing unit 53, processing is performed on the B image 91 so that these are closer to the A image 90. Since there may also be a difference in the shape of the point light source due to the difference in image height from the A image 90, in some cases, processing for changing the shape of the B image 91 is also performed. 【0042】 The A image 90 corrected for distortion by the A distortion correction unit 51A and the B image 91 processed by the B image processing unit 53 are subjected to synthesis processing by the image synthesis unit 52. For example, as shown in FIG. 4 for A / B synthesis, a composite image 98 obtained by aligning and synthesizing the A image 90 and the B image 91 is shown. In this case, taking the A image 90 as the main image, registration of the A image 90 and the B image 91 is performed to determine the positional relationship, and pixel information of the surrounding subject that is not projected onto the A image 90, that is, outside the angle of view, is acquired from the B image 91 and connected to the periphery of the A image 90 for the synthesis process. In the figure, although the composite image 98 is not an area where the A image 90 is in the center with respect to the B image 91, this is a deviation affected by the parallax between the A imaging system and the B imaging system. 【0043】 A composite image 98 further trimmed from this state is also shown. For example, trimming is performed to absorb the deviation due to parallax so that regions of the B image 91 are arranged substantially evenly around the A image 90. The image synthesis unit 52 in FIG. 3 outputs a composite image 98 like this, for example. 【0044】 The distortion correction unit 54 in FIG. 3 performs distortion correction on the composite image 98 according to a user operation. The distortion correction amount adjustment unit 57 sets the distortion correction amount according to the user operation by the operation unit 33 in FIG. 2 and instructs the set distortion correction amount to the distortion correction unit 54. The distortion correction unit 54 performs distortion correction with the instructed correction amount. FIG. 4 shows a composite image 98 in which distortion correction is performed with a correction amount according to a user operation, for example, with the sense of perspective emphasized. 【0045】Then, the distortion correction unit 54 supplies the composite image 98 before and after distortion correction to the preview display control unit 55 and the captured data storage control unit 56. 【0046】 The preview display control unit 55 controls the composite image 98 supplied from the distortion correction unit 54 to be displayed on the display unit 32 in FIG. 2. It also controls the display of the user interface related to the distortion correction operation. The captured data storage control unit 56 controls the image supplied from the distortion correction unit 54 to be stored in the recording medium in the storage unit 36 in FIG. 2 according to the user operation. 【0047】 For example, in the shooting process, the user activates a mode in the imaging device 1 that allows variable distortion correction and sets it to the standby state for shooting the subject. Then, while checking the preview display, the user sets the desired amount of distortion correction and performs the shooting operation, and the image with the specified amount of distortion correction will be saved. 【0048】 An example of the change in the image due to the above processing will be described with reference to FIGS. 5 and 6. First, FIG. 5 shows an A image 90 and a B image 91 respectively. The frame of the A image 90 is shown by a dashed line and is designated as the A image frame 90W. 【0049】 The B image 91 is an image captured with a wider angle of view than the A image 90. Inside the B image 91, the A image frame 90Wa indicating the angle of view of the A image 90 is shown by a dashed line. For example, from the B image 91, the pixel information in the area around the A image frame 90Wal, that is, the pixel information in the range indicated by the composite image frame 92W, is cut out and connected around the A image 90 to generate the composite image 98. The composite image frame 92W that is the cut-out range is not necessarily at the center of the B image 91. This is because of the parallax and the like due to the lens arrangements of the A imaging system and the B imaging system. 【0050】Such a composite image 98 is displayed as a preview and presented to the user. Even though the composite image 98 is generated, the original image that the user intends to capture and store is image A 90. The area of ​​image B 91 in the composite image 98 becomes an unnecessary area if distortion correction is not performed by the distortion correction unit 54. Furthermore, the area of ​​image B 91 appears in the image depending on the amount of distortion correction, but because it is significantly distorted by the distortion correction, even if the image quality is inferior to the central image A 90, it does not create much of a sense of incongruity. 【0051】 The preview display in Figure 5 shows a state where the generation of the composite image 98 is not yet complete. For example, the output image frame mark 93 indicates to the user the range of the image to be stored as shooting data. In Figure 5, the area within the output image frame mark 93 is image A 90, and the surrounding area, i.e., the area to which image B 91 will be combined, is displayed blurred because the composite image has not yet been generated (it is shown in gray in the figure). When the generation of the composite image 98 is complete, the area of ​​image B 91 is displayed without blurring in the preview display. However, if the proper compositing process cannot be performed and the composite image 98 cannot be generated, the display as shown in the figure will continue. To inform the user of these situations, a message 94 is displayed indicating that compositing is in progress or that a compositing error has occurred. 【0052】 Situations where proper image compositing is not possible include, for example, when one lens is blocked, when the subject is complex and the compositing accuracy is low, or when there are occlusion objects due to parallax that prevent compositing. 【0053】 It should be added that if the user does not request distortion correction as an image effect, the A image 90 captured by the A imaging system will be displayed in preview or stored as is. In that case, the processing of the A distortion correction unit 51A, image synthesis unit 52, and distortion correction unit 54 in Figure 3 does not need to be performed on the A image 90. Figure 3 only shows the processing when the user performs distortion correction as an image effect. 【0054】Figure 6 shows the preview display, which includes the composite image 98, the output image frame mark 93, and the slider bar 96 as the distortion correction control. The top of Figure 6 shows the state when the user moves the pointer 96P of the slider bar 96 all the way up, setting the pincushion distortion intensity (pincushion distortion correction amount) to the maximum value. The middle of Figure 6 shows the state when the pointer 96P of the slider bar 96 is in the center, specifying zero distortion. The bottom of Figure 6 shows the state when the user moves the pointer 96P of the slider bar 96 all the way down, setting the barrel distortion intensity (barrel distortion correction amount) to the maximum value. 【0055】 For example, the user can arbitrarily specify the amount of correction for pincushion distortion and barrel distortion by operating the pointer 96P on the slider bar 96. The user's operation of the pointer 96P is detected by the distortion correction amount adjustment unit 57 in Figure 3, and the distortion correction unit 54 is instructed to either correct for pincushion distortion, correct for barrel distortion, or not correct for distortion. The distortion correction unit 54 performs pincushion distortion correction, barrel distortion correction, or no distortion correction according to the instruction, and supplies the resulting composite image 98 to the preview display control unit 55. The preview display control unit 55 displays the composite image 98 as a preview display. 【0056】 As a result, as shown in Figure 6, a composite image 98 with distortion correction applied in conjunction with the operation of the slider bar 96 is displayed. In the upper part of Figure 6, a composite image 98 with pincushion distortion is displayed, and in the lower part of Figure 6, a composite image 98 with barrel distortion is displayed. The composite image 98 with distortion correction applied by an amount corresponding to the operation position of the pointer 96P is displayed. Therefore, the user can find the desired amount of distortion correction by operating the pointer 96P of the slider bar 96. 【0057】If the user specifies the amount of distortion correction, or if they leave the distortion correction at zero, and then performs an image storage operation (e.g., shutter operation), the output image from the distortion correction unit 54, which is currently displayed in the preview, is sent to the image data storage control unit 56, and storage processing is performed. The image data storage control unit 56 controls the storage unit 36 ​​in Figure 2 to store the output image supplied from the distortion correction unit 54, that is, the image within the output image frame mark 93 in the composite image 98, on the recording medium. 【0058】 Looking at the upper part of Figure 6, even in the image with pincushion distortion applied, no black areas are found within the output image frame mark 93. The dashed line shows the A image frame 90Wa, which indicates the range of A image 90. If pincushion distortion is applied to A image 90 only, the A image frame 90Wa will curve inward from the output image frame mark 93. This means that black areas will occur because there are no pixels of A image 90 outside of the A image frame 90Wa. 【0059】 However, in the processing shown in Figure 3, the distortion correction unit 54 applies pincushion distortion to the composite image 98, and pixels acquired from image B 91 exist outside the A image frame 90Wa. Therefore, in the state shown in the upper part of Figure 6, the subject image exists outside the A image frame 90Wa within the output image frame mark 93. In other words, even if output (image storage or transmission) is performed in this state, no blackouts occur in the output image. That is, even if the user instructs the maximum correction amount for distortion correction by operating the slider bar 96, no blackouts will occur in the image data transmitted to and stored in the shooting data storage control unit 56. 【0060】In the upper part of Figure 6, the blacked-out area represents a region where image B 91 does not exist, i.e., an area outside the composite image 98. Therefore, if the correction strength is further increased, the blacked-out area may encroach upon the output image frame mark 93. This would result in blacked-out areas in the output image. For this reason, it is desirable for the preview display control unit 55 to limit the range of correction amounts that can be specified by the slider bar 96 in the UI display control to a correction amount that does not cause blacked-out areas. In other words, even if the user specifies the maximum correction amount with the slider bar 96, the correction amount will be as shown in the upper part of Figure 6. By doing this, it is possible to ensure that blacked-out areas do not always occur in the output image after distortion correction. 【0061】 Blackout occurs when the correction intensity is set to include the output image up to the point where image B 91 no longer exists. Therefore, the limit of the correction intensity can be calculated according to the width of the region of image B 91. The preview display control unit 55 can set this limit of correction intensity to the upper limit of the slider bar 96. 【0062】 In the above embodiment, the user can specify the distortion correction amount for pincushion distortion and barrel distortion. However, during this preview display, the user may specify the field of view, and the distortion correction unit 54 may change the field of view accordingly. If the field of view is specified to be widened, the area of ​​image B 91 can also be used, thus widening the range in which the field of view can be changed in the wide-angle direction. In this embodiment, the user can specify the distortion correction amount for both pincushion distortion and barrel distortion. However, it is also conceivable that the user can specify the distortion correction amount for only one of pincushion distortion or barrel distortion. 【0063】 Figure 3 further shows the shooting setting unit 58 and the B setting calculation unit 59. The shooting setting unit 58 performs various shooting settings according to user operation. Shooting settings include, for example, shutter speed, aperture value, focus position, sensitivity, and white balance. The shooting setting unit 58 reflects the shooting settings in the imager setting unit 25A and lens setting unit 24A of the A imaging system, and also in the imager setting unit 25B and lens setting unit 24B of the B imaging system. 【0064】However, the settings for the B imaging system are reflected via the B setting calculation unit 59. Since the A imaging system and the B imaging system have different focal lengths (different depths of field), if the B imaging system has exactly the same shooting settings as the A imaging system, an unnatural appearance will occur when the A image 90 and B image 91 are combined. Therefore, it is advisable to adjust the depth of field of the B imaging system to match that of the A imaging system by changing the aperture value. Also, if the image sensors 22A and 22B are not identical, their sensitivity and color reproduction will differ. Therefore, it is advisable to adjust the sensitivity and color reproduction of the B imaging system to match those of the A imaging system. Note that, since the B image processing unit 53 mentioned above adjusts the color reproduction to match that of the A imaging system, it is desirable to match the sensitivity. 【0065】 For these reasons, the B setting calculation unit 59 does not use the shooting setting values ​​from the shooting setting unit 58 as they are, but instead performs calculations to modify the shooting setting values ​​so that the aperture value, focus position, curtain speed, sensitivity, white balance, etc. of the B imaging system are similar to those of the A imaging system, and then applies these modified settings to the B imaging system. For this calculation, it is preferable to store the sensitivity difference between the A and B imaging systems as design values ​​in advance, for example, in the non-volatile memory of the memory unit 37. This ensures that no discrepancy occurs between the regions of the A image 90 and the B image 91 in the composite image 98. 【0066】 <3. Image Processing of the Second Embodiment> The image processing of the second embodiment will be explained with reference to Figure 7. The same parts as in Figure 3 are denoted by the same reference numerals to avoid redundant explanation. 【0067】 In the case of Figure 7, the image processing unit 30a is provided with an image synthesis unit 52P for preview display and a distortion correction unit 54P, in addition to the configuration of Figure 3. 【0068】 Image A 90, which has been distorted by the distortion correction unit A 51A, and Image B 91, which has been distorted by the distortion correction unit B 51B and further processed by the Image B processing unit B 53, are supplied to the image synthesis unit 52 to generate a composite image 98. As in Figure 3, distortion correction is applied to this composite image 98 by the distortion correction unit 54 according to user operation, and the output image from the distortion correction unit 54 is stored in the shooting data storage control unit 56 according to the user's storage operation. 【0069】 In addition, the A image 90, which has been distorted by the A distortion correction unit 51A, and the B image 91, which has been distorted by the B distortion correction unit 51B, are supplied to the image synthesis unit 52P, and a composite image for preview is generated. The distortion correction unit 54P then performs distortion correction on this composite image for preview by an amount corresponding to the user operation, i.e., the operation of the slider bar 96, and the resulting image is sent to the preview display control unit 55. 【0070】 In other words, in the configuration shown in Figure 7, the image compositing and distortion correction processes for preview display are performed independently of the image compositing and distortion correction processes for the image to be stored. For this reason, the image compositing unit 52P and the distortion correction unit 54P use a simplified compositing process. 【0071】 Specifically, for the image synthesis processing for preview display by the image synthesis unit 52P, image quality matching between image A 90 and image B 91 is omitted. That is, the processing of image B processing unit 53 for image B 91 is omitted. Also, the processing near the synthesis boundary between image A 90 and image B 91 is simplified. Furthermore, since it is for preview display, lower-resolution data is used for image A 90 and image B 91. In addition, the evaluation in the synthesis process is made more lenient, and synthesis accuracy is not required. This means that the threshold used in the evaluation in step S170 of Figure 11, which will be described later, is lowered. In the distortion correction unit 54P, low-resolution data is used, and the distortion correction processing is simplified. 【0072】 For example, by simplifying the synthesis and distortion correction processes in this way, and by stopping all operations except those related to recording in the image synthesis unit 52 and distortion correction unit 54, high-speed processing can be achieved, making it easy to display images that are responsive to the user's operation of the slider bar 96 in the preview display. In other words, the immediacy of the preview display can be improved. 【0073】 Even if the processing of image data for storage by the distortion correction unit 54 takes longer than the processing by the distortion correction unit 54P, this is not a problem because the output image with distortion correction applied can be supplied to the shooting data storage control unit 56 at a timing corresponding to the user's storage operation. 【0074】<4. Image Processing of the Third Embodiment> The image processing of the third embodiment will be explained in Figure 8. This is an example of combining image A 90 from the A imaging system and image B 91 from the distance measuring sensor unit 35. 【0075】 Figure 8 shows the distance measuring sensor unit 35 in place of the B imaging system in Figure 3. The distance measuring sensor unit 35 has a ToF sensor 9 and a ToF output processing unit 41. The field of view of the ToF sensor 9 is wider than that of the A imaging system. The light reception information obtained by the ToF sensor 9 is output by the ToF output processing unit 41 as distance information for each pixel of the subject, a so-called depth map distance image. This corresponds to the B image 91. 【0076】 Since the distance image contains information about the distance to the subject, it can also be said to contain information that shows the shape of the subject, and in that sense, the output of the ToF output processing unit 41 can be said to be information about the subject image without color information. 【0077】 In the image processing unit 30a, distortion correction is performed on the distance image (B image 91) from the distance measuring sensor unit 35 using the ToF distortion correction unit 60. This is a process that corrects the B image 91 in a direction that eliminates distortion, similar to the B distortion correction unit 51B in Figure 3. 【0078】 The distance image (B image 91), which has been distorted by the ToF distortion correction unit 60, is then color-added by the color information addition unit 61. Although the shape of the subject can be determined from the distance image, color information is not present. Therefore, for example, the color information addition unit 61 prepares a color estimation model using a neural network in advance, performs scene inference using the A image 90 and the distance image as input, and complements the color information of the distance image. 【0079】 Then, the depth image with interpolated color information is processed by the B image processing unit 53 and supplied to the image synthesis unit 52 for synthesis with the A image 90. Subsequent processing is the same as in Figure 3. 【0080】As in this third embodiment, it is also possible to use the information from the distance measuring sensor unit 35 as the B image 91. In this sense, even a device equipped only with an A imaging system for normal shooting purposes can perform appropriate synthesis processing and distortion correction as long as it has a distance measuring configuration. 【0081】 Figure 8 shows an example of a distance measuring sensor unit 35 using a ToF sensor 9, but similar processing can be performed using a LiDAR or the like instead. 【0082】 <5. Image Processing of the Fourth Embodiment> An example of image processing of the fourth embodiment is shown in Figure 9. This is an example where image A 90 and image B 91 are moving images. The same parts as in Figure 3 are denoted by the same reference numerals. In addition to the configuration of Figure 3, Figure 9 is equipped with an inter-frame motion estimation unit 63 that calculates optical flow. 【0083】 The inter-frame motion estimation unit 63 stores the image from the previous frame and determines the flow of moving objects. The area targeted for determining moving objects may be limited to the periphery of image A 90, that is, the area near the boundary with image B 91, and the area near the distortion correction permission area. By not performing motion object flow determination on the entire image area, the computational burden can be reduced. 【0084】 Furthermore, since image B 91 encompasses the shooting area of ​​image A 90, motion estimation only needs to be performed using image B 91. This is because cases where only the subject in image A 90 is visible are unlikely to occur unless the object is at a very close distance. For this reason, only image B 91 from the output processing unit 23B is input to the inter-frame motion estimation unit 63. 【0085】 The inter-frame motion estimation unit 63 estimates the amount of motion of each moving subject in the target area and inputs this information to the image synthesis unit 52. The image synthesis unit 52 determines the magnitude of the motion and decides whether or not to recalculate the synthesis process. By registering image A 90 and image B 91 using the direction and amount of motion of the moving subjects, a synthesis process suitable for video can be performed. 【0086】In the case of video, it is advisable to limit the amount of distortion transitions during image storage. Furthermore, to avoid the risk of image synthesis failure during image storage, it is recommended to store both the composite image and image B 91 (ultra-wide-angle image). 【0087】 <6. Processing Examples> Below, we will describe specific processing examples, including the synthesis process and distortion correction process described in the first to fourth embodiments. Figures 10, 11, 12, and 17 show processing examples executed by the image processing unit 30a (control unit 30) based on the program. 【0088】 Figure 10 shows the process from the moment the user activates the variable distortion mode in the imaging device 1 until the captured data is saved. 【0089】 When the image processing unit 30a detects a user instruction for variable distortion mode in step S101, it proceeds to step S102 and first performs control to automatically start the A imaging system. In step S103, the image processing unit 30a starts preview display on the display unit 32. In this case, the image processing unit 30a displays image A 90, for example, as shown in Figure 5, while blurring the area of ​​image B 91. It also displays a standby message 94 to inform the user that it is starting up. 【0090】 In step S104, the image processing unit 30a loads the settings of the A imaging system and the design difference information between the A imaging system and the B imaging system into the B image processing unit 53, as shown in Figure 3. For example, the design difference information between the A imaging system and the B imaging system is stored in the non-volatile memory or ROM of the memory unit 37. Here, the B image processing unit 53 is able to perform processing on the B image 91, such as adjusting color reproduction, gradation, gamma characteristics, black level, etc., to match the A image 90, based on this design difference information and the current settings of the A imaging system. If there are any changes in settings due to user operations at any given time, such as a change in shutter speed, these changes are reflected in the B image processing unit 53 in real time each time. 【0091】In step S105, the image processing unit 30a focuses on the subject using the A imaging system. For example, it transmits the lens setting corresponding to the user's manual focus operation to the lens setting unit 24A and drives the focus lens 13A. Autofocus control may also be performed. 【0092】 In step S106, the image processing unit 30a controls the focus position of the B imaging system based on the focal length of the A imaging system. Specifically, the B setting calculation unit 59 calculates the focus position setting for the B imaging system according to the focal length of the A imaging system, notifies the lens setting unit 24B, and drives the focus lens 13B. 【0093】 In step S110, the image processing unit 30a performs a composite processing of image A 90 and image B 91 using the image compositing unit 52. The composite processing will be described in detail in Figure 11 and subsequent figures. 【0094】 In step S180, the image processing unit 30a monitors the stability of the synthesis process, and proceeds to step S181 when the output of the synthesis process, i.e., the synthesized image 98, is output stably. In step S181, the image processing unit 30a controls the preview display control unit 55 to display the synthesized image 98 and the slider bar 96 as a preview display on the display unit 32. 【0095】 During preview display, the image processing unit 30a monitors the user's correction amount operation, i.e., the operation of the slider bar 96, in step S182. It also monitors the user's shooting instructions (shutter operation, etc.) in step S184. 【0096】 When the image processing unit 30a detects a user's adjustment of the correction amount, it proceeds to step S183 and changes the distortion correction strength. Specifically, the distortion correction amount adjustment unit 57 instructs the distortion correction unit 54 to adjust the correction amount according to the position of the pointer 96P on the slider bar 96, and the distortion correction unit 54 performs distortion correction of the new correction amount on the composite image 98. Then the image processing unit 30a returns to step S181, and the preview display control unit 55 performs preview display control. As a result, the degree of distortion of the displayed composite image 98 changes. 【0097】When the image processing unit 30a detects a shooting instruction operation by the user, such as shutter operation, it proceeds from step S184 to step S185, and controls the shooting data storage control unit 56 so that the image within the output image frame mark 93 of the composite image 98 at that time is stored in the storage unit 36. This completes the processing for the variable distortion mode. As long as the variable distortion mode is not deactivated, the process from step S105 onwards in Figure 10 should be repeated. 【0098】 The synthesis process in step S110 will be explained in detail from Figure 11 onward. In Figure 11, the image processing unit 30a acquires image A 90 from the A imaging system in step S111, and performs distortion correction on image A 90 in step S112. This is the processing of the A distortion correction unit 51A. In parallel, the image processing unit 30a acquires image B 91 from the B imaging system (or distance sensor unit 35) in step S115, and performs distortion correction on image B 91 in step S116. This is the processing of the B distortion correction unit 51B (or ToF distortion correction unit 60). Furthermore, in step S117, the B image processing unit 53 performs image processing on image B 91 to make it similar to the image composition of image A 90. 【0099】 After the above processing, in step S120, the image processing unit 30a performs a composite processing of image A 90 and image B 91 using the image synthesis unit 52. This process in step S120 is shown in Figure 12. 【0100】 When actually combining image A 90 and image B 91, the image processing unit 30a first performs a design difference correction on image B 91 in step S121 of Figure 12. This correction absorbs the parallax between image A 90 and image B 91, as well as differences in specifications of the image sensor and optical system. Then, in step S122, the image processing unit 30a sets a calculation area for registration. 【0101】Figure 13 shows examples of image A 90 and image B 91. The calculation area 70 for image A 90 and the calculation area 71 for image B 91 are shown as shaded areas. First, for image A 90, a band-shaped calculation area 70 is set around the periphery of the image. This is because the periphery is the region that will be combined with image B 91 during synthesis. Then, for image B 91, the subject area corresponding to the calculation area 70 of image A 90 is set as the calculation area 71. 【0102】 By detecting edges, areas, feature points, etc., within these calculation areas 70 and 71, registration becomes possible that matches the positional relationship between image A 90 and image B 91. The reason for using only a portion of the image, rather than the entire image, as the calculation areas 70 and 71 is to reduce the amount of computation. Furthermore, by making the calculation area 70 the peripheral part of image A 90, it is easier to maintain the accuracy of alignment with image B 91 even in a portion of the image. 【0103】 In step S123, the image processing unit 30a selects a synthesis method. Here, we show an example of selecting one of the following four synthesis methods: a synthesis method using edge detection, a synthesis method using area detection, a synthesis method using feature detection, and a method using image extrapolation. 【0104】 This synthesis method is expected to be selected by the image processing unit 30a depending on the configuration and function of, for example, the A imaging system and the B imaging system (or distance measuring sensor, etc.). For example, if the configuration does not allow for subject distance detection, edge detection can be selected; if the configuration allows for subject distance detection, area detection can be selected; and if a CNN (Convolutional Neural Network) model is available, feature detection can be used. 【0105】 Furthermore, if multiple synthesis methods are possible in terms of configuration, the image processing unit 30a can select one according to user specifications, mode, etc. Also, even if feature detection is possible, this increases the power load, so it is possible to select a synthesis method according to the state of the imaging device 1, such as selecting edge detection or area detection depending on the battery state. 【0106】Furthermore, the image processing unit 30a can select a suitable synthesis method based on the subject conditions of image A 90 and image B 91. For example, it can select a suitable synthesis method based on the overall brightness, average brightness, peak brightness, and contrast difference within the image. It can also select based on whether the subject is complex or simple, or on the type of subject, such as a person, face, object, or landscape. In other words, it can select based on whether the image is suitable for edge detection or feature detection. 【0107】 Furthermore, the synthesis method may be selected according to the current state of the optical system 10A and the image sensor 22A of the A imaging system. Current state includes, for example, temperature, the shift position and orientation of the lens and image sensor 22A, and the distance to the main subject. 【0108】 Furthermore, depending on the registration results, if registration is repeated due to insufficient accuracy or other reasons, the synthesis method can be changed. For example, if the registration accuracy is insufficient even after multiple attempts, image extrapolation can be used. 【0109】 Thus, various selection criteria can be considered depending on the function, situation, and user preferences, and the image processing unit 30a should select the synthesis method according to the predetermined conditions. 【0110】 If edge detection is selected, the image processing unit 30a proceeds from step S123 to step S130 and performs edge detection in the calculation areas 70 and 71. Figure 14 shows examples of edges 72 and 73 detected in the calculation areas 70 and 71 in image A 90 and image B 91. 【0111】 In step S131, the image processing unit 30a performs registration based on the detected edges 72 and 73 to generate a composite image 98. That is, it aligns image A 90 and image B 91 based on the edges 72 and 73 and performs synthesis by concatenating pixels of image B 91 corresponding to the area around image A 90 with image A 90. 【0112】 If area detection is selected, the image processing unit 30a proceeds from step S123 to step S140, and first performs edge detection within the calculation areas 70 and 71. For example, edges 72 and 73 are detected as shown in Figure 14. 【0113】 In step S141, the image processing unit 30a performs depth detection within the calculation areas 70 and 71. That is, it determines the distance information to the subject that appears in each pixel within the calculation areas 70 and 71. In the case of the imaging device 1 as a smartphone shown in Figure 1, the distance information to the subject can be detected because the distance measuring sensor unit 35, which includes a ToF sensor 9, is provided. Therefore, it can be detected even in the configurations shown in Figures 3, 7, and 9. In the configuration shown in Figure 8, the B image 91 itself functions as a depth map, so distance information can be detected. In all cases, if the image sensors 22A and 22B of the imaging units 31A and 31B are configured to perform image plane phase difference detection, focal length information can be obtained, and distance information can be obtained. 【0114】 In step S142, the image processing unit 30a divides the area based on the edge detection results and depth detection results within the calculation areas 70 and 71. In other words, it detects an object within a common depth range that includes edges 72 and 73, respectively. For example, Figure 15 shows the state in which objects 74 and 75 have been detected. 【0115】 In step S143, the image processing unit 30a performs registration area by area based on edge and depth information to generate a composite image 98. That is, it aligns image A 90 and image B 91 based on the areas of objects 74 and 75, and performs a composite image by concatenating pixels of image B 91 corresponding to the area surrounding image A 90 with image A 90. 【0116】 Furthermore, registration based on area detection can achieve higher synthesis accuracy than edge detection because it allows for alignment across the entire object area. Therefore, if distance information is available, it is preferable to choose the area detection synthesis method over edge detection. On the other hand, if you want to simplify the synthesis process considering the computational load and processing time, the edge detection synthesis method is more suitable. 【0117】If feature detection is selected, the image processing unit 30a proceeds from step S123 to step S150 and performs CNN-based feature extraction on image A 90 and image B 91. In this case, the image processing unit 30a also performs preprocessing such as image resizing and normalization. Feature extraction itself is performed on the entire image, and the calculation area is limited when performing feature matching. 【0118】 Feature point extraction can also be performed using algorithms such as SIFT (Scale-invariant feature transform), SURF (Speeded Up Robust Features), and ORB (Oriented FAST and Rotated BRIEF). Texture features using CNNs, or even higher-level semantic features, may be extracted. 【0119】 In step S151, the image processing unit 30a compares the feature vectors between the two images and creates matching pairs. In this case, outlier exclusion is performed using RANSAC (Random Sample Consensus) or similar methods. 【0120】 In step S152, the image processing unit 30a performs registration based on matching pairs and generates a composite image 98. That is, it aligns image A 90 and image B 91 by matching the matching pairs and performs a composite image by concatenating pixels of image B 91 corresponding to the area around image A 90 with image A 90. 【0121】This feature-based synthesis method is considered the most desirable because it can achieve higher registration accuracy than the area detection synthesis method. However, it requires an equipment environment capable of performing CNN calculations, or an equipment environment that can utilize CNN processing via cloud services, etc. It also has a high power load. Therefore, implementation may be difficult when considering smartphones and other mobile devices. However, depending on the size of the learning model and the network environment, it may be possible to implement it even on the processor installed in a mobile device. Furthermore, with future improvements in processor processing power, implementation on mobile devices will be quite possible. Therefore, for imaging device 1, it is best to select the feature-based synthesis method if possible, depending on the processing power, communication environment, and battery status. 【0122】 If extrapolation is selected, the image processing unit 30a proceeds from step S123 to step S160 to perform preprocessing on image A 90. This process involves setting a mask 76 in the extrapolation area as shown in Figure 16, and resizing the image so that the final output image, including the extrapolation area, fits within the range of the frame 77. 【0123】 In step S161, the image processing unit 30a inputs the pre-processed A image 90 to the CNN model. For example, a pre-trained model using a context encoder or partial convolution is assumed. The above pre-processing is performed in order to understand the context around the mask 76 and generate the region of the mask 76. 【0124】 In step S162, the image processing unit 30a acquires the extrapolated image. That is, the image in which the pixel values ​​generated by the CNN processing are extrapolated into the area of ​​the mask 76. Post-processing such as blending may be performed as needed. Alternatively, the A image 90 and the setting values ​​of the distortion correction adjustment unit 57 may be input to the CNN model, extrapolated, and the image after the specified distortion processing may be output as is. 【0125】This image extrapolation can be used, for example, as an alternative method when a composite image 98 with sufficient accuracy cannot be generated by the edge detection, area detection, and feature detection processes described above. 【0126】 After performing the synthesis process using one of the above methods, the image processing unit 30a calculates an evaluation value of the registration result in step S124. For example, the evaluation is performed using an evaluation function such as MSE (Mean Squared Error) or SSIM (Structural SIMilarity). If the subject change is not significant and the same synthesis method is selected for the next step, it is advisable to feed back the evaluation value and make appropriate parameter adjustments to the subsequent registration. 【0127】 The process shown in Figure 12 is performed in step S120 of Figure 11. In step S170 of Figure 11, the image processing unit 30a determines the similarity evaluation value of the composite image 98. In this case, the evaluation value of the cross-correlation function or an evaluation value such as the structural similarity index between image A 90 and the composite image 98 is compared with a predetermined threshold as the similarity evaluation of the overlapping parts of the images. Alternatively, the evaluation value of the registration result in step S124 may be compared with a predetermined threshold. 【0128】 If the evaluation value is above the threshold, the image processing unit 30a proceeds to step S171 and performs trimming of the composite image 98 (see Figure 4). Then, in step S172, the image processing unit 30a starts preview display control of the composite image 98 on the display unit 32 using the functions of the preview display control unit 55. In step S173, the image processing unit 30a allows user operation of variable distortion correction. 【0129】 Then the process proceeds to step S180 in Figure 10. Consequently, as described above, distortion correction and shooting data storage are performed in steps S181 and beyond, in accordance with user operations. 【0130】In step S170 of Figure 11, if the evaluation value is less than a threshold, and in particular, if the number of retries for the synthesis process when the evaluation value falls below a predetermined level is less than a predetermined number, the image processing unit 30a proceeds to step S174 and controls the display unit 32 to display the preview shown in Figure 5. That is, since the synthesis process is not yet complete, the outer peripheral region 95 surrounding image A 90 is displayed in a blurred state, and a message 94, such as "Registration in progress," is displayed. 【0131】 Then the process returns to steps S111 and S115. Consequently, the synthesis process of image A 90 and image B 91 will be retried. 【0132】 If the evaluation value remains below the threshold for a predetermined number of consecutive steps in step S170, and the number of synthesis processes exceeds a predetermined number, the image processing unit 30a proceeds to step S176. In this case, the image processing unit 30a changes the synthesis method. As described above, the synthesis methods for registration include edge detection, area detection, and feature detection. For example, the image processing unit 30a switches the synthesis method from the previous one to another. Alternatively, the image processing unit 30a may switch from edge detection, area detection, or feature detection to image extrapolation. 【0133】 In step S177, the image processing unit 30a controls the display unit 32 to display the preview shown in Figure 5. That is, since the synthesis process is not yet complete in this case, the outer peripheral region 95 surrounding image A 90 is displayed in a blurred state, and a message 94 such as "Registration in progress" is displayed. At the same time, a message 94 may also be displayed indicating that the synthesis method has been switched. For example, when switching to image extrapolation, a message such as "Switching to generation AI" may be displayed. 【0134】 The image processing unit 30a then returns to steps S111 and S115. Consequently, the compositing process of image A 90 and image B 91 is retried by switching the processing method. 【0135】As described above, regarding the synthesis process, if a synthesized image 98 with sufficient accuracy cannot be obtained, a retry is performed, and the synthesis method may also be switched. This ensures that a synthesized image 98 with the highest possible accuracy is generated. 【0136】 If sufficient synthesis accuracy cannot be obtained after a certain number of retries, the area in which distortion correction can be performed may be narrowed to improve the synthesis accuracy. For example, the range of image B 91 may be narrowed. 【0137】 Next, Figures 17 and 18 will explain an example of processing when image A 90 and image B 91 are videos. Figure 17 shows an alternative processing example to Figure 12, with steps S200 and S201 added to Figure 12. 【0138】 In step S200, the image processing unit 30a performs optical flow estimation on the current frame (t) and the previous frame (t-1) of image B 91. Then, in the calculation area setting in step S122, the results of the optical flow estimation are also used. 【0139】 For example, in image B 91 shown in Figure 18, the overlapping region 78 is used as the reference. The overlapping region 78 is the region corresponding to the periphery of image A 90, and may be the same region as the calculation area 71 in Figure 13. The arrows in the image in Figure 18 indicate the movement of the subject between frames. Optical flow is determined targeting the overlapping region 78 and the extended region 79 which is an extension of the overlapping region 78. Then, within the range of this overlapping region 78 and extended region 79, the optical flow, that is, the movement of the subject indicated by the arrows, is determined. In step S122, the areas where the optical flow is greater than a predetermined value, for example, the areas enclosed by ellipses in Figure 18, are designated as calculation areas 70 and 71. In other words, the calculation areas are determined by how much the optical flow has changed since the last calculation. After setting the calculation areas 70 and 71 in this way, a synthesis method is selected and the synthesis process is performed. 【0140】In step S201, the current B image 91 is temporarily saved as the image of the previous frame (t-1) for optical flow estimation in the next frame. The values ​​to be temporarily saved for optical flow estimation are those that allow us to determine the change in optical flow since the last recalculation. For example, if the last calculation was n frames ago, the image of frame (t-n) is needed. Therefore, for example, the current B image 91 is saved as the (t-1) frame, and images up to (t-n) frames are also temporarily stored, for a total of n frames. Alternatively, the image of the (t-1) frame and the change in each region since the last recalculation may be saved. 【0141】 By utilizing optical flow estimation in this way, it becomes possible to perform registration that responds to the movement of the subject and emphasizes areas that are likely to attract attention within the image. 【0142】 In the case of video capture, the following considerations are possible. It is advisable to prohibit changing the distortion correction amount using the slider bar 96 while video is being recorded. Alternatively, the range of correction may be limited. Furthermore, while video is being recorded, the distortion-corrected image (video) from the distortion correction unit 54 may be recorded simultaneously with the A image 90 (video) or the B image 91 from the B imaging system. This allows for later editing in case of failure in the synthesis / distortion correction process. 【0143】 Regarding the image synthesis process, the recalculation frequency may be reduced for subjects with little change and increased for subjects with a change greater than a predetermined value. The predetermined value may also be changed depending on the complexity of the subject's shape. 【0144】 <7. Summary and Modifications> According to the above embodiments, the following effects can be obtained. 【0145】The imaging device 1 of this embodiment includes an A imaging system (first imaging unit) equipped with an optical system 10A and an image sensor 22A, a B imaging system and a distance measuring sensor unit 35 (sensor unit) that include the imaging range of the A imaging system and are capable of sensing a wider range than the imaging range of the A imaging system, and an image processing unit 30a. The image processing unit 30a generates a composite image 98 by combining the A image 90 (first image) obtained by the A imaging system with the B image 91, which contains images outside the imaging range of the A image 90, and performs distortion correction on the composite image 98. As a result, the image after distortion correction is less likely to have black areas where there is no image around it, and a distortion-corrected image suitable for the user can be provided. For example, in situations where the intensity of distortion needs to be changed according to the subject and expression when shooting with a wide angle, normally, the A image 90 does not contain information outside the image circle of the A imaging system's shooting lens, so depending on the distortion intensity, black areas like those in a circular fisheye lens may occur. In contrast, in the example of this embodiment, information outside the wide-angle field remains as image B 91, making it possible to provide a diagonal fisheye image without blackout even with strong distortion intensity. 【0146】 In this embodiment, a configuration is shown that includes an optical system 10B and an image sensor 22B, which comprises a B imaging system (second imaging unit). Compared to the A image 90 taken by the imaging unit 31A (first imaging unit) of the A imaging system, a wider-angle B image 91 is obtained using the imaging unit 31B (second imaging unit) of the B imaging system, which is capable of wider-angle imaging. In this case, the B image 91 can be an image taken under conditions similar to the A image 90, making it easier to obtain a composite image 98 with a natural outer edge. 【0147】In this embodiment, an example is given in which the lens in the optical system 10A of imaging system A (imaging unit 31A) and the lens in the optical system 10B of imaging system B (imaging unit 31B) are fixedly arranged on the housing at a distance from each other. The optical system 10A of imaging unit 31A constitutes, for example, the wide lens 6W in the smartphone shown in Figure 1, and the optical system 10B of imaging unit 31B constitutes, for example, the ultra-wide lens 6U. In this configuration in which the optical systems 10A and 10B are fixedly arranged on the housing at a distance from each other, the imaging units 31A and 31B have parallax, but this parallax is known. Therefore, the amount of shift in the subject range due to the parallax of image A 90 and image B 91 is known. Consequently, registration can be performed with higher precision, and it is easier to generate a natural composite image 98. 【0148】 In this embodiment, an example is given in which a sensor device capable of recognizing shape is used as the distance measuring sensor unit 35. For example, Figure 8 shows an example in which image A 90 is obtained by the imaging unit 31A of the A imaging system, and image B 91 is obtained by a sensor device capable of sensing at a wider angle, such as a ToF sensor 9. It is also possible to acquire and combine images of the surrounding area of ​​image A 90 using such a sensor device. This makes it possible to perform the processing of the embodiment even with an imaging device that does not have two imaging systems. Examples of sensor devices include ToF sensors, LiDAR, and spectral sensors. 【0149】 In this embodiment, an example was given in which the image processing unit 30a generates a composite image 98 after performing a process to add color information to the B image 91 obtained by the distance measuring sensor unit 35. In the case of a sensor device such as a ToF sensor or LiDAR, if the output B image 91 does not contain color information, color is added to the B image 91. As a result, in the composite image 98, the surrounding area of ​​the A image 90 (the area of ​​the B image 91) can also be a color image, just like the A image 90. 【0150】In this embodiment, one or both of the first imaging unit (imaging unit 31A) and the sensor unit (imaging unit 31B or distance measuring sensor unit 35) are capable of detecting distance information to an object that has become a subject within the imaging range. Knowing the distance information to the object that has become a subject in image A 90 and image B 91 makes it suitable for image synthesis. For example, it becomes possible to synthesize images with high accuracy based on area-based registration. 【0151】 In this embodiment, there is a memory unit 37 that stores parallax information between the first imaging unit (imaging unit 31A) and the sensor unit (imaging unit 31B or distance measuring sensor unit 35), and the image processing unit 30a performs image synthesis based on the parallax information stored in the memory unit 37 and the in-image information of the overlapping region of image A 90 and image B 91. As shown in the processing of step S104 in Figure 10 and step S121 in Figure 12, by loading the parallax information appearing in image A 90 and image B 91 and adjusting image B 91, the overlapping region can be appropriately determined. This makes it possible to perform registration using in-image information within the overlapping region. 【0152】 In this embodiment, an example was given in which the image processing unit 30a performs registration and image synthesis using at least one of the following as in-image information of the overlapping region of image A 90 and image B 91: edge information of the subject, distance-based area information determined from distance information to the subject object within the imaging range, or arbitrary feature quantity information. As shown in Figure 12, registration can be performed by using, for example, edges, area information using distance (depth) information, or feature quantities as in-image information within the overlapping region. 【0153】 In this embodiment, an example was given in which the image processing unit 30a takes a pre-trained generation model and image A 90 as input, generates an extrapolated image of image A 90 from the generation model, and generates a composite image 98 using the extrapolated image. This is the process shown in steps S160 to S162 in Figure 12 and in Figure 16. Image extrapolation can be used, for example, as a method when appropriate registration cannot be performed based on the information within the image. 【0154】In this embodiment, the image processing unit 30a evaluates the synthesis accuracy of the composite image 98 generated by the first image synthesis process on image A 90 and image B 91. If it determines that the accuracy is insufficient, it switches to a second image synthesis process to generate the composite image 98. For example, in step S170 of Figure 11, the composite image 98 is evaluated, and if it is determined to be insufficient, the image synthesis process is changed in step S176. For example, it may be switched from edge-based or area-based synthesis to feature-based synthesis as shown in Figure 12, or to an image extrapolation method. By making it possible to switch between multiple image synthesis processes in this way, it is possible to contribute to the generation of a highly accurate composite image 98. 【0155】 In this embodiment, the image processing unit 30a sets the outer edge region of image A 90 and the region in image B 91 where the same subject as the outer edge region of image A 90 appears as calculation areas 70 and 71, respectively, and performs image synthesis based on the image information within the calculation areas 71 and 71. For example, as shown in Figures 13 to 15, instead of using the entire image A 90 as the calculation area, the outer edge region of image A 90 and the corresponding region in image B 91 are used as calculation areas 70 and 71 (step S122 in Figure 12). By using the image information within these calculation areas, the amount of calculation required for registration can be reduced, thereby reducing the processing load. Furthermore, by using the outer edge region of image A 90 as calculation area 70, the image information in the region that is the coupling critical with the wider-angle image B 91 is used, which is suitable in terms of registration accuracy. 【0156】 In this embodiment, when the image processing unit 30a generates a composite image 98 of video images A 90 and B 91, it sets calculation areas 70 and 71 based on the amount of change between frames obtained using optical flow, and performs image synthesis based on the image information within the calculation areas. For example, as explained in Figures 17 and 18, it is possible to focus on an object that moves between frames and set a calculation area that matches that movement. Therefore, registration suitable for video becomes possible. 【0157】In this embodiment, an example is given in which the image processing unit 30a performs distortion correction on image A 90 and image B 91 respectively, and generates a composite image 98 using the distortion-corrected images A 90 and B 91. The image processing unit 30a performs the processing shown in Figures 3, 7, 8, and 9 as the A distortion correction unit 51A and B distortion correction unit 51B (or ToF distortion correction unit 60). As a result, the distortion-corrected images A 90 and B 91 are combined, as shown in Figure 4. Correcting the distortion makes it easier to align the two images, making them suitable for the combination process. 【0158】 In this embodiment, an example is given in which the image processing unit 30a generates a first composite image 98 for use in image recording using image A 90 and image B 91, performs distortion correction on the first composite image 98, and generates a second composite image for use in display using image A 90 and image B 91, and performs distortion correction on the second composite image. As shown in Figure 7, the image processing unit 30a can also generate a composite image for preview display using an image synthesis unit 52P, separate from the composite image data for storage, and perform distortion correction using a distortion correction unit 54P. This allows for the display of a simplified correction result in the preview display, meeting the requirement for immediacy. 【0159】 In this embodiment, an example is given in which the image processing unit 30a performs distortion correction on the composite image 98 by a correction amount specified by the user. In the image processing unit 30a, the distortion correction amount adjustment unit 57 shown in Figures 3, 7, 8, and 9 sets the correction amount according to the user's operation and instructs the distortion correction unit 54. As a result, the distortion correction desired by the user is performed on the composite image 98. This is particularly suitable for distortion correction that results in an image effect without blackout areas. 【0160】In this embodiment, an example was given in which the image processing unit 30a performs a process to display the composite image 98 and the operator as a preview image, and sets a correction amount according to the operation of the operator to perform distortion correction on the composite image 98. As shown in Figures 3, 7, 8, and 9, the preview display control unit 55 of the image processing unit 30a displays the composite image 98, which is composed of image A 90 and image B 91, on the display unit 32, as shown in Figure 6, and also displays a slider bar 96 and a pointer 96P as operators. This provides the user with an operating environment in which they can specify the amount of distortion correction for the composite image 98. 【0161】 In this embodiment, the composite image 98 displayed in the preview is an image that has undergone distortion correction processing based on the operation of the control element. This allows the user to operate the pointer 96P, and the image will be previewed with distortion correction applied according to the amount of operation. The user can specify the desired distortion correction amount by operating the pointer 96P while observing the degree of distortion in the composite image 98. This makes it convenient for the user to adjust the desired distortion correction amount. 【0162】 In this embodiment, an example is described in which the image processing unit 30a allows the amount of distortion correction to be adjusted by an operator, within a range where no image areas are generated in the distortion-corrected composite image 98. For example, even if the pointer 96P is operated to the upper or lower end of the slider bar 96, the specified amount of distortion correction is such that no image areas (blacked-out areas) are generated in the output image. This allows the user to arbitrarily adjust the amount of distortion correction within a range where blacked-out areas are not generated. Note that the no-image-area refers to what is commonly known as a blacked-out area, but it is not limited to "blacked-out areas" and refers to any area where the subject image is not displayed. 【0163】 Many variations are possible for the configuration that enables the synthesis process and variable distortion correction process described in the embodiment. 【0164】A variety of devices are envisioned to be imaging devices 1 to which this technology can be applied, including smartphones, compact digital cameras, interchangeable lens cameras, tablet terminals, personal computer terminals, video cameras, professional cameras, surveillance cameras, fixed-mount cameras, wearable cameras, and in-vehicle cameras. These devices will have a first imaging unit that obtains image A 90 and a sensor unit (second imaging unit) that obtains image B 91. 【0165】 The imaging unit that obtains image A 90 is referred to as the "first imaging unit" because it is assumed to be a camera that captures still images or videos subject to distortion correction. However, the unit that obtains image B 91 may also be a camera that captures still images or videos, or it may be a ToF sensor 9 or LiDAR, etc., so it is collectively referred to as the "sensor unit." This means that the second imaging unit may be used as one form of the sensor unit. 【0166】 The following are examples of combinations of the first imaging unit and the sensor unit (second imaging unit): • Imaging unit and wide-angle imaging unit • Wide-angle imaging unit and ultra-wide-angle imaging unit • Imaging unit and ToF sensor • Imaging unit and LiDAR • Imaging unit and spectral sensor 【0167】 Each of the imaging units—the wide-angle imaging unit and the ultra-wide-angle imaging unit—may or may not be equipped with focus distance detection and subject distance detection functions using PD (Phase Detect) or SPAD (Single Photon Avalanche Diode) sensors. For example, registration is possible using edge detection methods even with a combination of the imaging unit and the wide-angle imaging unit. Furthermore, if subject distance detection is possible with the wider-angle imaging unit, it is possible to detect the subject distance of image B 91, which includes the range of image A 90. If subject distance detection is possible with both imaging units, more accurate registration becomes possible. 【0168】An example using an imaging unit and a ToF sensor is illustrated in Figure 8. Since the distance image, which becomes image B 91, does not have color information, color information is added during synthesis. Because it is difficult to add color to areas that do not exist in image A 90, in such cases, processing by generation AI may be applied, or the distortion correction area may be narrowed so that the area is not included (making it impossible to set on the UI). 【0169】 LiDAR is a type of ToF sensor in a broad sense, but it is a more accurate, long-range distance sensor and is suitable for applications such as cameras mounted on vehicles. Spectral sensors analyze the wavelength of objects and are useful for object labeling. 【0170】 Alternatively, the system may have a generation model that takes image A 90, image B 91, and a distortion correction amount as inputs and generates a distortion-corrected image of image A 90 based on the distortion correction amount. 【0171】 The program of the embodiment is a program that causes the processing shown in Figures 10, 11, 12, and 17 above to be executed by the processing unit of the control unit 30 of the imaging device 1, such as a CPU, DSP, or other processor, or a device including these. Specifically, the program of the embodiment is a program that causes the processor in an imaging device 1, which includes a first imaging unit equipped with an optical system and an image sensor, and a sensor unit that includes the imaging range of the first imaging unit and is capable of sensing a wider range than the imaging range, to execute the following processes: generating a composite image 98 by combining the first image obtained by the first imaging unit with an image outside the imaging range of the first image using a second image obtained by the sensor unit, and performing distortion correction processing on the composite image 98. 【0172】 With such a program, the imaging device 1 that performs the processing of the embodiment can be realized using various information processing devices. 【0173】The programs of the embodiments described above can be pre-recorded on recording media such as HDDs built into computer devices, or on ROMs in microcomputers with CPUs. Furthermore, such programs can be temporarily or permanently stored (recorded) on removable recording media such as flexible disks, CD-ROMs (Compact Disc Read Only Memory), MO (Magneto Optical) disks, DVDs (Digital Versatile Discs), Blu-ray Discs (registered trademark), magnetic disks, semiconductor memory, and memory cards. Such removable recording media can be provided as so-called packaged software. In addition to being installed from the removable recording media to a personal computer, such programs can also be downloaded from download sites via networks such as LANs (Local Area Networks) and the Internet. 【0174】 Furthermore, such a program is suitable for providing the imaging device 1 of the embodiment to a wide range of users. For example, by downloading the program to devices with imaging capabilities, such as personal computers, communication devices, mobile terminal devices such as smartphones and tablets, mobile phones, game consoles, video equipment, and PDAs (Personal Digital Assistants), these devices can be made to function as the imaging device 1 of this disclosure. 【0175】 Furthermore, the effects described herein are merely illustrative and not limited to those described herein, and other effects may also occur. 【0176】The technology can also be configured as follows: (1) An imaging device comprising: a first imaging unit equipped with an optical system and an image sensor; a sensor unit that includes the imaging range of the first imaging unit and is capable of sensing a wider range than the imaging range; and an image processing unit that generates a composite image by combining a first image obtained by the first imaging unit with an image outside the imaging range of the first image obtained by the sensor unit, and performs distortion correction on the composite image. (2) The imaging device according to (1) above, wherein the sensor unit is a second imaging unit equipped with an optical system and an image sensor. (3) The imaging device according to (2) above, wherein the lens in the optical system of the first imaging unit and the lens in the optical system of the second imaging unit are fixedly arranged on the housing at a distance from each other. (4) The imaging device according to (1) above, wherein the sensor unit is a sensor device capable of recognizing shape. (5) The imaging device according to (4) above, wherein the image processing unit generates the composite image after performing a process to add color information to the second image obtained by the sensor unit. (6) An imaging device according to any one of (1) to (5) above, wherein one or both of the first imaging unit and the sensor unit are capable of detecting distance information to an object that has become a subject within the imaging range. (7) An imaging device according to any one of (1) to (6) above, having a memory unit for storing disparity information between the first imaging unit and the sensor unit, wherein the image processing unit performs image synthesis based on the disparity information stored in the memory unit and the in-image information of the overlapping region of the first image and the second image. (8) An imaging device according to any one of (1) to (7) above, wherein the image processing unit performs registration using at least one of the following as the in-image information of the overlapping region of the first image and the second image: edge information of the subject, area information based on distance that can be determined from the distance information to an object that has become a subject within the imaging range, or arbitrary feature information, and performs image synthesis. (9) An imaging device according to any one of (1) to (8) above, wherein the image processing unit takes a pre-trained generative model and the first image as input, generates an extrapolated image of the first image from the generative model, and generates the composite image using the extrapolated image.(10) The imaging apparatus according to any one of (1) to (9) above, wherein the image processing unit evaluates the synthesis accuracy of the composite image generated by the first image synthesis process on the first image and the second image, and if it determines that the accuracy is insufficient, switches to the second image synthesis process to generate the composite image. (11) The imaging apparatus according to any one of (1) to (10) above, wherein the image processing unit sets the outer edge region of the first image and the region in the second image where the same subject as the outer edge region appears as calculation areas, and performs image synthesis based on the image information within the calculation areas. (12) The imaging apparatus according to any one of (1) to (11) above, wherein when generating a composite image of the first image and the second image which are moving images, the image processing unit sets the calculation area based on the amount of change between frames obtained using optical flow, and performs image synthesis based on the image information within the calculation areas. (13) The imaging apparatus according to any one of (1) to (12) above, wherein the image processing unit performs distortion correction on the first image and the second image, and generates a composite image using the first image and the second image after distortion correction. (14) The imaging apparatus according to any one of (1) to (13) above, wherein the image processing unit performs the following processes: generating a first composite image to be used for image recording using the first image and the second image, and performing distortion correction on the first composite image; and generating a second composite image to be used for display using the first image and the second image, and performing distortion correction on the second composite image. (15) The imaging apparatus according to any one of (1) to (14) above, wherein the image processing unit performs distortion correction on the composite image by a correction amount specified by user operation. (16) The imaging apparatus according to any one of (1) to (15) above, wherein the image processing unit performs the process of displaying the composite image and the operator as a preview image, and sets a correction amount according to the operation of the operator to perform distortion correction on the composite image. (17) The imaging apparatus according to (16) above, wherein the composite image displayed as the preview image is an image on which distortion correction processing has been performed based on the operation of the operator.(18) The imaging apparatus according to (17) above, wherein the image processing unit is configured such that the amount of distortion correction can be adjusted by the operator within a range in which no image area occurs in the distortion-corrected composite image. (19) An image processing method comprising: a first imaging unit equipped with an optical system and an image sensor; and a sensor unit that includes the imaging range of the first imaging unit and is capable of sensing a range wider than the imaging range, wherein the imaging apparatus generates a composite image by combining a first image obtained by the first imaging unit with an image outside the imaging range of the first image using a second image obtained by the sensor unit, and performs distortion correction on the composite image. (20) A program that causes a processor in an imaging device comprising: a first imaging unit equipped with an optical system and an image sensor; a sensor unit that includes the imaging range of the first imaging unit and is capable of sensing a wider range than the imaging range; to perform the following processes: generate a composite image by combining a first image obtained by the first imaging unit with an image outside the imaging range of the first image using a second image obtained by the sensor unit; and perform distortion correction processing on the composite image. 【0177】 1 Imaging device 6U Ultra-wide lens 6W Wide lens 9 TOF sensors 22A, 22B Image sensors 30 Control unit 30a Image processing unit 31A, 31B Imaging unit 32 Display unit 35 Distance measuring unit 36 ​​Storage unit 37 Memory unit 51A A Distortion correction unit 51B B Distortion correction unit 52, 52P Image synthesis unit 53 B Image processing unit 54 Distortion correction unit 55 Preview display control unit 56 Shooting data storage control unit 57 Distortion correction amount adjustment unit 58 Shooting setting unit 59 B Setting calculation unit 60 ToF distortion correction unit 61 Color information assignment unit 90 A image 91 B image 98 Composite image

Claims

1. An imaging device comprising: a first imaging unit equipped with an optical system and an image sensor; a sensor unit that includes the imaging range of the first imaging unit and is capable of sensing a wider range than the imaging range; and an image processing unit that generates a composite image by combining a first image obtained by the first imaging unit with an image outside the imaging range of the first image using a second image obtained by the sensor unit, and performs distortion correction on the composite image.

2. The imaging apparatus according to claim 1, wherein the sensor unit is a second imaging unit equipped with an optical system and an image sensor.

3. The imaging apparatus according to claim 2, wherein the lens in the optical system of the first imaging unit and the lens in the optical system of the second imaging unit are fixedly arranged on the housing at a distance from each other.

4. The imaging device according to claim 1, wherein the sensor unit is a sensor device capable of recognizing shape.

5. The imaging apparatus according to claim 4, wherein the image processing unit performs a process to add color information to the second image obtained by the sensor unit, and then generates the composite image.

6. The imaging device according to claim 1, wherein one or both of the first imaging unit and the sensor unit are capable of detecting distance information to an object that has become a subject within the imaging range.

7. The imaging apparatus according to claim 1, further comprising a memory unit for storing disparity information between the first imaging unit and the sensor unit, wherein the image processing unit performs image synthesis based on the disparity information stored in the memory unit and image information of the overlapping region of the first image and the second image.

8. The imaging apparatus according to claim 1, wherein the image processing unit performs registration using at least one of the following as in-image information of the overlapping region of the first image and the second image: edge information of the subject, area information based on distance determined from distance information to the object that has become the subject within the imaging range, or arbitrary feature information, and performs image synthesis.

9. The imaging apparatus according to claim 1, wherein the image processing unit takes a pre-trained generation model and the first image as input, generates an extrapolated image of the first image from the generation model, and generates the composite image using the extrapolated image.

10. The imaging apparatus according to claim 1, wherein the image processing unit evaluates the synthesis accuracy of the composite image generated by the first image synthesis process using the first image and the second image, and if it determines that the accuracy is insufficient, switches to the second image synthesis process to generate the composite image.

11. The imaging apparatus according to claim 1, wherein the image processing unit sets the outer edge region of the first image and the region in the second image in which the same subject as the outer edge region appears as calculation areas, and performs image synthesis based on the image information within the calculation areas.

12. The imaging apparatus according to claim 1, wherein the image processing unit, when generating a composite image of the first image and the second image which are moving images, sets a calculation area based on the amount of change between frames obtained using optical flow, and performs image synthesis based on the image information within the calculation area.

13. The imaging apparatus according to claim 1, wherein the image processing unit performs distortion correction on the first image and the second image, and performs a process of generating a composite image using the first image and the second image after distortion correction.

14. The imaging apparatus according to claim 1, wherein the image processing unit performs the following processes: generating a first composite image for use in image recording using the first image and the second image, and performing distortion correction on the first composite image; and generating a second composite image for use in display using the first image and the second image, and performing distortion correction on the second composite image.

15. The imaging apparatus according to claim 1, wherein the image processing unit performs distortion correction on the composite image by a correction amount specified by user operation.

16. The imaging apparatus according to claim 1, wherein the image processing unit performs a process to display the composite image and the operator as preview images, and sets a correction amount according to the operation of the operator to correct the distortion of the composite image.

17. The imaging apparatus according to claim 16, wherein the composite image displayed as the preview image is an image on which distortion correction processing has been performed based on the operation of the control element.

18. The imaging apparatus according to claim 17, wherein the image processing unit is configured such that the amount of distortion correction can be adjusted by the operator within a range in which no image area occurs in the distortion-corrected composite image.

19. An imaging device comprising: a first imaging unit equipped with an optical system and an image sensor; and a sensor unit that includes the imaging range of the first imaging unit and is capable of sensing a wider range than the imaging range; an imaging device comprising: generating a composite image by combining a first image obtained by the first imaging unit with an image outside the imaging range of the first image using a second image obtained by the sensor unit; and performing distortion correction on the composite image.

20. A program that causes a processor in an imaging device comprising: a first imaging unit equipped with an optical system and an image sensor; a sensor unit that includes the imaging range of the first imaging unit and is capable of sensing a wider range than the imaging range; to perform the following processes: generate a composite image by combining a first image obtained by the first imaging unit with an image outside the imaging range of the first image using a second image obtained by the sensor unit; and perform distortion correction processing on the composite image.

Images