Imaging device and imaging system

Abstract

Provided is an imaging device capable of generating a 3D image with reduced cutoff.　This imaging device is provided with: an imaging unit that images, on the basis of light from a subject entering through an optical system, an input image, in which a left-eye input image and a right-eye input image having parallax relative to each other are arranged; and an image processing unit that processes the input image imaged by the imaging unit and generates a left-eye output image and a right-eye output image. The image processing unit has a first mode for extracting, from the input image, a first left-eye extracted image that includes at least a part of the left-eye input image and has a ratio of the short direction to the long direction greater than the display aspect ratio of a display device, and a first right-eye extracted image that includes at least a part of the right-eye input image and has a ratio of the short direction to the long direction greater than the display aspect ratio, and joining combined images to the outer edges of the first left-eye extracted image and the first right-eye extracted image, thereby generating a first left-eye output image and a first right-eye output image with the display aspect ratio.

Description

Imaging Apparatus and Imaging System

[0001] The present invention relates to an imaging apparatus and an imaging system.

[0002] Conventionally, there has been known an imaging apparatus including: an imaging unit that captures a left-eye image and a right-eye image having parallax therebetween; and an image processing unit that processes the left-eye image and the right-eye image captured by the imaging unit to generate a 3D image (see, for example, Patent Document 1).

[0003] Japanese Patent No. 6912313

[0004] Here, the image processing unit described in Patent Document 1 cuts out and outputs a part of the left-eye image and the right-eye image in accordance with the aspect ratio of a display device. Therefore, there is a problem that a part of an imaging range is not displayed (that is, cut off) in a 3D image displayed on the display device.

[0005] The present invention has been made in view of the above circumstances, and an object of the present invention is to provide an imaging apparatus capable of generating a 3D image with less cut-off.

[0006] In order to solve the above problem, an imaging apparatus according to the present invention includes: an imaging unit that captures an input image in which a left-eye input image and a right-eye input image having parallax therebetween are arranged based on light from a subject incident through an optical system; and an image processing unit that processes the input image captured by the imaging unit to generate a left-eye output image and a right-eye output image. The image processing unit has a first mode in which: a first left-eye extracted image including at least a part of the left-eye input image and having a larger ratio of a short side direction to a longitudinal direction than a display aspect ratio of a display device, and a first right-eye extracted image including at least a part of the right-eye input image and having a larger ratio of a short side direction to a longitudinal direction than the display aspect ratio are extracted from the input image; and a combined image is combined with an outer edge of each of the first left-eye extracted image and the first right-eye extracted image to generate the first left-eye output image and the first right-eye output image having the display aspect ratio.

[0007] According to the present invention, it is possible to obtain an imaging apparatus capable of generating a 3D image with less cut-off. Problems, configurations, and effects other than those described above will be clarified by the following description of embodiments.

[0008] This is a hardware configuration diagram of the imaging system. This is a functional block diagram of the image processing unit. This is a flowchart showing the image processing procedure in vertical resolution priority mode. This is a diagram showing the image image at each step in vertical resolution priority mode. This is a flowchart showing the image processing procedure in horizontal display size priority mode. This is a diagram showing the image image at each step in horizontal display size priority mode.

[0009] The embodiments of the invention will be described below with reference to the drawings. These embodiments contribute to "Goal 9: Build resilient infrastructure, promote inclusive and sustainable industrialization and foster innovation," one of the Sustainable Development Goals (SDGs) advocated by the United Nations, by realizing a highly versatile imaging device and imaging system.

[0010] [Configuration of Imaging System 1] Figure 1 is a hardware configuration diagram of the imaging system 1. The imaging system 1 is a system that captures images of a subject O and displays a 3D image. The image includes a plurality of images (still images) arranged in chronological order. The imaging system 1 generates and displays a 3D image by processing each of the plurality of images. As shown in Figure 1, the imaging system 1 includes, for example, an optical system 10, an imaging device 20, and a 3D monitor 30.

[0011] The imaging system 1 can be used in a wide range of fields, including the medical and industrial sectors. Typical applications of the imaging system 1 include, for example, rigid endoscope systems and surgical microscope systems. The configurations of rigid endoscope systems and surgical microscope systems are already well known, as described in Patent Document 1, so a detailed explanation will be omitted.

[0012] In this specification, the vertical direction is an example of the short-side direction, and the left-right direction is an example of the long-side direction. Furthermore, the vertical direction corresponds to the vertical direction (short-side direction) of the display surface of the 3D monitor 30, and the left-right direction corresponds to the horizontal direction (long-side direction) of the display surface of the 3D monitor 30. However, the absolute directions of the short-side direction and the long-side direction are not particularly limited as long as they are orthogonal to each other.

[0013] The optical system 10 is a general term for optical components (e.g., light source, lens, prism) used to focus the reflected light from the subject O onto the imaging surface of the image sensor 23. More specifically, the optical system 10 splits the reflected light to form images of the left eye input image L1 and the right eye input image R1, which have parallax with respect to each other, onto the imaging surface of the image sensor 23. The optical system 10 used in a rigid endoscope system includes, for example, a scope 11 and a coupler 12, as shown in Figure 1.

[0014] The scope 11 is the part that is inserted into the patient's body. The scope 11 comprises a light source (not shown), a left eye scope 13L, and a right eye scope 13R. The light source emits light toward the subject O. The left eye scope 13L and the right eye scope 13R are positioned at a distance from the eyes of a typical person. The left eye scope 13L and the right eye scope 13R guide the reflected light from the same subject O to the coupler 12.

[0015] The coupler 12 is an adapter that connects the scope 11 and the camera head 21. The coupler 12 includes, for example, a left eye lens 14L and a right eye lens 14R, as shown in Figure 1. The left eye lens 14L focuses (images) the reflected light that has passed through the left eye scope 13L onto the imaging surface of the image sensor 23 as the left eye input image L1. The right eye lens 14R focuses (images) the reflected light that has passed through the right eye scope 13R onto the imaging surface of the image sensor 23 as the right eye input image R1. On the imaging surface of the image sensor 23, the left eye input image L1 and the right eye input image R1 are imaged at different positions.

[0016] The specific configuration of the optical system 10 is not limited to the example shown in Figure 1. As another example, the optical system 10 may use a single lens (or a combination of a lens and a prism) instead of the left eye lens 14L and the right eye lens 14R to form images of the left eye input image L1 and the right eye input image R1 on the imaging surface. In addition, in the optical system 10 of a surgical microscope system, the optical component corresponding to the scope 11 is the objective lens, and the optical component corresponding to the coupler 12 is the camera adapter. As yet another example, the optical system 10 may employ a 3D optical system that integrates the scope 11 and the coupler 12.

[0017] The imaging device 20 generates input image data from reflected light from a subject O incident via the optical system 10, processes the input image data to generate output image data, and outputs the generated output image data to the 3D monitor 30. The imaging device 20 includes, for example, a camera head 21 and an image processing unit 22.

[0018] The camera head 21 is an imaging unit that converts reflected light from a subject O, which is incident on the optical system 10, into photoelectric data to generate an input image I. The camera head 21 also transmits the input image data representing the generated input image I to the image processing unit 22 via a cable. The camera head 21 is equipped with an image sensor 23. The image sensor 23 is, for example, a CMOS (Complementary Metal-Oxide-Semiconductor) or a CCD (Charge-Coupled Device).

[0019] The surface of the image sensor 23 facing the optical system 10 is the imaging surface. The imaging surface is a horizontally elongated rectangle having multiple pixels arranged in a matrix. The size of the imaging surface (combination of pixel count and aspect ratio) may be the same as that of the 3D monitor 30. In this embodiment, the pixel count (resolution) of the image sensor 23 is set to 4k (3840 pixels in the horizontal direction and 2160 pixels in the vertical direction), but it may also be 2k (1920 pixels × 1080 pixels), 8k (7680 pixels × 4320 pixels), etc.

[0020] For example, the image sensor 23 may be a color-capable sensor having a Bayer array. That is, the image sensor 23 includes an R filter that transmits only R light, a G filter that transmits only G light, and a B filter that transmits only B light. The image sensor 23 has one of the R filter, G filter, or B filter placed in front of each of its multiple pixels. More specifically, among the multiple pixels constituting the image sensor 23, an R filter is placed in front of the first pixel, a G filter is placed in front of the second pixel which is different from the first pixel, and a B filter is placed in front of the third pixel which is different from the first and second pixels. As a result, the image sensor 23 can generate a color image from the reflected light that has passed through the optical system 10.

[0021] As another example, the camera head 21 may include a prism (not shown) that spectrally separates the reflected light by wavelength, and a plurality of image sensors that convert the spectrally separated light of each color into photoelectric signals to generate an input image. The camera head 21 may be a two-chip type corresponding to visible light and infrared light, a three-chip type corresponding to red light, green light and blue light, or a four-chip type corresponding to red light, green light, blue light and infrared light.

[0022] An optical system 10 is detachably attached to the camera head 21. That is, the camera head 21 is configured to allow interchangeable optical systems 10 with different specifications. The specifications (type) of the optical system 10 affect the input image I (more specifically, the size of the left eye input image L1 and the right eye input image R1). For example, the relationship between the specifications of the optical system 10 and the position, shape, and size of the left eye input image L1 and the right eye input image R1 may be pre-stored in the memory 24 of the image processing unit 22. As another example, the position, shape, and size of the left eye input image L1 and the right eye input image R1 may be manually set (adjusted) through the operation unit 27 to match the attached optical system 10.

[0023] The image processing unit 22 processes the input image I captured by the camera head 21 to generate a first left eye output image L4 (or a second left eye output image L6) and a first right eye output image R4 (or a second right eye output image R6). The image processing unit 22 also transmits the output image data, including the first left eye output image L4 (or a second left eye output image L6) and the first right eye output image R4 (or a second right eye output image R6), to the 3D monitor 30 via a cable. The image processing unit 22 includes, for example, a memory 24, a CPU (Central Processing Unit) 25, an ISP (Image Signal Processing) 26, and an operation unit 27.

[0024] The memory 24 includes at least RAM (Random Access Memory) and ROM (Read Only Memory). The memory 24 temporarily stores programs and control data necessary for the operation of the image processing unit 22, as well as data or information generated during the operation of each part of the image processing unit 22. The RAM is, for example, a work memory used when each part of the image processing unit 22 is in operation. The ROM stores and holds programs and control data for controlling each part of the image processing unit 22 in advance.

[0025] The CPU 25 is a processor that controls the overall operation of the image processing unit 22. The CPU 25 performs control processing to coordinate the operation of each part of the image processing unit 22, data input / output processing between the image processing unit 22 and each part, data calculation processing, and data storage processing. The CPU 25 operates according to the program and control data stored in the memory 24. When operating, the CPU 25 uses the memory 24 and transfers data that the CPU 25 generates or acquires to the memory 24 for temporary storage.

[0026] ISP26 is a processor that performs various image processing operations within the image processing unit 22. ISP26 reads image data from memory 24 and uses the read image data to perform various image processing operations (for example, white balance and gradation correction, image cropping, enlargement, and masking, as described later). During operation, ISP26 uses memory 24 to transfer data or information generated or acquired by ISP26 to memory 24 for temporary storage.

[0027] The operation unit 27 receives user input to switch the operation of the image processing unit 22. The operation unit 27 can employ any known configuration, such as a push button, switch, dial, or touch panel. For example, the operation unit 27 receives user input to specify the specifications (type) of the optical system 10 attached to the camera head 21. As another example, the operation unit 27 receives user input to specify the image processing mode (vertical resolution priority mode, horizontal display size priority mode) of the image processing unit 22. As another example, the operation unit 27 accepts user input to specify the vertical length (number of pixels) of the first left eye extracted image L2 and the first right eye extracted image R2, the aspect ratio of the display surface of the 3D monitor 30 (hereinafter referred to as "display aspect ratio"), and the method of transmitting 3D images to the 3D monitor 30 (for example, Simultaneous (Dual Stream) method, Side-by-Side method, Line-by-Line method, frame packing method, Top-and-Bottom method).

[0028] The 3D monitor 30 is a display device that displays a 3D image (3D video) of the subject O based on output image data output from the image processing unit 22. As an example, the 3D monitor 30 may be a frame rate type monitor that alternately displays the first left eye output image L4 (or the second left eye output image L6) and the first right eye output image R4 (or the second right eye output image R6). As another example, the 3D monitor 30 may be a polarizing filter type monitor that displays a composite image in which the first left eye output image L4 (or the second left eye output image L6) and the first right eye output image R4 (or the second right eye output image R6) are combined line by line. As yet another example, the 3D monitor 30 may be an HMD type monitor that displays the first left eye output image L4 (or the second left eye output image L6) on the left lens and the first right eye output image R4 (or the second right eye output image R6) on the right lens.

[0029] The user positions the optical system 10 (scope 11) facing the subject O and views the 3D image of the subject O in a manner consistent with the display method of the 3D monitor 30. For example, in the case of a frame rate type 3D monitor 30, the user views the 3D image by wearing glasses with a shutter that opens and closes the left and right lenses alternately. As another example, in the case of a polarizing filter type 3D monitor 30, the user views the 3D image by wearing glasses with polarizing filters attached to the left and right lenses. As yet another example, in the case of an HMD type 3D monitor 30, the user views the 3D image by wearing the HMD on their head.

[0030] [Functional Blocks of Image Processing Unit 22] Figure 2 is a functional block diagram of the image processing unit 22. As shown in Figure 2, the image processing unit 22 includes a switching unit 221, a first extraction processing unit 222, a first magnification processing unit 223, a mask processing unit 224, a second extraction processing unit 225, a second magnification processing unit 226, and an output processing unit 227. The CPU 25 and ISP 26 execute a program stored in the memory 24 to realize each of the functional blocks (221 to 227) shown in Figure 2. However, the functional blocks (221 to 227) of the image processing unit 22 are not limited to the example in Figure 2.

[0031] The switching unit 221 switches the image processing mode of the image processing unit 22 according to user operations via the operation unit 27. The image processing modes according to this embodiment include a vertical resolution priority mode (first mode) and a horizontal display size priority mode (second mode). In the vertical resolution priority mode, the switching unit 221 operates the first extraction processing unit 222, the first magnification processing unit 223, the mask processing unit 224, and the output processing unit 227. In the horizontal display size priority mode, the switching unit 221 operates the second extraction processing unit 225, the second magnification processing unit 226, and the output processing unit 227.

[0032] The vertical resolution priority mode is an image processing mode that extracts an image from the input image I that is longer vertically than the display aspect ratio to generate the first left eye output image L4 and the first right eye output image R4. In other words, the vertical resolution priority mode is an image processing mode that extracts an image in which the ratio of the vertical direction to the horizontal direction is larger than the display aspect ratio. Compared to the horizontal display size priority mode, the vertical resolution priority mode has the advantage of reducing the vertical cropping of the subject O. On the other hand, compared to the horizontal display size priority mode, the vertical resolution priority mode has the disadvantage that the image is not displayed on part of the display surface of the 3D monitor 30 (both the left and right sides). Details of the processing in the vertical resolution priority mode will be described later with reference to Figures 3 and 4.

[0033] The horizontal display size priority mode is an image processing mode that extracts an image with a display aspect ratio from the input image I to generate the second left eye output image L6 and the second right eye output image R6. Compared to the vertical resolution priority mode, the horizontal display size priority mode has the advantage of being able to display the 3D image of the subject O across the entire display surface of the 3D monitor 30. On the other hand, the horizontal display size priority mode has the disadvantage of having a larger vertical cropping of the subject O compared to the vertical resolution priority mode. Details of the processing in the horizontal display size priority mode will be described later with reference to Figures 5 and 6.

[0034] In vertical resolution priority mode, the first extraction processing unit 222 extracts from the input image I a first left eye extracted image L2 which includes at least a portion (more preferably the entirety) of the left eye input image L1 and has a larger ratio of vertical to horizontal than the display aspect ratio, and a first right eye extracted image R2 which includes at least a portion (more preferably the entirety) of the right eye input image R1 and has a larger ratio of vertical to horizontal than the display aspect ratio. In other words, the aspect ratios of the first left eye extracted image L2 and the first right eye extracted image R2 have a longer vertical length (larger ratio) compared to the display aspect ratio.

[0035] In vertical resolution priority mode, the first magnification processing unit 223 enlarges the first left eye extracted image L2 and the first right eye extracted image R2, respectively, to match the number of pixels in the vertical direction of the 3D monitor 30, thereby generating a left eye magnified image L3 and a right eye magnified image R3.

[0036] In vertical resolution priority mode, the mask processing unit 224 combines mask images ML and MR with the outer edges of the left eye magnified image L3 (or first left eye extracted image L2) and the right eye magnified image R3 (or first right eye extracted image R2), respectively, to generate a first left eye output image L4 and a first right eye output image R4 that correspond to the display size of the 3D monitor 30. More specifically, the mask processing unit 224 combines mask images ML and MR with both sides in the left-right direction of the left eye magnified image L3 and the right eye magnified image R3, respectively, to generate a first left eye output image L4 and a first right eye output image R4.

[0037] In horizontal display size priority mode, the second extraction processing unit 225 extracts a second left eye extracted image L5 that includes a portion of the left eye input image L1 and has the same display aspect ratio, and a second right eye extracted image R5 that includes a portion of the right eye input image R1 and has the same display aspect ratio, from the input image I. More specifically, the second extraction processing unit 225 extracts a second left eye extracted image L5 in which the number of pixels in the vertical direction is smaller than that of the left eye input image L1, and the number of pixels in the horizontal direction is the same as that of the left eye input image L1. The second extraction processing unit 225 also extracts a second right eye extracted image R5 in which the number of pixels in the vertical direction is smaller than that of the right eye input image R1, and the number of pixels in the horizontal direction is the same as that of the right eye input image R1.

[0038] In horizontal display size priority mode, the second magnification processing unit 226 enlarges the second left eye extracted image L5 and the second right eye extracted image R5, respectively, to match the display size of the 3D monitor 30, and generates the second left eye output image L6 and the second right eye output image R6.

[0039] The output processing unit 227 generates output image data including the first left eye output image L4 (or second left eye output image L6) and the first right eye output image R4 (or second right eye output image R6) in accordance with the transmission method to the 3D monitor 30, and outputs the generated output image data to the 3D monitor 30.

[0040] [Vertical Resolution Priority Mode] The image processing procedure in vertical resolution priority mode will be explained with reference to Figures 3 and 4. Figure 3 is a flowchart showing the image processing procedure in vertical resolution priority mode. Figure 4 is a diagram showing the image images at each step of vertical resolution priority mode. When vertical resolution priority mode is selected via the operation unit 27, the image processing unit 22 performs the processing shown in Figures 3 and 4 on the input image data output from the camera head 21.

[0041] First, the switching unit 221 loads the input image I, indicated by the input image data, into the memory 24 (S11). As shown in Figure 4, the size of the input image I is the same as the imaging surface of the image sensor 23. The input image I also includes the left eye input image L1 and the right eye input image R1. Within the input image I, the left eye input image L1 and the right eye input image R1 are arranged adjacent to each other in the left-right direction. Also, the left eye input image L1 and the right eye input image R1 do not overlap.

[0042] The left eye input image L1 and the right eye input image R1 are circular images (typically perfect circles) with a diameter (in other words, the maximum number of pixels in the vertical and horizontal directions) of 1920 pixels. On the other hand, in the input image I, the area outside the left eye input image L1 and the right eye input image R1 (the area of ​​dot hatching, hereinafter referred to as the "peripheral area") is either an area where little light from the optical system 10 reaches, or an image with very large aberrations.

[0043] The shape, position, and size of the left-eye input image L1 and the right-eye input image R1 in the input image I are not limited to the example shown in FIG. 4. The shape, position, and size of the left-eye input image L1 and the right-eye input image R1 in the input image I are uniquely determined according to the specifications of the optical system 10. That is, the image processing unit 22 can specify the shape, position, and size of the left-eye input image L1 and the right-eye input image R1 in the input image I through a user's operation (input) via the operation unit 27.

[0044] Next, the first extraction processing unit 222 extracts a first left-eye extracted image L2 and a first right-eye extracted image R2 from the input image I expanded in the memory 24 (S12). Then, the first extraction processing unit 222 expands the extracted first left-eye extracted image L2 and first right-eye extracted image R2 in the memory 24. The first left-eye extracted image L2 is an image that includes at least a part (the whole in the example of FIG. 4) of the left-eye input image L1, and has a larger ratio of the vertical direction to the horizontal direction than the display aspect ratio. The first right-eye extracted image R2 is an image that includes at least a part (the whole in the example of FIG. 4) of the right-eye input image R1, and has a larger ratio of the vertical direction to the horizontal direction than the display aspect ratio. Further, the first left-eye extracted image L2 and the first right-eye extracted image R2 are images extracted from different positions (positions where portions do not overlap each other) in the input image I.

[0045] As an example, the shape, position, and size (typically, the length in the vertical direction) of the first left-eye extracted image L2 and the first right-eye extracted image R2 may be predetermined. As another example, the image processing unit 22 may specify the shape, position, and size (typically, the length in the vertical direction) of the first left-eye extracted image L2 and the first right-eye extracted image R2 through a user's operation (input) via the operation unit 27.

[0046] The first left-eye extracted image L2 is a rectangular image configured by four sides that do not intersect the left-eye input image L1. More specifically, the first left-eye extracted image L2 is a rectangular image configured by four sides in contact with the top edge, bottom edge, left edge and right edge of the left-eye input image L1. In other words, the first left-eye extracted image L2 is a quadrilateral (square) image circumscribed around the perfect circular left-eye input image L1. That is, the first left-eye extracted image L2 is a square image in which the intersection of diagonal lines coincides with the center of the left-eye input image L1, and the number of pixels in the vertical direction and the horizontal direction is equal to the diameter (1920 pix) of the left-eye input image L1. Furthermore, the first left-eye extracted image L2 is a square image obtained by adding peripheral area images to the four corners of the circular left-eye input image L1.

[0047] The first right-eye extracted image R2 is a rectangular image configured by four sides that do not intersect the right-eye input image R1. More specifically, the first right-eye extracted image R2 is a rectangular image configured by four sides in contact with the top edge, bottom edge, left edge and right edge of the right-eye input image R1. In other words, the first right-eye extracted image R2 is a quadrilateral (square) image circumscribed around the perfect circular right-eye input image R1. That is, the first right-eye extracted image R2 is a square image in which the intersection of diagonal lines coincides with the center of the right-eye input image R1, and the number of pixels in the vertical direction and the horizontal direction is equal to the diameter (1920 pix) of the right-eye input image R1. Furthermore, the first right-eye extracted image R2 is a square image obtained by adding peripheral area images to the four corners of the circular right-eye input image R1.

[0048] Next, the first enlargement processing unit 223 enlarges each of the first left-eye extracted image L2 and the first right-eye extracted image R2 developed in the memory 24 in accordance with the number of pixels in the vertical direction of the display surface of the 3D monitor 30, thereby generating an enlarged left-eye image L3 and an enlarged right-eye image R3 (S13). Then, the first enlargement processing unit 223 develops the enlarged left-eye image L3 and the enlarged right-eye image R3 in the memory 24.

[0049] In this embodiment, the first magnification processing unit 223 magnifies the 1920 × 1920 pixel first left eye extracted image L2 and the first right eye extracted image R2 into a 2160 × 2160 pixel left eye magnified image L3 and right eye magnified image R3 (i.e., a magnification ratio of 1.125). The image magnification is achieved by interpolating the increased pixels using a well-known upconversion technique. On the other hand, the number of pixels in the left-right direction of the left eye magnified image L3 and the right eye magnified image R3 (2160 pixels) is less than the number of pixels in the left-right direction of the display surface of the 3D monitor 30 (3840 pixels).

[0050] Next, the mask processing unit 224 combines the mask images ML and MR on both sides in the left-right direction of the left eye magnified image L3 and the right eye magnified image R3, respectively, to generate the first left eye output image L4 and the first right eye output image R4 (S14). Then, the mask processing unit 224 loads the first left eye output image L4 and the first right eye output image R4 into the memory 24.

[0051] The mask images ML and MR are rectangular images with a resolution of 840 x 2160 pixels. More specifically, the number of pixels in the left-right direction of the mask images ML and MR (840 pixels) is half the difference between the number of pixels in the left-right direction of the display surface of the 3D monitor 30 (3840 pixels) and the number of pixels in the left-right direction of the left-eye magnified image L3 and the right-eye magnified image R3 (2160 pixels). Also, the number of pixels in the up-down direction of the mask images ML and MR (2160 pixels) is the same as the number of pixels in the up-down direction of the left-eye magnified image L3 and the right-eye magnified image R3 (in other words, the display surface of the 3D monitor 30).

[0052] In other words, the mask images ML and MR are congruent (identical in size and shape). Furthermore, the mask images ML and MR are filled with a specific color (for example, black) (i.e., all pixels are assigned the same pixel value). The mask processing unit 224 then combines the mask images ML and MR with the left eye magnified image L3 and the right eye magnified image R3 to generate a first left eye output image L4 and a first right eye output image R4 that correspond to the display size of the 3D monitor 30 (more specifically, are identical to the display size of the 3D monitor 30).

[0053] The mask images ML and MR are examples of combined images that are joined to the outer edges of the first left eye extracted image L2 (or left eye magnified image L3) and the first right eye extracted image R2 (or right eye magnified image R3). However, the combined image is not limited to being symmetrical; it may be asymmetrical, or joined to only one side. Furthermore, the combined image is not limited to an image filled with a specific color; it may include a part of the input image I generated by the image sensor 23, or it may include menu icons, etc.

[0054] Next, the output processing unit 227 generates output image data, including the first left eye output image L4 and the first right eye output image R4, according to the transmission method to the 3D monitor 30, and outputs it to the 3D monitor 30 (S15). For example, in the Side-by-Side method, the output processing unit 227 outputs output image data in which the first left eye output image L4 and the first right eye output image R4 are arranged side by side. As another example, in the Top-and-Bottom method, the output processing unit 227 outputs output image data in which the first left eye output image L4 and the first right eye output image R4 are arranged vertically. For other methods, the output image data is generated using well-known methods.

[0055] As a result, the 3D monitor 30 displays a 3D image that includes at least a portion of the left eye input image L1 and the right eye input image R1, and combines mask images ML and MR on the left and right sides, such as the first left eye output image L4 and the first right eye output image R4. In other words, with the vertical resolution priority mode, most of (more preferably, the entire) of the imaging range of the left eye input image L1 and the right eye input image R1 can be viewed as a 3D image. On the other hand, with the vertical resolution priority mode, substantial images are not displayed on the left and right portions of the display surface of the 3D monitor 30, so the horizontal display size is smaller compared to the horizontal display size priority mode.

[0056] [Horizontal Display Size Priority Mode] The image processing procedure in horizontal display size priority mode will be explained with reference to Figures 5 and 6. Figure 5 is a flowchart showing the image processing procedure in horizontal display size priority mode. Figure 6 is a diagram showing the image images at each step in horizontal display size priority mode. When horizontal display size priority mode is selected via the operation unit 27, the image processing unit 22 performs the processing shown in Figures 5 and 6 on the input image data output from the camera head 21. Note that a detailed explanation of the similarities with vertical resolution priority mode will be omitted, and the explanation will focus on the differences.

[0057] First, the switching unit 221 loads the input image I, indicated by the input image data, into the memory 24 (S21). The details of the input image I are the same as in the vertical resolution priority mode.

[0058] Next, the second extraction processing unit 225 extracts the second left eye image L5 and the second right eye image R5 from the input image I expanded in the memory 24 (S22). Then, the second extraction processing unit 225 expands the extracted second left eye image L5 and the second right eye image R5 into the memory 24. The second left eye image L5 is an image that includes a portion of the left eye input image L1 and has a display aspect ratio. The second right eye image R5 is an image that includes a portion of the right eye input image R1 and has a display aspect ratio. Furthermore, the second left eye image L5 and the second right eye image R5 are images extracted from different positions within the input image I (positions where parts of them do not overlap).

[0059] The second left eye extracted image L5 is a rectangular image composed of a pair of horizontal sides that intersect with the left eye input image L1 and a pair of vertical sides that do not intersect with the left eye input image L1. More specifically, the second left eye extracted image L5 is a rectangular image composed of a pair of horizontal sides extending horizontally between the upper and lower ends of the left eye input image L1 and a pair of vertical sides touching the left and right ends of the left eye input image L1. In other words, the second left eye extracted image L5 is a rectangular image in which the intersection of the diagonals coincides with the center of the left eye input image L1, the number of pixels in the vertical direction (1080 pixels) is smaller than that of the left eye input image L1, and the number of pixels in the horizontal direction (1920 pixels) is the same as that of the left eye input image L1. Furthermore, the second left eye extracted image L5 is a rectangular image in which images of the peripheral region are added to the four corners of the central part in the vertical direction of the circular left eye input image L1. Furthermore, the aspect ratio of the second left eye extracted image L5 is the same as the display aspect ratio (i.e., similar in shape to the display surface of the 3D monitor 30).

[0060] The second right eye extracted image R5 is a rectangular image composed of a pair of horizontal sides that intersect with the right eye input image R1 and a pair of vertical sides that do not intersect with the right eye input image R1. More specifically, the second right eye extracted image R5 is a rectangular image composed of a pair of horizontal sides extending horizontally between the upper and lower ends of the right eye input image R1 and a pair of vertical sides touching the left and right ends of the right eye input image R1. In other words, the second right eye extracted image R5 is a rectangular image in which the intersection of the diagonals coincides with the center of the right eye input image R1, the number of pixels in the vertical direction (1080 pixels) is smaller than that of the right eye input image R1, and the number of pixels in the horizontal direction (1920 pixels) is the same as that of the right eye input image R1. Furthermore, the second right eye extracted image R5 is a rectangular image in which images of the peripheral region are added to the four corners of the central part in the vertical direction of the circular right eye input image R1. Furthermore, the aspect ratio of the second right eye extracted image R5 is the same as the display aspect ratio (i.e., similar in shape to the display surface of the 3D monitor 30).

[0061] Next, the second magnification processing unit 226 enlarges the second left eye extracted image L5 and the second right eye extracted image R5, which have been expanded in the memory 24, to match the display size of the display surface of the 3D monitor 30, thereby generating the second left eye output image L6 and the second right eye output image R6 (S23). Then, the second magnification processing unit 226 expands the enlarged second left eye output image L6 and the second right eye output image R6 into the memory 24.

[0062] The second magnification processing unit 226 in this embodiment magnifies the second left eye extracted image L5 and the second right eye extracted image R5, which are 1920 × 1080 pixels, into the second left eye output image L6 and the second right eye output image R6, which are 3840 × 2160 pixels (i.e., magnification ratio 2x). The image magnification is achieved by interpolating the increased pixels using a well-known upconversion technique. Furthermore, the second left eye output image L6 and the second right eye output image R6 are congruent (identical in size and shape) to the display surface of the 3D monitor 30.

[0063] Next, the output processing unit 227 generates output image data, including the second left eye output image L6 and the second right eye output image R6, in accordance with the transmission method to the 3D monitor 30, and outputs it to the 3D monitor 30 (S24). The format of the output data is the same as that of the vertical resolution priority mode.

[0064] As a result, the 3D monitor 30 displays a 3D image that includes parts of the left eye input image L1 and the right eye input image R1, such as the second left eye output image L6 and the second right eye output image R6. In other words, in the horizontal display size priority mode, the output image is displayed across the entire display surface of the 3D monitor 30, resulting in a larger horizontal display size compared to the vertical resolution priority mode. On the other hand, in the horizontal display size priority mode, the vertical cropping of the imaging range of the left eye input image L1 and the right eye input image R1 is greater than in the vertical resolution priority mode.

[0065] [Effects of the Embodiment] According to the above embodiment, by selecting the vertical resolution priority mode, a 3D image including most of the imaging range (more preferably the entire range) can be generated. Furthermore, if the number of pixels in the left-right direction of the extracted image (1920 pixels) is the same, the magnification ratio in the vertical resolution priority mode (= 1.125) becomes smaller than the magnification ratio in the horizontal display size priority mode (= 2). As a result, in the vertical resolution priority mode, the proportion of pixels captured by the image sensor 23 is larger (in other words, the proportion of interpolated pixels is smaller) compared to the horizontal display size priority mode, so that the user can view high-resolution 3D images.

[0066] Furthermore, according to the above embodiment, in vertical resolution priority mode, the display size of the 3D image can be maximized by enlarging the first left eye extracted image L2 and the first right eye extracted image R2 to match the number of pixels in the vertical direction of the 3D monitor 30. However, if the size of the imaging surface of the image sensor 23 is made larger than the size of the display surface of the 3D monitor 30, the processing of the first enlargement processing unit 223 can be omitted. In this case, the mask processing unit 224 only needs to combine the mask images ML and MR with the first left eye extracted image L2 and the first right eye extracted image R2.

[0067] Furthermore, according to the above embodiment, by further providing a horizontal display size priority mode, although the vertical cropping of the imaging range becomes larger, it is possible to display the 3D image across the entire display surface of the 3D monitor 30. In addition, by setting the display aspect ratio of the second left eye extracted image L5 and the second right eye extracted image R5, the magnification in the vertical and horizontal directions in step S23 can be made the same. This makes it possible to allow the user to view a distortion-free 3D image.

[0068] Furthermore, according to the above embodiment, by allowing the user to select between a vertical resolution priority mode and a horizontal display size priority mode via the operation unit 27, 3D images can be displayed that are tailored to the application and the user's preferences.

[0069] Furthermore, according to the above embodiment, since the left eye input image L1 and the right eye input image R1 are imaged onto the image sensor 23 which is the same size as the display surface of the 3D monitor 30, a large-sized 3D image can be displayed with a relatively small image sensor 23.

[0070] Note that in vertical resolution priority mode, the order of image extraction, enlargement, and mask image is not limited to the example in Figure 3. Similarly, in horizontal display size priority mode, the order of image extraction and enlargement is not limited to the example in Figure 5.

[0071] [Other Embodiments] Some or all of the means implemented by the program can also be implemented by hardware such as integrated circuits. Furthermore, the program may be provided by being recorded on a non-transient recording medium that can be read by a computer. Recording medium refers to, for example, a hard disk, an SD card, a DVD, or a server on the Internet.

[0072] Although various embodiments have been described above with reference to the drawings, it goes without saying that this disclosure is not limited to such examples. It is clear to those skilled in the art that various modifications, alterations, substitutions, additions, deletions, and equivalents can be conceived within the scope of the claims, and these are also understood to fall within the technical scope of this disclosure. Furthermore, the components of the various embodiments described above can be combined arbitrarily without departing from the spirit of the invention.

[0073] (Note 1) An imaging device comprising: an imaging unit that captures an input image in which a left-eye input image and a right-eye input image having parallax with respect to each other are arranged based on light from a subject incident through an optical system; and an image processing unit that processes the input image captured by the imaging unit to generate a left-eye output image and a right-eye output image, wherein the image processing unit has a first mode that extracts from the input image a first left-eye extracted image which includes at least a part of the left-eye input image and has a ratio of the short side to the long side that is greater than the display aspect ratio of the display device, and a first right-eye extracted image which includes at least a part of the right-eye input image and has a ratio of the short side to the long side that is greater than the display aspect ratio, and combines a combined image with the outer edges of the first left-eye extracted image and the first right-eye extracted image respectively to generate a first left-eye output image and a first right-eye output image with the display aspect ratio.

[0074] The imaging device described in Appendix 2, wherein the image processing unit extracts a first left eye extracted image, which includes the entire left eye input image, and a first right eye extracted image, which includes the entire right eye input image, from the input image in the first mode.

[0075] The imaging apparatus described in Appendix 2, wherein the image processing unit, in the first mode, enlarges the first left eye extracted image and the first right eye extracted image, respectively, to match the number of pixels in the short direction of the display device to generate a left eye enlarged image and a right eye enlarged image, and combines the combined image on both sides of the long direction of the left eye enlarged image and the right eye enlarged image, respectively, to generate a first left eye output image and a first right eye output image.

[0076] The imaging device described in Appendix 1, wherein the image processing unit is configured to adjust the length of the first left eye extracted image and the first right eye extracted image in the short direction according to the user's operation.

[0077] The imaging apparatus described in Appendix 1, wherein the image processing unit extracts a second left-eye extracted image from the input image that includes a portion of the left-eye input image and has the display aspect ratio, and a second right-eye extracted image that includes a portion of the right-eye input image and has the display aspect ratio, and has a second mode that enlarges the second left-eye extracted image and the second right-eye extracted image, respectively, to match the display size of the display device to generate a second left-eye output image and a second right-eye output image.

[0078] The imaging device described in Appendix 5, wherein the image processing unit extracts a second left eye extracted image in the second mode in which the number of pixels in the short direction is smaller than that of the left eye input image and the number of pixels in the long direction is the same as that of the left eye input image, and extracts a second right eye extracted image in which the number of pixels in the short direction is smaller than that of the right eye input image and the number of pixels in the long direction is the same as that of the right eye input image.

[0079] The imaging device described in Appendix 5, wherein the image processing unit is configured to switch between the first mode and the second mode according to user operation.

[0080] An imaging device as described in Appendix 1, wherein the size of the imaging surface of the imaging unit is the same as the size of the display device.

[0081] An imaging system comprising: an imaging device as described in Appendix 1; an optical system for forming an image of light from the subject as the input image on the imaging unit; and a display device for displaying a 3D image of the subject based on the left eye output image and the right eye output image generated by the image processing unit.

[0082] This disclosure is useful as an imaging system 1 and imaging device 20 capable of generating 3D images with minimal cropping.

[0083] 1: Imaging system 10: Optical system 11: Scope 12: Coupler 13L: Left eye scope 13R: Right eye scope 14L: Left eye lens 14R: Right eye lens 20: Imaging device 21: Camera head 22: Image processing unit 23: Image sensor 24: Memory 25: CPU 27: Operation unit 30: 3D monitor 221: Switching unit 222: First extraction processing unit 223: First magnification processing unit 224: Mask processing unit 225: Second extraction processing unit 226: Second magnification processing unit 227: Output processing unit

Claims

1. An imaging device comprising: an imaging unit that captures an input image in which a left-eye input image and a right-eye input image having parallax with respect to each other are arranged based on light from a subject incident through an optical system; and an image processing unit that processes the input image captured by the imaging unit to generate a left-eye output image and a right-eye output image, wherein the image processing unit has a first mode that extracts from the input image a first left-eye extracted image which includes at least a portion of the left-eye input image and has a ratio of the short-width direction to the long-width direction greater than the display aspect ratio of the display device, and a first right-eye extracted image which includes at least a portion of the right-eye input image and has a ratio of the short-width direction to the long-width direction greater than the display aspect ratio, and combines a combined image with the outer edges of the first left-eye extracted image and the first right-eye extracted image respectively to generate a first left-eye output image and a first right-eye output image with the display aspect ratio.

2. An imaging device according to claim 1, wherein the image processing unit extracts a first left eye extracted image, which includes the entire left eye input image, and a first right eye extracted image, which includes the entire right eye input image, from the input image in the first mode.

3. The imaging apparatus according to claim 2, wherein the image processing unit, in the first mode, enlarges the first left eye extracted image and the first right eye extracted image, respectively, to match the number of pixels in the short direction of the display device to generate a left eye enlarged image and a right eye enlarged image, and combines the combined image on both sides in the long direction of the left eye enlarged image and the right eye enlarged image, respectively, to generate a first left eye output image and a first right eye output image.

4. The imaging apparatus according to claim 1, wherein the image processing unit is configured to adjust the length in the short direction of the first left eye extracted image and the first right eye extracted image according to the user's operation.

5. The imaging apparatus according to claim 1, wherein the image processing unit has a second mode in which it extracts a second left eye extracted image from the input image that includes a portion of the left eye input image and has the display aspect ratio, and a second right eye extracted image that includes a portion of the right eye input image and has the display aspect ratio, and enlarges the second left eye extracted image and the second right eye extracted image, respectively, to match the display size of the display device to generate a second left eye output image and a second right eye output image.

6. The imaging apparatus according to claim 5, wherein the image processing unit extracts a second left eye extracted image in the second mode, wherein the number of pixels in the short direction is smaller than that of the left eye input image and the number of pixels in the long direction is the same as that of the left eye input image, and extracts a second right eye extracted image, wherein the number of pixels in the short direction is smaller than that of the right eye input image and the number of pixels in the long direction is the same as that of the right eye input image.

7. The imaging apparatus according to claim 5, wherein the image processing unit is configured to switch between the first mode and the second mode according to user operation.

8. An imaging device according to claim 1, wherein the size of the imaging surface of the imaging unit is the same as the size of the display device.

9. An imaging system comprising: an imaging device according to claim 1; an optical system for forming an image of light from a subject as an input image on the imaging unit; and a display device for displaying a 3D image of the subject based on the left eye output image and the right eye output image generated by the image processing unit.

Images