Three-dimensional information acquisition device

Abstract

Provided is a three-dimensional information acquisition device that enables observation by compensating for a missing portion of an image. The three-dimensional information acquisition device comprises: a lens array (110) that is disposed so as to face an object (1) and in which a plurality of lens elements (112) are disposed in a matrix; a light receiving unit (142) that receives, on an imaging surface, a light flux passing through the lens elements; an imaging unit (120) in which a plurality of imaging elements (125) for receiving the light flux are disposed in a matrix on the light receiving unit (142) so as to correspond to each of the plurality of lens elements; a housing unit (145) in which gaps (140, 141) are formed in each of a surface of the lens array on which the lens elements are disposed and a surface of the light receiving unit so as to be oriented toward the object, the housing unit (145) forming a space that tubularly connects the gaps each other ; and an information processing unit (150) that processes optical information acquired by the imaging unit for each imaging element.

Description

3D Information Acquisition Device

[0001] The present invention relates to a 3D information acquisition device, and more particularly to a 3D information acquisition device that utilizes the principle of optical integral photography (Integral Photography (IP)).

[0002] There is a method called integral photography (hereinafter referred to as IP), which is attracting attention as one of the methods for displaying 3D images. This method projects a natural and highly reproducible 3D image into space using a 2D planar lens array. Since this method forms an image of a point in space, an observer can obtain an effect as if the target object exists in space. Furthermore, a method for generating an IP image on a computer (Computer Generated IP) has been developed for creating and displaying a 3D moving image system that combines a lens array, a moving image display device, and a computer. The video display method based on the principle of this IP is called integral videography (hereinafter referred to as IV).

[0003] For example, as IP and IV, which are optical 3D shape measurement methods, a technique using a lens array in which a plurality of lenses are arranged in a multi-eye manner in one camera and intensity-modulated light that is spatially or temporally intensity-modulated has been disclosed (see Patent Document 1).

[0004] According to this technique, images of a subject viewed from different viewpoints obtained by the lens array are each imaged on the imaging surface of a 2D imaging element and captured. As a result, a group of subject images viewed from different viewpoints corresponding to the positions of the plurality of lenses of the lens array is obtained, and 3D information of the subject is detected based on the group of subject images.

[0005] Also, regarding IP, a method for complementing the images of the gap portions between a plurality of imaging elements arranged in a matrix with a gap between the imaging surfaces has been disclosed (Patent Document 2).

[0006] This technology assumes a virtual element lens for the element image to be interpolated, and for each of multiple viewpoints in order of increasing distance from the principal point, the value of the pixel to be interpolated is the average value of the average of multiple pixel values ​​when the rate of change of the average value is below a threshold. This configuration for interpolating gaps provides an imaging device that generates integral image data for obtaining high-resolution spatial images.

[0007] Meanwhile, in the medical field, devices have been developed that arrange the observation unit and the laser irradiation unit in parallel or coaxially to enable treatment while observing the laser irradiation position. For example, a technology has been disclosed that enables laser irradiation to always be directed to the center of the observation field by arranging multiple observation optical fibers around a single laser transmission optical fiber (see Non-Patent Literature 1).

[0008] In the medical field, another technology has been disclosed (Patent Document 3) relating to an endoscope system that uses a small-diameter composite optical fiber with a structure in which an optical fiber for image transmission used for detecting and diagnosing the affected area and a large-diameter optical fiber for laser light transmission used for treating the affected area are integrated, the composite optical fiber having a large number of optical fibers for image transmission focused and integrated around a large-diameter optical fiber for laser light transmission, a laser irradiation and image observation optical system connected to the eyepiece of the composite optical fiber which causes laser light to be incident on the large-diameter optical fiber and forms an image on a camera so that it can be observed, and an illumination light transmission device which transmits and irradiates illumination light to the tip of the objective end of the composite optical fiber.

[0009] This technology provides an endoscopic system with an extremely small diameter that facilitates observation within the body, and enables simultaneous image observation and treatment with laser light irradiation.

[0010] Furthermore, a technique has been disclosed in which observation and laser irradiation can be performed using the same lens optical system by reflecting only the laser light using a hot mirror and transmitting the visible light acquired as an observation image (see Non-Patent Document 2).

[0011] Japanese Patent Publication No. 2009-300268, Japanese Patent Publication No. 2015-103846, Japanese Patent Publication No. 2005-237436

[0012] Kiyoshi Oka, Takeshi Seki, Tomohiro Akatsu, "A combined optical fiber endoscope system enabling simultaneous diagnosis and treatment," Optics, Vol. 41, No. 11, pp. 576-578, 2012. Noriaki Yamanaka, Hiromasa Yamashita, Ken Masamune, Toshio Chiba, Takeyoshi Dohi, "An endoscope With 2 DOFs Steering of Coaxial Nd: YAG Laser Beam for Fetal Surgery," IEEE / ASME Transactions on Mechatronics, Vol. 15 No. 6, pp. 898-905, 2010.

[0013] In the technology disclosed in Patent Document 1, when the observation unit and the laser irradiation unit are arranged in parallel or coaxially, the portion occupied by the device becomes a gap between the observation unit and the device, making it difficult to obtain accurate information.

[0014] The technology disclosed in Patent Document 2 provides a process that prevents gaps that occur when arranging lens arrays from affecting integral image data, and does not utilize the gaps themselves.

[0015] In the technology disclosed in Non-Patent Document 1, the developed device has an optical fiber for laser transmission located at the center of the device, making it difficult to observe the laser irradiation position during laser irradiation.

[0016] The technology disclosed in Patent Document 3 is configured as an endoscope system, but it does not generate image information three-dimensionally using IP.

[0017] Furthermore, in the technology disclosed in Non-Patent Document 2, although the developed device allows observation of the laser irradiation position during laser irradiation, there were concerns about poor observation and errors in the laser irradiation position due to damage to the coating applied to the lens optical system or thermal damage caused by the use of high-power lasers and long-term laser transmission.

[0018] This invention has been made in view of the circumstances described above, and aims to provide a three-dimensional information acquisition device that enables observation using the principles of optical integral photography (IP) and integral videography (IV), even when the observation unit and the laser irradiation unit are arranged in parallel or coaxially.

[0019] The present invention is a three-dimensional information acquisition device that acquires three-dimensional information of an object arranged in a predetermined space of a three-dimensional coordinate system. One embodiment thereof comprises: a lens array arranged to face the object and having a plurality of element lenses arranged in a matrix; a light receiving unit that receives light beams passing through the element lenses on an imaging surface; an imaging unit arranged in the light receiving unit corresponding to each of the plurality of element lenses and having a plurality of image sensors arranged in a matrix that receive the light beams; a housing unit in which gaps are formed on the surface of the lens array on which the element lenses are arranged and on the surface of the light receiving unit, respectively, so as to point toward the object, and a space is formed that connects the gaps in a tubular shape; and an information processing unit that processes the optical information acquired by the imaging unit for each of the image sensors.

[0020] In IP and IV optical systems, elemental images of IP and IV, consisting of elements from multiple viewpoints, are acquired by using a lens array. Even if some elemental pixels are missing due to gaps or other reasons in some images, the surrounding images can be configured to include the pixels of the missing elements. Therefore, in this configuration, when displaying elemental images, the information processing unit performs calculations on the missing parts (gaps) of the image using the surrounding pixels, making it possible to process the acquired optical information into positional information and image information on a per-image sensor basis.

[0021] This configuration makes it possible to observe the gaps between the lens array surface and the light-receiving surface, which would normally be considered missing images. By enabling observation of these gaps, which are normally considered missing images, it becomes possible to position remotely acting energy devices such as laser fibers and ultrasound (HIFU) probes, as well as physical treatment instruments such as electrosurgical units and forceps, into these gaps.

[0022] Furthermore, this configuration includes a housing section that utilizes the gaps between the surfaces of the lens array and the light-receiving section to form a tubular space connecting the gaps. This housing section can be used, for example, to house optical equipment for acquiring clearer images, to arrange actuators to perform tasks using the generated IP and IV image information, and to arrange liquid delivery and suction equipment to maintain a stable environment for the target. Thus, by including this housing section, this configuration not only allows for obtaining clearer image information but also enables the improvement and enhancement of the imaging environment and the advanced use of the image information.

[0023] Furthermore, many applications have recently become available for processing positional and image information at the image sensor level, and these can be selected as appropriate. In addition, image recognition technologies such as AI can also be applied.

[0024] In the above configuration, the information processing unit can acquire optical information obtained from image sensors arranged around the gap of the light-receiving unit, and generate image information corresponding to the position of the gap of the light-receiving unit from the image correlation with the optical information.

[0025] This configuration presents one method of image processing. Digital image correlation is a technique that compares pixels with several regions or reference pixels, calculates correlation values, and finds identical pixels. The pixels of the IP and IV element images, which consist of elements from multiple viewpoints, contain a lot of information about characteristic attributes, regions, contours, and colors. From this information, image correlation can generate a complementary image for the gaps. Many image correlation techniques have been developed, and it is possible to select and apply the appropriate image correlation method depending on the situation to recognize and extract images.

[0026] According to the above configuration, by performing image correlation, IP and IV images that can be observed from the front can be complemented and displayed.

[0027] In the above configuration, an illumination unit that irradiates light toward the object can be arranged in the housing unit.

[0028] According to the above configuration, it becomes possible to observe the laser irradiation position during laser irradiation, which was difficult in the conventional technology disclosed in Non-Patent Document 1. Furthermore, it eliminates concerns such as poor observation and errors in the laser irradiation position due to damage to the coating applied to the lens optical system or thermal damage caused by the use of high-power lasers or long-term laser transmission, as in the technology disclosed in Non-Patent Document 2.

[0029] In the above configuration, the imaging unit is composed of a plurality of rectangular segmented imaging units, and each of the segmented imaging units can be arranged around the gap between the light-receiving units.

[0030] This configuration utilizes the multi-viewpoint advantage of IP and IV optical systems, and divides the imaging section on the image sensor side, which is easier to process by computer, without compromising the optical information on the lens array side that directly observes the object. Image interpolation can be performed by applying the image correlation technique described above. Even when the imaging section is divided as in this configuration, there are element lenses arranged in the lens array corresponding to each image sensor, so there is no image loss due to the image sensor. Here, the angle may be set with respect to the angular optical axis to facilitate the acquisition of information about the object.

[0031] This configuration allows for the use of rectangular, segmented imaging units, enabling the application of readily available rectangular image sensor panels in various sizes, while also reducing costs. Furthermore, selecting an image sensor panel of an appropriate size simplifies the layout.

[0032] In the above configuration, the generation of image information corresponding to the position of the gap in the light-receiving unit by image correlation can also be configured to generate virtual optical information from the virtual element lens based on the optical information acquired from the image sensor arranged around the gap in the light-receiving unit, using the optical information acquired from the image sensor arranged around the gap in the light-receiving unit, based on the angle difference between the viewing angle connecting the element lens arranged around the gap in the lens array and the object, and the viewing angle connecting the virtual element lens and the object, assuming that a virtual element lens is arranged in the gap in the light-receiving unit.

[0033] When the object is far away, the optical information from adjacent element lenses is almost the same due to the characteristics of the multi-viewpoint elements in the IP and IV optical systems. When the object is close, the optical information from adjacent element lenses relative to the object is shifted by the respective field of view angle. This shift due to the angle difference can be compensated for in a lens array where element lenses of the same shape are arranged side by side, by considering the thickness of the element lenses in the ratio of the pitch of adjacent element lenses to the distance to the object.

[0034] With this configuration, even without applying generally complex and advanced image correlation methods, it is possible to easily generate interpolated images from the relationship between the angular difference, i.e., the pitch amount between element lenses, and the distance to the object (subject distance), by taking advantage of the multi-view characteristic of IP and IV optical systems. Because the computational load is low, it is possible to handle IV video with a simple configuration.

[0035] In the above configuration, the information processing unit can be configured to compare the pixels of each of the multiple virtual optical information pieces generated from optical information based on image sensors arranged around the gap between at least two light-receiving units, and if there is a difference in the pixels, it can average one of the chromaticity, brightness, luminance, or illuminance of the pixels, or perform a weighted processing according to the arrangement of the image sensors, so to speak, the layout of the image sensors, and set it as the virtual optical information of the gap.

[0036] This configuration generates a complementary image using optical information such as chromaticity, brightness, and luminosity contained in the pixels, in addition to the region and contour. As mentioned above, in IP and IV optical systems, images of objects with different viewpoints are captured from the back of each single lens in the lens array, and the image difference becomes smaller, especially in closely spaced regions. For image interpolation, the pixels of the images from the lenses surrounding the gap that constitutes the missing area are superimposed, and the average value of the chromaticity, brightness, luminosity, and illuminance of the corresponding pixels is taken, or a weighted processing is applied according to the layout to perform image interpolation.

[0037] Chromaticity, luminance, luminosity, and illuminance can be displayed as digital values ​​through general image analysis. For example, even a typical personal computer can obtain the RGB chromaticity for each pixel, making it easy to perform calculations for averaging or weighted processing according to the layout. The choice of which of chromaticity, luminance, luminosity, or illuminance to select depends on the state of the image, and a combination of two or more of these can also be used.

[0038] This configuration allows for better reproduction of missing portions. Unlike conventional optical methods, IP and IV optical systems make it easier to detect anomalies, even when conditions differ from the surrounding area in gaps where images are missing, due to the characteristics of elements from other viewpoints. By applying processing using this configuration, the reliability of observation can be improved.

[0039] In the above configuration, the target object can be biological tissue, and this configuration can be mounted on an endoscope or microscope. This configuration can be replaced in the same form as Non-Patent Documents 1 and 2, and despite being compact, this configuration can acquire more reliable images than conventional technology, with image defects being filled in.

[0040] The present invention provides a three-dimensional information acquisition device that enables observation by complementing missing parts of an image using the principles of optical integral photography (IP) and integral videography (IV), even when the observation unit and laser irradiation unit are arranged in parallel or coaxially.

[0041] This is an overall configuration diagram of a three-dimensional information acquisition device according to one embodiment of the present invention. This is an explanatory diagram of the lens array viewed along the axis CL direction according to one embodiment of the present invention. This is an explanatory diagram of the imaging unit viewed along the axis CL direction according to one embodiment of the present invention. This is an explanatory diagram of the multi-viewpoint feature of the IP, IV optical system. This is an explanatory diagram of the situation when image defects occur due to gaps, etc. This is an explanatory diagram showing an example of an image interpolation method. This is the configuration of a verification test of a three-dimensional information acquisition device according to one embodiment of the present invention. This is a sample image of an image defect used in the verification test. This is an example of image interpolation, where (A) is an IV element image with a defect, and (B) is an IV element image with the defect interpolated. This is an example of image interpolation, where (A) is an IV image when an IV element image without a defect is displayed, (B) is an IV image when an IV element image with a defect is displayed, and (C) is an IV image when an IV element image with the defect interpolated is displayed. This is an example of image interpolation, and it is an example of correction by brightness (comparison of pixel values ​​on the horizontal line of the image in Figure 11), comparing the IV image when an IV element image with missing parts is displayed with the IV image without missing parts displayed with the IV image with the missing parts interpolated and the IV image when an IV element image without missing parts is displayed.

[0042] Hereinafter, preferred embodiments of the three-dimensional information acquisition device of the present invention will be described with reference to the drawings. In the following description, components that are denoted by the same reference numerals in different drawings are considered to be the same, and their descriptions may be omitted.

[0043] One aspect of the three-dimensional information acquisition device according to the present invention is as follows. There is a lens array arranged so as to face the object, in which a plurality of element lenses are arranged in a matrix. There is a light receiving unit that receives the light beam passing through the element lenses on an imaging surface. There is an imaging unit in which a plurality of imaging elements that are arranged in a matrix and receive the light beam are arranged corresponding to each of the plurality of element lenses. There is a housing unit in which a gap is formed in each of the surface of the lens array on which the element lenses are arranged and the surface of the light receiving unit so as to face the object, and a space that connects the gaps in a tubular shape is formed. There is an information processing unit that processes the optical information acquired by the imaging unit for each imaging element. If it is characterized by this, any specific aspect may be adopted.

[0044] In the following description, one aspect of the three-dimensional information acquisition device according to the present invention will be described. Needless to say, the present invention can be applied to quantitative information acquisition in general video media such as security, games, robot vision, and surveying, as well as medical image analysis in which IP and IV technologies are utilized.

[0045] (Description of Embodiment) First, referring to FIGS. 1 to 3, the main configuration of the three-dimensional information acquisition device according to an embodiment of the present invention will be described. FIG. 1 is an overall configuration diagram of the three-dimensional information acquisition device according to an embodiment of the present invention. FIG. 2 is an explanatory diagram of the lens array viewed along the axis CL direction according to an embodiment of the present invention. FIG. 3 is an explanatory diagram of the imaging unit viewed along the axis CL direction according to an embodiment of the present invention.

[0046] Referring to FIG. 1, the three-dimensional information acquisition device 100 according to an embodiment of the present invention acquires the three-dimensional information of the object 1 arranged in a predetermined space of a three-dimensional coordinate system. In the present embodiment, the configuration of the optical system excluding the information processing unit 150 that processes the optical information of the three-dimensional information acquisition device 100 is mounted on the endoscope 105. And the irradiation unit 130 is arranged so as to extend on the axis CL that is the center of the endoscope 105.

[0047] Here, in the present embodiment, the irradiation unit 130 is arranged to extend on the axis CL that is the center of the endoscope 105. However, the arrangement of the irradiation unit 130 is not limited to being centered as long as it is directed at the object 1. For example, when the cross-section is circular, it can be arranged to contact the outer periphery, or when the cross-section is rectangular, it can be arranged near the edge. Thus, the position of the irradiation unit 130 can also be selected according to the form of the irradiation unit 130 and the power / energy supply layout.

[0048] In addition, the irradiation unit 130 can be applied not only to the lasers disclosed in Non-Patent Documents 1 and 2, but also to ordinary light sources for illuminating dark places, energy devices that act remotely such as ultrasonic (HIFU) probes, and physical treatment tools such as electric scalpels and forceps.

[0049] This three-dimensional information acquisition device 100 is arranged to face the object 1, and includes a lens array 110 in which a plurality of element lenses 112 are arranged in a matrix, a light receiving unit 142 that receives the light beam passing through the element lenses 112 on the imaging surface, an imaging unit 120 in which a plurality of imaging elements 125 that receive the light beam and are arranged in a matrix corresponding to each of the plurality of element lenses 112 are arranged, a housing unit 145 in which gaps 140 and 141 are formed on the surfaces of the lens array 110 and the light receiving unit 142 where the element lenses 112 are arranged so as to face the object 1, and a space that forms a tube connecting the two gaps 140 and 141, and an information processing unit 150 that processes the optical information acquired by the imaging unit 120 for each imaging element 125.

[0050] The predetermined space where the object 1 is arranged is an area that can be measured by the three-dimensional information acquisition device, and is a range where the light beam reaches the lens array 110.

[0051] The lens array 110 is an optical device in which a large number of minute element lenses 112 are arranged in a matrix on a single substrate. By arranging a large number of element lenses 112 in parallel in this way, the lens array 110 can form an inverted image for each element lens 112, and by superimposing these images, it can form a single continuous image. In addition, the lens array 110 has the advantage of being able to reduce the distance between objects compared to ordinary lenses.

[0052] Referring to Figure 2, the lens array 110 in this embodiment is donut-shaped, with a gap 140 in the central part that serves as an opening for an irradiation unit 130 and a housing unit 145 for mounting the irradiation unit 130. Thus, the irradiation unit 130 in this embodiment is a device in which an observation unit and a laser irradiation unit are arranged in parallel or coaxially for treatment while observing the laser irradiation position. For example, by arranging multiple observation optical fibers around a single laser transmission optical fiber, laser irradiation can always be directed to the center of the observation field of view.

[0053] Furthermore, as mentioned above, the position of the irradiation unit 130 does not need to be on the axis CL, which is the center of the lens array 110; it can be offset in any direction (up, down, left, or right) within the housing unit 145 shown in Figure 2. Also, the arrangement of the element lenses 112 can be set in a matrix-like arrangement as appropriate to match the arrangement of the irradiation unit 130. Moreover, the size, number, shape, etc. of the element lenses are not limited to the form shown in Figure 2.

[0054] The lens array 110 is widely used not only in photocopiers and printers, but also in applications in fields such as medicine, communications, measurement, astronomy, aerospace, and security. The lens array 110 is made of materials such as glass, quartz, silicon, and polymer, and the element lenses 112 can be formed in a square, triangular, or random arrangement, with a lens pitch (distance between adjacent lenses) depending on the application.

[0055] The irradiation unit 130 located in the housing unit 145 can be fitted with an optical fiber for laser transmission. High-power optical fibers used in medical applications and the like generate a lot of heat, so it is preferable to set the position of the aforementioned irradiation unit 130 according to heat dissipation and cooling.

[0056] In this embodiment, the illumination unit 130 is arranged in the housing unit 145. However, for example, the housing unit 145 could be configured to allow light to enter from the outside through a space such as a hole, or actuators could be placed to provide liquid delivery equipment and suction equipment to support work using the IP and IV image information generated, or air conditioning equipment could be placed to maintain a stable environment for the target. Thus, by including the housing unit 145, this configuration not only provides clear image information but can also be used for a variety of purposes, such as work support using the image information, cleaning the imaging environment, and advanced utilization of the image information.

[0057] The reflected light from object 1 passes through element lens 112 and is then focused onto the image sensor 125 of imaging unit 120 to form an image. In this way, an inverted image of object 1 is formed for each element lens 112.

[0058] Referring to Figure 3, the imaging unit 120 in this embodiment consists of four segmented imaging units 121, 122, 123, and 124 arranged on a disc-shaped light-receiving unit 142. The segmented imaging units 121, 122, 123, and 124 are formed in a rectangular shape, and image sensors 125 are arranged on each of them.

[0059] The image sensor 125 can be an image sensor such as a CCD or CMOS. Furthermore, the image sensor 125 can also be an element sensitive not only to visible light, but also to infrared, ultraviolet, or X-rays.

[0060] The image sensors 125 that make up the segmented imaging units 121, 122, 123, and 124 each receive a light beam from the corresponding element lens 112. The light beam incident on the image sensor 125 is converted into electrons for each image on the image sensor 125 using a photodetector called a photodiode, which passes through the color filter of the image sensor 125. This electronic information (charge, etc.) is transmitted as a signal to the information processing unit 150.

[0061] The information processing unit 150 is a type of computer and includes a processor (CPU) that performs calculations, random access memory (RAM) that provides a storage area for temporarily storing various data and a workspace for calculations performed by the processor, read-only memory (ROM) that pre-stores programs executed by the processor and various data used for calculations, and rewritable non-volatile memory that stores the results of calculations performed by the processor and data obtained from various parts of the engine system that should be saved. The non-volatile memory can be implemented as RAM with a backup function that is constantly supplied with voltage even after the system has stopped.

[0062] The information processing unit 150 receives the transmitted electronic information (electric charge, etc.) and passes it as analog data such as color intensity to the image processing engine within the information processing unit 150, where it is digitally converted and stored in memory as color space values.

[0063] In this way, the information processing unit 150 stores the color space values ​​as optical information for each image sensor 125 that constitutes the direction of the light beam, along with the direction of the light beam. Here, the main functions of the information processing unit 150 have been described, but the system may also be configured to include a display device (such as a display) that shows the results of the information processing.

[0064] Here, the color space value refers to a color space value that can be recorded as a color profile, and is not particularly limited to, for example, RGB, RGBA, YCbCr, CMYK, Lab color, etc. The color space value can be appropriately selected according to the conditions of the predetermined space and the brightness and color state of object 1.

[0065] For example, in the RGB color space used in computer displays and other devices that utilize the three primary colors of light—red, green, and blue—24 bits are allocated (8 bits each for RGB) to enable the display of 16,777,216 colors, allowing for precise identification.

[0066] The information processing unit 150, in this manner, compares the pixels of each of the multiple virtual optical information generated from the optical information based on the image sensor 125 that receives light beams from element lenses 112 arranged around at least two gaps 140. If there are differences in the pixels, it averages one of the pixel's chromaticity, brightness, luminous intensity, or illuminance and sets it as the virtual optical information for the gap.

[0067] Then, in addition to regions and contours, an interpolated image is generated using optical information contained in the pixels, such as chromaticity, brightness, and luminosity. For image interpolation, the pixels of the image captured by the lens around the gap that represents the missing area are superimposed, and the average value of the chromaticity, brightness, luminosity, and illuminance of the corresponding pixels is taken, or a weighted process is performed according to the layout to perform image interpolation.

[0068] Chromaticity, luminance, luminosity, and illuminance can be displayed as digital values ​​through general image analysis. For example, even a typical personal computer can obtain the RGB chromaticity for each pixel, making it easy to perform calculations for averaging. The choice of which of chromaticity, luminance, luminosity, or illuminance to use depends on the state of the image, and a combination of two or more of these values ​​may also be used.

[0069] According to this embodiment, missing images can be reproduced more effectively. Unlike conventional optical methods, the IP and IV optical systems make it easier to detect anomalies in the gaps where images are missing, even if conditions differ from the surroundings, based on the characteristics of elements from other viewpoints. By applying processing related to chromaticity, brightness, luminosity, and illuminance, the reliability of observation can be improved.

[0070] Furthermore, the information processing unit 150 acquires optical information from the image sensor 125 arranged around the gap 141 of the light receiving unit 142, and generates image information corresponding to the position of the gap 141 from the image correlation with the image information facing the gap 141.

[0071] Image correlation is a digital image processing method that finds a reference image based on the similarity between multiple images. It includes (1) image preprocessing to remove blur, indistinctness due to lighting, noise, etc., (2) feature extraction to extract characteristic attributes, regions, and contours, (3) region segmentation to divide the image into regions with the same brightness and color, (4) shape recognition to examine the characteristics of object regions obtained from the image, such as size and direction, and (5) object extraction to extract objects with similar attributes from the image. In short, it is a method that compares pixels with several regions or reference pixels, calculates correlation values, and finds similar ones.

[0072] In the IP and IV optical systems, elemental images of IP and IV, which consist of elements from multiple viewpoints, are acquired by using the lens array 110. Even if some elemental pixels are missing in some images due to gaps 140, 141, etc., it is possible to configure the system so that the pixels of the missing parts are included in the peripheral images.

[0073] In this embodiment, when displaying elemental images, surrounding pixels are used to perform the image correlation described above on areas where the image is missing (gaps 140, 141), thereby complementing and displaying IP and IV images that can be observed from the front.

[0074] The pixels in the IP and IV element images, which consist of elements from multiple viewpoints, contain a wealth of information about characteristic attributes, regions, contours, and colors. From this information, image correlation can be used to generate complementary images for gaps. Many image correlation techniques have been developed, and by selecting and applying the appropriate image correlation method for the situation, images can be recognized and extracted.

[0075] Thus, according to this embodiment, observation is possible at the position of the irradiation unit 130, which is the gap 141. Furthermore, as mentioned above, even when a high-power laser, which is a concern due to heat generation, is applied as the irradiation unit 130, the arrangement of the irradiation unit 130 can be freely selected on a plane, thus eliminating concerns such as damage to the coating applied to the lens optical system, poor observation due to thermal damage, and errors in the laser irradiation position.

[0076] (Explanation of an example of complementary image generation) Next, an example of complementary image generation will be explained with reference to Figures 4 to 6. Figure 4 is an explanatory diagram illustrating the multi-view characteristic of IP and IV optical systems. Figure 5 is an explanatory diagram illustrating the situation when image loss occurs due to gaps, etc. Figure 6 is an explanatory diagram showing an example of an image interpolation method.

[0077] Referring to Figure 4, when the object 1 is observed in the IP, IV optical system, element lens 112a forms an image of viewpoint image 114a with light beam R1, element lens 112b forms an image of viewpoint image 114b with light beam R2, and element lens 112c forms an image of viewpoint image 114c with light beam R3. Viewpoint images 114a, 114b, and 114c are each different viewpoint images.

[0078] Thus, the acquired multi-view images are obtained from different viewpoints for each back surface of a single lens. However, for distant observations or interpolation using close-up images, the angular difference is small, and they can be considered as images from the same viewpoint. Therefore, it becomes possible to interpolate missing parts by moving pixels while ignoring the angular difference.

[0079] On the other hand, as shown in Figure 5, when the object 1 is close to the lens array 110, the shape, contour, etc., of the missing image 170 and the extracted image 180 are clearly different, and in such cases the angular difference cannot be ignored.

[0080] In such cases, in the image relationship of each back surface of a single element lens 112 of the lens array 110 in adjacent regions in the viewpoint image 114, in order for a pixel indicating a certain point of object 1 to be displayed at the same point after pixel movement due to interpolation, angle correction is required to move by an interpolation amount x in addition to the lens pitch P.

[0081] Referring to Figure 6, the interpolation amount x can be expressed by the following equation (1) based on the geometric relationship, where l is the distance to the subject, h is the thickness of the imaging lens array, and P is the lens pitch.

[0082]

[0083] If the pixels in the extracted area are interpolated using only the movement amount of the lens pitch P, without including the interpolation amount x, a difference in display position will occur between the peripheral image and the interpolated image when displaying the interpolated image. In this case, as shown on the right side of Figure 6, the difference X in display position at the protruding surface 162, which is at the same distance H as the imaging position of the viewpoint image 114, is equal to the lens pitch P, which is the movement amount of the pixels, because the parameters of adjacent single lenses do not change. An interpolation amount x is determined such that this difference X in display position becomes 0.

[0084] This interpolation method allows for the easy generation of interpolated images from the relationship between the angular difference (i.e., the pitch between element lenses) and the distance to the object (subject distance), taking advantage of the multi-view characteristic of IP and IV optical systems, without the need to apply generally complex and advanced image correlation methods. Because of the low computational cost, it can also handle IV video with a simple configuration.

[0085] (Description of the Examples) Next, examples will be described with reference to Figures 7 to 12. Figure 7 is a configuration of a verification test which is an example of a 3D information acquisition device according to one embodiment of the present invention. Figure 8 is a sample image of an image defect used in the verification test. Figure 9 is an example of image interpolation, where (A) an IV element image with a defect and (B) an IV element image with the defect interpolated. Figure 10 is an example of image interpolation, where (A) an IV image when an IV element image without a defect is displayed, (B) an IV image when an IV element image with a defect is displayed, and (C) an IV image when an IV element image with the defect interpolated is displayed. Figure 11 is an example of image interpolation, where (A) a difference image between an IV image when an IV element image with a defect is displayed and an IV image when an IV element image without a defect is displayed, and (B) a difference image between an IV image when an IV element image with the defect interpolated is displayed and an IV image when an IV element image without a defect is displayed. Figure 12 shows an example of correction by brightness (comparison of pixel values ​​along the horizontal line in the image of Figure 11).

[0086] Referring to Figure 7, this is an overall diagram of a test to verify the acquisition of element images containing image defects, in order to perform image interpolation using pixels in the surrounding area of ​​an element image where image defects occur, and to evaluate the interpolated image obtained by displaying the newly constructed element image.

[0087] The configuration for this verification test consists of a subject 501 corresponding to object 1, an imaging lens array 510 corresponding to lens array 110, a relay lens 515 for transmitting the light beam from imaging lens array 510, and a CCD camera 520 corresponding to imaging unit 120.

[0088] For the imaging method, elemental images with defects were acquired by moving the imaging lens array 510, which was divided into two parts, perpendicular to the optical axis. A moving stage with a micrometer was used to move the imaging lens array 510. Since the diameter of a single lens (corresponding to the element lens 112) of the imaging lens array 510 is 1 mm, the distance between them was increased by 0.5 mm each, changing the distance from 0 to 10 pitches of the single lenses of the imaging lens array 510. The pitch is explained in Figure 6.

[0089] The imaging lens array 510 was prepared by processing and dividing a single lens array into two pieces, each measuring 25 mm in length, 15 mm in width, and 3.0 mm in thickness. A single CCD camera 520 with an effective pixel count of 640 x 480 was used. This was done to suppress the effects of pixel-level alignment of the elements and the potential for interference between substrates when using multiple image sensors.

[0090] Referring to Figure 8, for subject 501, in order to observe the effect of lateral image interpolation, a set of vertical stripes with intervals of 0.5 mm and 3 mm in the range of 0.5 mm to 3 mm was applied.

[0091] Figure 9 shows the steps for interpolating the elemental images. First, elemental images of size 640 x 480 pixels are obtained from the imaging experimental system.

[0092] Next, element images are acquired when the distance between lenses is set to two lens pitches. When the imaging area of ​​the image sensor is divided, the image area including the center of the subject 501 cannot be acquired. Based on the acquired element images, Figure 9(A), which is an element image with a 22% missing portion in the image area including the center of the subject, and Figure 9(B), which is an element image interpolated using the surrounding image, were generated. Figure 9(A) is the IV element image with the missing portion, and Figure 9(B) is the IV element image with the missing portion interpolated. Adobe Photoshop®, an image editing software from Adobe Systems Incorporated, was used to create these element images.

[0093] In this case, looking at the relationship between the pixels and the imaging lens array 510, there are 50 pixels in the width direction on the back of a single lens. Also, if the distance to the subject 501 is 40 mm, the interpolation amount x becomes 3.75 pixels, and interpolation is performed by moving it by 4 pixels plus the lens pitch P after rounding.

[0094] Figure 10 shows the results of displaying each IV element image on a display (not shown) and acquiring the IV image displayed via a display lens array (not shown). Figure 10 shows (A) the IV image when an IV element image without defects is displayed, (B) the IV image when an IV element image with defects is displayed, and (C) the IV image when an IV element image with defects is displayed.

[0095] Based on these results, it was confirmed that the images were supplemented for parts that could not be observed from the front due to the presence of gaps 140 and 141.

[0096] To evaluate the interpolation method, the degree of agreement between the interpolated image of the missing portion displayed by interpolation and the image that would have been originally displayed was measured. Figure 11 shows the difference in pixel values ​​obtained by correlating the IV image with no missing parts (Figure 10(A)), the IV image with missing parts (Figure 10(B)), and the IV image with interpolated parts (Figure 10(C)). Figure 11(A) is the difference image between the IV image when displaying the IV image with missing parts and the IV image when displaying the IV image without missing parts, and Figure 11(B) is the difference image between the IV image when displaying the IV image with interpolated missing parts and the IV image when displaying the IV image without missing parts.

[0097] Furthermore, Figure 12 shows the results of comparing the luminance values ​​in the pixel value difference. From these results, it can be seen that the luminance value obtained from the difference with the interpolated element image is smaller than the difference with the element image with missing values, and that an image closer to the original displayed pixel value was obtained.

[0098] Furthermore, it was confirmed that by displaying the interpolated image, an image configuration was created that reproduced the original pixel values ​​for pixels that could not be observed from the front in the display of the missing image.

[0099] This concludes the explanation, but the embodiments of the present invention are not limited to those described above. For example, even if a defect occurs in the lens array 110 later, or if imaging becomes difficult due to dirt, the complementary means of the present invention can be applied to generate a replacement image.

[0100] Furthermore, although this embodiment uses the endoscope 105 as an example, it can be applied to a wide range of applications, including microscopes, as well as entertainment fields such as 3D television and games, educational displays in museums, and the medical field.

[0101] 1...Object 100...3D information acquisition device 105...Endoscope 110...Lens array 112, 112a, 112b, 112c...Element lenses 114a, 114b, 114c...Viewpoint image 120...Imaging unit 121, 122, 123, 124...Segmented imaging unit 125...Image sensor 130...Irradiation unit 140, 141...Gap 142...Light receiving unit 145...Housing unit 150...Information processing unit 160...Complementary image 162...Protruding surface 170...Defective image 180...Extracted image 501...Subject 510...Imaging lens array 515...Relay lens 520...CCD camera CL...Axis l...Distance to subject H...Distance h...Imaging lens array thickness P...Lens pitch R1, R2, R3...Light beam x... interpolation amount X... difference in display position

Claims

1. A three-dimensional information acquisition device for acquiring three-dimensional information of an object placed in a predetermined space of a three-dimensional coordinate system, comprising: a lens array positioned to face the object and having a plurality of element lenses arranged in a matrix; a light-receiving unit that receives light beams passing through the element lenses on its imaging surface; an imaging unit that is positioned in the light-receiving unit corresponding to each of the plurality of element lenses and has a plurality of image sensors arranged in a matrix to receive the light beams; a housing unit that forms a space connecting gaps in a tubular manner between the surfaces of the lens array on which the element lenses are arranged and the surface of the light-receiving unit, so as to point toward the object; and an information processing unit that processes the optical information acquired by the imaging unit for each of the image sensors.

2. The three-dimensional information acquisition device according to claim 1, wherein the information processing unit acquires optical information obtained from image sensors arranged around the gap of the light-receiving unit, and generates image information corresponding to the position of the gap of the light-receiving unit from the image correlation with the optical information.

3. The three-dimensional information acquisition device according to claim 1 or 2, wherein the housing section is provided with an irradiation section for irradiating light toward the object.

4. The imaging unit is composed of a plurality of rectangular divided imaging units, each of which is arranged around the gap between the light-receiving units, in the three-dimensional information acquisition device according to claim 1 or 2.

5. The three-dimensional information acquisition apparatus according to claim 2, wherein the generation of image information corresponding to the position of the gap in the light-receiving unit by image correlation is based on the angle difference between the viewing angle connecting the element lens arranged around the gap in the lens array and the object, and the viewing angle connecting the virtual element lens and the object, assuming that a virtual element lens is arranged in the gap in the light-receiving unit, and generates virtual optical information from the virtual element lens from optical information acquired from the image sensor arranged around the gap in the light-receiving unit.

6. The three-dimensional information acquisition device according to claim 5, wherein the information processing unit compares the pixels of each of the multiple virtual optical information pieces generated from optical information based on image sensors arranged around at least two of the gaps, and if there is a difference in the pixels, it averages the chromaticity, brightness, luminance, or illuminance of the pixels, or performs a weighting process according to the arrangement of the image sensors, and sets it as the virtual optical information of the gap.

7. The three-dimensional information acquisition device according to claim 1 or 2, wherein the object is biological tissue.

8. A three-dimensional information acquisition device according to claim 1 or 2, which is mounted on an endoscope.

9. A three-dimensional information acquisition device according to claim 1 or 2, which is mounted on a microscope.

Images