Image processing system, image processing method, and program
The image processing system addresses the loss of depth values in occluded areas by identifying and warping a second depth value from the ToF camera's output, ensuring accurate depth information association in superimposed images.
Patent Information
- Application Number
- PCT/JP2026/014587
- Authority / Receiving Office
- WO · WO
- Patent Type
- Applications
- Current Assignee / Owner
- Priority Date
- 2025-05-13
- Filing Date
- 2026-04-08
- Publication Date
- 2026-11-19
AI Technical Summary
Existing image processing methods suffer from the loss of depth values in occluded areas when superimposing depth images onto images captured by another camera, due to occluding objects blocking the line of sight.
An image processing system that identifies occluded edge pixels based on the output of a ToF camera and warps a second depth value, calculated from the second strongest reflected light, onto the image plane of another camera to associate depth values with appropriate pixel positions, thereby preventing the loss of depth information in occluded areas.
The system effectively fills in missing depth values in occluded regions, ensuring accurate depth information is associated with the correct pixel positions in the superimposed image, enhancing the reliability of depth-based image processing.
Smart Images

Figure JP2026014587_19112026_PF_FP_ABST
Abstract
Description
IMAGE PROCESSING SYSTEM, IMAGE PROCESSING METHOD, AND PROGRAM
[0001] The present technology relates to an image processing system, an image processing method, and a program, and more particularly, to a warping processing technology for warping a depth value obtained based on output information of a ToF camera onto a captured image of another camera whose sensing range overlaps with that of the ToF camera.
[0002] Depth cameras that obtain depth information of a subject for each two-dimensionally arranged pixel are known. For example, Patent Document 1 below discloses a technology of a ToF camera that obtains depth information for each pixel by a ToF (Time of Flight) method. Specifically, Patent Document 1 discloses a technology of a dToF camera that obtains depth information by a dToF (direct ToF) method.
[0003] A depth image obtained using a depth camera makes it possible to recognize the three-dimensional shape of a subject, or to recognize the distance to each subject when a plurality of subjects are present.
[0004] Patent Document 2 below discloses a technology of performing warping (alignment) on a pixel-by-pixel basis for a depth image obtained using a depth camera and a captured image of another camera whose sensing range overlaps with that of the depth camera, to obtain combination information of depth information and luminance information for each pixel. Specifically, Patent Document 2 discloses obtaining combination information of depth information and luminance information for each pixel by associating luminance information for each pixel obtained by the other camera with the pixel at the corresponding position in the depth image.
[0005] Hereinafter, the process of determining which pixel position in one image corresponds to which pixel position in the other image by warping a depth image obtained using a depth camera and an image captured by another camera whose sensing range overlaps with that of the depth camera, and then associating the pixel values of one image with the corresponding pixels in the other image, will be referred to as "superposition processing." The superposition processing disclosed in Patent Document 2 can be described as superimposing an image captured by another camera (luminance image) onto the depth image.
[0006] Japanese Patent Publication No. 2021-113743 Japanese Patent Publication No. 2011-123071
[0007] Incidentally, as a superposition process, instead of superimposing an image captured by another camera onto a depth image as disclosed in Patent Document 2, it is also conceivable to perform the opposite process: superimposing a depth image onto an image captured by another camera. By superimposing a depth image onto an image captured by another camera, depth information can be obtained for each pixel of the image captured by the other camera. Such pixel-by-pixel depth information of an image captured can be used, for example, as auxiliary information during focusing in a camera. Specifically, for example, when considering an AF (Auto Focus) function that focuses on a target subject specified by the user via a monitor image of the captured image, it is conceivable to calculate the target focus lens position based on the depth information of the target subject. Alternatively, the pixel-by-pixel depth information of the captured image can also be used as auxiliary information when superimposing a virtual object created by CG (Computer Graphics) onto an image captured by another camera. Specifically, the size of the virtual object can be determined based on the depth information of the subject onto which the virtual object is superimposed.
[0008] Here, even when superimposing a depth image onto an image captured by another camera, the occurrence of occlusion, as described in Patent Document 2, should be considered. Specifically, if there is an occluding object that obstructs the line of sight from the depth camera toward the position of the other camera, the occluded area caused by the occluding object becomes an area from which depth information cannot be obtained, even though it is visible to the other camera. In this case, superimposition may result in the loss of depth information for the occluded area in the captured image.
[0009] This technology was developed in light of the above circumstances, and aims to prevent the loss of depth values in occluded areas caused by occluded objects when superimposing depth images onto images captured by another camera.
[0010] The image processing system relating to this technology performs a warping process that warps depth values obtained based on the output information of a ToF camera, which performs light receiving signal processing for distance measurement using the ToF method for each pixel in a two-dimensional array, onto the image plane of another camera whose sensing range overlaps with that of the ToF camera. This warping process involves: identifying occluded edge pixels based on the output information of the ToF camera, and identifying pixels corresponding to the farther end of an occluded object that obstructs the line of sight from the viewpoint of the ToF camera toward the position of the other camera, as occluded edge pixels; and warping a second depth value, which is a depth value calculated based on the second strongest reflected light detected by the occluded edge pixels, onto the image plane of the other camera. In distance measurement using the ToF method, reflected light corresponding to both the near depth and the far depth can be observed at the depth boundary. Specifically, the near depth is observed as the reflected light with the highest intensity, and the far depth is observed as the reflected light with the second strongest intensity. As described above, by identifying the occluded edge pixels corresponding to the edge of the occluded subject that is farther from the ToF camera's viewpoint, and warping the second depth value based on the second intensity of reflected light at the occluded edge pixels onto the image plane of another camera, it becomes possible to associate the depth value of the region occluded by the occluded subject with the appropriate pixel position in the image captured by the other camera.
[0011] This figure shows an example configuration of an image processing system as an embodiment of this technology. This is a block diagram showing an example of the internal configuration of a ToF camera in the embodiment. This is an explanatory diagram of the distance histogram in the dToF method. This is a block diagram showing an example of the hardware configuration of an image processing device in the embodiment. This is a diagram for explaining the warping process. This is an explanatory diagram of the occluded area caused by the occluded subject. This is an explanatory diagram of the area of missing depth values caused by the occluded subject. This is a functional block diagram for explaining the functions of the image processing device in the embodiment. This is an explanatory diagram of a pixel identified as a depth boundary pixel. This is an explanatory diagram of the peak value of reflected light intensity shown in the distance histogram at the occluded edge pixel. This is a diagram illustrating how the second depth value of the occluded edge pixel is warped to the image plane captured by another camera. This is a flowchart showing a specific example of a processing procedure for realizing the superposition method as an embodiment. This is a flowchart showing an example of a processing procedure corresponding to the superposition method as a first modified example. This is a flowchart showing an example of a processing procedure corresponding to the superposition method as a second modified example.
[0012] The embodiments of this technology will be described below in the following order, with reference to the attached drawings. <1. Example of an image processing system configuration as an embodiment> <2. Example of a ToF camera configuration> <3. Example of an image processing device configuration> <4. About warping processing> <5. About the loss of depth values in the occluded region> <6. Superposition method as an embodiment> <7. Processing procedure> <8. Modification example> <9. Summary of embodiments> <10. This technology>
[0013] <1. Example Configuration of an Image Processing System as an Embodiment> Figure 1 is a diagram showing an example configuration of an image processing system 10, which is one embodiment of the image processing system according to the present technology. As shown in the figure, the image processing system 10 as an embodiment includes an image processing device 1, as well as a ToF camera 2 and a separate camera 3.
[0014] The ToF camera 2 is a camera that processes a received light signal for distance measurement using the ToF (Time of Flight) method for each pixel in a two-dimensional array. As is well known, there are two types of ToF methods: dToF (direct ToF) and iToF (indirect ToF). In this example, the ToF camera 2 is a dToF camera that supports the dToF method.
[0015] Camera 3 is a separate camera from ToF camera 2, and its sensing range overlaps with that of ToF camera 2. Camera 3 is configured as a digital camera device equipped with an image sensor such as a CCD (Charge Coupled Device) sensor or a CMOS (Complementary Metal Oxide Semiconductor) sensor. In other words, Camera 3 is configured to obtain a color image such as an RGB image or a grayscale image (monochrome image) as the captured image. A grayscale image, as used here, refers to an image in which information indicating the received light intensity is associated with a predetermined number of grayscale values for each pixel.
[0016] In this example, the pixel count of ToF camera 2 is assumed to be less than that of camera 3. However, this is merely an example, and the pixel count of ToF camera 2 may be greater than or equal to that of camera 3.
[0017] Image processing device 1 is configured as a computer device and performs the process of superimposing a depth image obtained based on the output information of ToF camera 2 onto an image captured by another camera 3. As mentioned above, "superimposition" means the process of identifying the correspondence between the pixels of one image and the corresponding pixel positions of the other image through warping, and associating the pixel values of one image with the pixels at the corresponding positions in the other image. In this embodiment, since the depth image is superimposed on the image captured (grayscale image), superimposition in this case refers to the process of identifying the correspondence between the depth values of each pixel in the depth image and the corresponding pixel positions of the image captured image through warping, and associating the depth values in the depth image with the pixels at the corresponding positions in the image captured image.
[0018] <2. Example of ToF Camera Configuration> Figure 2 is a diagram showing an example of the schematic internal configuration of ToF camera 2 as a dToF camera. ToF camera 2 has a light-emitting unit 25 and a light-receiving unit 26. The light-emitting unit 25 emits light that is reflected by the target object (subject) and the light-receiving unit 26 receives the light and performs light-receiving signal processing to measure the distance to the target object. Specifically, ToF camera 2 includes a measurement unit 27 and a histogram image generation unit 28 along with the light-emitting unit 25 and the light-receiving unit 26.
[0019] The light-emitting unit 25 emits light from multiple light sources. In this example, the light-emitting unit 25 has light-emitting elements, each of which is a VCSEL (Vertical Cavity Surface Emitting Laser), and these light-emitting elements are arranged in two dimensions in a predetermined manner, such as a matrix.
[0020] The light-receiving unit 26 is configured with a pixel array section in which a plurality of pixels (light-receiving pixels) are arranged, each having a light-receiving element that receives light emitted from the light-emitting unit 25 and reflected by the target object. These pixels are arranged in two dimensions in a predetermined manner, such as a matrix. In this example, the light-receiving unit 26 has a SPAD (Single Photon Avalanche Diode) element as the light-receiving element for each pixel.
[0021] In the following, the pixels of the light-receiving unit 26 of the ToF camera 2 may be referred to as "depth pixels" to distinguish them from the pixels of the image sensor of the separate camera 3.
[0022] The measurement unit 27 controls the emission of light from the light-emitting unit 25 and measures the time required from emission to reception based on the light-receiving signal from the light-receiving unit 26. Specifically, when the dToF method is adopted, the measurement unit 27 pulses the emission of light from each light-emitting element in the light-emitting unit 25. The measurement unit 27 then measures the time from the emission timing of the light-emitting unit 25 to the light-receiving timing of the light-receiving element for each pixel of the light-receiving unit 26. In the dToF method, a distance value, i.e., a depth value, is calculated for each pixel based on the measured time from emission to reception (Time of Flight) and the speed of light.
[0023] Here, the measurement unit 27 performs the above-described time measurement from light emission to light reception multiple times per unit time in order to obtain the final depth value for each pixel. Specifically, the number of such measurements per unit time is, for example, several hundred thousand to several million times. In other words, in the calculation of one depth value, the light emission of pulsed light by the light emission unit 25 and the reception of reflected light from the target object are performed several hundred thousand to several million times.
[0024] The reason for measuring the time from light emission to light reception multiple times per unit time is that measuring the flight time for only one pulse would result in unreliable distance measurement results.
[0025] The histogram image generation unit 28 receives the time information from light emission to light reception obtained multiple times per unit time in the measurement unit 27 as described above, for each pixel, and generates a histogram image. Here, a histogram image refers to information where a "distance histogram," which represents the time information from light emission to light reception obtained multiple times per unit time as described above, as frequency information for each time period from light emission to light reception, is assigned to each pixel position.
[0026] Figure 3 is an explanatory diagram of the distance histogram. As shown in the figure, the distance histogram is frequency distribution information where the class is the time from emission to reception, and the frequency for each class is the frequency. Here, the frequency on the vertical axis can be rephrased as the number of photons detected by the photodetector as a SPAD element, or more precisely, as a value indicating the intensity of reflected light received from the target object. In other words, the distance histogram can be rephrased as information indicating the intensity of reflected light received for each "time from emission to reception" corresponding to "distance".
[0027] <3. Example Configuration of Image Processing Device> Figure 2 is a block diagram showing an example of the hardware configuration of the image processing device 1. As shown in the figure, the image processing device 1 is equipped with a processor 11. The processor 11 is configured to have at least a CPU (Central Processing Unit) and executes various processes according to a program stored in the ROM (Read Only Memory) 12 or a program loaded from the storage unit 19 into the RAM (Random Access Memory) 13. Here, in order to efficiently perform various image signal processing, the processor 11 may also be configured to have a GPU (Graphics Processing Unit) in addition to the CPU.
[0028] In this embodiment, the superposition of the depth image onto the grayscale image is performed by the processor 11. During this superposition process, the processor 11 calculates the depth value based on the distance histogram. As illustrated in Figure 3, the distance histogram in this example is generated as information showing the frequency for each time interval from light emission to light reception, so the processor 11 calculates the depth value using information on the speed of light. It is also conceivable that the distance histogram be generated as information showing the frequency for each depth value, in which case the calculation of the depth value would be performed on the ToF camera 2 side, and the processor 11 would not need to calculate the depth value using information on the speed of light.
[0029] The RAM 13 also stores data necessary for the processor 11 to perform various processes. The processor 11, ROM 12, and RAM 13 are interconnected via a bus 14. An input / output interface (I / F) 15 is also connected to this bus 14.
[0030] An input unit 16, consisting of operators or operating devices, is connected to the input / output interface 15. For example, the input unit 16 can be various operators or operating devices such as a keyboard, mouse, keys, dial, touch panel, touchpad, or remote controller. User operations are detected by the input unit 16, and the signals corresponding to the input operations are interpreted by the processor 11.
[0031] Furthermore, a display unit 17, consisting of an LCD (Liquid Crystal Display) or an organic EL (Electro-Luminescence) panel, and an audio output unit 18, consisting of a speaker, are connected to the input / output interface 15, either as an integrated unit or as separate components. The display unit 17 is used for displaying various types of information and is composed of, for example, a display device provided on the housing of the image processing device 1, or a separate display device connected to the image processing device 1.
[0032] The display unit 17 displays images for various image processing tasks and videos to be processed on the display screen based on instructions from the processor 11. The display unit 17 also displays various operation menus, icons, messages, etc., i.e., a GUI (Graphical User Interface), based on instructions from the processor 11.
[0033] The input / output interface 15 may also be connected to a storage unit 19 consisting of an HDD (Hard Disk Drive) or solid memory, or a communication unit 20 consisting of a modem or the like. The storage unit 19 is used to store various types of data.
[0034] The communication unit 20 performs wired or wireless data communication with external devices. For example, it can be configured to perform wired data communication with external devices in accordance with a predetermined wired communication standard such as the USB (Universal Serial Bus) communication standard, wireless data communication with external devices in accordance with a predetermined wireless communication standard such as the Bluetooth® communication standard, or wireless or wired data communication with external devices via a predetermined communication network such as the Internet. The processor 11 can send and receive data with external devices via the communication unit 20. In particular, in this embodiment, it is possible to send and receive data with the ToF camera 2 and another camera 3 shown in Figure 1.
[0035] A drive 21 is also connected to the input / output interface 15 as needed, and a removable recording medium 22 such as a magnetic disk, optical disk, magneto-optical disk, or semiconductor memory is appropriately mounted there.
[0036] The drive 21 can read data (including computer programs, etc.) used for each process from the removable recording medium 22. The read data is stored in the storage unit 19, or, if the read data is image data or audio data, the image or audio is output by the display unit 17 or the audio output unit 18. The computer program read from the removable recording medium 22 is installed in the storage unit 19 as needed.
[0037] In the image processing apparatus 1 having the hardware configuration described above, for example, the software for processing in this embodiment can be installed via network communication by the communication unit 20 or via the removable recording medium 22. Alternatively, the software may be stored in advance in the ROM 12 or the storage unit 19. The processor 11 performs processing operations based on various programs, thereby executing the necessary information processing and communication processing for the image processing apparatus 1.
[0038] Specific examples of the image processing device 1 include PCs (personal computers) and server devices.
[0039] The image processing device 1, as an embodiment, is not limited to being composed of a single computer device as illustrated in Figure 2, but may be composed of multiple computer devices. These multiple computer devices may be systematized via a LAN (Local Area Network) or the like, or they may be located remotely via a VPN (Virtual Private Network) using the Internet or the like. The multiple computer devices may also include computer devices that function as a group of servers (cloud) available through cloud computing services.
[0040] <4. Warping Process> Referring to Figure 5, the warping process performed in the depth image superposition process will be explained. Figures 5A and 5B illustrate the positional relationship in three-dimensional space between the depth image plane, the depth-side viewpoint, the grayscale image plane, and the grayscale viewpoint. Here, the depth image plane and the grayscale image plane refer to the two-dimensional projection plane of the subject in the ToF camera 2 and the two-dimensional projection plane of the subject by the other camera 3, respectively. The depth-side viewpoint refers to the viewpoint of the ToF camera 2, and the grayscale viewpoint refers to the viewpoint of the other camera 3.
[0041] First, in the warping process, as shown in Figure 5A, a coordinate transformation process is performed to determine the coordinates in three-dimensional space for subject points (indicated by △, □, and ○ in the figure) projected onto each pixel position in the depth image plane. In this coordinate transformation process, the three-dimensional spatial coordinates of a subject point captured by a certain pixel on the depth image plane (hereinafter referred to as the "target pixel") can be determined as coordinates on a straight line passing through the target pixel from the depth viewpoint. Specifically, the coordinates can be determined as the coordinates of the position on that straight line where the distance from the depth viewpoint is the distance indicated by the depth value of the target pixel.
[0042] Then, in the warping process, each subject point obtained for each depth pixel through the coordinate transformation process described above is projected onto the grayscale image plane in a two-dimensional projection process, as shown in Figure 5B. As shown in the figure, the projection point of each subject point on the grayscale image plane can be obtained as the intersection point of the line connecting the subject point and the grayscale viewpoint and the grayscale image plane.
[0043] Here, the warping process consisting of the coordinate transformation process and the two-dimensional projection process described above can be performed using the internal parameters and external parameters of the ToF camera 2 and the separate camera 3 (see Patent Document 1).
[0044] By performing the warping process as described above, the correspondence relationship regarding which pixel position in the grayscale image each depth value of each pixel of the depth image corresponds to is specified, and the depth value of each pixel in the depth image is associated with the pixel at the corresponding position in the grayscale image. That is, the process of superimposing the depth image on the grayscale image is realized.
[0045] <5. Regarding Missing Depth Values in Occlusion Regions> Here, when an image obtained by a certain camera is superimposed on an image obtained by another camera with a overlapping sensing range as in the present embodiment, when there is an occluding subject that blocks the line of sight of the certain camera toward the arrangement position side of the other camera, it should be taken into consideration that an occlusion region that is visible from the other camera but not visible from the certain camera will occur.
[0046] Figure 6 is an explanatory diagram of an occlusion region caused by an occluding subject. For example, when viewed from camera A in the figure, the line of sight toward camera B among the lines of sight from the viewpoint of camera A is blocked by the occluding subject, so that an occlusion region b in the figure is generated. Further, when viewed from camera B, the line of sight toward camera A among the lines of sight from the viewpoint of camera B is blocked by the occluding subject, so that an occlusion region a in the figure is generated.
[0047] Since the occlusion region b is a region that is visible from camera B but not visible from camera A, in this case, when the image from camera A (referred to as A image) is superimposed on the image from camera B (referred to as B image), the pixel value of the A image is missing for the occlusion region b that is visible in the B image. Further, since the occlusion region a is a region that is visible from camera A but not visible from camera B, in this case, when the B image is superimposed on the A image, the pixel value of the B image is missing for the occlusion region a that is visible in the A image.
[0048] For example, assuming that camera A is the ToF camera 2 and camera B is the separate camera 3, when the depth image obtained by the ToF camera 2 is superimposed on the grayscale image obtained by the separate camera 3, the depth value will be missing for the occluded region b that is visible in the grayscale image.
[0049] At this time, the missing region of depth values occurring on the grayscale image side due to the occluding subject tends to become larger because the warping distance varies depending on the depth value.
[0050] This point will be described with reference to FIG. 7. FIG. 7A and FIG. 7B exemplify the positional relationship of the depth image plane, the depth-side viewpoint, the grayscale image plane, and the grayscale-side viewpoint in the three-dimensional space, similarly to the case of FIG. 5 described above. In FIG. 7A, subject points p1, p2, and p3 exemplify subject points on the front surface of the occluding subject illustrated in FIG. 6, and subject point p4 exemplifies a subject point on the surface farther back than the occluding subject in FIG. 6. Subject point po exemplifies a subject point in the occluded region b.
[0051] FIG. 7B exemplifies the two-dimensional projection results onto the grayscale image plane for each of the subject points p1 to p4. Comparing the respective projection points on the grayscale image plane, the respective intervals between the subject points p1 to p3 that are close in distance (small depth values) are relatively dense, whereas for the projection point of the subject point p4 that is far in distance (large depth value), the interval becomes sparse relative to the projection point of the subject point p3, which is the closest in the left-right direction position (the position in the direction of separation between the ToF camera 2 and the separate camera 3) among the projection points of the subject points p1 to p3. Therefore, in the superimposed image in this case, a relatively large missing region as indicated by "X" in the drawing will occur as the missing region of depth values.
[0052] <6. Superposition Method as an Embodiment> In this embodiment, when superimposing a depth image obtained by the ToF camera 2 onto an image captured by another camera 3, the objective is to prevent the loss of depth values in occluded areas caused by occluded subjects. To this end, in the image processing system 10 of this embodiment, the processor 11 of the image processing device 1 has the following functions.
[0053] Figure 8 is a functional block diagram illustrating the functions of the processor 11 as an embodiment. As shown in the figure, the processor 11 has the function of a warping processing unit F0. The warping processing unit F0 performs a process of warping the depth value obtained based on the output information of the ToF camera 2 to the image plane (grayscale image plane) of another camera 3.
[0054] The warping processing unit F0 has the functions of a first depth value warping processing unit F1, an occluded edge pixel identification processing unit F2, and a second depth value warping processing unit F3.
[0055] The first depth value warping processing unit F1 performs a warping process on the image plane of each depth pixel, where the first depth value is calculated based on the reflected light with the highest intensity. This is equivalent to performing superposition in the same way as in the conventional method, as superimposing the depth image onto the grayscale image. Here, the first depth value is determined by identifying the "time from emission to reception" with the highest frequency in the distance histogram, and is calculated based on the identified time from emission to reception and the speed of light.
[0056] The occluded edge pixel identification processing unit F2 identifies pixels corresponding to the edge of the occluded subject that is farther from the viewpoint of the ToF camera 2, based on the output information of the ToF camera 2, as occluded edge pixels. As mentioned earlier, an occluded subject refers to a subject that obstructs the line of sight from the viewpoint of the ToF camera 2 toward the position of the other camera 3. In the example of Figure 7A, the occluded edge pixel is the depth pixel that captures the subject point p3.
[0057] In this example, the occluded edge pixel identification processing unit F2 identifies occluded edge pixels as follows: First, based on the magnitude of the difference in the first depth value with an adjacent pixel, it identifies pixels corresponding to the depth boundary as depth boundary pixels. Then, for the identified depth boundary pixels, it determines whether or not the polarity of the difference in the first depth value with an adjacent pixel is a predetermined polarity, and uses this determination to determine whether or not they are occluded edge pixels.
[0058] The identification of depth boundary pixels is performed by sequentially selecting pixels on the depth image plane as processing pixels in a predetermined order, such as raster order. In this case, adjacent pixels are selected, for example, pixels adjacent to the right of the processing image (i.e., in this example, the direction from which the other camera 3 is located as seen from the ToF camera 2).
[0059] Whether or not a pixel is a depth boundary pixel is determined by checking whether the absolute value of the difference in the first depth value between it and an adjacent pixel is greater than or equal to a predetermined threshold. The threshold here can be determined, for example, according to the depth of the object assumed to be an occluding object. In this example, the difference in the first depth value between it and an adjacent pixel is calculated by subtracting the first depth value of the adjacent pixel from the first depth value of the pixel to be processed.
[0060] Figure 9 is an explanatory diagram of pixels identified as depth boundary pixels. In Figure 9A, two subjects that are convex towards the foreground (ToF camera 2 side) and are within the sensing range of ToF camera 2 are shown as examples: subject s1 located on the left side (i.e., opposite to the other camera 3) and subject s2 located on the right side (the other camera 3 side) when viewed from the depth viewpoint. Subject s1 is not an occluding subject, while subject s2 is an occluding subject.
[0061] Figure 9B schematically shows a depth image obtained when the ToF camera 2 performs sensing in the state shown in Figure 9A. As shown by the textured area in the figure, in the depth image, for each of the subjects s1 and s2, the inner surface, that is, the surface facing the optical axis extending from the depth viewpoint, is captured, as well as the top surface. Therefore, for the pixels at the inner edges of the top surface and the inner edges of the front surface of these subjects s1 and s2, the difference in the first depth value from adjacent pixels is not very large, and they are not identified as depth boundary pixels. On the other hand, for the pixels at the outer edges of the top surface and the outer edges of the front surface of subjects s1 and s2, shown by the thick lines in the figure, the difference in the first depth value from adjacent pixels is large, and they are identified as depth boundary pixels.
[0062] In the occluded edge pixel identification processing unit F2, the determination of whether an identified depth boundary pixel is an occluded edge pixel is made by determining whether the polarity of the difference in the first depth value with an adjacent pixel is a predetermined polarity determined by the positional relationship between the ToF camera 2 and the other camera 3. For example, in this example, if the adjacent pixel is a right-side adjacent pixel, and it is assumed that the difference in the first depth value between adjacent pixels is calculated by subtracting the first depth value of the adjacent pixel from the first depth value of the pixel to be processed, then if the other camera 3 is positioned to the right of the ToF camera 2, the "predetermined polarity" will be "negative polarity". On the other hand, under the same premise, if the other camera 3 is positioned to the left of the ToF camera 2, the "predetermined polarity" will be "positive polarity". Note that if the adjacent pixel is a left-side adjacent pixel, the "predetermined polarity" will be the opposite polarity to the above polarity. Similarly, if the difference in the first depth value between adjacent pixels is calculated by subtracting the first depth value of the pixel to be processed from the first depth value of the adjacent pixel, the "predetermined polarity" will also be the opposite polarity to the above polarity.
[0063] Furthermore, regarding the identification of depth boundary pixels and occluded edge pixels as described above, the difference in the first depth value between adjacent pixels can also be calculated using a differential filter. In other words, in this case, the value of the difference is the derivative calculated using a differential filter.
[0064] The second depth value warping processing unit F3 performs a warping operation on the image plane of the separate camera 3, which is a second depth value calculated based on the second intensity reflected light detected at the occluded edge pixel.
[0065] Figure 10 is an explanatory diagram of the peak values of reflected light intensity shown in the distance histogram at the occluded edge pixels. In the ToF method, since the light for distance measurement is shone with a certain degree of spread toward the subject, reflected light corresponding to both the near depth and the far depth can be observed at the depth boundary. Specifically, the near depth is observed as the reflected light with the highest intensity (first echo), and the far depth is observed as the reflected light with the second highest intensity (second echo). In the figure, the first peak P1 represents the peak value of the "time from emission to reception" corresponding to the reflected light with the highest intensity, and the second peak P2 represents the peak value of the "time from emission to reception" corresponding to the reflected light with the second highest intensity.
[0066] Here, the first peak P1 can be identified in the distance histogram as the "time from emission to reception" with the highest frequency. The second peak P2 can be identified in the distance histogram as the "time from emission to reception" with the highest frequency and above a predetermined value among the "time from emission to reception" after the frequency decreases from the first peak P1 and then begins to increase when scanning the frequency on the increasing side of the horizontal axis of "time from emission to reception".
[0067] Figure 11 illustrates how the second depth value of the occluded edge pixel is warped onto the image plane of another camera 3. Figure 11A shows the three-dimensional spatial coordinates of the subject points p1 to p4 mentioned above, and also illustrates the three-dimensional spatial coordinates of the subject point ps for the second depth value obtained at the occluded edge pixel. Since subject points p3 and ps are points determined based on the first and second depth values obtained at the same occluded edge pixel, both of these points lie on the straight line connecting the depth viewpoint and the occluded edge pixel. Subject point p3 corresponds to the depth on the near side, and subject point ps corresponds to the depth on the far side, so as shown in the figure, subject point p3 is located on the near side of the straight line, and subject point ps is located on the far side of the straight line.
[0068] Figure 11B schematically shows how each subject point shown in Figure 11A is projected onto the grayscale image plane. According to Figure 11B, the subject point ps based on the second depth value at the occluded edge pixel is warped to a position that fills in the missing region X exemplified in Figure 7B. In other words, the warping of the second depth value is performed so that the region where depth value is missing when only the first depth value is superimposed can be appropriately filled in.
[0069] <7. Processing Procedure> Referring to the flowchart in Figure 12, a specific example of a processing procedure that the processor 11 should execute to realize the superposition method as described above will be explained. Note that the processing shown in Figure 12 is executed by the processor 11 based on a program stored in a predetermined storage device such as ROM 12 or memory unit 19. Furthermore, it is assumed that the image processing device 1 has already received input of a histogram image from the ToF camera 2 and a grayscale image as an captured image from another camera 3 when the processing shown in Figure 12 is executed.
[0070] In Figure 12, in step S101, the processor 11 calculates a first depth value for each depth pixel. That is, for each depth pixel, it identifies the "time from emission to reception" which has the highest frequency in the distance histogram, and calculates a first depth value based on the identified "time from emission to reception" and the speed of light.
[0071] In step S102, following step S101, the processor 11 performs the superposition of the first depth value. Note that the details of the superposition of depth value have already been explained, so a redundant explanation will be avoided.
[0072] In step S103, following step S102, the processor 11 sets the identifier n of the depth pixel to be processed to "1".
[0073] In step S104, following step S103, the processor 11 determines whether the nth depth pixel is a depth boundary pixel. In other words, in this example, it determines whether the absolute value of the difference in the first depth value between it and an adjacent pixel is greater than or equal to a predetermined threshold.
[0074] In step S104, if the absolute value of the difference in the first depth value between adjacent pixels is greater than or equal to a predetermined threshold, and it is determined that the nth depth pixel is a depth boundary pixel, the processor 11 proceeds to step S105 to determine whether or not it is an occluded edge pixel. In other words, in this example, it is determined whether or not the polarity of the difference in the first depth value between adjacent pixels is a predetermined polarity.
[0075] In step S105, if the polarity of the difference in the first depth value between adjacent pixels is a predetermined polarity and it is determined that the nth depth pixel is an occluded edge pixel, the processor 11 proceeds to step S106 and calculates a second depth value for the nth depth pixel. That is, in the distance histogram of the nth depth pixel, the "time from emission to reception" corresponding to the second peak P2 mentioned above is identified, and the second depth value is calculated based on the identified "time from emission to reception" and the speed of light.
[0076] In step S107, following step S106, the processor 11 performs superposition processing of the second depth value. Note that the details of the superposition processing of the second depth value have already been explained, so a redundant explanation will be avoided.
[0077] The processor 11 proceeds to step S108 in accordance with having performed the superposition process in step S107.
[0078] Furthermore, if the processor 11 determines in step S104 that the nth depth pixel is not a depth boundary pixel, or if it determines in step S105 that it is not an occluded edge pixel, it skips steps S106 and S107 and proceeds to step S108.
[0079] In step S108, the processor 11 determines whether the identifier n is greater than or equal to the maximum value N. Here, the maximum value N represents the total number of depth pixels that are candidates for superposition processing.
[0080] In step S108, if it is determined that the identifier n is not greater than or equal to the maximum value N, the processor 11 proceeds to step S109, increments the identifier n by 1, and then returns to step S104. This allows for the determination of whether or not all depth pixels that are candidates for superposition processing are occluded edge pixels (S104 and S105), and for occluding the second depth value if they are occluded edge pixels (S106 and S107).
[0081] In step S108, the processor 11 determines that the identifier n is greater than or equal to the maximum value N, and then completes the series of processes shown in Figure 12.
[0082] <8. Modifications> The embodiments of this technology have been described above, but the embodiments are not limited to the specific examples described so far, and various modified configurations can be adopted. For example, in the above, an example was given in which the process of identifying occluded edge pixels, including the determination of whether or not they are depth boundary pixels, is performed for all depth pixels that are candidates for superposition processing, but it is also conceivable that the process of identifying occluded edge pixels is performed on pixels other than those corresponding to the background. In this case, the occluded edge pixel identification processing unit F2 identifies the pixels corresponding to the background among the depth pixels as background pixels based on the magnitude of the first depth value, and performs the process of identifying occluded edge pixels on depth pixels other than background pixels.
[0083] The flowchart in Figure 13 shows an example of a processing procedure corresponding to the superposition method, which is the first modified example that performs the process of identifying occluded edge pixels on pixels other than background pixels. In the following explanation, processes that are the same as those already explained will be given the same step number and their explanation will be omitted.
[0084] As can be seen by comparing with Figure 12 above, in the first modified example, the processes of steps S201 and S202 are added between steps S102 and S103.
[0085] In step S201, the processor 11 performs a process to identify background pixels based on the first depth value. The determination of whether or not a pixel is a background pixel can be made by determining whether or not the first depth value is greater than or equal to a predetermined background threshold. The background threshold can be determined according to the distance assumed to be the distance at which the background object exists.
[0086] In step S202, following step S201, the processor 11 sets depth pixels other than background pixels as depth pixels to be processed. As a result, the processing for identifying occluded edge pixels (S104 and S105) performed from step S104 onward will target only depth pixels other than background pixels.
[0087] Here, even if there is an occluding object in the background, the pixel area corresponding to the occluded region caused by that occluding object in the grayscale image is minute, so the loss of depth value can be ignored. According to the first modification described above, for occluding objects in the background where the loss of depth value can be ignored, the process of identifying occluded edge pixels is omitted, thus improving the efficiency of the process of identifying occluded edge pixels.
[0088] Furthermore, the above explanation was based on the assumption that the second peak P2 is always detected at occluded edge pixels. However, in reality, there may be cases where the second peak P2 cannot be detected at occluded edge pixels due to various factors.
[0089] Therefore, it is also possible to adopt the superposition method as a second modification, as described below. In this second modification, if the second peak P2 is not detected at the occluded edge pixel, that is, if the second intensity of reflected light is not detected, the second depth value at the occluded edge pixel is estimated by interpolation using the depth value calculated based on the second intensity of reflected light detected at the surrounding pixels of the occluded edge pixel.
[0090] The flowchart in Figure 14 shows an example of a processing procedure corresponding to the superposition method as a second modification. As can be seen by comparing it with Figure 12, in the second modification, the processing of step S301 is added between steps S105 and S106, and the processing of step S302 is also added.
[0091] Specifically, in this case, if the processor 11 determines in step S105 that it is an occluded edge pixel, it proceeds to step S301 to determine whether or not there is a second peak P2 in the nth depth pixel. Note that an example of the method for detecting the second peak P2 has already been explained, so a redundant explanation will be avoided.
[0092] If, in step S301, it is determined that there is a second peak P2 at the nth depth pixel, the processor 11 proceeds to step S106. The subsequent processing is as described in Figure 12.
[0093] On the other hand, if it is determined in step S301 that there is no second peak P2 at the nth depth pixel, the processor 11 proceeds to step S302 and estimates the second depth value of the nth depth pixel by interpolation processing based on the second peak values of the surrounding pixels. For example, targeting a pixel range of a predetermined number of pixels centered on the nth depth pixel, such as a 3x3 pixel range centered on the nth depth pixel, the processor detects the second peak P2 based on a distance histogram for each surrounding pixel of the nth depth pixel, and estimates the second depth value of the nth depth pixel by performing interpolation processing based on the "time from emission to reception" of the detected second peak P2. For example, for each of the surrounding pixels mentioned above, the "time from emission to reception" of the second peak P2 can be converted into a depth value using the speed of light, and the average value of these converted depth values can be obtained as the estimated second depth value of the nth depth pixel. It should be noted that other interpolation methods, such as bilinear interpolation, can also be used for the interpolation processing.
[0094] The processor 11 proceeds to step S107 in response to having performed the processing in step S302. This enables the warping process of the second depth value estimated from the surrounding pixels in step S302.
[0095] With the second modification described above, even if the second intensity of reflected light cannot be detected at an occluded edge pixel due to some factor, the second depth value of that occluded edge pixel can be estimated, and the estimated second depth value can be warped onto the grayscale image plane. Therefore, the effect of preventing loss of depth values in the occluded region can be improved.
[0096] In the above example, a dToF camera was used as the ToF camera, but this technology is also applicable to iToF (indirect ToF) cameras that perform light reception signal processing for distance measurement. In the iToF method, as in the dToF method, multiple pulses of light are emitted for one distance measurement. In the iToF method, a process is performed to determine the phase difference from emission to reception for each pulse of light emission. By generating a histogram of the multiple phase difference values obtained for one distance measurement, as in the dToF case, a second depth value can be calculated as a depth value based on the second reflected light intensity. Specifically, in the iToF method as well, the second depth value can be calculated based on the second peak value in the histogram.
[0097] Furthermore, while the above example illustrates the case where the ToF camera and another camera are arranged horizontally, it goes without saying that the superposition method (warping of the second depth value) related to this technology can also be applied when the ToF camera and another camera are arranged vertically.
[0098] Furthermore, while the above example shows that the image processing device 1 is the entity executing the warping processing unit F0, it is also conceivable that the warping processing unit F0 be executed by a device other than the image processing device 1, such as a separate camera 3. Additionally, while the example shows that the occluded edge pixel identification processing unit F2 and the second depth value warping processing unit F3 are executed within the same device, it is also conceivable that these two processes, the occluded edge pixel identification processing unit F2 and the second depth value warping processing unit F3, are executed by different devices.
[0099] <9. Summary of Embodiments> As described above, the image processing system (10) as an embodiment performs a warping process in which depth values obtained based on the output information of a ToF camera that performs light receiving signal processing for distance measurement by the ToF method for each two-dimensionally arranged pixel are warped to the image plane of another camera whose sensing range overlaps with that of the ToF camera. This process includes an occluded edge pixel identification process, in which pixels corresponding to the edge of an occluded object that obstructs the line of sight from the viewpoint of the ToF camera toward the position of the other camera are identified as occluded edge pixels based on the output information of the ToF camera, and a second depth value warping process, in which a second depth value, which is a depth value calculated based on the second intensity reflected light detected by the occluded edge pixels, is warped to the image plane of the other camera. In distance measurement by the ToF method, reflected light corresponding to both the near depth and the far depth can be observed at the depth boundary. Specifically, the depth on the near side is observed as the reflected light with the highest intensity, and the depth on the far side is observed as the reflected light with the second highest intensity. As described above, by identifying the occluded edge pixels corresponding to the edge of the occluded object that is farther from the ToF camera's viewpoint, and warping the second depth value based on the reflected light of the second highest intensity at the occluded edge pixels onto the image plane of another camera, it becomes possible to associate the depth value of the region occluded by the occluded object with the appropriate pixel position in the image captured by the other camera. Therefore, when superimposing the depth image onto the image captured by another camera, it is possible to prevent the loss of depth values in the occluded region due to the occluded object.
[0100] Furthermore, in the image processing system as an embodiment, during the warping process, a first depth value, which is a depth value calculated based on the reflected light with the highest intensity for each pixel, is warped onto the captured image plane. As a result, both the depth value of the portion visible to the ToF camera (first depth value) and the depth value of the occluded area due to the occluding object (second depth value) are warped onto the captured image plane. Therefore, both the depth value of the portion visible to the ToF camera and the depth value of the occluded area due to the occluding object can be superimposed on the captured image.
[0101] Furthermore, in the embodiment of the image processing system, when the depth value calculated based on the reflected light with the highest intensity is taken as the first depth value, the occluded edge pixel identification process identifies pixels corresponding to the depth boundary as depth boundary pixels based on the magnitude of the difference in the first depth value with adjacent pixels, and for the identified depth boundary pixels, it is determined whether or not they are occluded edge pixels by determining whether or not the polarity of the difference in the first depth value with adjacent pixels is a predetermined polarity. Pixels that can be occluded edge pixels are depth boundary pixels, and if the separation direction of another camera from the ToF camera is known, the polarity of the difference in the first depth value with adjacent pixels at the occluded edge pixels is also known. Therefore, by identifying depth boundary pixels as described above and determining whether or not the polarity of the difference in the first depth value with adjacent pixels is a predetermined polarity for the depth boundary pixels, occluded edge pixels can be appropriately identified.
[0102] Furthermore, in the embodiment of the image processing system, the ToF camera is configured as a dToF camera that performs light reception signal processing for distance measurement using the dToF method, and in the second depth value warping process, the second depth value is calculated based on the second peak value in the distance histogram output by the ToF camera. This makes it possible to appropriately calculate the second depth value in cases where the ToF camera is a dToF camera.
[0103] Furthermore, in the image processing system as an embodiment, when the depth value calculated based on the reflected light with the highest intensity is taken as the first depth value, the occluded edge pixel identification process identifies pixels that correspond to the background as background pixels based on the magnitude of the first depth value, and then performs a process to identify occluded edge pixels targeting pixels other than background pixels. Even if there is an occluding object in the background, the pixel area corresponding to the occluded region caused by the occluding object in the captured image is minute, so the loss of depth value can be ignored. With the above configuration, the occluded edge pixel identification process is omitted for occluding objects in the background where the loss of depth value can be ignored, thus improving the efficiency of the occluded edge pixel identification process.
[0104] Furthermore, in the image processing system as an embodiment, if the second depth value warping process does not detect reflected light of the second intensity at an occluded edge pixel, the second depth value at that occluded edge pixel is estimated by interpolation using a depth value calculated based on the second intensity reflected light detected at surrounding pixels of that occluded edge pixel. As a result, even if reflected light of the second intensity cannot be detected at an occluded edge pixel due to some factor, the second depth value of that occluded edge pixel can be estimated, and the estimated second depth value can be warped onto the image plane. Therefore, the effect of preventing loss of depth values in occluded regions can be improved.
[0105] The image processing method as an embodiment involves an image processing system that warps depth values obtained based on the output information of a ToF camera, which performs light reception signal processing for distance measurement using the ToF method for each pixel in a two-dimensional array, onto the image plane of another camera whose sensing range overlaps with that of the ToF camera. This warping process includes: an occluded edge pixel identification process, which identifies pixels corresponding to the edge of an occluded object that obstructs the line of sight from the ToF camera's viewpoint toward the location of the other camera, as occluded edge pixels, based on the output information of the ToF camera; and a second depth value warping process, which warps a second depth value, calculated based on the second intensity of reflected light detected by the occluded edge pixels, onto the image plane of the other camera. This image processing method can also obtain the same functions and effects as the image processing system as an embodiment described above.
[0106] Here, as an embodiment, we can consider a program that causes a CPU, DSP (Digital Signal Processor), or a device including these to execute the processing as an embodiment described with reference to Figure 12, etc. That is, the program of the embodiment is a program that can be read by a computer device and causes a computer device to execute the following: a warping process that warps depth values obtained based on the output information of a ToF camera that performs light receiving signal processing for distance measurement using the ToF method for each two-dimensionally arranged pixel, onto the image plane of another camera whose sensing range overlaps with that of the ToF camera; an occluded edge pixel identification process that identifies pixels corresponding to the edge of an occluded object that obstructs the line of sight from the viewpoint of the ToF camera toward the position of the other camera, on the side farther from the viewpoint, based on the output information of the ToF camera; and a second depth value warping process that warps a second depth value, which is a depth value calculated based on the second intensity of reflected light detected by the occluded edge pixel, onto the image plane of the other camera. Such a program allows the functions of the aforementioned occluded edge pixel identification processing unit F2 and second depth value warping processing unit F3 to be implemented in a computer device.
[0107] The programs described above can be pre-recorded on a storage medium built into a computer device, such as an HDD (Hard Disk Drive), or on ROM within a microcomputer with a CPU. Alternatively, they can be temporarily or permanently stored (recorded) on removable storage media such as flexible disks, CD-ROMs (Compact Disc Read Only Memory), MO (Magneto Optical) disks, DVDs (Digital Versatile Discs), Blu-ray Discs (registered trademark), magnetic disks, semiconductor memory, and memory cards. Such removable storage media can be provided as so-called packaged software. Furthermore, such programs can be installed from the removable storage media to a personal computer, or downloaded from a download site via a network such as a LAN (Local Area Network) or the Internet.
[0108] Furthermore, such a program is suitable for providing a wide range of functions for the occluded edge pixel identification processing unit and the second depth value warping processing unit of this disclosure. For example, by downloading the program to a personal computer, portable information processing device, mobile phone, game console, video equipment, PDA (Personal Digital Assistant), etc., the personal computer, etc. can be used as a device to implement the occluded edge pixel identification processing and the second depth value warping processing of this disclosure.
[0109] Furthermore, the effects described herein are merely illustrative and not limited to those described herein, and other effects may also occur.
[0110] <10. This Technology> This technology can also be configured as follows: (1) An image processing system that warps depth values obtained based on the output information of a ToF camera that performs light receiving signal processing for distance measurement by ToF method for each two-dimensionally arranged pixel to the image plane of another camera whose sensing range overlaps with that of the ToF camera, wherein the warping process includes: an occluded edge pixel identification process that identifies pixels corresponding to the edge of an occluded object that obstructs the line of sight from the viewpoint of the ToF camera toward the position of the other camera, on the side furthest from the viewpoint, as occluded edge pixels, and a second depth value warping process that warps a second depth value, which is a depth value calculated based on the second strongest reflected light detected by the occluded edge pixel, to the image plane of the other camera. (2) The image processing system according to (1) above, wherein in the warping process, for each of the pixels, a first depth value, which is a depth value calculated based on the strongest reflected light, is warped to the image plane. (3) The image processing system according to (1) or (2), wherein the first depth value is calculated based on the reflected light with the highest intensity, and in the occluded edge pixel identification process, the pixel corresponding to the depth boundary is identified as a depth boundary pixel based on the magnitude of the difference in the first depth value with an adjacent pixel, and for the identified depth boundary pixel, the determination of whether or not it is an occluded edge pixel is made as the determination of whether or not it is an occluded edge pixel. (4) The image processing system according to any one of (1) to (3), wherein the ToF camera is configured as a dToF camera that performs light reception signal processing for distance measurement using the dToF method, and in the second depth value warping process, the second depth value is calculated based on the second peak value in the distance histogram output by the ToF camera. (5) The image processing system according to any one of (1) to (4) above, in which the depth value calculated based on the reflected light with the highest intensity is taken as the first depth value, the occluded edge pixel identification process identifies pixels that correspond to the background among the pixels as background pixels based on the magnitude of the first depth value, and performs the process of identifying the occluded edge pixels with respect to the pixels other than the background pixels.(6) In the second depth value warping process, if the second intensity of reflected light is not detected at the occluded edge pixel, the image processing system according to any one of (1) to (5) above, wherein the second depth value at the occluded edge pixel is estimated by interpolation using a depth value calculated based on the second intensity of reflected light detected at the surrounding pixels of the occluded edge pixel. (7) An image processing method in which an image processing system warps depth values obtained based on output information of a ToF camera that performs light receiving signal processing for distance measurement by ToF method for each two-dimensionally arranged pixel to the image plane of another camera whose sensing range overlaps with that of the ToF camera, the warping process comprising: an occluded edge pixel identification process in which, based on the output information of the ToF camera, pixels corresponding to the edge of an occluded object that obstructs the line of sight from the viewpoint of the ToF camera toward the position of the other camera, on the side furthest from the viewpoint, are identified as occluded edge pixels; and a second depth value warping process in which a second depth value, which is a depth value calculated based on the second intensity reflected light detected by the occluded edge pixel, is warped to the image plane of the other camera. (8) The image processing method according to (7) in which, in the warping process, a first depth value, which is a depth value calculated based on the highest intensity reflected light, is warped to the image plane for each of the pixels. (9) The image processing method according to (7) or (8), wherein the first depth value is calculated based on the reflected light with the highest intensity, and in the occluded edge pixel identification process, the pixels corresponding to the depth boundary are identified as depth boundary pixels based on the magnitude of the difference in the first depth value with adjacent pixels, and for the identified depth boundary pixels, the determination of whether or not the polarity of the difference in the first depth value with adjacent pixels is a predetermined polarity is made as the determination of whether or not it is an occluded edge pixel. (10) The image processing method according to any one of (7) to (9), wherein the ToF camera is configured as a dToF camera that performs light reception signal processing for distance measurement by the dToF method, and in the second depth value warping process, the second depth value is calculated based on the second peak value in the distance histogram output by the ToF camera.(11) The image processing method according to any one of (7) to (10), wherein, when the depth value calculated based on the reflected light with the highest intensity is taken as the first depth value, the occluded edge pixel identification process identifies pixels that correspond to the background among the pixels based on the magnitude of the first depth value as background pixels, and performs a process to identify the occluded edge pixels with respect to the pixels other than the background pixels. (12) The image processing method according to any one of (7) to (11), wherein, in the second depth value warping process, if the reflected light of the second intensity is not detected at the occluded edge pixel, the second depth value at the occluded edge pixel is estimated by interpolation using a depth value calculated based on the reflected light of the second intensity detected at the surrounding pixels of the occluded edge pixel. (13) A program readable by a computer device, which causes the computer device to perform a warping process in which depth values obtained based on output information of a ToF camera that performs light receiving signal processing for distance measurement by ToF method for each two-dimensionally arranged pixel, are warped to the image plane of another camera whose sensing range overlaps with that of the ToF camera, the program comprising: an occluded edge pixel identification process which identifies, based on the output information of the ToF camera, the pixels corresponding to the edge of an occluded object that obstructs the line of sight from the viewpoint of the ToF camera toward the position of the other camera, on the side furthest from the viewpoint, as occluded edge pixels; and a second depth value warping process which warps a second depth value, which is a depth value calculated based on the second intensity reflected light detected by the occluded edge pixel, to the image plane of the other camera. (14) The program according to (13), wherein, in the warping process, a first depth value, which is a depth value calculated based on the highest intensity reflected light, is warped to the image plane for each of the pixels.(15) The program according to (13) or (14), wherein the first depth value is calculated based on the reflected light with the highest intensity, and in the occluded edge pixel identification process, the program identifies the pixel corresponding to the depth boundary as a depth boundary pixel based on the magnitude of the difference in the first depth value with an adjacent pixel, and for the identified depth boundary pixel, the program determines whether or not the polarity of the difference in the first depth value with an adjacent pixel is a predetermined polarity, and determines whether or not it is an occluded edge pixel. (16) The program according to any one of (13) to (15), wherein the ToF camera is configured as a dToF camera that performs light reception signal processing for distance measurement using the dToF method, and in the second depth value warping process, the program calculates the second depth value based on the second peak value in the distance histogram output by the ToF camera. (17) The program according to any one of (13) to (16) above, wherein the first depth value is calculated based on the reflected light with the highest intensity, and in the occluded edge pixel identification process, based on the magnitude of the first depth value, the program identifies the pixels that correspond to the background among the pixels as background pixels, and performs a process to identify the occluded edge pixels targeting the pixels other than the background pixels. (18) The program according to any one of (13) to (17) above, wherein in the second depth value warping process, if the reflected light of the second intensity is not detected at the occluded edge pixel, the program estimates the second depth value at the occluded edge pixel by interpolation using a depth value calculated based on the reflected light of the second intensity detected at the surrounding pixels of the occluded edge pixel.
[0111] 1 Image processing device 2 ToF camera 3 Separate camera 10 Image processing system 11 Processor 12 ROM 13 RAM 15 Input / Output Interface (I / F) 16 Input unit 17 Display unit 18 Audio output unit 19 Storage unit 20 Communication unit 21 Drive 22 Removable recording medium 23 Bus 25 Light receiving unit 26 Light emitting unit 27 Measurement unit 28 Histogram image generation unit p1 to p4, po Subject point F0 Warping processing unit F1 First depth value warping processing unit F2 Occlusion edge pixel identification processing unit F3 Second depth value warping processing unit P1 First peak P2 Second peak
Claims
1. An image processing system that warps depth values obtained based on output information of a ToF camera, which performs light receiving signal processing for distance measurement by the ToF method for each pixel in a two-dimensional array, onto the image plane of another camera whose sensing range overlaps with that of the ToF camera, comprising: an occluded edge pixel identification process that identifies pixels corresponding to the edge of an occluded object that obstructs the line of sight from the viewpoint of the ToF camera toward the position of the other camera, on the side furthest from the viewpoint, as occluded edge pixels, based on the output information of the ToF camera; and a second depth value warping process that warps a second depth value, which is a depth value calculated based on the second intensity reflected light detected by the occluded edge pixel, onto the image plane of the other camera.
2. The image processing system according to claim 1, wherein, in the warping process, for each pixel, a first depth value, which is a depth value calculated based on the reflected light with the highest intensity, is warped onto the image plane.
3. When the depth value calculated based on the reflected light with the highest intensity is taken as the first depth value, the occluded edge pixel identification process identifies the pixel corresponding to the depth boundary as a depth boundary pixel based on the magnitude of the difference in the first depth value with an adjacent pixel, and for the identified depth boundary pixel, the determination of whether or not it is an occluded edge pixel is made by determining whether or not the polarity of the difference in the first depth value with an adjacent pixel is a predetermined polarity.
4. The image processing system according to claim 1, wherein the ToF camera is configured as a dToF camera that performs light reception signal processing for distance measurement using the dToF method, and in the second depth value warping process, the second depth value is calculated based on the second peak value in the distance histogram output by the ToF camera.
5. The image processing system according to claim 1, wherein, when the depth value calculated based on the reflected light with the highest intensity is taken as the first depth value, the occluded edge pixel identification process identifies pixels that correspond to the background among the pixels as background pixels based on the magnitude of the first depth value, and performs a process to identify the occluded edge pixels targeting the pixels other than the background pixels.
6. The image processing system according to claim 1, wherein, in the second depth value warping process, if reflected light of the second intensity is not detected at the occluded edge pixel, the second depth value at the occluded edge pixel is estimated by interpolation using a depth value calculated based on the reflected light of the second intensity detected at the surrounding pixels of the occluded edge pixel.
7. An image processing method in which an image processing system warps depth values obtained based on output information of a ToF camera that performs light receiving signal processing for distance measurement by ToF method for each pixel in a two-dimensional array onto the image plane of another camera whose sensing range overlaps with that of the ToF camera, comprising: an occluded edge pixel identification process based on the output information of the ToF camera, in which pixels corresponding to the edge of an occluded object that obstructs the line of sight from the viewpoint of the ToF camera toward the position of the other camera are identified as occluded edge pixels; and a second depth value warping process in which a second depth value, which is a depth value calculated based on the second intensity reflected light detected by the occluded edge pixel, is warped onto the image plane of the other camera.
8. The image processing method according to claim 7, wherein, in the warping process, for each pixel, a first depth value, which is a depth value calculated based on the reflected light with the highest intensity, is warped onto the image plane.
9. The image processing method according to claim 7, in which, when the depth value calculated based on the reflected light with the highest intensity is taken as the first depth value, the occluded edge pixel identification process identifies the pixel corresponding to the depth boundary as a depth boundary pixel based on the magnitude of the difference in the first depth value with an adjacent pixel, and for the identified depth boundary pixel, the determination of whether or not it is an occluded edge pixel is made by determining whether or not the polarity of the difference in the first depth value with an adjacent pixel is a predetermined polarity.
10. The image processing method according to claim 7, wherein the ToF camera is configured as a dToF camera that performs light reception signal processing for distance measurement using the dToF method, and in the second depth value warping process, the second depth value is calculated based on the second peak value in the distance histogram output by the ToF camera.
11. The image processing method according to claim 7, wherein, when the depth value calculated based on the reflected light with the highest intensity is taken as the first depth value, the occluded edge pixel identification process identifies pixels that correspond to the background among the pixels as background pixels based on the magnitude of the first depth value, and performs the process of identifying the occluded edge pixels with respect to the pixels other than the background pixels.
12. The image processing method according to claim 7, wherein, in the second depth value warping process, if reflected light of the second intensity is not detected at the occluded edge pixel, the second depth value at the occluded edge pixel is estimated by interpolation using depth values calculated based on reflected light of the second intensity detected at surrounding pixels of the occluded edge pixel.
13. A program readable by a computer device, which causes the computer device to perform a warping process that warps depth values obtained based on output information of a ToF camera that performs light receiving signal processing for distance measurement by ToF method for each two-dimensionally arranged pixel, onto the image plane of another camera whose sensing range overlaps with that of the ToF camera, the warping process being: an occluded edge pixel identification process that identifies pixels corresponding to the edge of an occluded object that obstructs the line of sight from the viewpoint of the ToF camera toward the position of the other camera, on the side furthest from the viewpoint, as occluded edge pixels, and a second depth value warping process that warps a second depth value, which is a depth value calculated based on the second intensity reflected light detected by the occluded edge pixel, onto the image plane of the other camera.
14. The program according to claim 13, wherein, in the warping process, for each pixel, a first depth value, which is a depth value calculated based on the reflected light with the highest intensity, is warped onto the image plane.
15. The program according to claim 13, in which, when the depth value calculated based on the reflected light with the highest intensity is taken as the first depth value, the occluded edge pixel identification process identifies the pixel corresponding to the depth boundary as a depth boundary pixel based on the magnitude of the difference in the first depth value with an adjacent pixel, and for the identified depth boundary pixel, the program determines whether or not the polarity of the difference in the first depth value with an adjacent pixel is a predetermined polarity, thereby determining whether or not it is an occluded edge pixel.
16. The ToF camera is configured as a dToF camera that performs light reception signal processing for distance measurement using the dToF method, and in the second depth value warping process, the second depth value is calculated based on the second peak value in the distance histogram output by the ToF camera, according to claim 13.
17. The program according to claim 13, wherein, when the depth value calculated based on the reflected light with the highest intensity is taken as the first depth value, the occluded edge pixel identification process identifies pixels that correspond to the background among the pixels as background pixels based on the magnitude of the first depth value, and performs a process to identify the occluded edge pixels targeting the pixels other than the background pixels.
18. The program according to claim 13, wherein, in the second depth value warping process, if reflected light of the second intensity is not detected at the occluded edge pixel, the second depth value at the occluded edge pixel is estimated by interpolation using depth values calculated based on reflected light of the second intensity detected at surrounding pixels of the occluded edge pixel.
