Information processing device and method

Abstract

The present disclosure relates to an information processing device and a method that make it possible to estimate a depth at a shorter distance while suppressing an increase in load. The present disclosure executes: setting an offset of disparity of a stereo image; obtaining the disparity of the stereo image by applying the offset; and estimating the depth of a subject of the stereo image by using the disparity to which the offset has been applied. The present disclosure can be applied to, for example, an information processing device, an imaging device, an electronic device, an information processing method, a program, or the like.

Description

Information Processing Apparatus and Method ,

[0006] ,

[0005] ,

[0001] The present disclosure relates to an information processing apparatus and method, and more particularly to an information processing apparatus and method capable of estimating a depth in a closer distance while suppressing an increase in load.

[0002] Conventionally, various studies have been conducted on image processing techniques for generating a depth map from parallax images of two viewpoints on the left and right or multiple viewpoints. For example, as a framework of a passive sensing algorithm, a method has been proposed in which after calculating a matching cost, each makes a contrivance to leave highly reliable information, and finally outputs a depth map while making corrections (see Non-Patent Document 1). Also, in stereo matching, it has been proposed to calibrate the focus, angle of view, etc. of each camera in order to make the relative position and angle of view of the cameras common (see Non-Patent Document 2). By performing stereo matching after such calibration of two cameras, it becomes possible to estimate the depth of the photographed space. That is, the depth information is estimated using the parallax information of the two calibrated cameras.

[0003] The parallax information (disparity) of a stereo image is smaller for a subject farther away and becomes zero at infinity. Conversely, the disparity becomes larger as the subject is closer.

[0004] Rostam Affendi, Haidi Ibrahim, "Literature Survey on Stereo Vision Disparity Map Algorithms", 2016Zhengyou Zhang, "A Flexible New Technique for Camera Calibration", Technical Report MSR-TR-98-71, Aug. 13, 2008

[0005] However, generally, the larger the maximum disparity is, the larger the size of the image required for depth estimation becomes. Therefore, there has been a risk that the processing load of depth estimation increases as the depth estimation for a closer distance is performed.

[0006] This disclosure was made in light of these circumstances, and aims to enable the estimation of depth at closer distances while suppressing an increase in load.

[0007] One aspect of this technology is an information processing device comprising: an offset setting unit that sets an offset for the disparity of a stereo image; and a depth estimation unit that applies the offset to determine the disparity of the stereo image and estimates the depth of the subject in the stereo image using the disparity to which the offset has been applied.

[0008] One aspect of this technology is an information processing method that includes setting an offset for the disparity of a stereo image, determining the disparity of the stereo image by applying the offset, and estimating the depth of the subject in the stereo image using the disparity to which the offset has been applied.

[0009] An information processing device for another aspect of this technology includes an acquisition unit that acquires a stereo image, the disparity of the stereo image, and the offset of the disparity; a 3D conversion unit that estimates the depth of the subject in the stereo image using the acquired disparity and the offset and converts the stereo image into 3D data; and a 3D synthesis unit that synthesizes the obtained 3D data with other 3D data.

[0010] Another aspect of this technology is an information processing method that includes acquiring a stereo image, the disparity of the stereo image, and the offset of the disparity; estimating the depth of the subject in the stereo image using the acquired disparity and offset; converting the stereo image into 3D data; and synthesizing the obtained 3D data with other 3D data.

[0011] In one aspect of this technology, the information processing device and method include the following steps: setting an offset for the disparity of a stereo image; applying that offset to obtain the disparity of the stereo image; and using the disparity obtained by applying that offset to estimate the depth of the subject in the stereo image.

[0012] In other aspects of this technology, the information processing device and method include the following processes: obtaining a stereo image, the disparity of the stereo image, and the offset of the disparity; estimating the depth of the subject in the stereo image using the obtained disparity and offset; converting the stereo image into 3D data; and synthesizing the obtained 3D data with other 3D data.

[0013] This figure shows the relationship between subject distance and disparity. This figure shows an example of the depth estimation range. This figure shows an example of applying an offset to disparity. This is a block diagram showing an example of the main configuration of a depth map generation device. This is a flowchart explaining an example of the depth map generation process flow. This figure shows an example of calibration. This is a block diagram showing an example of the main configuration of an imaging device. This is a flowchart explaining an example of the calibration process flow. This figure shows an example of controlling the depth estimation range. This is a block diagram showing an example of the main configuration of an imaging device. This is a flowchart explaining an example of the imaging process flow. This is a block diagram showing an example of the main configuration of an imaging device. This is a block diagram showing an example of the main configuration of a 3D reconstruction device. This is a flowchart explaining an example of the imaging process flow. This is a flowchart explaining an example of the 3D reconstruction process flow. This figure shows an example of controlling the depth estimation range. This is a block diagram showing an example of the main configuration of a depth map generation device. This is a flowchart explaining an example of the depth map generation process flow. This is a block diagram showing an example of the main configuration of a computer.

[0014] The following describes the embodiments for implementing this disclosure. The description will be given in the following order: 1. Technical content and supporting literature for technical terms 2. Depth estimation using stereo images 3. First embodiment (Application of disparity offset in depth map generation) 4. Second embodiment (Application of disparity offset in calibration) 5. Third embodiment (Depth estimation range position control that follows the subject) 6. Fourth embodiment (3D synthesis when disparity offset is applied) 7. Fifth embodiment (Control of depth estimation range when disparity offset is applied) 8. Appendix

[0015] <1. Supporting Documents for Technical Content and Terminology> The scope disclosed in this technology includes not only the contents described in the embodiments, but also the contents described in the following non-patent documents that were publicly known at the time of filing, as well as the contents of other documents referenced in the following non-patent documents.

[0016] Non-patent document 1: (described above) Non-patent document 2: (described above)

[0017] In other words, the content described in the aforementioned non-patent literature, as well as the content of other documents referenced in the aforementioned non-patent literature, can also serve as a basis for determining the support requirements.

[0018] <2. Depth Estimation Using Stereo Images> <Disparity and Depth Estimation Range> Conventionally, various studies have been conducted on image processing techniques for generating depth maps from disparity images of two or more viewpoints. For example, Non-Patent Literature 1 proposed a framework for passive sensing algorithms in which, after calculating the matching cost, each user makes efforts to retain highly reliable information and makes corrections to ultimately output a depth map. Non-Patent Literature 2 proposed calibrating the focus and field of view of each camera in order to standardize the relative position and field of view of the cameras in stereo matching. By performing stereo matching after such calibration of two cameras, it becomes possible to estimate the depth of the captured space. In other words, depth information was estimated using the disparity information of the two calibrated cameras.

[0019] Here, "parallax" between images refers to the shift in the pixels where the same subject is located between images. In this specification, a pair of captured images having such parallax is also called a "stereo image." This stereo image is generated by multiple cameras that capture the same subject from different positions. The parallax between images is formed by the difference in position (baseline length) between the cameras. There can be any number of cameras, as long as there are multiple cameras. It may be a "two-lens camera" consisting of two cameras, or a "multi-lens camera" consisting of three or more cameras. In the case of a multi-lens camera, multiple stereo images may be generated.

[0020] Here, "number of eyes" refers to the number of regions that receive light from the subject. For example, "number of eyes" may refer to the number of light-receiving lenses on the light-receiving surface. In other words, this "number of eyes" indicates the number of captured images that are generated. In a "two-lens camera" or "multi-lens camera" that generates stereo images in this way, there may be any number of image sensors that receive light from the subject and convert it into photoelectric light. A single image sensor may receive light from a subject input through different light-receiving lenses in multiple regions. In other words, a stereo image may be generated by a single image sensor. Alternatively, an independent image sensor may be formed for each light-receiving lens. In other words, a stereo image may be generated by multiple image sensors.

[0021] Furthermore, in this specification, the parallax (pixel shift) between images in a stereo image as described above is also referred to as "parallax information" or "disparity." This disparity is obtained by performing matching (detection of the same subject) between images in a stereo image. In this specification, such matching between images in a stereo image is also referred to as "stereo matching." As shown in the table in Figure 1, the parallax information (disparity) of a stereo image is smaller the further away the subject is and larger the closer the subject is. In other words, disparity is smaller the longer the subject distance (distance from the camera to the subject) is and larger the shorter the subject distance is. For example, disparity is 0 when the subject distance is infinity and is at its maximum value when the subject distance is shortest. Utilizing these characteristics, the depth of the subject is estimated using disparity in a manner similar to triangulation.

[0022] Because of the relationship between disparity and subject distance as described above, if there is an upper limit (maximum value) to disparity, the depth of subjects closer than the subject distance corresponding to that maximum value cannot be estimated. In this specification, this upper limit (maximum value) of disparity is also referred to as maximum disparity (or DM). In other words, this maximum disparity (DM) can be said to correspond to the closest (shortest distance) of the range in which depth estimation is performed (also referred to as the depth estimation range). Therefore, in order to estimate the depth of closer subjects, it was necessary to increase this maximum disparity (DM). For example, if the maximum disparity (DM) is set to 120 pixels, the depth of subjects whose disparity is 120 pixels or less can only be estimated. In other words, in this case, the shortest distance in the depth estimation range is the subject distance corresponding to a disparity of 120 pixels. In the example in the table in Figure 1, the shortest distance is approximately 11.4 meters. In other words, depth estimation is possible when the subject distance is 11.4 meters or more (the gray area in the table in Figure 1). Therefore, in order to estimate the depth of a subject with a distance shorter than 11.4 meters, it was necessary to increase this maximum disparity (DM).

[0023] For example, consider the case shown in Figure 2, where a multi-lens camera 11 captures an image of an object 21, and the distance (depth) from the multi-lens camera 11 to the object 21 is estimated. If the depth estimation range is set to the range indicated by the double arrow 31 due to the setting of maximum disparity (DM), then it is difficult to estimate the depth of object 21 because it is not included in that depth estimation range. In other words, in order to estimate the depth of object 21, it was necessary to extend the depth estimation range toward the front in the depth direction from the multi-lens camera 11, as shown by the double arrow 32. To do this, it was necessary to increase the maximum disparity (DM) as described above.

[0024] However, the larger the maximum disparity (DM), the larger the image size that needed to be stored in memory and used for depth estimation. Therefore, the closer the depth estimation was performed, the greater the processing load (memory usage and computational load) of the depth estimation could become. In addition, for example, if the image size of the stereo image generated by a multi-camera was not large enough, or if the hardware specifications of the information processing unit that performs depth estimation were not high enough, it became difficult to sufficiently increase the maximum disparity (DM), and thus difficult to sufficiently expand the depth estimation range. In other words, it became difficult to perform depth estimation in a certain range on the near side in the depth direction.

[0025] <3. First Embodiment> <Application of Disparity Offset in Depth Map Generation> Therefore, the depth estimation range is controlled by applying a disparity offset. By applying a disparity offset, for example, as shown in Figure 3, the disparity at a predetermined shooting distance D in front of infinity can be set to "0". In other words, the depth estimation range can be shifted from the range indicated by the double arrow 31 to the range indicated by the double arrow 33. In this specification, this disparity offset may be referred to as "CALSHIFT".

[0026] By shifting the depth estimation range in this way, object 21 can be included in the depth estimation range, as shown in the example in Figure 3. This shift in the depth estimation range is achieved by applying an offset (CALSHIFT), so the increase in maximum disparity (DM) described above is unnecessary. In other words, it is possible to estimate the depth of closer objects while suppressing the increase in the processing load of depth estimation.

[0027] <Depth Map Generation Device> Figure 4 is a block diagram showing an example of the configuration of a depth map generation device, which is one embodiment of an information processing device to which this technology is applied. The depth map generation device 100 shown in Figure 4 estimates the depth of an object from an input stereo image, generates a depth map, and outputs it. At that time, the depth map generation device 100 sets a disparity offset (CALSHIFT) and applies that offset to the disparity to perform depth estimation.

[0028] As shown in Figure 4, the depth map generation device 100 has an offset setting unit 111 and a depth estimation unit 112.

[0029] The offset setting unit 111 performs processing related to setting the disparity offset. For example, the offset setting unit 111 may set the disparity offset (CALSHIFT) of the stereo image input to the depth map generation device 100. For example, the offset setting unit 111 may set an initial value for the disparity offset (CALSHIFT) and update the offset (CALSHIFT) as needed. The offset setting unit 111 may supply the set (or updated) offset (CALSHIFT) to the depth estimation unit 112.

[0030] The depth estimation unit 112 performs processing related to depth estimation. For example, the depth estimation unit 112 may acquire a stereo image supplied from outside the depth map generation device 100 (for example, a preceding processing unit). The depth estimation unit 112 may acquire a disparity offset (CALSHIFT) supplied from the offset setting unit 111. The depth estimation unit 112 may apply this offset to determine the disparity of the stereo image. The depth estimation unit 112 may use the disparity obtained by applying this offset to estimate the depth of the subject in the stereo image. The depth estimation unit 112 may generate a depth map as the depth estimation result and output it to outside the depth map generation device 100 (for example, a subsequent processing unit).

[0031] <Depth Map Generation Process Flow> An example of the depth map generation process flow performed by this depth map generation device 100 will be explained with reference to the flowchart in Figure 5.

[0032] When the depth map generation process starts, the offset setting unit 111 sets CALSHIFT to its initial value in step S101.

[0033] In step S102, the depth estimation unit 112 acquires a stereo image.

[0034] In step S103, the depth estimation unit 112 estimates the depth using CALSHIFT. That is, the depth estimation unit 112 determines the disparity of the stereo image using the CALSHIFT set in step S101 or step S105 described later, and estimates the depth of the subject in the stereo image using the disparity to which CALSHIFT has been applied.

[0035] In step S104, the depth estimation unit 112 determines whether the maximum disparity (DM) is appropriate. That is, the depth estimation unit 112 determines whether the desired subject for depth estimation is located within the depth estimation range. If the desired subject for depth estimation is located outside the depth estimation range and it is determined that the maximum disparity (DM) is inappropriate, the process proceeds to step S105.

[0036] In step S105, the offset setting unit 111 adjusts CALSHIFT. That is, the offset setting unit 111 updates CALSHIFT so that the desired subject for which depth estimation is to be estimated is located within the depth estimation range. In other words, the offset setting unit 111 adjusts CALSHIFT if the shortest distance (DM) of the depth estimation range based on disparity to which CALSHIFT is applied is not appropriate. For example, the offset setting unit 111 may increase the value of CALSHIFT when moving the depth estimation range toward the front in the depth direction. Conversely, the offset setting unit 111 may decrease the value of CALSHIFT when moving the depth estimation range toward the back in the depth direction.

[0037] Once the processing in step S105 is completed, the process returns to step S102 and executes the subsequent processes. In other words, each process from step S102 to step S105 is repeatedly executed until it is determined in step S104 that the maximum disparity (DM) is appropriate.

[0038] Then, if it is determined in step S104 that the maximum disparity (DM) is appropriate, the process proceeds to step S106.

[0039] In step S106, the depth estimation unit 112 generates a depth map as the depth estimation result and supplies (outputs) the depth map to the outside.

[0040] Once the processing in step S106 is complete, the depth map generation process ends.

[0041] By performing each process as described above, the depth map generation device 100 can apply an offset (CALSHIFT) to shift the depth estimation range while suppressing an increase in maximum disparity (DM). Therefore, the depth map generation device 100 can estimate the depth of closer distances while suppressing an increase in the processing load (memory usage and computational processing load) of depth estimation.

[0042] <Disparity Offset> In the above, an example was described in which the initial value and updated CALSHIFT are stored in the CALSHIFT value, which is treated as a single variable, with respect to the disparity offset. Embodiments of this disclosure are not limited to this, and an offset (initial value) for representing the disparity of a predetermined shooting distance D in front of infinity as "0" may be set as the "first offset," and an offset representing the change from the set first offset (i.e., the updated "CALSHIFT") may be set as the "second offset."

[0043] This second offset is a variable in which the first offset is not stored as a value. In other words, it is a variable set based on infinity corresponding to the first offset. In this case, since the shooting distance set for the first offset corresponds to the disparity 0 representing infinity, a value of 0 or more is stored in the second offset representing the change from the first offset. More specifically, when the depth estimation range is moved toward the front side in the depth direction, the magnitude of the value of the second offset always increases, and when the depth estimation range is moved toward the back side in the depth direction, the magnitude of the value of the second offset always decreases.

[0044] For example, in the flowchart (depth map generation process) of FIG. 5, the offset setting unit 111 may set the first offset in step S101 and set the second offset in step S105. Then, the depth estimation unit 112 may estimate the depth using the first offset or the second offset in step S103. That is, the offset setting unit 111 may set the initial value of the disparity offset as the first offset and set the second offset by adjusting the first offset. Then, when the second offset is not set, the depth estimation unit 112 estimates the depth of the subject using the disparity to which the first offset is applied, and when the second offset is set, the depth estimation unit 112 may estimate the depth of the subject using the disparity to which the second offset is applied. For example, in the flowchart of FIG. 5, the depth estimation unit 112 may estimate the depth using the first offset in the first processing of step S103 and estimate the depth using the second offset in the subsequent processing. The method for setting the second offset may be executed in the same manner as in the case of updating CALSHIFT (step S105) described above with reference to the flowchart of FIG. 5.

[0045] Note that the positive or negative of the second offset may be arbitrarily set, and it may be set so that a value of 0 or less is stored instead of a value of 0 or more.

[0046] As described above, by setting the first and second offsets, the control direction of the second offset always corresponds one-to-one with the movement direction of the depth estimation range, regardless of the initial value, thus enabling more efficient control of the depth estimation range.

[0047] The "first offset" may be referred to as "CALSHIFT," and the "second offset" as "CALOFFSET." Alternatively, the change from the first offset (for example, the difference between the initial value and the updated value of CALSHIFT) may be used as the second offset. In that case, in step S103 of the flowchart (depth map generation process) in Figure 5, the depth estimation unit 112 estimates the depth using the first offset and the second offset (for example, the sum of the first and second offsets). The second offset is initially set to "0" at the start of the depth map generation process, and in step S105, the offset setting unit 111 updates the second offset. The method for updating the second offset can be performed in the same way as the CALSHIFT update (step S105) described above, referring to the flowchart in Figure 5. The second offset can be any value that indicates the change from the first offset, and is not limited to the "difference" between the initial value and the updated value of CALSHIFT. For example, the coefficient multiplied by the initial value of CALSHIFT may be used as the second offset.

[0048] <4. Second Embodiment> <Application of Disparity Offset in Calibration>Note that this disparity offset (CALSHIFT) may be applied during camera calibration (setting of internal parameters). This calibration is generally performed using a calibration chart (hereinafter also referred to as a calibration chart). That is, the calibration chart is placed at a predetermined distance from the camera and imaged, and the calibration is performed using the captured image. In the case of a stereo image, since there is a disparity, calibration is performed by applying corrections as shown in Non-Patent Document 1 and Non-Patent Document 2. Therefore, there was a risk that the processing load (memory usage, calculation processing amount, processing time, etc.) of the calibration would increase.

[0049] In order to make the disparity of the calibration chart "0", it was necessary to place the calibration chart at an infinite object distance and image it. In that case, a huge calibration chart had to be used, which was unrealistic.

[0050] Therefore, the above-described disparity offset (CALSHIFT) is applied to make the disparity of the calibration chart "0". That is, as shown in FIG. 6, CALSHIFT is set so that the disparity at the shooting distance (object distance) where the calibration chart 121 is placed becomes "0". By doing so, calibration can be executed without the need for correction for disparity, and an increase in its processing load (memory usage, calculation processing amount, processing time, etc.) can be suppressed.

[0051] <Imaging Device>FIG. 7 is a block diagram showing an example of the configuration of an imaging device, which is an aspect of an information processing device to which this technology is applied. The imaging device 200 shown in FIG. 7 images an object to generate a stereo image. Further, the imaging device 200 performs calibration (setting of internal parameters) on the stereo image (captured image).

[0052] As shown in Figure 7, the imaging device 200 includes an imaging unit 211, a calibration unit 212, and a parameter storage unit 213.

[0053] The imaging unit 211 has a dual-lens camera or a multi-lens camera and performs processing related to imaging a subject using the camera. For example, the imaging unit 211 may image a subject and generate a stereo image. Alternatively, the imaging unit 211 may supply the generated stereo image to the calibration unit 212.

[0054] The calibration unit 212 performs calibration (setting internal parameters). For example, the calibration unit 212 may acquire a stereo image supplied from the imaging unit 211. The calibration unit 212 may perform calibration using the stereo image and set internal parameters. The calibration unit 212 may supply the set internal parameters to the parameter storage unit 213.

[0055] The parameter storage unit 213 has a storage medium and uses its storage area to perform processing related to the storage of internal parameters. For example, the parameter storage unit 213 may acquire internal parameters supplied from the calibration unit 212. The parameter storage unit 213 may store the acquired internal parameters.

[0056] The calibration unit 212 includes a focus setting unit 221, a depth map generation unit 222, and a lens distortion correction unit 223.

[0057] The focus setting unit 221 performs processing related to setting the focus position. For example, the focus setting unit 221 adjusts the focus of each image in the stereo image supplied from the imaging unit 211 to the calibration chart. In other words, the focus setting unit 221 sets the focus position for each camera of the imaging unit 211 to the distance on the calibration chart.

[0058] The depth map generation unit 222 is a processing unit similar to the depth map generation device 100 (Figure 4), has the same configuration as the depth map generation device 100, and performs the same processing. In other words, the depth map generation unit 222 has an offset setting unit 111 and a depth estimation unit 112 (Figure 4). Therefore, for example, the depth map generation unit 222 (offset setting unit 111) may set the disparity offset (CALSHIFT) of the stereo image supplied from the imaging unit 211. In this case, the depth map generation unit 222 (offset setting unit 111) may set the offset (CALSHIFT) to make the disparity of the distance in the calibration chart "0". Alternatively, the depth map generation unit 222 (depth estimation unit 112) may apply its offset (CALSHIFT) to determine the disparity of the stereo image supplied from the imaging unit 211, and use the disparity obtained by applying the offset (CALSHIFT) to estimate the depth of the calibration chart, which is the subject of the stereo image, and generate a depth map.

[0059] The lens distortion correction unit 223 performs processing related to correcting lens distortion. For example, the lens distortion correction unit 223 may set internal parameters to correct the lens distortion of the stereo image using the generated depth map. Alternatively, the lens distortion correction unit 223 may supply the set internal parameters to the parameter storage unit 213.

[0060] <Calibration Process Flow> An example of the calibration process flow performed by this imaging device 200 will be explained with reference to the flowchart in Figure 8.

[0061] When the calibration process is started, the imaging unit 211 captures an image of a calibration chart placed at a predetermined shooting distance in step S201 and generates a stereo image.

[0062] In step S202, the focus setting unit 221 uses the stereo image to adjust the focus position for each camera to the calibration chart.

[0063] In step S203, the depth map generation unit 222 estimates the depth by setting CALSHIFT to set the disparity of the calibration chart to 0. For example, the depth map generation unit 222 may perform the depth map generation process and estimate the depth in a flow similar to the example flowchart in Figure 5.

[0064] In step S204, the lens distortion correction unit 223 sets its internal parameters to correct lens distortion using its depth map.

[0065] In step S205, the parameter storage unit 213 may store its internal parameters.

[0066] The calibration process ends when step S205 is completed.

[0067] By performing each process in this manner, the imaging device 200 can perform calibration without requiring compensation for disparity, thereby suppressing an increase in its processing load (memory usage, computational processing load, processing time, etc.).

[0068] <5. Third Embodiment> <Depth Estimation Range Position Control that Tracks the Subject> For example, when generating a moving image as a stereo image, it is possible to capture a moving subject. If the subject of interest is moving, there is a risk that the subject may move outside the depth estimation range due to its movement. In that case, there is a risk that it will become impossible to estimate the depth of the subject of interest.

[0069] Therefore, the shift of the depth estimation range, which is performed by applying a disparity offset (CALSHIFT), may be controlled to follow the subject. For example, as shown in Figure 9, if the object 251 is moving, the position of the depth estimation range may be switched to match its movement, such as double arrows 261, 262, and 263. This switching of the depth estimation range position can be easily achieved by changing the value of the disparity offset (CALSHIFT), as described above. In this way, even if the movement range of the object 251 extends outside the depth estimation range, the position of the depth estimation range can be made to follow its movement. Therefore, the depth of the moving object 251 can be estimated more accurately as described above. In other words, the depth can be estimated not only at close range but also at long range, while suppressing an increase in the processing load of depth estimation.

[0070] In other words, by changing the value of the disparity offset (CALSHIFT), the range of depth estimation can be expanded. That is, compared to changing the value of the maximum disparity, the range of depth estimation can be expanded while suppressing the increase in processing load for depth estimation.

[0071] <Imaging Device> Figure 10 is a block diagram showing an example of the configuration of an imaging device, which is one aspect of an information processing device to which this technology is applied. The imaging device 300 shown in Figure 10 captures an image of a subject and generates a stereo image (moving image). The imaging device 300 also uses the stereo image to perform depth estimation and other operations to generate a depth map. At that time, the imaging device 300 detects the subject of interest in the stereo image and, as explained with reference to Figure 9, uses a disparity offset (CALSHIFT) to control the depth estimation range in accordance with the movement of the subject.

[0072] As shown in Figure 10, the imaging device 300 includes an imaging unit 311, a subject detection unit 312, a depth map generation unit 313, a display unit 314, a storage unit 315, and an output unit 316.

[0073] The imaging unit 311 has a dual-lens camera or a multi-lens camera and performs processing related to imaging a subject using the camera. For example, the imaging unit 311 may image a subject and generate a stereo image. Alternatively, the imaging unit 311 may supply the generated stereo image to the subject detection unit 312.

[0074] The subject detection unit 312 performs processing related to the detection of subjects included in the stereo image. For example, the subject detection unit 312 may acquire a stereo image supplied from the imaging unit 311. The subject detection unit 312 may perform image analysis on the stereo image and detect a subject of interest included in the stereo image. The subject detection unit 312 may supply the result of the subject detection and the stereo image to the depth map generation unit 313.

[0075] The depth map generation unit 313 is a processing unit similar to the depth map generation device 100 (Figure 4), has the same configuration as the depth map generation device 100, and performs the same processing. In other words, the depth map generation unit 313 has an offset setting unit 111 and a depth estimation unit 112 (Figure 4). Therefore, for example, the depth map generation unit 313 (offset setting unit 111) may set the disparity offset (CALSHIFT) of the stereo image supplied from the subject detection unit 312. In this case, the depth map generation unit 313 (offset setting unit 111) may set the offset (CALSHIFT) based on the subject detection result supplied from the subject detection unit 312 so that the subject is included in the depth estimation range (i.e., the subject is located within the specified depth range) in accordance with the movement of the detected subject of interest. Alternatively, the depth map generation unit 313 (depth estimation unit 112) may apply its offset (CALSHIFT) to obtain the disparity of the stereo image supplied from the subject detection unit 312, and use the disparity obtained by applying the offset (CALSHIFT) to estimate the depth of the detected subject of interest and generate a depth map.

[0076] Regarding the adjustment of the CALSHIFT value, it may be acceptable to set a certain threshold, for example, adjusting it when the shortest distance (DM) of the depth estimation range becomes 100, if it was 120. This ensures that depth estimation of the subject is always possible, even when there are limitations on available memory or output frame rate due to hardware constraints.

[0077] The depth map generation unit 313 may supply the generated depth map (a depth map including the estimated depth of the subject) and the stereo image to the display unit 314. The depth map generation unit 313 may supply the generated depth map (a depth map including the estimated depth of the subject) and the stereo image to the storage unit 315. The depth map generation unit 313 may supply the generated depth map (a depth map including the estimated depth of the subject) and the stereo image to the output unit 316.

[0078] The display unit 314 has a monitor (display device) and performs display-related processing. For example, the display unit 314 may acquire depth maps and stereo images supplied from the depth map generation unit 313. The display unit 314 may also display the acquired depth maps and stereo images on the monitor (display device) as appropriate.

[0079] The storage unit 315 has a storage medium and performs storage-related processing using its storage area. For example, the storage unit 315 may acquire depth maps or stereo images supplied from the depth map generation unit 313. The storage unit 315 may store the acquired depth maps or stereo images.

[0080] The output unit 316 performs processing related to the output of information. For example, the output unit 316 may acquire a depth map or stereo image supplied from the depth map generation unit 313. The output unit 316 may also output the acquired depth map or stereo image to the outside of the imaging device 300.

[0081] <Flow of Imaging Process> An example of the flow of the imaging process performed by this imaging device 300 will be explained with reference to the flowchart in Figure 11.

[0082] When the imaging process is started, the imaging unit 311 captures an image of the subject and generates a stereo image in step S301.

[0083] In step S302, the subject detection unit 312 performs image analysis on the stereo image and detects the subject of interest.

[0084] In step S303, the depth map generation unit 313 executes the depth map generation process described in the first embodiment with reference to the flowchart in Figure 5, and generates a depth map by setting CALSHIFT to include the detected subject in the depth estimation range. In other words, the depth map generation unit 313 generates a depth map by setting CALSHIFT in the same manner as the depth map generation device 100.

[0085] In step S304, the display unit 314 displays the generated depth map and stereo image.

[0086] In step S305, the storage unit 315 stores the generated depth map and stereo image.

[0087] In step S306, the output unit 316 outputs the generated depth map and stereo image.

[0088] In step S307, the imaging unit 311 determines whether or not to terminate the imaging process. If it is determined not to terminate, the process returns to step S301, and the subsequent processes are executed. In other words, each process from step S301 to step S307 is repeatedly executed frame by frame until it is determined that the imaging process has terminated.

[0089] Then, if it is determined in step S307 to terminate the imaging process, the imaging process is terminated. The criteria for determining whether or not to terminate the imaging process can be anything. For example, if the user performs an optional operation indicating the end of shooting, it may be determined that the imaging process is terminated. Also, if the subject is completely out of the field of view, it may be determined that the imaging process is terminated.

[0090] By doing so, even when the subject's movement extends beyond the depth estimation range, the imaging device 300 can adjust the position of the depth estimation range to track the subject's movement, thereby more accurately estimating the subject's depth. In other words, real-time depth output corresponding to moving subjects becomes possible.

[0091] In other words, the imaging device 300 can estimate depth not only at close range but also at long range while suppressing an increase in the processing load for depth estimation. In other words, the imaging device 300 can expand the range of depth estimation by changing the value of the disparity offset (CALSHIFT). That is, compared to changing the value of the maximum disparity, the imaging device 300 can expand the range of depth estimation while suppressing an increase in the processing load for depth estimation.

[0092] <Disparity Offset> In this embodiment as well, as in the first embodiment, the offset (initial value) for representing the disparity of a predetermined shooting distance D in front of infinity as "0" may be set as the "first offset," and the offset representing the change from the set first offset (i.e., the updated "CALSHIFT") may be set as the "second offset." Alternatively, the "first offset" may be called "CALSHIFT" and the "second offset" may be called "CALOFFSET." Alternatively, the change from the first offset may be set as the second offset.

[0093] <6. Fourth Embodiment> <3D Synthesis When Disparity Offset is Applied> In addition, for the imaging process of a moving subject as described in the third embodiment, 3D data may be generated for each frame as a post-processing step, and the 3D data of each frame may be synthesized. In that case, as described above, if a variable offset (CALSHIFT) is applied, there is a risk that the resulting disparity value will not be consistent between frames. For example, even if the disparity value is the same, if the applied offset (CALSHIFT) value is different, the depth (subject distance) corresponding to that disparity may be different. Therefore, for example, there is a risk that the depth of the same subject may shift between frames.

[0094] Therefore, the disparity of each frame is corrected according to the disparity offset (CALSHIFT) applied in each frame, enabling disparity consistency between frames. By doing so, the 3D data of each frame can be correctly combined, and more accurate combined 3D data can be generated.

[0095] For example, suppose that in the initial frame, DM is 120 pixels and the initial CALSHIFT (also called CALSHIFT0) is 100 pixels. Then, after several frames (after N frames), suppose CALSHIFT (CALSHIFT1) is changed to 120. The DM itself after N frames remains unchanged at 120, but from the perspective of the initial frame, this value of 120 becomes 120 + (CALSHIFT1 - CALSHIFT0), which in this example represents a position shift of 140 pixels from the initial frame. Therefore, by correcting the disparity according to CALSHIFT, it is possible to construct a space that is consistent in 3D space with the initial frame. By processing in this way, it becomes possible to reconstruct a 3D space in which objects moving in a space that is consistent from the initial frame to the final frame are detected.

[0096] <Imaging Device> Figure 12 is a block diagram showing an example of the configuration of an imaging device, which is one aspect of an information processing device to which this technology is applied. The imaging device 400 shown in Figure 12 is basically the same as the imaging device 300 (Figure 10), and captures an image of a subject to generate a stereo image (moving image). The imaging device 400 then uses the stereo image to perform depth estimation and the like. In this case, the imaging device 400, as with the imaging device 300, detects the subject of interest in the stereo image and, as explained with reference to Figure 9, uses a disparity offset (CALSHIFT) to control the depth estimation range in accordance with the movement of the subject.

[0097] However, in depth estimation, the imaging device 400 generates a detected disparity map instead of a depth map and stores or outputs it along with the offset (CALSHIFT) and stereo image. This information is used to generate 3D data in the 3D reconstruction device 450, which will be described later.

[0098] As shown in Figure 12, the imaging device 400 includes an imaging unit 411, a subject detection unit 412, a depth map generation unit 413, a display unit 414, a storage unit 415, and an output unit 416.

[0099] The imaging unit 411 has a dual-lens camera or a multi-lens camera and performs processing related to imaging a subject using the camera. For example, the imaging unit 411 may image a subject and generate a stereo image. Alternatively, the imaging unit 411 may supply the generated stereo image to the subject detection unit 412.

[0100] The subject detection unit 412 performs processing related to the detection of subjects included in the stereo image. For example, the subject detection unit 412 may acquire a stereo image supplied from the imaging unit 411. The subject detection unit 412 may perform image analysis on the stereo image and detect a subject of interest included in the stereo image. The subject detection unit 412 may supply the result of the subject detection and the stereo image to the depth map generation unit 413.

[0101] The depth map generation unit 413 is a processing unit similar to the depth map generation device 100 (Figure 4), has the same configuration as the depth map generation device 100, and performs the same processing. In other words, the depth map generation unit 413 has an offset setting unit 111 and a depth estimation unit 112 (Figure 4). Therefore, for example, the depth map generation unit 413 (offset setting unit 111) may set the disparity offset (CALSHIFT) of the stereo image supplied from the subject detection unit 312. In this case, the depth map generation unit 413 (offset setting unit 111) may set the offset (CALSHIFT) based on the subject detection result supplied from the subject detection unit 412 so that the subject is included in the depth estimation range (i.e., the subject is located within the specified depth range) in accordance with the movement of the detected subject of interest. Alternatively, the depth map generation unit 413 (depth estimation unit 112) may apply its offset (CALSHIFT) to determine the disparity of the stereo image supplied from the subject detection unit 412 and generate a disparity map.

[0102] Regarding the adjustment of the CALSHIFT value, it may be acceptable to set a certain threshold, for example, adjusting it when the shortest distance (DM) of the depth estimation range becomes 100, if it was 120. This ensures that depth estimation of the subject is always possible, even when there are limitations on available memory or output frame rate due to hardware constraints.

[0103] The depth map generation unit 413 may supply the generated disparity map (a disparity map including the estimated parallax of the subject), the stereo image, and the offset (CALSHIFT) to the display unit 414. The depth map generation unit 413 may supply the generated disparity map (a disparity map including the estimated parallax of the subject), the stereo image, and the offset (CALSHIFT) to the storage unit 415. The depth map generation unit 413 may supply the generated disparity map (a disparity map including the estimated parallax of the subject), the stereo image, and the offset (CALSHIFT) to the output unit 416.

[0104] The display unit 414 has a monitor (display device) and performs display-related processing. For example, the display unit 414 may acquire a disparity map, offset (CALSHIFT), and stereo image supplied from the depth map generation unit 413. The display unit 414 may also display the acquired disparity map, offset (CALSHIFT), and stereo image on the monitor (display device) as appropriate.

[0105] The storage unit 415 has a storage medium and performs storage-related processing using its storage area. For example, the storage unit 415 may acquire a disparity map, an offset (CALSHIFT), and a stereo image supplied from the depth map generation unit 413. The storage unit 415 may store the acquired disparity map, offset (CALSHIFT), and stereo image.

[0106] The output unit 416 performs processing related to the output of information. For example, the output unit 416 may acquire a disparity map, offset (CALSHIFT), and stereo image supplied from the depth map generation unit 413. The output unit 416 may output the acquired disparity map, offset (CALSHIFT), and stereo image to the outside of the imaging device 400.

[0107] <3D Reconstruction Apparatus> Figure 13 is a block diagram showing an example of the configuration of a 3D reconstruction apparatus, which is one embodiment of an information processing device to which this technology is applied. The 3D reconstruction apparatus 450 shown in Figure 13 converts stereo images into 3D data using the disparity map and offset (CALSHIFT) obtained in the imaging apparatus 400 (Figure 12). The 3D reconstruction apparatus 450 also synthesizes the 3D data of each frame to generate composite 3D data.

[0108] As shown in Figure 13, the 3D reconstruction device 450 includes an acquisition unit 461, a 3D conversion unit 462, a 3D synthesis unit 463, a display unit 464, a storage unit 465, and an output unit 466.

[0109] The acquisition unit 461 performs processing related to the acquisition of the disparity map, offset (CALSHIFT), and stereo image generated by the imaging device 400. For example, the acquisition unit 461 may acquire the disparity map, offset (CALSHIFT), and stereo image stored in a storage medium (e.g., removable media) by the storage unit 315 of the imaging device 400. The acquisition unit 461 may also acquire the disparity map, offset (CALSHIFT), and stereo image output by the output unit 316 of the imaging device 400. The acquisition unit 461 supplies the acquired information to the 3D conversion unit 462.

[0110] The 3D conversion unit 462 performs processing related to 3D conversion, converting 2D data into 3D data. For example, the 3D conversion unit 462 may acquire a disparity map, an offset (CALSHIFT), and a stereo image supplied from the acquisition unit 461. The 3D conversion unit 462 may correct the disparity using the offset (CALSHIFT), estimate the depth using the corrected disparity, and generate a depth map. The 3D conversion unit 462 may convert the stereo image (2D data) into 3D data using the generated depth map. The 3D conversion unit 462 may supply the converted 3D data to the 3D synthesis unit 463.

[0111] The 3D synthesis unit 463 performs processing related to the synthesis of 3D data. For example, the 3D synthesis unit 463 may acquire 3D data for each frame supplied from the 3D conversion unit 462. The 3D synthesis unit 463 may synthesize the 3D data of each frame and generate synthesized 3D data. At this time, by reading the viewpoint image of the central image, it is possible to add color information such as RGB to the data. The 3D synthesis unit 463 may supply the synthesized data to the display unit 464. The 3D synthesis unit 463 may supply the synthesized data to the storage unit 465. The 3D synthesis unit 463 may supply the synthesized data to the output unit 466.

[0112] The display unit 464 has a monitor (display device) and performs display-related processing. For example, the display unit 464 may acquire a disparity map, offset (CALSHIFT), and stereo image supplied from the 3D synthesis unit 463. The display unit 464 may also display the acquired disparity map, offset (CALSHIFT), and stereo image on the monitor (display device) as appropriate.

[0113] The storage unit 465 has a storage medium and performs storage-related processing using its storage area. For example, the storage unit 465 may acquire a disparity map, an offset (CALSHIFT), and a stereo image supplied from the 3D synthesis unit 463. The storage unit 465 may store the acquired disparity map, offset (CALSHIFT), and stereo image.

[0114] The output unit 466 performs processing related to the output of information. For example, the output unit 466 may acquire a disparity map, offset (CALSHIFT), and stereo image supplied from the 3D synthesis unit 463. The output unit 466 may output the acquired disparity map, offset (CALSHIFT), and stereo image to the outside of the 3D reconstruction device 450.

[0115] <Flow of Imaging Process> An example of the flow of the imaging process performed by this imaging device 400 will be explained with reference to the flowchart in Figure 14.

[0116] When the imaging process is started, the imaging unit 411 captures an image of the subject and generates a stereo image in step S401.

[0117] In step S402, the subject detection unit 412 performs image analysis on the stereo image and detects the subject of interest.

[0118] In step S403, the depth map generation unit 413 executes the depth map generation process described in the first embodiment with reference to the flowchart in Figure 5, and generates a disparity map by setting CALSHIFT to include the detected subject in the depth estimation range. In other words, the depth map generation unit 413 generates a disparity map by setting CALSHIFT in the same manner as the depth map generation device 100.

[0119] In step S404, the display unit 414 displays the generated disparity map and stereo image.

[0120] In step S405, the storage unit 415 stores the generated disparity map, CALSHIFT, and stereo image.

[0121] In step S406, the output unit 416 outputs the generated disparity map, CALSHIFT, and stereo image.

[0122] In step S407, the imaging unit 411 determines whether or not to terminate the imaging process. If it is determined not to terminate, the process returns to step S401, and the subsequent processes are executed. In other words, each process from step S401 to step S407 is repeatedly executed frame by frame until it is determined that the imaging process has terminated.

[0123] Then, if it is determined in step S407 to terminate the imaging process, the imaging process is terminated. The criteria for determining whether or not to terminate the imaging process can be anything. For example, if the user performs an optional operation indicating the end of shooting, it may be determined that the imaging process is terminated. Also, if the subject is completely out of the field of view, it may be determined that the imaging process is terminated.

[0124] <Flow of 3D Reconstruction Processing> An example of the flow of 3D reconstruction processing performed by the 3D reconstruction device 450 will be explained with reference to the flowchart in Figure 15.

[0125] When the 3D reconstruction process is started, the acquisition unit 461 acquires a disparity map, an offset (CALSHIFT), and a stereo image in step S451.

[0126] In step S452, the 3D conversion unit 462 converts the 2D data of the frame to be processed into 3D data. That is, the 3D conversion unit 462 uses the acquired disparity and offset (CALSHIFT) of the frame to be processed to convert the stereo image of the frame to be processed into 3D data.

[0127] In step S453, the 3D synthesis unit 463 synthesizes 3D data between frames to generate synthesized 3D data. For example, the 3D synthesis unit 463 synthesizes the obtained 3D data of the frame to be processed with the 3D data of the reference frame, or with synthesized 3D data which is the result of synthesizing the 3D data up to the present.

[0128] In step S454, the 3D compositing unit 463 determines whether all frames have been processed. If it is determined that there are unprocessed frames, the process returns to step S451 and subsequent processing is executed. If it is determined in step S454 that all frames have been processed, the process proceeds to step S455.

[0129] In step S455, the display unit 464 displays the generated composite 3D data.

[0130] In step S456, the storage unit 465 stores the generated composite 3D data.

[0131] In step S457, the output unit 466 outputs the generated composite 3D data.

[0132] When the process in step S457 is completed, the 3D reconstruction process is finished.

[0133] For example, let's set the CALSHIFT value at the reference frame to CALSHIFT0. Next, to perform 3D conversion of the recorded data from other frames, we load the recorded data from other frames. There are no particular rules at this time; depending on the function, the frames can be in ascending, descending, or random order. An example of loading the Nth frame is described below. The disparity map of the Nth frame and the CALSHIFT value at this time are treated as CALSHIFT_N, and the resulting disparity map is treated as disparity map_N with an offset of CALSHIFT_N - CALSHIFT0, and 3D conversion is performed based on disparity map_N. After that, data merging is performed with the reference frame. At this point, the reference frame and the Nth frame match in 3D space, so simply overlapping positions can be superimposed, or noise reduction can be performed. After that, by repeating until all frames are finished, data of an object moving in a wide-range 3D space using the data from all frames is reconstructed. After that, animation or CG compositing can be performed.

[0134] By performing each process in this manner, the imaging device 400 and the 3D reconstruction device 450 can correctly combine the 3D data of each frame and generate more accurate combined 3D data.

[0135] <Disparity Offset> In this embodiment as well, as in the first embodiment, the offset (initial value) for representing the disparity of a predetermined shooting distance D in front of infinity as "0" may be set as the "first offset," and the offset representing the change from the set first offset (i.e., the updated "CALSHIFT") may be set as the "second offset." Alternatively, the "first offset" may be called "CALSHIFT" and the "second offset" may be called "CALOFFSET." Alternatively, the change from the first offset may be set as the second offset.

[0136] <7. Fifth Embodiment> <Control of Depth Estimation Range When Applying Disparity Offset> In the above, we have explained how to control the position of the depth estimation range using a disparity offset (CALSHIFT), but we are not limited to this, and the width of the depth estimation range may also be controlled using a disparity offset (CALSHIFT).

[0137] In so-called passive sensing, which estimates depth information by detecting the pixel displacement of each camera, from the perspective of computational complexity, both real-time depth processing and offline processing tend to increase memory usage and decrease processing frame rates as DM increases. Therefore, minimizing DM as much as possible can reduce memory usage during computation and improve processing frame rates.

[0138] For example, in Figure 16, if the depth estimation range for the subject 481 is the range indicated by the double arrow 491, let the offset be CALSHIFT1, and the shortest distance within that depth estimation range be DM1. Also, if the depth estimation range for the subject 481 is the range indicated by the double arrow 492, let the offset be CALSHIFT2, and the shortest distance within that depth estimation range be DM2. In this case, since CALSHIFT1 < CALSHIFT2, DM1 > DM2 holds true. In other words, by setting the depth estimation range for the subject 481 to the range indicated by the double arrow 492, the value of DM can be made smaller. Therefore, it is possible to estimate the depth of a shorter distance while suppressing an increase in the load.

[0139] <Depth Map Generation Device> Figure 17 is a block diagram showing an example of the configuration of a depth map generation device, which is one embodiment of an information processing device to which this technology is applied. The depth map generation device 500 shown in Figure 17 estimates the depth of an object for an input stereo image, generates a depth map, and outputs it. At that time, the depth map generation device 500 sets a disparity offset (CALSHIFT), sets DM based on that offset, and performs depth estimation by applying that offset and DM to the disparity.

[0140] As shown in Figure 17, the depth map generation device 500 includes an offset setting unit 511, a DM setting unit 512, and a depth estimation unit 513.

[0141] The offset setting unit 511 performs processing related to setting the disparity offset. For example, the offset setting unit 511 may set the disparity offset (CALSHIFT) of the stereo image input to the depth map generation device 500. After determining the initial DM value for a static subject, the offset setting unit 511 changes the offset (CALSHIFT) and supplies the changed CALSHIFT value (let's call it CALSHIFT_N) to the DM setting unit 512. The offset setting unit 511 may also supply the set offset (CALSHIFT) to the DM setting unit 512.

[0142] The DM setting unit 512 performs processing related to the setting of DM. For example, the DM setting unit 512 may obtain an offset (CALSHIFT) supplied from the offset setting unit 511. The DM setting unit 512 may set DM using the obtained offset (CALSHIFT). For example, the DM setting unit 512 decreases or increases the DM value by referring to the difference between the initial CALSHIFT value (= CALSHIFT0) and the obtained CALSHIFT_N, and determines the DM_N value. The DM setting unit 512 may supply the offset (CALSHIFT) and the set DM to the depth estimation unit 513.

[0143] The depth estimation unit 513 performs processing related to depth estimation. For example, the depth estimation unit 513 may acquire a stereo image supplied from outside the depth map generation device 500 (for example, a preceding processing unit). The depth estimation unit 513 may acquire a disparity offset (CALSHIFT) and DM supplied from the DM setting unit 512. The depth estimation unit 513 may apply the offset to determine the disparity of the stereo image. The depth estimation unit 513 may use the disparity obtained by applying the offset to estimate the depth of the subject in the stereo image. In other words, the depth estimation unit 513 can determine the DM value by performing calculations with DM as DM_N and CALISHIFT value as CALISHIFT0. The depth estimation unit 513 may generate a depth map as the depth estimation result and output it to outside the depth map generation device 500 (for example, a subsequent processing unit).

[0144] <Depth Map Generation Process Flow> An example of the depth map generation process flow performed by the depth map generation device 500 will be explained with reference to the flowchart in Figure 18.

[0145] When the depth map generation process starts, in step S501, the offset setting unit 511 sets the offset (CALSHIFT) to an initial value. For example, the offset setting unit 511 changes the CALISHIFT value to generate CALISHIFT_N. The initial value set here can be anything, but for simplicity, it is set to the initial CALISHIFT.

[0146] In step S502, the DM setting unit 512 sets DM based on its offset (CALSHIFT). That is, the DM setting unit 512 uses CALISHIFT_N and the value of CALISHIFT to change the maximum disparity detection range to DM_N.

[0147] In step S503, the depth estimation unit 513 acquires a stereo image.

[0148] In step S504, the depth estimation unit 513 estimates the depth of the stereo image using the set offset (CALSHIFT) and DM.

[0149] In step S505, the depth estimation unit 513 determines whether the set offset (CALSHIFT) is appropriate. The depth estimation unit 513 performs depth calculation using DB_N and determines whether the initial value is appropriate. There are many ways to make this determination, but for example, the degree of dispersion of depth within a certain range may be used as an evaluation axis. If it is determined that it is not appropriate, the process proceeds to step S506.

[0150] In step S506, the offset setting unit 511 adjusts CALSHIFT. Once the processing in step S506 is complete, the process returns to step S502, and the subsequent processing is repeatedly executed. In other words, the DM setting unit 512 sets DM using the adjusted offset. The depth estimation unit 513 estimates the depth of the subject within a depth estimation range where the set DM value is the shortest distance.

[0151] If it is determined in step S505 that the offset (CALSHIFT) is appropriate, the process proceeds to step S507.

[0152] In step S507, the depth estimation unit 513 supplies the generated depth map to the outside of the depth map generation device 500.

[0153] In step S508, the depth estimation unit 513 determines whether or not to terminate the process. If it is determined not to terminate, the process returns to step S504, and the subsequent processes are repeatedly executed. That is, for example, when the depth estimation range becomes appropriate (i.e., when CALISHIFT_N and DB_N become appropriate values), the depth estimation using DB_N is repeatedly executed without changing CALISHIFT_N and DB_N until the process is terminated. If the depth estimation range becomes inappropriate during this repetition, the CALSHIFT may be adjusted and the DM set for the adjusted CALSHIFT may be performed. That is, if it is determined in step S505 that CALISHIFT is inappropriate, the process may return to step S502 after adjusting CALSHIFT in step S506.

[0154] Furthermore, if it is determined in step S508 that the depth map generation process should be terminated, the depth map generation process will be terminated.

[0155] By performing each process as described above, the depth map generation device 500 can estimate the depth of shorter distances while suppressing an increase in the processing load for depth estimation.

[0156] <Disparity Offset> In this embodiment as well, as in the first embodiment, the offset (initial value) for representing the disparity of a predetermined shooting distance D in front of infinity as "0" may be set as the "first offset," and the offset representing the change from the set first offset (i.e., the updated "CALSHIFT") may be set as the "second offset." Alternatively, the "first offset" may be called "CALSHIFT" and the "second offset" may be called "CALOFFSET." Alternatively, the change from the first offset may be set as the second offset.

[0157] <8. Addendum> <Computer> The series of processes described above can be executed by hardware or by software. When the series of processes are executed by software, the programs that make up the software are installed on a computer. Here, a computer includes computers built into dedicated hardware, as well as general-purpose personal computers, for example, that can perform various functions by installing various programs.

[0158] Figure 19 is a block diagram showing an example of the hardware configuration of a computer that executes the series of processes described above using a program.

[0159] In the computer 900 shown in Figure 19, the CPU (Central Processing Unit) 901, ROM (Read Only Memory) 902, and RAM (Random Access Memory) 903 are interconnected via a bus 904.

[0160] An input / output interface 910 is also connected to the bus 904. An input / output interface 910 is connected to an input unit 911, an output unit 912, a storage unit 913, a communication unit 914, and a drive 915.

[0161] The input unit 911 consists of, for example, a keyboard, mouse, microphone, touch panel, and input terminals. The output unit 912 consists of, for example, a display, speaker, and output terminals. The storage unit 913 consists of, for example, a hard disk, RAM disk, and non-volatile memory. The communication unit 914 consists of, for example, a network interface. The drive 915 drives removable media 921 such as a magnetic disk, optical disk, magneto-optical disk, or semiconductor memory.

[0162] In a computer configured as described above, the CPU 901 loads, for example, a program stored in the memory unit 913 into the RAM 903 via the input / output interface 910 and the bus 904, and executes it, thereby performing the series of processes described above. The RAM 903 also appropriately stores data necessary for the CPU 901 to perform various processes.

[0163] The program executed by the computer can be recorded and applied, for example, on removable media 921 such as a package medium. In this case, the program can be installed in the storage unit 913 via the input / output interface 910 by inserting the removable media 921 into the drive 915.

[0164] Furthermore, this program can also be provided via wired or wireless transmission media such as a local area network, the internet, or digital satellite broadcasting. In that case, the program can be received by the communication unit 914 and installed in the storage unit 913.

[0165] In addition, this program can be pre-installed in ROM 902 or memory unit 913.

[0166] <Applications of this technology> This technology can be applied to any configuration. For example, this technology can be applied to various electronic devices.

[0167] Furthermore, this technology can also be implemented as part of a device, such as a processor as a system LSI (Large Scale Integration) (e.g., a video processor), a module using multiple processors (e.g., a video module), a unit using multiple modules (e.g., a video unit), or a set with additional functions added to a unit (e.g., a video set).

[0168] Furthermore, this technology can also be applied to network systems composed of multiple devices. For example, this technology may be implemented as cloud computing, where multiple devices share and collaborate on processing via a network. For example, this technology may be implemented in a cloud service that provides image (video) related services to any terminal such as computers, AV (Audio Visual) equipment, portable information processing terminals, and IoT (Internet of Things) devices.

[0169] In this specification, a system refers to a collection of multiple components (devices, modules (parts), etc.), regardless of whether all components are located in the same enclosure. Therefore, multiple devices housed in separate enclosures and connected via a network, and a single device containing multiple modules within a single enclosure, are both considered systems.

[0170] <Applicable Fields and Applications of This Technology> Systems, devices, and processing units incorporating this technology can be used in any field, such as transportation, medical care, security, agriculture, livestock farming, mining, beauty, factories, home appliances, weather, and nature monitoring. Furthermore, the applications are entirely arbitrary.

[0171] <Other> In this specification, "flag" refers to information used to identify multiple states, and includes not only information used to identify two states, true (1) or false (0), but also information capable of identifying three or more states. Therefore, the values ​​that this "flag" can take are, for example, two values, 1 / 0, or three or more values. In other words, the number of bits that constitute this "flag" is arbitrary, and can be 1 bit or multiple bits. Furthermore, identification information (including flags) is envisioned not only in the form of including the identification information itself in the bitstream, but also in the form of including difference information of the identification information relative to a certain reference information in the bitstream. Therefore, in this specification, "flag" and "identification information" include not only the information itself, but also difference information relative to the reference information.

[0172] Furthermore, various types of information (metadata, etc.) related to encoded data (bitstream) may be transmitted or recorded in any form as long as they are associated with the encoded data. Here, the term "associate" means, for example, making it possible to use (link) one data when processing the other. In other words, associated data may be combined into a single data, or they may be individual data. For example, information associated with encoded data (image) may be transmitted on a different transmission path than the encoded data (image). Also, for example, information associated with encoded data (image) may be recorded on a different recording medium (or a different recording area on the same recording medium) than the encoded data (image). Note that this "association" may not apply to the entire data, but only to a part of it. For example, an image and the information corresponding to that image may be associated with each other in any unit, such as multiple frames, one frame, or a part within a frame.

[0173] In this specification, terms such as "combine," "multiplex," "add," "integrate," "include," "store," "insert," "insert," and "place" mean combining multiple things into one, such as combining encoded data and metadata into a single data, and represent one method of "associating" as described above.

[0174] Furthermore, the embodiments of this technology are not limited to those described above, and various modifications are possible without departing from the gist of this technology.

[0175] For example, the configuration described as a single device (or processing unit) may be divided and configured as multiple devices (or processing units). Conversely, the configurations described above as multiple devices (or processing units) may be combined and configured as a single device (or processing unit). Furthermore, it is also possible to add configurations other than those described above to the configuration of each device (or each processing unit). In addition, if the overall system configuration and operation are substantially the same, a part of the configuration of one device (or processing unit) may be included in the configuration of another device (or other processing unit).

[0176] Furthermore, for example, the program described above may be executed on any device. In that case, the device should have the necessary functions (such as functional blocks) and be able to obtain the necessary information.

[0177] Furthermore, for example, each step of a flowchart may be executed by one device, or it may be divided among multiple devices. Additionally, if a single step includes multiple processes, these processes may be executed by one device, or they may be divided among multiple devices. In other words, multiple processes included in a single step can be executed as multiple steps. Conversely, processes described as multiple steps can be combined and executed as a single step.

[0178] Furthermore, for example, a program executed by a computer may be structured so that the steps of the program are executed chronologically in the order described herein, or they may be executed in parallel or individually at necessary times, such as when a call is made. In other words, the steps may be executed in an order different from the order described above, as long as no inconsistencies arise. Moreover, the steps of this program may be executed in parallel with the processing of other programs, or in combination with the processing of other programs.

[0179] Furthermore, for example, multiple technologies relating to this technology can be implemented independently, as long as they do not create a contradiction. Of course, any multiple technologies can also be implemented in combination. For example, some or all of the technologies described in one embodiment can be implemented in combination with some or all of the technologies described in another embodiment. Also, some or all of the above-mentioned technologies can be implemented in combination with other technologies not mentioned above.

[0180] Furthermore, this technology can also be configured as follows: (1) An information processing device comprising: an offset setting unit that sets an offset for the disparity of a stereo image; and a depth estimation unit that determines the disparity of the stereo image by applying the offset and estimates the depth of the subject in the stereo image using the disparity to which the offset has been applied. (2) The information processing device according to (1), wherein the offset setting unit adjusts the offset if the shortest distance of the depth estimation range based on the disparity to which the offset has been applied is not appropriate. (3) The information processing device according to (2), wherein the offset setting unit increases the value of the offset when moving the depth estimation range toward the front in the depth direction, and decreases the value of the offset when moving the depth estimation range toward the back in the depth direction. (4) The information processing apparatus according to (2) or (3), wherein the offset setting unit sets the initial value of the offset as the first offset, and the depth estimation unit sets the second offset by adjusting the first offset, and if the second offset is not set, estimates the depth of the subject using the disparity to which the first offset has been applied, and if the second offset is set, estimates the depth of the subject using the disparity to which the second offset has been applied.(5) The information processing apparatus according to any one of (1) to (4), further comprising: an imaging unit that images the subject and generates the stereo image; a focus setting unit that sets the focus position of the stereo image; and a lens distortion correction unit that corrects the lens distortion of the stereo image, wherein the imaging unit generates the stereo image by imaging, with a calibration chart placed at a predetermined distance as the subject; the focus setting unit sets the focus position of the stereo image to the distance of the calibration chart; the offset setting unit sets the offset so that the disparity of the distance of the calibration chart is "0"; the depth estimation unit applies the offset to obtain the disparity of the stereo image, and uses the disparity obtained by applying the offset to estimate the depth of the calibration chart, which is the subject of the stereo image; and the lens distortion correction unit uses the result of the estimation to set an internal parameter for correcting the lens distortion. (6) An information processing device according to any one of (1) to (5), further comprising: an imaging unit that images the subject and generates the stereo image; and a subject detection unit that detects the subject included in the stereo image, wherein the offset setting unit sets the offset so that the subject is included in the depth estimation range based on the disparity to which the offset has been applied; and the depth estimation unit determines the disparity of the stereo image by applying the offset, and estimates the depth of the subject using the disparity to which the offset has been applied. (7) An information processing device according to (6), further comprising: a storage unit that stores a depth map including the estimated depth of the subject. (8) An information processing device according to (6) or (7), further comprising: an output unit that outputs a depth map including the estimated depth of the subject. (9) An information processing device according to any one of (6) to (8), further comprising: an imaging unit that images the subject and generates the stereo image; and a subject detection unit that detects the subject included in the stereo image, wherein the offset setting unit sets the offset so that the subject is included in the depth estimation range based on the disparity to which the offset has been applied; and a depth estimation unit that determines the disparity of the stereo image by applying the offset, and estimates the depth of the subject using the disparity to which the offset has been applied.(10) An information processing device according to any one of (1) to (9), further comprising: an imaging unit that images the subject and generates the stereo image; a subject detection unit that detects the subject included in the stereo image; a storage unit that stores the stereo image, the disparity of the stereo image, and the offset, wherein the offset setting unit sets the offset so that the subject is included in the depth estimation range based on the disparity to which the offset has been applied; and the depth estimation unit determines the disparity of the stereo image by applying the offset. (11) An information processing device according to any one of (1) to (10), further comprising: an imaging unit that images the subject and generates the stereo image; a subject detection unit that detects the subject included in the stereo image; and an output unit that outputs the stereo image, the disparity of the stereo image, and the offset, wherein the offset setting unit sets the offset so that the subject is included in the depth estimation range based on the disparity to which the offset has been applied; and the depth estimation unit determines the disparity of the stereo image by applying the offset. (12) The information processing device according to any one of (1) to (11), further comprising a DM setting unit that sets a parameter indicating the shortest distance of the depth estimation range using the offset, wherein the depth estimation unit estimates the depth of the subject in the depth estimation range where the value of the set parameter is the shortest distance. (13) The information processing device according to (12), wherein if the offset is not appropriate, the offset setting unit adjusts the offset, the DM setting unit sets the parameter using the adjusted offset, and the depth estimation unit estimates the depth of the subject in the depth estimation range where the value of the set parameter is the shortest distance. (14) An information processing method comprising setting an offset for the disparity of a stereo image, obtaining the disparity of the stereo image by applying the offset, and estimating the depth of the subject of the stereo image using the disparity to which the offset has been applied.(15) A program for causing a computer to perform a process that includes setting an offset for the disparity of a stereo image, applying the offset to obtain the disparity of the stereo image, and using the disparity obtained by applying the offset to estimate the depth of the subject in the stereo image.

[0181] (21) An information processing apparatus comprising: an acquisition unit that acquires a stereo image, the disparity of the stereo image, and the offset of the disparity; a 3D conversion unit that estimates the depth of the subject in the stereo image using the acquired disparity and the offset, and converts the stereo image into 3D data; and a 3D synthesis unit that synthesizes the obtained 3D data with other 3D data. (22) The information processing apparatus according to (21), wherein the acquisition unit acquires the stereo image, the disparity and the offset for each frame; the 3D conversion unit converts the stereo image of the processing target frame into 3D data using the acquired disparity and the offset of the processing target frame; and the 3D synthesis unit synthesizes the obtained 3D data of the processing target frame with the 3D data of a reference frame, or with synthesized 3D data which is the result of synthesizing the 3D data up to the present. (23) The information processing apparatus according to (22), further comprising a storage unit that stores the synthesized 3D data after synthesizing the 3D data of the processing target frame. (24) The information processing apparatus according to (22) or (23), further comprising an output unit that outputs the synthesized 3D data obtained by combining the 3D data of the frame to be processed. (25) The information processing apparatus according to any one of (22) to (24), further comprising a display unit that displays the synthesized 3D data obtained by combining the 3D data of the frame to be processed. (26) An information processing method comprising: acquiring a stereo image, the disparity of the stereo image, and the offset of the disparity; estimating the depth of the subject in the stereo image using the acquired disparity and the offset, converting the stereo image into 3D data; and combining the obtained 3D data with other 3D data.(27) A program for causing a computer to perform a process that includes obtaining a stereo image, the disparity of the stereo image, and the offset of the disparity; estimating the depth of the subject in the stereo image using the obtained disparity and offset, converting the stereo image into 3D data; and synthesizing the obtained 3D data with other 3D data.

[0182] 100 Depth map generation device, 111 Offset setting unit, 112 Depth estimation unit, 200 Imaging device, 211 Imaging unit, 212 Calibration unit, 213 Parameter storage unit, 221 Focus setting unit, 222 Depth map generation unit, 223 Lens distortion correction unit, 300 Imaging device, 311 Imaging unit, 312 Subject detection unit, 313 Depth map generation unit, 314 Display unit, 315 Storage unit, 316 Output unit, 400 Imaging device, 411 Imaging unit, 412 Subject detection unit, 413 Depth map generation unit, 414 Display unit, 415 Storage unit, 416 Output unit, 450 3D reconstruction device, 461 Acquisition unit, 462 3D conversion unit, 463 3D synthesis unit, 464 display unit, 465 memory unit, 466 output unit, 500 depth map generation device, 511 offset setting unit, 512 DM setting unit, 513 depth estimation unit, 900 computer

Claims

1. An information processing device comprising: an offset setting unit for setting an offset for the disparity of a stereo image; and a depth estimation unit for determining the disparity of the stereo image by applying the offset, and estimating the depth of the subject in the stereo image using the disparity to which the offset has been applied.

2. The information processing apparatus according to claim 1, wherein the offset setting unit adjusts the offset if the shortest distance of the depth estimation range based on the disparity to which the offset has been applied is not appropriate.

3. The information processing apparatus according to claim 2, wherein the offset setting unit increases the value of the offset when moving the depth estimation range toward the front in the depth direction, and decreases the value of the offset when moving the depth estimation range toward the back in the depth direction.

4. The information processing apparatus according to claim 2, wherein the offset setting unit sets the initial value of the offset as a first offset, and the depth estimation unit sets a second offset by adjusting the first offset, and if the second offset is not set, estimates the depth of the subject using the disparity to which the first offset has been applied, and if the second offset is set, estimates the depth of the subject using the disparity to which the second offset has been applied.

5. The information processing apparatus according to claim 1, further comprising: an imaging unit that images the subject and generates the stereo image; a focus setting unit that sets the focus position of the stereo image; and a lens distortion correction unit that corrects the lens distortion of the stereo image, wherein the imaging unit generates the stereo image by imaging, with a calibration chart placed at a predetermined distance as the subject; the focus setting unit sets the focus position of the stereo image to the distance of the calibration chart; the offset setting unit sets the offset so that the disparity of the distance of the calibration chart is "0"; the depth estimation unit obtains the disparity of the stereo image by applying the offset, and estimates the depth of the calibration chart, which is the subject of the stereo image, using the disparity obtained by applying the offset; and the lens distortion correction unit sets an internal parameter for correcting the lens distortion using the result of the estimation.

6. The information processing apparatus according to claim 1, further comprising: an imaging unit that images the subject and generates the stereo image; and a subject detection unit that detects the subject included in the stereo image, wherein the offset setting unit sets the offset so that the subject is included in the depth estimation range based on the disparity to which the offset is applied; and the depth estimation unit determines the disparity of the stereo image by applying the offset and estimates the depth of the subject using the disparity to which the offset is applied.

7. The information processing apparatus according to claim 6, further comprising a storage unit for storing a depth map including the estimated depth of the subject.

8. The information processing apparatus according to claim 6, further comprising an output unit that outputs a depth map including the estimated depth of the subject.

9. The information processing apparatus according to claim 6, further comprising a display unit that displays a depth map including the estimated depth of the subject.

10. An information processing apparatus according to claim 1, further comprising: an imaging unit that images the subject and generates the stereo image; a subject detection unit that detects the subject included in the stereo image; and a storage unit that stores the stereo image, the disparity of the stereo image, and the offset, wherein the offset setting unit sets the offset so that the subject is included in the depth estimation range based on the disparity to which the offset has been applied; and the depth estimation unit determines the disparity of the stereo image by applying the offset.

11. The information processing apparatus according to claim 1, further comprising: an imaging unit that images the subject and generates the stereo image; a subject detection unit that detects the subject included in the stereo image; and an output unit that outputs the stereo image, the disparity of the stereo image, and the offset, wherein the offset setting unit sets the offset so that the subject is included in the depth estimation range based on the disparity to which the offset has been applied, and the depth estimation unit determines the disparity of the stereo image by applying the offset.

12. The information processing apparatus according to claim 1, further comprising a DM setting unit that sets a parameter indicating the shortest distance of the depth estimation range using the offset, wherein the depth estimation unit estimates the depth of the subject in the depth estimation range where the value of the set parameter is the shortest distance.

13. If the offset is not appropriate, the information processing apparatus according to claim 12, wherein the offset setting unit adjusts the offset, the DM setting unit sets the parameter using the adjusted offset, and the depth estimation unit estimates the depth of the subject in the depth estimation range where the set parameter value is the shortest distance.

14. An information processing method comprising setting an offset for the disparity of a stereo image, obtaining the disparity of the stereo image by applying the offset, and estimating the depth of the subject in the stereo image using the disparity obtained by applying the offset.

15. An information processing device comprising: an acquisition unit that acquires a stereo image, the disparity of the stereo image, and the offset of the disparity; a 3D conversion unit that estimates the depth of the subject in the stereo image using the acquired disparity and the offset, and converts the stereo image into 3D data; and a 3D synthesis unit that synthesizes the obtained 3D data with other 3D data.

16. The information processing apparatus according to claim 15, wherein the acquisition unit acquires the stereo image, the disparity, and the offset for each frame; the 3D conversion unit converts the stereo image of the processing target frame into 3D data using the acquired disparity and the offset of the processing target frame; and the 3D synthesis unit synthesizes the obtained 3D data of the processing target frame with the 3D data of the reference frame, or synthesized 3D data which is the result of synthesizing the 3D data up to the present.

17. The information processing apparatus according to claim 16, further comprising a storage unit for storing the synthesized 3D data after the 3D data of the frame to be processed has been synthesized.

18. The information processing apparatus according to claim 16, further comprising an output unit that outputs the synthesized 3D data after the 3D data of the frame to be processed has been synthesized.

19. The information processing apparatus according to claim 16, further comprising a display unit that displays the synthesized 3D data after the 3D data of the frame to be processed has been synthesized.

20. An information processing method comprising: obtaining a stereo image, the disparity of the stereo image, and the offset of the disparity; estimating the depth of the subject in the stereo image using the obtained disparity and offset; converting the stereo image into 3D data; and synthesizing the obtained 3D data with other 3D data.

Images