Imaging apparatus and method

Abstract

The present disclosure relates to an imaging apparatus and a method with which it is possible to more easily associate a rangefinding position corresponding to rangefinding data and a pixel in a captured image. Calibration information relating to a correlation between imaging achieved using a lens unit and a coordinate system of rangefinding performed using a rangefinding sensor is acquired, and, by using the acquired calibration information, a rangefinding position corresponding to rangefinding data that is generated through rangefinding is associated with a pixel of a captured image that is generated by the imaging. The present disclosure can be applied to, for example, an imaging apparatus, an electronic device, an information processing apparatus, an image processing apparatus, an imaging method, an information processing method, an image processing method, or a program.

Description

Imaging apparatus and method

[0001] This disclosure relates to an imaging apparatus and method, and more particularly to an imaging apparatus and method that makes it easier to associate the distance measurement position of distance measurement data with the pixels of an captured image.

[0002] Conventionally, there were systems that measured the distance of a subject in an image and used the image and the measurement data in combination. In such systems, when combining and using the data, it was necessary to correctly associate the measurement values ​​that constitute the measurement data with the measured distance positions and the pixels in the image.

[0003] For example, there was a system that, based on information obtained from an imager (image sensor), generated not only captured images but also distance measurement data using phase difference detection technology or DFD (Depth From Defocus), and stored them (see, for example, Patent Document 1). In this system, the coordinate systems (reference position and reference direction) for imaging and distance measurement coincide with each other. Therefore, the distance measurement position in the distance measurement data naturally corresponds to a predetermined pixel in the captured image.

[0004] In contrast, there was a system that combined a camera that generates captured images with an external distance measuring sensor that generates distance measurement data. In this system, the coordinate systems (reference position and reference direction) of the camera's image capture and the external distance measuring sensor were different. Therefore, the distance measurement position in the distance measurement data did not naturally correspond to the pixels in the captured image. Consequently, it was necessary to derive the correspondence between them.

[0005] Japanese Patent Publication No. 2022-181027

[0006] However, correctly matching the distance measurement location in the distance measurement data with the pixels in the captured image could potentially require a complicated process.

[0007] This disclosure is made in view of these circumstances and aims to make it easier to associate the distance measurement position in distance measurement data with the pixels in the captured image.

[0008] One aspect of this technology is an imaging device comprising: a calibration information acquisition unit that acquires calibration information regarding the correspondence between the coordinate systems of imaging via a lens unit and distance measurement by a distance measuring sensor; and a correspondence unit that uses the acquired calibration information to associate the distance measurement position of the distance measurement data generated by the distance measurement with the pixels of the image image generated by the imaging.

[0009] One aspect of this technology is an imaging method that includes acquiring calibration information regarding the correspondence between the coordinate systems of imaging via a lens unit and distance measurement by a distance measuring sensor, and using the acquired calibration information to associate the distance measurement position of the distance measurement data generated by the distance measurement with the pixels of the image image generated by the imaging.

[0010] In one aspect of this technology, the imaging apparatus and method acquire calibration information regarding the correspondence between the coordinate systems of imaging via a lens unit and distance measurement by a distance measuring sensor. This acquired calibration information is then used to associate the distance measurement position of the distance measurement data generated by the distance measurement with the pixels of the image image generated by the imaging.

[0011] This figure shows an example of the correspondence between captured images and distance measurement data. This figure shows an example of a method for associating the distance measurement position in the distance measurement data with the pixels in the captured image. This figure illustrates an example of the camera reference position and camera reference direction. This figure illustrates an example of optical calibration information and imager calibration information. This figure illustrates an example of distance measurement calibration information. This figure illustrates an example of distance measurement data. This figure illustrates an example of the overview of distance measurement data conversion. This figure illustrates an example of polar coordinate conversion. This figure illustrates an example of coordinate transformation. This figure illustrates an example of projection transformation. This figure shows an example of comparing UV point cloud distance measurement data and depth map. This figure shows an example of UV point cloud distance measurement data. This figure illustrates an example of correction processing. This figure illustrates an example of storage in a content file. This figure illustrates an example of storage in a content file. This figure illustrates an example of storage in a content file. This figure illustrates an example of storage in a content file. This figure illustrates an example of processing methods for distance measurement data and captured images. This figure shows an example of distance measurement utilization processing (post-processing). This figure illustrates an example of distance measurement utilization processing (post-processing). This block diagram shows an example of the main configuration of the imaging system. This block diagram shows an example of the main configuration of the imaging system. This block diagram shows an example of the main configuration of the imaging system. This is a flowchart illustrating an example of the image processing flow. This is a flowchart illustrating an example of the calibration information acquisition flow. This is a flowchart illustrating an example of the calibration information acquisition flow. This is a flowchart illustrating an example of the mapping process flow. This is a block diagram illustrating an example of the main configuration of an image processing device. This is a flowchart illustrating an example of the image processing flow. This is a flowchart illustrating an example of the readout process flow. This is a flowchart illustrating an example of the distance measurement utilization process flow. This is a block diagram illustrating an example of the main configuration of a computer.

[0012] The following describes the embodiments for implementing this disclosure. The explanation will be given in the following order: 1. Supporting literature, etc., for technical content and technical terminology 2. Acquired images and distance measurement data 3. Correspondence between distance measurement position and pixels using calibration information 4. Use of correspondence information and acquired images 5. First embodiment (imaging system) 6. Second embodiment (image processing device) 7. Appendix

[0013] <1. Supporting Documents for Technical Content and Terminology> The scope disclosed in this technology includes not only the contents described in the embodiments, but also the contents described in the following non-patent documents that were publicly known at the time of filing, as well as the contents of other documents referenced in the following non-patent documents.

[0014] Patent Document 1: (mentioned above)

[0015] In other words, the contents described in the aforementioned patent documents, as well as the contents of other documents referenced in those patent documents, will also serve as a basis for determining the support requirements.

[0016] <2. Captured Images and Distance Measurement Data> <Correspondence between Image Capture and Distance Measurement> Conventionally, there have been systems that measure the distance of the subject in an captured image and use the captured image and distance measurement data in combination. For example, as shown in Figure 1A, an image of an object 12 is captured from a camera 11 to generate an captured image 13, and at the same time, the depth value D (double arrow 15) from the camera 11 to a point 14 on the surface of the object 12 is measured to generate distance measurement data. By using this distance measurement data in combination with the captured image 13, for example, the three-dimensional shape of the object 12 captured in the captured image 13 can be represented by the distribution of the depth value D.

[0017] In such systems, when combining and using them, it was necessary to correctly associate the distance values ​​that constitute the distance measurement data with the measured distance position and the pixels in the captured image. For example, as shown in Figure 1B, it was necessary to clarify which pixel in the captured image 13 corresponds to the measured point 14 (which pixel's depth value D corresponds to).

[0018] For example, there was a system that, based on information obtained from an imager (image sensor), generated not only captured images but also distance measurement data using techniques such as phase difference detection and DFD (Depth From Defocus), and stored them (see, for example, Patent Document 1). Phase difference detection is a technique that divides the lens into pupils by dividing the pixels and estimates the distance from the amount of shift of each image. DFD is a technique that estimates the distance from the change in sharpness of the image when the focus position of the lens is changed. In addition, there was also a technique that estimated depth by image recognition. In these methods, the coordinate systems (reference position and reference direction) for imaging and distance measurement coincide with each other. Therefore, the distance measurement position of the distance measurement data naturally corresponds to a predetermined pixel in the captured image. In other words, the relationship shown in Figure 1B was naturally obtained.

[0019] In contrast, there were systems that combined a camera that generates captured images with an external rangefinder that generates distance measurement data. Examples include active rangefinders using invisible light (such as ToF (Time of Flight) sensors), rangefinders using a stereo optical system dedicated to distance measurement, and systems that use wireless devices (such as beacons) attached to the object to be measured. In this system, the coordinate systems (reference position and reference direction) of the image captured by the camera and the distance measured by the external rangefinder are different from each other. Therefore, the distance measurement position in the distance measurement data does not naturally correspond to the pixels in the captured image.

[0020] For example, as shown in Figure 1C, if a distance measuring sensor 21, which is different from the camera 11, measures the distance to point 14 from a different position than the camera 11, the measured distance R will be the distance from the distance measuring sensor 21 to point 14, as shown by the double arrow 22, and may differ from the depth value D from the camera. Furthermore, which pixel of the captured image 13 the obtained measured distance R corresponds to may change depending on the position and direction relationship between imaging and distance measurement. In other words, a relationship like that shown in Figure 1B cannot be obtained automatically. Therefore, it was necessary to derive these correspondences.

[0021] However, correctly mapping the distance measurement position in the distance measurement data to the pixels in the captured image could require complicated procedures. In the case of an external distance measurement sensor, as mentioned above, the coordinate systems (reference position and reference direction) for imaging and distance measurement are different from each other. Therefore, calibration is necessary to determine the accurate correspondence between the coordinate systems of imaging and distance measurement. However, this calibration could require complicated procedures, such as imaging a chart of the calibration pattern. For example, if the relative relationship of the reference position and reference direction or camera parameters change due to changes in the equipment configuration, calibration would be required each time these changes occur, potentially requiring even more complicated procedures. In short, mapping the distance measurement position in the distance measurement data to the pixels required complicated procedures.

[0022] <3. Correspondence between distance measurement position and pixels using calibration information> <Method 1> As shown in the top row of the table in Figure 2, calibration information is acquired, and this calibration information is used to associate the distance measurement position of the distance measurement data with the pixels of the captured image (Method 1). Here, the calibration information is information regarding the correspondence between the coordinate systems of imaging via an external lens unit and distance measurement by an external distance measurement sensor.

[0023] A lens unit is installed in front of the imager (image sensor) of an imaging device and has a lens barrel that forms the optical path for light incident on the imager. This lens barrel is composed of one or more lenses, an aperture, etc. The lens unit is an accessory to the imaging device and is detachable from it. Lens units exist with various specifications, and the imaging device can obtain diverse optical specifications by changing the attached lens unit. Note that the multiple lenses (lens group) in a lens unit can be optically represented as a single lens; therefore, unless otherwise specified, the lenses in a lens unit will be described as a single lens below. However, this description can also be applied to cases where a lens unit has multiple lenses.

[0024] A distance measuring sensor is a sensor that measures the distance from a distance measuring reference position to an object. The distance measuring reference position is the position that serves as the reference for distance measurement. For example, there are TOF sensors that measure the time it takes to receive the reflected light after irradiating with laser light and measure the distance based on that time. In the following explanation, a TOF sensor will be used as an example, but any distance measuring method may be used. In this specification, a distance measuring sensor that is attached to an imaging device will also be referred to as an external distance measuring sensor. An external distance measuring sensor is an accessory device of an imaging device and is detachable from the imaging device.

[0025] The lens unit and external distance measuring sensor are mounted to a predetermined mounting part of the imaging device in a predetermined manner. The mounting position may be fixed or variable. For example, the imaging device may have multiple mounting parts for external distance measuring sensors, and an external distance measuring sensor may be mounted to any of them. Furthermore, the mounting orientation of the lens unit and external distance measuring sensor relative to the imaging device (the relative orientation to the imaging device when mounted) may be fixed or variable. In any case, the lens unit and external distance measuring sensor are used while mounted on the imaging device, and the relative position and relative direction of imaging and distance measurement are determined when the lens unit and external distance measuring sensor are mounted on the imaging device. In addition, since the lens unit and external distance measuring sensor are replaceable, the relative position and relative direction of imaging and distance measurement may change depending on the specifications of the mounted equipment.

[0026] In other words, in order to associate the distance measurement position in the distance measurement data with the pixels, it is necessary to determine the relative position and direction of the image and the distance measurement while the lens unit or external distance measurement sensor is attached to the imaging device. As mentioned above, methods that use calibration pattern charts to determine this relative relationship require complicated work. Therefore, calibration information regarding the correspondence between the coordinate systems of the image taken via the attached external lens unit and the distance measurement by the external distance measurement sensor is acquired and used to associate the distance measurement position in the distance measurement data with the pixels of the captured image.

[0027] For example, the imaging device may include a calibration information acquisition unit that acquires calibration information regarding the correspondence between the coordinate systems of imaging via a lens unit and distance measurement by a distance measuring sensor, and a correspondence unit that uses the acquired calibration information to associate the distance measurement position of the distance measurement data generated by the distance measurement with the pixels of the image image generated by the imaging.

[0028] For example, the imaging method performed by the imaging device may include acquiring calibration information regarding the correspondence between the coordinate systems of imaging via the lens unit and distance measurement by the distance measuring sensor, and using the acquired calibration information to associate the distance measurement position of the distance measurement data generated by the distance measurement with the pixels of the image image generated by the imaging.

[0029] For example, the first program executed by the imaging device may be a program that causes a computer to perform a process that includes acquiring calibration information regarding the correspondence between the coordinate systems of imaging via the lens unit and distance measurement by the distance measurement sensor, and using the acquired calibration information to associate the distance measurement positions of the distance measurement data generated by the distance measurement with the pixels of the image image generated by the imaging.

[0030] By utilizing calibration information in this way, the relative relationship between imaging and distance measurement can be determined without the need for cumbersome tasks such as capturing calibration pattern charts. Therefore, the distance measurement position in the distance measurement data can be more easily associated with the pixels in the captured image. The method for associating the distance measurement position in the distance measurement data with the pixels in the captured image can be any method. For example, the distance measurement position in the distance measurement data can be associated with the pixels in the captured image by generating a depth map. A depth map is map information that shows the distance (depth value) from the camera to the subject for each pixel in the captured image. In other words, in a depth map, the depth value from the camera to the distance measurement position is shown at the pixel corresponding to that distance measurement position. That is, by generating such a depth map, the distance measurement position in the distance measurement data can be associated with the pixels in the captured image.

[0031] <Camera Reference Position and Reference Direction> For imaging and distance measurement, a reference position and reference direction are defined, and data is generated based on this reference position and reference direction. This reference position and reference direction can be expressed as a coordinate system. In other words, the captured image is expressed in the imaging coordinate system, and the distance measurement data is expressed in the distance measurement coordinate system. Therefore, in order to associate the distance measurement position in the distance measurement data with the pixels in the captured image, it is necessary to determine the correspondence between these coordinate systems. In calibration information, the coordinate systems for imaging and distance measurement (reference position and reference direction) are shown as relative information with respect to the reference position and reference direction of the imaging device. In other words, the correspondence between the coordinate systems for imaging and distance measurement can be determined via the reference position and reference direction of the imaging device.

[0032] Figure 3A shows an example of the mounting surfaces of the imaging device 31 and the lens unit 32. The lens unit 32 is mounted on the mounting portion 33 (mount portion) provided on the front (light-receiving side) of the imaging device 31, so that the parts indicated in gray on each side are aligned. Figure 3B shows the imaging device 31 and the lens unit 32 viewed from the side. The lens unit 32 is mounted on the mounting portion 33 so that its gray-indicated part aligns with the gray-indicated part of the mounting portion 33, as indicated by the dotted arrow 35. For example, when the lens unit 32 is mounted, the mounting portion 33 is designed so that the optical axis of the lens unit 32 and the optical axis of the imaging device 31 coincide, as indicated by the dotted line. However, even if they coincide in the design, there may be a misalignment in the manufactured units. The mounting portion 33 also has electrical contacts on its inside, so that when the lens unit 32 is mounted, the imaging device 31 and the lens unit 32 are electrically connected to each other. For example, the communication interfaces of the imaging device 31 and the lens unit 32 are connected, enabling the imaging device 31 and the lens unit 32 to communicate with each other via this mounting section 33. This connection may also enable power supply from the imaging device 31 to the lens unit 32.

[0033] In this specification, the reference position of the imaging device is also referred to as the camera reference position (or overall reference position). This camera reference position may be anywhere, but is generally set on the housing of the imaging device. For example, in the imaging device 31 of Figure 3A, the center of the mounting portion 33 on which the lens unit 32 is attached (the point indicated by the cross) may be set as the camera reference position 34.

[0034] Furthermore, the reference direction of the imaging device is also referred to as the camera reference direction (or overall reference direction). This camera reference direction can be any direction. For example, in the imaging device 31 of Figure 3B, the optical axis direction indicated by the dotted line (for example, the direction perpendicular to the mounting surface of the mounting part 33) may be designated as the camera reference direction 36.

[0035] <Optical Calibration Information> The calibration information described above may include any information. For example, the calibration information may include optical calibration information relating to the correspondence between the lens unit and the optical system of the imaging device. For example, the optical calibration information may include at least one of the following: optical reference position, optical reference direction, and intrinsic parameters.

[0036] The optical reference position is the reference position of the lens unit. This optical reference position can be anywhere, but for example, it may be set at the center of the lens of the lens unit. For example, the optical center (No Parallax Point) of the lens unit may be used as the optical reference position. The optical center position is given as the object-side nodal point (Front Nodal Point) of the lens. In a uniform medium, the position of the object-side nodal point coincides with the position of the rear principal point.

[0037] In Fig. 4, A shows a pinhole camera model representing the imaging range of an imaging device and a lens unit. In this pinhole camera model, the Z-axis indicated by the dotted arrow represents the optical axis. Plane P represents the subject, and plane Q represents the light-receiving surface (image plane) of the image sensor of the imaging device. That is, when the imaging device images the subject, the image of plane P is projected onto plane Q through the pinhole C. Plane R is a virtual plane equivalent to plane Q that has been shifted toward the subject side. As shown by the thick line in this figure, a frustum of a cone including plane P and plane R is formed. The pinhole C corresponds to the apex of the frustum of the cone. The optical center (object-side nodal point) of the lens unit corresponds to the position of this pinhole C. That is, the position of the pinhole C in the pinhole camera model representing the imaging range of such an imaging device and lens unit may be used as the optical reference position.

[0038] In the optical calibration information, this optical reference position is shown as the amount of displacement from the camera reference position. That is, in the cases of the examples of A in Fig. 3 and A in Fig. 4, the optical reference position (pinhole C) is shown as relative coordinates with the camera reference position 34 as the reference (origin).

[0039] The optical reference direction is the reference direction of the lens unit. This optical reference direction may be in any direction. For example, the optical axis direction of the lens unit may be used as the optical reference direction. In the case of the example of A in Fig. 4, the optical axis direction (Z-axis) of the pinhole camera model may be used as the optical reference direction. In the optical calibration information, this optical reference direction is shown as the amount of displacement from the camera reference direction. That is, in the cases of the examples of A in Fig. 3 and A in Fig. 4, the optical reference direction is shown as a relative direction with the camera reference direction 36 as the reference. In the following, the case where the optical axis direction (Z-axis) of the lens of the lens unit is used as the optical reference direction will be used as an example for explanation. When an optical reference direction different from this optical axis direction is used, the difference (amount of displacement) may be taken into account in the calculation.

[0040] The internal parameters may include parameters indicating the angle of view and focal length of the lens of the lens unit. Also, the internal parameters may include information (related parameters) for deriving the focal length. For example, the zoom, focus, aperture, etc. of the lens may be used as these related parameters. Also, the readout resolution of the imager (image sensor) may be included in these related parameters.

[0041] Also, the internal parameters may include parameters indicating the amount of distortion aberration of the lens of the lens unit. For example, the center position and coefficient of the distortion aberration may be included in the internal parameters. Also, the internal parameters may include information (related parameters) for deriving the amount of distortion aberration. For example, the zoom, focus, aperture, etc. of the lens may be used as these related parameters.

[0042] Also, when the lens of the lens unit is an anamorphic lens, the internal parameters may include parameters indicating the squeeze ratio of the anamorphic lens. Note that an anamorphic lens is a lens used when it is desired to capture an image that is wider than the aspect ratio of the imager (image sensor), and the squeeze ratio is the ratio for fitting the aspect ratio of the image to the aspect ratio of the imager.

[0043] That is, the imaging device can obtain information such as the operation status (zoom position, focus, iris), the design values of the lens (focal length, principal point position, pupil distance, imaging characteristics), the individual differences measured during manufacturing (tilt of the optical axis, deviation amount from the design values), and the connection status (directly or via a teleconverter) from the lens unit as optical calibration information. Then, the imaging device can use this information to associate the distance measurement position of the distance measurement data with the pixels of the captured image.

[0044] The optical calibration information described above is designed or measured in advance, for example, before shipment from the factory. Therefore, by acquiring and utilizing this optical calibration information, the imaging device can more easily grasp the correct correspondence between the lens unit and the optical system of the imaging device, such as the optical reference position and optical reference direction. Consequently, the imaging device can associate the distance measurement position of the distance measurement data with the pixels of the captured image without requiring complicated work (i.e., more easily).

[0045] Alternatively, the lens unit may store this optical calibration information, and when the lens unit is mounted on the imaging device, the imaging device may read and use the optical calibration information from the lens unit. For example, in the imaging device, a calibration information acquisition unit may acquire the optical calibration information stored in the memory unit of the lens unit. By doing so, the imaging device can acquire the optical calibration information more easily. Therefore, the imaging device can more easily associate the distance measurement position of the distance measurement data with the pixels of the captured image.

[0046] The imaging device may also obtain this optical calibration information from other information processing devices such as servers using communication functions or removable media. Furthermore, the imaging device may pre-store this optical calibration information for multiple lens units and select and use the optical calibration information corresponding to the mounted lens unit.

[0047] <Imager Calibration Information> The calibration information may also include imager calibration information relating to the imager (image sensor) of the imaging device. For example, the imager calibration information may include at least one of the following: the position of the imager, the orientation of the imager, and the readout resolution of the imager.

[0048] In this specification, "imager" refers to an image sensor of an imaging device that has a pixel array with photoelectric conversion elements and is used to capture an image of a subject and generate an image.

[0049] For example, Figure 4B shows the state in which the lens unit 52 is attached to the imaging device 51. The imaging device 51 has a built-in imager 53, and uses this imager 53 to generate captured images. For example, if the imaging device 51 has a function to correct camera shake by moving the imager 53 (imager shift type image stabilization function), the position and orientation of the imager 53 relative to the camera reference position and reference direction may change. Therefore, the imaging device 51 acquires imager calibration information and uses it to associate the distance measurement position of the distance measurement data with the pixels of the captured image.

[0050] The imager position indicates the horizontal and vertical position of the imager's reference point (e.g., center) at the time of imaging or distance measurement. In the imager calibration information, this imager position is shown as a displacement (relative position) from the camera's reference position. The imager direction indicates the direction (roll rotation amount or angle) of the imager's reference point (e.g., center) at the time of imaging or distance measurement. In the imager calibration information, this imager angle is shown as a displacement (relative direction) from the camera's reference direction. The imager readout resolution indicates the imager's readout resolution when generating the captured image.

[0051] By acquiring and utilizing this imager calibration information, the imaging device can determine the imager's position more accurately. Therefore, the imaging device can more accurately correlate the distance measurement position in the distance measurement data with the pixels in the captured image.

[0052] Furthermore, the imaging device may store such imager calibration information (information from imaging and distance measurement) and read it out when needed. For example, the imaging device may further include a storage unit for storing imager calibration information. The calibration information acquisition unit may then acquire the imager calibration information stored in the storage unit. In this way, the imaging device can use the imager calibration information at a desired timing. Therefore, the imaging device can more accurately associate the distance measurement position in the distance measurement data with the pixels in the captured image.

[0053] <Distance Measurement Calibration Information> The calibration information may also include distance measurement calibration information relating to the correspondence between the distance measurement system of the distance measurement sensor and the optical system of the imaging device. For example, the distance measurement calibration information may include at least one of the following: distance measurement reference position, distance measurement reference direction, and distance measurement direction.

[0054] The distance measurement reference position is the reference position for the external distance measurement sensor. This distance measurement reference position can be anywhere. For example, it may be set to the center of the lens of the external distance measurement sensor. For example, as shown in Figure 5A, the center of the lens 63 into which the laser light of the external distance measurement sensor 62, when mounted on the imaging device 61, is input and output may be set as the distance measurement reference position 64. In the distance measurement calibration information, this distance measurement reference position is shown as a displacement (relative position) from the camera reference position. In the following, we will explain using the case where the center of the lens of the external distance measurement sensor is set as the distance measurement reference position as an example. If a position different from the center of the lens of the external distance measurement sensor is set as the distance measurement reference position, the difference should be shifted accordingly.

[0055] The distance measurement reference direction is the reference direction for the external distance measuring sensor. This distance measurement reference direction can be any direction. For example, it may be set to the optical axis direction of the lens of the external distance measuring sensor. For example, as shown in Figure 5A, the optical axis direction (dotted line) of the lens 63 of the external distance measuring sensor 62 may be set as the distance measurement reference direction 65. In the distance measurement calibration information, this distance measurement reference direction is shown as the displacement amount (relative direction) from the camera reference direction. In the following explanation, we will use the case where the optical axis direction of the lens of the external distance measuring sensor is set as the distance measurement reference direction as an example. If a direction different from this optical axis direction is set as the distance measurement reference direction, the difference (displacement amount) should be added to the calculation.

[0056] The external distance measuring sensor can be mounted anywhere on the imaging device. For example, as shown in Figure 5B with the imaging device 61A and external distance measuring sensor 62A, the external distance measuring sensor may be mounted on the underside of the imaging device. Alternatively, as shown with the imaging device 61B and external distance measuring sensor 62B, the external distance measuring sensor may be mounted on the top surface of the imaging device. Furthermore, as shown with the imaging device 61C and external distance measuring sensor 62C, the external distance measuring sensor may be mounted on the side of the imaging device. Of course, the external distance measuring sensor is not limited to these examples, and may be mounted in locations other than those shown.

[0057] When the external distance measuring sensor is mounted on the imaging device, if its relative orientation (relative position and relative direction) to the imaging device is fixed, the imaging device can acquire the distance measurement reference position and distance measurement reference direction as fixed parameters (fixed values). When the external distance measuring sensor is mounted on the imaging device, if its relative orientation (relative position and relative direction) to the imaging device can be changed, the distance measurement reference position and distance measurement reference direction become variable parameters (variable values). In that case, the imaging device should be able to grasp its relative orientation and reflect it in the distance measurement reference position and distance measurement reference direction, or the imaging device should be able to grasp the distance measurement reference position and distance measurement reference direction that reflect its relative orientation.

[0058] Furthermore, mounting points for external distance measuring sensors may be provided at multiple locations on the imaging device, allowing the mounting position of the external distance measuring sensor to be selected. For example, the imaging device 61A and external distance measuring sensor 62A may be connected as shown in Figure 5B, the imaging device 61B and external distance measuring sensor 62B, or the imaging device 61C and external distance measuring sensor 62C. In such cases, the distance measurement reference position and distance measurement reference direction may change depending on the mounting position of the external distance measuring sensor. For example, a distance measurement reference position and distance measurement reference direction may be prepared in advance for each mounting position (mounting part), and the distance measurement reference position and distance measurement reference direction corresponding to the mounting position of the external distance measuring sensor may be selected from among them according to the mounting position of the external distance measuring sensor.

[0059] The distance measurement direction indicates the direction of the measurement point as seen from the distance measurement sensor (distance measurement reference position). The measurement point is the object of measurement as seen from the distance measurement sensor (distance measurement reference position). For example, as shown in Figure 6A, when the external distance measurement sensor 62 measures the distance (distance measurement value R) from object 71 to the measurement point 72, it irradiates laser light from the distance measurement reference position 64 toward the measurement point 72 as shown by arrow 73, and receives the reflected light. In other words, the distance measurement direction indicates the direction in which the distance measurement sensor (TOF sensor) irradiates laser light. Note that in the distance measurement calibration information, the distance measurement direction is shown as the amount of displacement (relative direction) from the distance measurement reference direction. In the example of Figure 6A, the distance measurement direction is expressed as the relative direction of arrow 73 with respect to the distance measurement reference direction 65.

[0060] Here, we will explain the distance measurement data output by the distance measuring sensor. Distance measurement data is information that shows the result (distance measurement value) of the distance measurement sensor measuring the distance point. In other words, the distance measurement value indicates the distance from the distance measurement reference position to the distance measurement position. To put it another way, the distance measurement position indicates the position where the distance from the distance measurement reference position was measured by the distance measuring sensor. Distance measurement is performed on the distance measurement point, but if the distance measurement point is set on a material that transmits laser light, such as glass, multiple distance measurement values ​​may be obtained for a single distance measurement point. In such cases, even if there is only one distance measurement point (distance measurement direction as seen from the distance measurement reference position), the distance to multiple distance measurement positions in 3D space is measured. In other words, the illumination light for distance measurement has a spot size. As a result, when multiple subjects are within the spot, multiple distance measurement values ​​(e.g., R1, R2, ..., Rn) may be obtained for a single distance measurement point. Furthermore, the accuracy of the expected angle of distance measurement has an error due to the size of the spot. (The smaller the spot, the smaller the error in the estimated angle.) Since the distance measurement data includes each obtained distance measurement value, it can also be said to be information indicating the distance measurement location.

[0061] Generally, a distance measuring sensor can measure distance in multiple distance directions, as shown in the example in Figure 6B. In Figure 6B, the black dots schematically represent the distance measuring points (distance measuring directions as viewed from the distance measuring sensor (distance measuring reference position)). In this example, the distance measuring sensor measures distance in an 8x8 grid of distance measuring points (distance measuring directions). In the distance measuring data, a distance measuring point ID, which is the identification information of the distance measuring point, is assigned to each distance measuring value, indicating which distance measuring point the distance measuring value corresponds to. In the example in Figure 6B, distance measuring point ID #1 is assigned to the distance measuring value of distance measuring point 81. Similarly, distance measuring point ID #2 is assigned to the distance measuring value of distance measuring point 82. Similarly, distance measuring point ID #8 is assigned to the distance measuring value of distance measuring point 83. Similarly, distance measuring point ID #64 is assigned to the distance measuring value of distance measuring point 84. In other words, the distance measuring data is composed of a combination of distance measuring point ID and distance measuring value R (ID,R). As mentioned above, it is possible to obtain multiple distance values ​​for a single distance measurement point. In such cases, the same distance measurement point ID is assigned to the multiple distance values ​​R.

[0062] When distance measurements are taken in multiple distance measurement directions, the distance measurement calibration information indicates each distance measurement direction (relative direction with respect to the distance measurement reference direction).

[0063] In other words, the imaging device can obtain information from the external distance measuring sensor as distance measurement calibration information, such as the distance measurement method, distance measurement capability, operating mode, module design values ​​(distance measurement direction, distance measurement reference position, distance measurement reference direction), and individual differences measured during manufacturing (optical axis tilt, temperature characteristics). The imaging device can then use this information to associate the distance measurement position in the distance measurement data with the pixels in the captured image.

[0064] The distance measurement calibration information described above is designed or measured in advance, for example, before shipment from the factory. Therefore, by acquiring and utilizing this distance measurement calibration information, the imaging device can more easily grasp the correct correspondence between the external distance measurement sensor and the optical system of the imaging device, such as the distance measurement reference position and distance measurement reference direction. Consequently, the imaging device can associate the distance measurement position of the distance measurement data with the pixels of the captured image without requiring complicated work (i.e., more easily).

[0065] Alternatively, the external distance measuring sensor may store this distance measurement calibration information, and when the external distance measuring sensor is attached to the imaging device, the imaging device may read and use the distance measurement calibration information from the external distance measuring sensor. For example, in the imaging device, the calibration information acquisition unit may acquire the distance measurement calibration information stored in the memory unit of the distance measuring sensor. By doing so, the imaging device can acquire the distance measurement calibration information more easily. Therefore, the imaging device can more easily associate the distance measurement position of the distance measurement data with the pixels of the captured image.

[0066] The imaging device may also obtain this distance measurement calibration information from other information processing devices such as servers using communication functions or removable media. Alternatively, the imaging device may pre-store this distance measurement calibration information for multiple external distance measurement sensors and select and use the distance measurement calibration information corresponding to the attached external distance measurement sensor.

[0067] <Signal Processing Calibration Information> The calibration information may also include signal processing calibration information related to signal processing on the captured image. When signal processing is applied to the captured image, it may be necessary to reflect the changes in the captured image caused by that signal processing in the distance measurement data. Signal processing calibration information is information used to achieve such reflection. For example, the signal processing calibration information may include at least one of the following: rolling shutter distortion correction amount, distortion correction amount, desqueeze ratio, parameter specifying the crop area for cropping, and recording magnification.

[0068] Rolling shutter distortion is distortion caused by a shift in exposure timing within the image plane and the movement of the imaging device. Rolling shutter distortion correction is a correction to reduce this distortion. The rolling shutter distortion correction amount is the amount of this correction. The rolling shutter distortion correction amount may be set for a sub-region of the captured image or for each pixel. In other words, the rolling shutter distortion correction amount may be configured as map information. Note that the signal processing calibration information may include information (related parameters) for deriving this rolling shutter distortion correction amount instead of the rolling shutter distortion correction amount itself. For example, the imager readout speed and the amount of camera movement during exposure may be these related parameters.

[0069] Distortion correction is a correction method used to reduce image distortion caused by the lens of a lens unit. The distortion correction amount is the amount of this distortion correction. For example, by combining this distortion correction amount with the lens's distortion aberration amount, it is possible to understand the characteristics of the distortion remaining in the saved captured image. For example, this distortion correction amount may consist of information indicating the center position of the distortion correction and a coefficient applied to the distortion correction.

[0070] The desqueeze ratio indicates the amount of correction applied to reduce the impact of squeeze magnification on the captured image when an anamorphic lens is used (i.e., a correction to restore the aspect ratio of the compressed image). In other words, this desqueeze ratio is applied when an anamorphic lens is used.

[0071] The parameter that specifies the cropping area for cropping is a parameter that indicates the coordinate range that represents the cropping area. Cropping is the process of cutting out a portion (the part of interest) from an captured image. In other words, cropping is the process of cutting out the unnecessary parts of an captured image. The cropping area is the area (the part of interest) that is cut out. The image of this cut-out cropping area is also called the cropped image. The size of this cropping area is also called the crop amount.

[0072] Cropping is performed, for example, to generate an image with a narrower field of view than the lens unit. In other words, cropping is performed to implement a function known as digital zoom or electronic zoom. Cropping may also be performed to correct the so-called focus breathing phenomenon, in which the field of view of the image changes depending on the focusing distance. Cropping may also be performed to achieve framing that tracks a recognized subject within the image, by cropping the area of ​​focus that includes the subject. Cropping may also be performed to reduce camera shake correction. In other words, cropping may be performed to implement a so-called electronic image stabilization function. Cropping may also be performed to change the aspect ratio of the image.

[0073] This cropping process may be performed with the center of the captured image as the center of the cropping region. In this case, the parameter specifying this cropping region may be the cropping amount. This cropping amount may be indicated, for example, by a magnification ratio relative to the original image, or by the image size. Alternatively, this cropping process may be performed with any position in the captured image as the center of the cropping region. In other words, any portion of the captured image may be cropped. In this case, the parameter specifying this cropping region may consist of the coordinates of the center of the cropping region and the cropping amount. This cropping amount may be indicated, for example, by a magnification ratio relative to the original image, or by the image size. Furthermore, the parameter specifying this cropping region may consist of coordinates indicating the outer frame of the cropping region.

[0074] The recording magnification parameter indicates the magnification of the captured image at the time of recording compared to the time of generation (capture). For example, when generating and processing high-resolution captured images to improve image quality and recording them to a recording medium, there is a technique called oversampling, which reduces the resolution of the captured image. The recording magnification parameter indicates the reduction ratio of the captured image when such oversampling is applied. Alternatively, the recording magnification parameter may indicate the magnification ratio when the captured image is enlarged.

[0075] By acquiring and utilizing this signal processing calibration information, the imaging device can more accurately reflect the changes in the captured image due to signal processing to the distance measurement data. Therefore, the imaging device can more accurately correlate the distance measurement position in the distance measurement data with the pixels in the captured image.

[0076] Furthermore, the imaging device may store such signal processing calibration information and read it out when needed. For example, the imaging device may further include a storage unit for storing signal processing calibration information. The calibration information acquisition unit may then acquire the signal processing calibration information stored in the storage unit. In this way, the imaging device can utilize the signal processing calibration information at a desired timing. Therefore, the imaging device can more accurately associate the distance measurement position in the distance measurement data with the pixels in the captured image.

[0077] <Method 1-1> When Method 1 is applied, the method for associating the distance measurement position in the distance measurement data with the pixels in the captured image may be any method as described above. For example, a depth map may be generated. Alternatively, as shown in the second row from the top of the table in Figure 2, UV point cloud distance measurement data that associates the distance measurement position with pixels may be generated (Method 1-1).

[0078] For example, in an imaging device, the correspondence unit may generate UV point cloud distance measurement data that associates the distance measurement position of the distance measurement data with the pixels of the captured image in a UV plane parallel to the image plane. UV point cloud distance measurement data is distance measurement information that associates the distance measurement position with pixels in a UV plane by projecting the distance measurement position in 3D space onto a UV plane parallel to the image plane. The distance measurement position projected onto the UV plane (distance measurement position in the UV plane) is expressed using UV coordinates (u,v) of the UV plane. These UV coordinates (u,v) can be set independently of the pixel positions in the captured image. For example, these u and v coordinates may each take values ​​with decimal precision.

[0079] This UV point cloud distance measurement data may be generated from distance measurement data using, for example, a camera reference position, a camera reference direction, and calibration information (optical calibration information, imager calibration information, distance measurement calibration information). For example, in an imaging device, a mapping unit may convert distance measurement data into UV point cloud distance measurement data using the camera reference position, camera reference direction, and calibration information.

[0080] For example, the matching unit may convert the distance measurement data obtained from the external distance measuring sensor into polar coordinate distance measurement data, where the measurement position is shown in polar coordinates. The matching unit may then convert the polar coordinate distance measurement data into 3D point cloud distance measurement data, where the measurement position is shown in 3D Cartesian coordinates. The matching unit may then convert the 3D point cloud distance measurement data into UV point cloud distance measurement data by projecting the measurement position shown in 3D Cartesian coordinates onto the UV plane.

[0081] <Distance Measurement> For example, distance measurement data may be converted to UV point cloud distance measurement data in the manner shown in Figure 7. First, when "distance measurement" is performed by an external distance measurement sensor, distance measurement data (ID,R) consisting of a distance measurement point ID and a distance measurement value R is obtained.

[0082] This distance measurement data (ID, R) is output information from the distance measurement sensor and indicates the measurement location. In terms of data structure, a distance measurement value R is assigned to each distance measurement point ID. In this distance measurement data, multiple distance measurement values ​​R may be assigned to one distance measurement point (one distance measurement point ID). Furthermore, in the distance measurement data, information such as the distance measurement start time (timing) and distance measurement end time (timing) may also be assigned to the distance measurement point ID.

[0083] <Polar Coordinate Conversion of Optical Reference> Next, the distance measurement data is expressed using a polar coordinate system through "polar coordinate conversion (optical reference)". In other words, the distance measurement direction (θ,φ) is added to the distance measurement data (ID,R) of each distance measurement point. However, this distance measurement direction (θ,φ) is the direction in the polar coordinate system of the distance reference, which is based on the distance measurement reference direction, and may differ from the direction in the image coordinate system (optical reference coordinate system). Therefore, with this representation, it is not possible to establish a correspondence between the distance measurement point (distance measurement position) and the pixels of the captured image. For example, as shown in Figure 8A, suppose an external distance measurement sensor 151 measures the distance to the distance measurement point 153 on the surface of object 152. In this case, the distance measurement direction of the distance measurement point 153 is the direction of the distance measurement point 153 as seen from the distance measurement reference position (ID,θ,φ), as shown by the dotted arrow.

[0084] In system cameras, there are countless combinations of imaging devices, lens units, and external distance measuring sensors. Therefore, in order to generate such UV point cloud distance measurement data, a camera reference that is independent of the combination is defined on the housing of the imaging device, and the optical reference and distance measuring reference are expressed in relation to that camera reference.

[0085] The projection characteristics of an image are represented as camera parameters. These parameters are mainly determined by the optical axis of the lens and the nodal points on the object side. To match the image, we first want to express the orientation of each distance measurement data as the orientation relative to the optical axis of the lens. Therefore, we examine the angular difference between the optical axis and the reference orientation of the distance measurement and add it to the orientation of each distance measurement point. In this case, since both are fixed via the camera, the angular difference is calculated via the overall reference orientation on the camera body, as shown in equation (1) below. In equation (1), M represents the rotation matrix for matching the angles.

[0086] (θ optical axis reference, Φ optical axis reference) T=M (θ, Φ) T ... (1)

[0087] In the calibration information, the distance measurement direction is shown as a relative direction from the distance measurement reference direction, and the distance measurement reference direction and optical reference direction are shown as relative directions from the camera reference direction. Therefore, using "Distance Measurement Direction Derivation (Optical Reference)," this information is used to convert the distance measurement direction of the distance measurement reference (ID, θ, φ) into an expression using the polar coordinate system of the optical reference (ID, θ', φ'). In other words, as shown in Figure 8B, a first relative direction indicating the camera reference direction with respect to the optical axis reference direction is derived, and by combining this first relative direction with a second relative direction indicating the distance measurement reference direction with respect to the camera reference direction, and a third relative direction indicating the distance measurement direction with respect to the distance measurement reference direction, the distance measurement direction is expressed as a relative direction with respect to the optical axis reference direction.

[0088] Then, in the "polar coordinate conversion (optical reference)" process, the distance measurement direction (ID, θ', φ') of the derived optical reference is added to the distance measurement data (ID, R), thereby generating distance measurement data (ID, θ', φ', R) expressed using the polar coordinate system of the optical reference.

[0089] In the following, distance measurement data (ID, θ', φ', R) expressed using this optical reference polar coordinate system will also be referred to as polar coordinate distance measurement data. This polar coordinate distance measurement data (ID, θ', φ', R) can be said to be distance measurement data (ID, R) to which the optical reference distance measurement direction (θ', φ') is assigned. In this polar coordinate distance measurement data, as with distance measurement data (ID, R), multiple distance measurement values ​​R may be assigned to one distance measurement point (one distance measurement point ID). Furthermore, in polar coordinate distance measurement data, the distance measurement start time (timing) and distance measurement end time (timing) may also be assigned to the distance measurement point ID. In addition, in polar coordinate distance measurement data, information such as the distance measurement direction and the error range of the distance measurement value (φe, θe, Re) may also be assigned to the distance measurement point ID.

[0090] <Coordinate System Transformation> Next, the coordinate system is transformed from a polar coordinate system to a 3D Cartesian coordinate system using "Coordinate System Transformation (Cartesian Coordinate Conversion)," and the distance measurement point (distance measurement position) is expressed in 3D Cartesian coordinates (X, Y, Z). In other words, after aligning the direction with the optical axis, the spatial coordinates of the distance measurement point are determined. First, the reference position for distance measurement is transformed into a Cartesian coordinate system with the origin. At this time, the direction in the direction of the optical axis (the direction corresponding to the depth direction relative to the image) is taken as the z axis, the direction mainly corresponding to the horizontal direction of the image is taken as the x axis, and the direction mainly corresponding to the vertical direction of the image is taken as the y axis. Through this process, the polar coordinate distance measurement data (ID, θ', φ', R) is transformed into distance measurement data (ID, X, Y, Z) expressed using a 3D Cartesian coordinate system. Any method can be used for this coordinate system transformation. Existing methods may also be applied to perform this coordinate system transformation.

[0091] <Reference Position Correction> The origin of this 3D Cartesian coordinate system is the distance measurement reference position, which may differ from the optical reference position. Therefore, in this representation, there is no correspondence between the distance measurement point (distance measurement position) and the pixels of the captured image. In the calibration information, the distance measurement reference position and the optical reference position are shown as relative positions from the camera reference position. Therefore, using "Correction Amount Derivation (Optical Reference)", the origin of the 3D Cartesian coordinate system is transformed (shifted) from the distance measurement reference position to the optical reference position using this information. In other words, the shift amount (ΔX, ΔY, ΔZ) of the 3D Cartesian coordinates is derived. Then, in "Reference Position Correction (Optical Reference)", the 3D Cartesian coordinates (X,Y,Z) of each distance measurement position in the "distance measurement data (ID,X,Y,Z) expressed using the 3D Cartesian coordinate system" generated by "Coordinate System Transformation (Cartesian Coordinate Conversion)" are shifted by the shift amount (ΔX, ΔY, ΔZ). This generates distance measurement data (ID, X', Y', Z') expressed using an optically-based 3D Cartesian coordinate system.

[0092] In the following, the distance measurement data (ID,X',Y',Z') expressed using this optically referenced 3D Cartesian coordinate system will also be referred to as 3D point cloud distance measurement data. This 3D point cloud distance measurement data (ID,X',Y',Z') consists of a distance measurement point ID and the optically referenced 3D Cartesian coordinates (X',Y',Z') that indicate the distance measurement position, which are assigned to that distance measurement point ID. In other words, the distance measurement position can be said to be located in 3D space (3D Cartesian coordinate system). The point that indicates the distance measurement position in this 3D space (3D Cartesian coordinate system) is also referred to as a 3D distance measurement point. Therefore, 3D point cloud distance measurement data can also be said to be distance measurement data composed of 3D distance measurement points. Note that if multiple distance measurement values ​​R can be assigned to one distance measurement point ID (one distance measurement point), it will be represented as multiple 3D distance measurement points. In other words, the same distance measurement point ID can be assigned to multiple 3D distance measurement points. Furthermore, in 3D point cloud distance measurement data, the distance measurement start time (timing) and distance measurement end time (timing) may be assigned to the distance measurement point ID. Also, in 3D point cloud distance measurement data, information such as the error range (X'e, Y'e, Z'e) of the 3D Cartesian coordinates (X', Y', Z') may be assigned to the distance measurement point ID.

[0093] For example, through the "coordinate system transformation (orthogonal coordinate conversion)" and "reference position correction (optical reference)" described above, the distance measurement point 153 of the polar coordinate distance measurement data (ID, θ', φ', R) is converted to the 3D distance measurement point 163 of the 3D point cloud distance measurement data (ID, X', Y', Z'), as shown in Figure 9A. In this case, for example, let's assume that the camera reference position is (0,0,0), the optical reference position is (0,0,+50), and the distance measurement reference position is (-10,-30,+10), as shown in Figure 9B. In this case, if the optical reference position is (0,0,0), then the camera reference position is (0,0,-50) and the distance measurement reference position is (10,-30,-40). In this way, the distance measurement reference position can be expressed as a relative position with respect to the optical reference position. Therefore, in "reference position correction (optical reference)," the coordinate system is shifted so that the distance measurement reference position is shifted to the origin (i.e., the optical reference position) (i.e., the 3D Cartesian coordinates of each distance measurement point are shifted).

[0094] Next, the coordinate values ​​are converted to a coordinate system with the optical center as the origin. Again, the conversion relationship between the optical center and the distance measurement reference position is calculated via the reference position on the camera.

[0095] The above series of calculations yields coordinate values ​​(X', Y', Z') relative to the optical axis and optical center position. The parameters used in the calculations are derived from values ​​obtained from an external distance measuring sensor or lens unit. The external distance measuring sensor and lens unit have design values ​​and calibration values. The CPU in the imaging device obtains these values ​​and, by further considering the connection information (position and direction) with the imaging device itself, obtains correction parameters.

[0096] <Projection Transformation> Next, the 3D distance measurement point is projected onto the UV plane parallel to the image plane using "projection transformation". In other words, the 3D distance measurement point is represented as a point on the UV plane. The UV plane is a plane parallel to the image plane of the imager, and can be said to correspond to the image plane of the imager. Furthermore, the position on the UV plane is indicated by UV coordinates (u,v), which are in a Cartesian coordinate system. These UV coordinates (u,v) can be set independently of the pixel position of the captured image. For example, these u and v coordinates may each take values ​​with decimal precision. Note that the z axis of the optical reference 3D Cartesian coordinate system corresponds to the optical reference direction (optical axis direction) and is orthogonal to the image plane (UV plane). In other words, the xy plane is parallel to the image plane (i.e., the UV plane). That is, by projecting the 3D distance measurement point onto the UV plane, the x coordinate "X'" and y coordinate "Y'" of the 3D distance measurement point are transformed into UV coordinates (u,v). In this way, "projection transformation" converts 3D point cloud distance measurement data (ID, X', Y', Z') into distance measurement data (ID, u, v, Z') expressed using UV coordinates.

[0097] In the following, the distance measurement data (ID,u,v,Z') expressed using these UV coordinates will also be referred to as UV point cloud distance measurement data. In other words, UV point cloud distance measurement data (ID,u,v,Z') can indicate the position of a 3D distance measurement point on the image plane using UV coordinates (u,v). That is, in UV point cloud distance measurement data (ID,u,v,Z'), the position of the 3D distance measurement point on the image plane can be set independently of the pixel position of the captured image. For example, the position of this 3D distance measurement point on the image plane may be made to take a value with decimal precision. Furthermore, the 3D distance measurement point projected onto this UV plane will also be referred to as a UV distance measurement point. In other words, UV point cloud distance measurement data can be said to be data that shows the position (u,v) of the UV distance measurement point and the distance measurement point ID and depth value Z' corresponding to the UV distance measurement point.

[0098] The result of this projection transformation (the position of the UV distance measurement point) depends on the focal length of the lens. Therefore, internal parameters of the optical calibration information (parameters related to the focal length) may be applied in the calculation of this projection transformation. For example, the zoom, focus, aperture of the lens, and the readout resolution of the imager may be applied.

[0099] Furthermore, if the imaging device has an image stabilization function that changes the position and orientation of the imager, the position and orientation of the imager are variable. Since this projection transformation can also be described as projection onto the image plane, the result of the projection transformation (position of the UV distance measurement point) may depend on the position and orientation of the imager. Therefore, imager calibration information (imager position, imager orientation, and imager readout resolution, etc.) may be applied in the calculation of the projection transformation.

[0100] <Lens aberration application> Also, the result of the projective transformation (position of the UV distance measurement point) depends on the distortion characteristics of the lens. Therefore, by "applying lens aberration", the influence of the distortion characteristics of the lens is reflected on the UV point cloud distance measurement data (ID, u, v, Z'). That is, using the internal parameters of the optical calibration information (parameters indicating the amount of distortion aberration of the lens of the lens unit, or related parameters thereof), the UV coordinates (u, v) of the UV point cloud distance measurement data are corrected. That is, the corrected UV coordinates (u', v') are derived. For example, parameters indicating the center position and coefficient of the distortion aberration of the lens may be applied to the correction. Also, zooming, focusing, aperture, etc. of the lens may be applied to the correction. By such "application of lens aberration", UV point cloud distance measurement data (ID, u', v', Z') with corrected UV coordinates is generated.

[0101] The above "projective transformation" and "application of lens aberration" may be performed together. For example, these processes may be performed using the following formulas (2) to (6).

[0102] x' = X' / Z'... (2) y' = Y' / Z'... (3) r 2 = x' 2 + y' 2 ... (4) ... (5) ... (6)

[0103] In the above formulas (2) to (6), the parameters k n , p n , and s n are coefficients of the distortion characteristics of the lens. This value changes with the settings of zooming, focusing, and aperture of the lens. Also, the parameters f x and f y are coefficients for converting to coordinate values on the image. This value is proportional to the focal length of the lens. This value changes depending on the settings of zooming, focusing, and aperture of the lens, and the readout resolution of the imager. Also, the parameters c x and c yThis value represents the positional relationship between the optical axis and the imager coordinates. This value changes depending on the readout resolution of the camera body's imager and the state of imager-shift image stabilization.

[0104] This calculation allows us to determine the coordinates of the autofocus point within the image readout from the imager. Furthermore, if the imager uses roll rotation image stabilization or a tilt lens, the calculation becomes even more complex.

[0105] Through the "projection transformation" and "lens aberration application" described above, for example, as shown in Figure 10A, the 3D distance measurement point 163 on the object 152 is projected onto the UV plane 172 toward the optical reference position 171. In other words, the 3D distance measurement point 163 is projected as indicated by arrow 173, and the projection destination is the UV distance measurement point 174. In other words, the UV point cloud distance measurement data (ID,u',v',Z') shows the position of each UV distance measurement point on the UV plane, as shown in Figure 10B. This UV plane corresponds to the image plane of the imager, and can therefore be said to correspond to the captured image. In other words, the UV point cloud distance measurement data (ID,u',v',Z') can be said to indicate the position (i.e., pixel) of the UV distance measurement point (i.e., distance measurement position) on the captured image. In short, the UV point cloud distance measurement data (ID,u',v',Z') directly associates the distance measurement position of the distance measurement data with the pixels of the captured image.

[0106] Therefore, by utilizing such UV point cloud ranging data, it becomes easier to associate the ranging position in the ranging data with the pixels in the captured image.

[0107] <Comparison of UV Point Cloud Distancing Data and Depth Maps> Traditionally, one method of recording distance measurement data was to record it as a depth map, which shows the depth value in the captured image. This depth map (xc,yc,Z') is generated by projecting 3D distance measurement points onto the captured image, similar to the case of UV point cloud distance measurement data (ID,u',v',Z'). Therefore, the correspondence between the depth map and the image can be directly seen compared to the point cloud representation using general spatial coordinates (XYZ coordinates). The same applies to UV point cloud distance measurement data. However, in the case of UV point cloud distance measurement data, the position on the UV plane can be represented by UV coordinates (u',v') independent of the pixel position, whereas in the case of depth maps, the position on the captured image can only be represented by the pixel position (xc,yc). Therefore, for example, as shown in Figure 11A, when the 3D distance measurement point 163 is projected onto the depth map 181, there was a risk that its projection position 182 (xc,yc) would differ from the UV distance measurement point 174 on the UV plane 172. For example, in the case of UV point cloud distance measurement data, it is possible to make the UV coordinates take values ​​with decimal precision. In other words, UV point cloud distance measurement data can more accurately correlate the distance measurement position of the distance measurement data with the pixels of the captured image than depth maps. In other words, by applying UV point cloud distance measurement data, it is possible to suppress the reduction in the accuracy of the correlation between the distance measurement position of the distance measurement data and the pixels of the captured image.

[0108] Furthermore, if there is distortion in the image, the correspondence between space and the image will not be a linear mapping, so it is necessary to correct the distortion of the captured image before associating it with the distance measurement data. However, there may be applications where the captured image distortion is not corrected, so both cases are possible: recording with distortion correction and recording with distortion remaining. Therefore, in order to more accurately associate the distance measurement position with the pixels in the subsequent stage, it is necessary to set the association method according to the application and whether or not distortion correction of the captured image is performed. In the case of depth maps, the correspondence between the distance measurement position of the distance measurement data and the pixels of the captured image is fixed, so there was a problem that the accuracy would decrease if such control was applied. For example, considering the distance at the boundary between two objects placed one in front of the other, the depth map records either the distance of the object in the foreground or the distance of the object in the background. If distortion correction processing of the image is performed here, interpolation calculations between pixels occur. When interpolation processing is applied to the depth map, the distance value at the boundary becomes inaccurate (a distance intermediate between the foreground and background appears). In this way, there was a risk that the accuracy of the correspondence between the distance measurement position of the distance measurement data and the pixels of the captured image would decrease.

[0109] In contrast, UV point cloud distance measurement data consists of data for each UV distance measurement point, and its position (UV coordinate) can be represented independently of the pixel position. Therefore, correction processing such as distortion correction can be applied more easily and appropriately at the necessary timing. In other words, in this respect as well, the reduction in the accuracy of the correspondence between the distance measurement position in the distance measurement data and the pixels in the captured image can be suppressed.

[0110] Furthermore, because depth maps cannot set multiple depth values ​​for a single pixel, their expressive capabilities are limited, and there was a risk that they could not represent all 3D autofocus points. For example, if multiple 3D autofocus points corresponded to the same pixel on a depth map, it was difficult to represent the depth values ​​of all those points on a single depth map. In other words, there was a risk that the accuracy of the correspondence between the autofocus position in the autofocus data and the pixels in the captured image would be reduced.

[0111] In contrast, UV point cloud ranging data consists of UV ranging points projected onto the UV plane from 3D ranging points. However, it is also possible to project multiple 3D ranging points onto the same UV coordinates. In other words, UV point cloud ranging data can contain multiple UV ranging points with the same UV coordinates. This means that UV point cloud ranging data can be considered data for each 3D ranging point. Therefore, UV point cloud ranging data can represent all 3D ranging points. This also helps to suppress the reduction in accuracy of the correspondence between the ranging position in the ranging data and the pixels in the captured image.

[0112] Furthermore, since the depth map is map information of depth values ​​(image data where depth values ​​are pixel values), it has depth values ​​for the same number of pixels as the captured image. In other words, as shown in Figure 11B, the depth map 181 also has pixel values ​​for pixels where 3D distance measurement points are not projected. Therefore, there was a risk that the data size of the depth map would increase unnecessarily compared to UV point cloud distance measurement data. Also, for example, as mentioned above, if there are multiple depth values ​​for the same pixel, it is necessary to provide multiple depth maps in order to represent all of them, which could further increase the data size.

[0113] In contrast, as mentioned above, UV point cloud ranging data consists of data for each UV ranging point (3D ranging point), so information about locations other than the UV ranging points is unnecessary. Furthermore, there is no need to create multiple map information sets. An example of UV point cloud ranging data is shown in Figure 12A. As in this example, the data for one UV ranging point consists of the ranging point ID, u-coordinate, v-coordinate, and depth value Z, and the UV point cloud ranging data contains data for each UV ranging point (3D ranging point). For example, if there are 64 3D ranging points, 64 data points for each UV ranging point are sufficient, regardless of the resolution of the captured image. Therefore, the increase in data volume can be suppressed compared to the case of depth maps. Also, UV point cloud ranging data can handle situations where the frame rate differs from that of the captured image.

[0114] As shown in Figure 12B, in UV point cloud ranging data, the ranging point ID may include information such as the error range, ranging start timing, and ranging end timing, in addition to the UV coordinates and depth value.

[0115] The above describes the various conversion processes shown in Figure 7. These conversion processes may be executed sequentially according to the flow shown in Figure 7, or they may be executed together. For example, all the conversion processes shown in Figure 7 may be combined into a single conversion process that derives UV point cloud distance measurement data from distance measurement data. Even in this case, only the conversion processes are combined, and the elements are the same as in the example in Figure 7. In other words, the conversion process that derives UV point cloud distance measurement data from distance measurement data is also executed using the various parameters used in the example in Figure 7. For example, the coefficients for the calculation of the conversion process that derives UV point cloud distance measurement data from distance measurement data are derived using the various parameters used in the example in Figure 7, and the calculation of the conversion process is executed by applying these derived coefficients.

[0116] <Method 1-1-1> In some cases, a predetermined signal processing (also called RAW correction) may be applied to the captured image (RAW image) generated by the imager. If the image size changes due to such signal processing, for example, the correspondence between the distance measurement position in the distance measurement data and the pixels in the captured image may be disrupted. Therefore, when Method 1-1 is applied, the UV point cloud distance measurement data may be corrected according to the RAW correction applied to the captured image, as shown in the third row from the top of Figure 2 (Method 1-1-1).

[0117] For example, in an imaging device, the correspondence unit may correct the UV point cloud distance measurement data in accordance with the RAW correction applied to the captured image. By doing so, it is possible to suppress the disruption of the correspondence between the distance measurement position in the distance measurement data and the pixels in the captured image due to RAW correction. In other words, it is possible to suppress a reduction in the accuracy of the correspondence between the distance measurement position in the distance measurement data and the pixels in the captured image.

[0118] Furthermore, this RAW correction can be any processing applied to the captured image (RAW image). For example, RAW correction may be cropping, which cuts out a portion of the captured image. Alternatively, RAW correction may be padding, which expands the area around the captured image. RAW correction may also be electronic image stabilization, which utilizes cropping or padding. Additionally, RAW correction may include other processing methods, or a combination of multiple processing methods. In short, RAW correction may include at least one of the following: cropping, padding, and electronic image stabilization, which utilizes both cropping and padding.

[0119] Furthermore, the correction process for UV point cloud distance measurement data performed in response to the RAW correction (RAW correction-compatible correction process) can be any type of process. For example, the origin position of the UV coordinates may be corrected as part of this RAW correction-compatible correction process. In other words, in an imaging device, the mapping unit may correct the origin position of the coordinates of the captured image when RAW correction is performed on the captured image.

[0120] For example, in cropping, the origin of the image coordinates changes due to the trimming of the edges of the image, and this change is reflected in the distance measurement coordinate values. The same applies to padding, which expands the edges of the image, and to electronic image stabilization, which uses cropping and padding.

[0121] This RAW correction-compatible correction process may be performed, for example, using the signal processing calibration information described above. For example, in an imaging device, the mapping unit may correct the origin position using signal processing calibration information related to signal processing on the captured image.

[0122] By doing so, RAW correction-compatible correction processing can be realized, and the reduction in the accuracy of the correspondence between the distance measurement position in the distance measurement data and the pixels in the captured image can be suppressed.

[0123] For example, as shown in Figure 13, if the captured image (RAW image) is output without RAW correction being performed, the RAW correction-compatible correction process is not executed, and UV point cloud distance measurement data showing the distance measurement coordinate values ​​on the imager of the UV distance measurement points is output. Conversely, if the captured image (RAW image) is output after RAW correction, the RAW correction-compatible correction process is executed on the UV point cloud distance measurement data, and UV point cloud distance measurement data showing the distance measurement coordinate values ​​for RAW recording of the UV distance measurement points is output.

[0124] Since these conversion coefficients are known internally by the imaging device, even if the parameters of the conversion process change frequently due to camera control, there is no need to recalibrate, reducing the burden on the user and minimizing the need for mutually exclusive functions.

[0125] <Method 1-1-2> In addition, the captured image (RAW image) generated in the imager may be converted into a YC image consisting of luminance and chrominance components, and a predetermined signal processing (also called YC correction) may be applied to the YC image. If such signal processing changes the image size or corrects distortion, the correspondence between the distance measurement position in the distance measurement data and the pixels in the captured image may be disrupted. Therefore, when Method 1-1 is applied, the UV point cloud distance measurement data may be corrected according to the YC correction applied to the captured image, as shown in the fourth row from the top of Figure 2 (Method 1-1-2). Here, YC correction applied to a YC image, which is commonly used in signal processing inside the camera, is given as an example. RGB correction applied to an RGB image may also be used.

[0126] For example, in an imaging device, the correspondence unit may correct the UV point cloud distance measurement data in accordance with the YC correction applied to the captured image. By doing so, it is possible to suppress the disruption of the correspondence between the distance measurement position in the distance measurement data and the pixels in the captured image due to YC correction. In other words, it is possible to suppress a reduction in the accuracy of the correspondence between the distance measurement position in the distance measurement data and the pixels in the captured image.

[0127] Furthermore, this YC correction can be any processing applied to the captured image (YC image). For example, YC correction may be lens distortion correction. Also, YC correction may be focal plane distortion correction. Also, YC correction may be cropping. Also, YC correction may be padding. Also, YC correction may be electronic image stabilization that applies cropping or padding. Also, YC correction may be resolution conversion. Also, YC correction may be electronic zoom (digital zoom) that applies cropping. Also, YC correction may be any other processing, or a combination of multiple processing. In other words, YC correction may include at least one of the following: lens distortion correction, focal plane distortion correction, cropping / electronic image stabilization, resolution conversion, and electronic zoom.

[0128] Furthermore, the correction process for UV point cloud distance measurement data performed in response to the YC correction (YC correction-compatible correction process) can be any type of process. For example, the origin position of the UV coordinates may be corrected as part of this YC correction-compatible correction process. Coordinate transformations or magnifications may also be applied as part of this YC correction-compatible correction process. In other words, in an imaging device, the mapping unit may apply coordinate transformations or magnifications to the captured image when YC correction is performed on the captured image.

[0129] For example, in cropping, the origin of the image coordinates changes due to the trimming of the edges of the image, and this change is reflected in the distance measurement coordinate values. The same applies to padding, which expands the edges of the image, and to electronic image stabilization, which uses cropping and padding. In addition, in correcting lens distortion and distortion due to imager readout time (focal plane distortion), the coordinate transformation process used for correction within the camera is applied to the distance measurement coordinates. Furthermore, in resolution conversion and electronic zoom, the magnification is applied to the distance measurement coordinates.

[0130] This YC correction-compatible correction process may be performed, for example, using the signal processing calibration information described above. For example, in an imaging device, the mapping unit may use signal processing calibration information related to signal processing on the captured image to perform coordinate transformations and magnifications on the captured image.

[0131] By doing so, it is possible to implement correction processing that supports YC correction, and to suppress the reduction in the accuracy of the correspondence between the distance measurement position in the distance measurement data and the pixels in the captured image.

[0132] For example, as shown in Figure 13, when a YC-corrected image (YC image) is output, a YC correction-compatible correction process is performed on the UV point cloud distance measurement data, and UV point cloud distance measurement data indicating the distance measurement coordinate values ​​for YC recording of the UV distance measurement points is output.

[0133] Since these conversion coefficients are known internally by the imaging device, even if the parameters of the conversion process change frequently due to camera control, there is no need to recalibrate, reducing the burden on the user and minimizing the need for mutually exclusive functions.

[0134] <Method 1-2> Alternatively, instead of generating UV point cloud ranging data as in Method 1-1, information applied to the generation of that UV point cloud ranging data may be generated. For example, when Method 1 is applied, auxiliary information may be generated and associated with the ranging data, as shown in the fifth row from the top of the table in Figure 2 (Method 1-2). Hereafter, this auxiliary information may be referred to as point cloud transformation parameters.

[0135] For example, in an imaging device, the correspondence unit may generate auxiliary information (point cloud conversion parameters) related to the conversion of distance measurement data into UV point cloud distance measurement data that associates distance measurement positions with pixels in a UV plane parallel to the image plane, and associate the generated auxiliary information with the distance measurement data.

[0136] Furthermore, this auxiliary information may include any information that is used to generate UV point cloud distance measurement data. For example, the auxiliary information may include at least one of the following: camera reference position, camera reference direction, and calibration information. In other words, the various types of information described with reference to Figure 7 may be associated with the distance measurement data as auxiliary information (point cloud conversion parameters).

[0137] As shown in Figure 7, UV point cloud distance measurement data can be generated using such distance measurement data and auxiliary information. Therefore, by utilizing such distance measurement data and auxiliary information, it is possible to more easily associate the distance measurement position in the distance measurement data with the pixels in the captured image, similar to the case of UV point cloud distance measurement data.

[0138] <Method 1-3> In Method 1-1, UV point cloud distance measurement data is generated as information that associates the distance measurement position of the distance measurement data with the pixels of the captured image (hereinafter also referred to as association information). In Method 1-2, distance measurement data and auxiliary information (point cloud transformation parameters) are generated as the association information. When Method 1 is applied, as shown in the sixth row from the top of Figure 2, such association information that associates the distance measurement position of the distance measurement data with the pixels of the captured image may be stored in association with the captured image (Method 1-3).

[0139] For example, the imaging device may further include a storage unit that stores association information, which associates the distance measurement position with a pixel, in relation to the captured image.

[0140] Alternatively, the association between the mapping information and the captured image may be established by storing the mapping information and the captured image in a single file. For example, the imaging device may further include a file generation unit that generates a content file to store the captured image and stores the mapping information in the generated content file. The storage unit may then store the generated content file.

[0141] In this way, by storing the correspondence information and the captured image in a single file, it becomes easier to associate the correspondence information with the captured image.

[0142] Alternatively, instead of storing the mapping information in the content file that stores the captured image, it can be saved as a separate file with the same name as the image, called a sidecar file.

[0143] <Method 1-3-1> When Method 1-3 is applied, UV point cloud distance measurement data may be stored as correspondence information, as shown in the seventh row from the top of the table in Figure 2 (Method 1-3-1). UV point cloud distance measurement data can be generated, for example, by applying Method 1-1.

[0144] For example, UV point cloud ranging data and captured images may be associated by storing them in a single file. For instance, in an imaging device, a file generation unit may store UV point cloud ranging data, which associates ranging positions with pixels in a UV plane parallel to the image plane, as association information in a content file that stores captured images. A storage unit may then store this content file.

[0145] In that case, UV point cloud distance measurement data may be stored in the metadata area of ​​the content file. The captured image may be a still image or a moving image consisting of multiple frames. Content file 231 in Figure 14 schematically shows a main example of the configuration of a content file that stores captured still images. Content file 232 in Figure 14 schematically shows a main example of the configuration of a content file that stores captured moving images.

[0146] As shown in Figure 14, the content file 231 has a metadata area. UV point cloud distance measurement data may be stored in this metadata area. In other words, if the captured image is a still image, the UV point cloud distance measurement data may be stored in the metadata area of ​​the content file that stores the captured image.

[0147] Furthermore, as shown in Figure 14, the content file 232 has a real-time metadata area and a non-real-time metadata area. The real-time metadata area is an area where metadata whose values ​​may change over time is stored. The non-real-time metadata area is an area where metadata whose values ​​cannot change over time (i.e., fixed values) is stored. UV point cloud distance measurement data can change over time, so it may be stored in this real-time metadata area. In other words, if the captured image is a moving image, the UV point cloud distance measurement data may be stored in the real-time metadata area of ​​the content file that stores the captured image.

[0148] Furthermore, if the captured image is a moving image, both the image and the distance measurement data will have multiple frames, requiring mapping (timing synchronization) between the frames. Therefore, metadata such as timestamps and distance measurement timings may be added to facilitate timing synchronization.

[0149] By storing UV point cloud ranging data and captured images in a single file in this way, it becomes easier to associate the UV point cloud ranging data with the captured images.

[0150] <Method 1-3-2> In addition, when Method 1-3 is applied, distance measurement data and auxiliary information (point cloud conversion parameters) may be stored as correspondence information, as shown in the bottom row of the table in Figure 2 (Method 1-3-2). The auxiliary information can be generated, for example, by applying Method 1-2.

[0151] For example, distance measurement data, auxiliary information, and captured images may be associated by storing them in a single file. For instance, in an imaging device, a file generation unit may store auxiliary information relating to the conversion of distance measurement data into UV point cloud distance measurement data that associates distance measurement positions with pixels in a UV plane parallel to the image plane, along with the distance measurement data itself, as association information in a content file. A storage unit may then store this content file.

[0152] In that case, the distance measurement data and auxiliary information may be stored in the metadata area of ​​the content file. The captured image may be a still image or a moving image consisting of multiple frames. Content file 241 in Figure 15 schematically shows a main example of the configuration of a content file that stores captured still images. Content file 242 in Figure 15 schematically shows a main example of the configuration of a content file that stores captured moving images.

[0153] Similar to content file 231, content file 241 has a metadata area. Distance measurement data and auxiliary information may be stored in this metadata area. In other words, if the captured image is a still image, distance measurement data and auxiliary information may be stored in the metadata area of ​​the content file that stores the captured image.

[0154] Similar to content file 232, content file 242 has a real-time metadata area and a non-real-time metadata area. Since distance measurement data can change over time, it may be stored in the real-time metadata area. Also, auxiliary information (point cloud transformation parameters) may include both variable-value parameters that can change over time and fixed-value parameters that cannot change over time. Therefore, variable-value parameters and time codes (TC) within the auxiliary information (point cloud transformation parameters) may be stored in the real-time metadata area. Also, fixed-value parameters within the auxiliary information (point cloud transformation parameters) may be stored in the non-real-time metadata area.

[0155] For example, in an imaging device, if the captured image is a moving image, the file generation unit may store the variable values ​​of the distance measurement data and auxiliary information in the real-time metadata area of ​​the content file that stores the captured image, and store the fixed values ​​of the auxiliary information in the non-real-time metadata area of ​​the same content file.

[0156] The fixed auxiliary information (point cloud conversion parameters) can be any parameter. For example, the fixed value may include at least one of the following: camera reference position, camera reference direction, distance measurement reference position, distance measurement reference direction, and distance measurement direction. The variable auxiliary information (point cloud conversion parameters) can also be any parameter. For example, the variable value may include at least one of the following: optical reference position, optical reference direction, internal parameters, imager position, imager attitude, rolling shutter distortion correction amount, distortion correction amount, desqueeze magnification, parameter specifying the crop area for cropping, recording magnification, and distance measurement value.

[0157] Furthermore, if the captured image is a moving image, both the image and the distance measurement data will have multiple frames, requiring mapping (timing synchronization) between the frames. Therefore, metadata such as timestamps and distance measurement timings may be added to facilitate timing synchronization.

[0158] In this way, by storing the distance measurement data, auxiliary information, and captured images in a single file, it becomes easier to associate the UV point cloud distance measurement data with the captured images.

[0159] <Specific Examples of Content Files> Figure 16 schematically shows a typical configuration example of a file container with a TIFF (Tagged Image File Format) structure for storing still images. This file container is used, for example, to store still images in JPEG (Joint Photographic Experts Group) format. When stored in such a file container, the mapping information may be recorded in the metadata's IFD (Image File Directory). An existing metadata IFD may be used, or an IFD dedicated to distance measurement information may be defined. For example, when using an existing IFD, the information may be stored in the MakerNote section of the Exif IFD (the gray area in Figure 16).

[0160] Figure 17 schematically shows a typical configuration example of a BOX-structured file container for storing moving images. This file container is used to store, for example, moving images in MP4 format or still images in HEIF (High Efficiency Image File Format) format. When stored in such a file container, mapping information may be stored in the metadata section or the stream data section. For example, in the case of ISOBMFF (International Organization for Standardization Base Media File Format), mapping information may be stored in moov or mdat (gray areas in Figure 16). Generally, data that changes within a file is stored in the mdat section, and immutable data is stored in the moov section.

[0161] <4. Use of Correspondence Information and Captured Images> <Method 2> When processing captured images, distance measurement information, in which the distance measurement position is associated with pixels in the captured image, may be used for processing (Method 2). In this specification, distance measurement information refers to information related to distance measurement, and indicates distance measurement data or data converted from distance measurement data. In other words, distance measurement information may include, for example, the distance measurement data, polar coordinate distance measurement data, 3D point cloud distance measurement data, UV point cloud distance measurement data, depth map, etc.

[0162] <Method 2-1> When Method 2 is applied, as shown in the bottom row of the table in Figure 18, distance measurement information may be read from the captured image file and converted to an appropriate format depending on the format and intended use of the distance measurement information.

[0163] For example, the image processing device may include a reading processing unit that reads distance measurement information relating to distance measurement data generated by distance measurement by a distance measuring sensor from a content file that stores captured images generated by imaging via a lens unit, and converts the format of the distance measurement information as necessary based on at least one of the format and application of the distance measurement information, and a distance measurement utilization processing unit that performs predetermined processing using the distance measurement information whose format has been converted as necessary and the captured image.

[0164] For example, an image processing method performed by an image processing device may include reading distance measurement information relating to distance measurement data generated by distance measurement by a distance measuring sensor from a content file that stores an image captured by imaging via a lens unit, converting the format of the distance measurement information as necessary based on at least one of the format and application of the distance measurement information, and performing predetermined processing using the distance measurement information whose format has been converted as necessary and the image captured.

[0165] For example, the second program executed by the image processing device may be a program that causes a computer to perform a process that includes reading distance measurement information relating to distance measurement data generated by distance measurement by a distance measuring sensor from a content file that stores an image captured by imaging via a lens unit, converting the format of the distance measurement information as necessary based on at least one of the format and purpose of the distance measurement information, and performing a predetermined process using the distance measurement information whose format has been converted as necessary and the image captured.

[0166] By reading and utilizing the distance measurement information in this way, it becomes easier to associate the distance measurement position in the distance measurement data with the pixels in the captured image for a wider range of applications.

[0167] <Readout Processing> For example, if the distance measurement information is 3D point cloud distance measurement data or a depth map, that distance measurement information may be used as is. For example, in an image processing device, if the readout processing unit finds that the distance measurement information is 3D point cloud distance measurement data or a depth map that shows the distance measurement position of the distance measurement data in 3D Cartesian coordinates, it may omit the format conversion. In this way, 3D point cloud distance measurement data and depth maps can be used for processing the captured image.

[0168] Furthermore, if the distance measurement information is UV point cloud distance measurement data and 3D orthogonal coordinates are not required for the application, the UV point cloud distance measurement data may be used as is. For example, in an image processing device, if the reading processing unit determines that the distance measurement information is UV point cloud distance measurement data that associates the distance measurement position of the distance measurement data with the pixels of the captured image on a UV plane parallel to the image plane, and 3D orthogonal transformed coordinates of the distance measurement position are not required for the application, then the format conversion may be omitted. In this way, UV point cloud distance measurement data can be used for processing the captured image.

[0169] Furthermore, if the distance measurement information is UV point cloud distance measurement data and 3D Cartesian coordinates are required for the application, the UV point cloud distance measurement data may be converted into 3D point cloud distance measurement data. For example, in an image processing device, if the distance measurement information is UV point cloud distance measurement data that associates the distance measurement position of the distance measurement data with the pixels of the captured image on a UV plane parallel to the image plane, and 3D Cartesian coordinates of the distance measurement position are required for the application, the readout processing unit may convert the UV point cloud distance measurement data into 3D point cloud distance measurement data that shows the distance measurement position in 3D Cartesian coordinates. In this way, when 3D point cloud distance measurement data is required for processing the captured image, the UV point cloud distance measurement data can be converted to generate and use 3D point cloud distance measurement data.

[0170] Furthermore, in applications where distortion-free images are not required, the captured images may be used as is. For example, in an image processing device, the readout processing unit may omit distortion correction of the captured images if distortion-free images are not required for that application. Conversely, in applications where distortion-free images are required, the captured images may be used after distortion correction. For example, in an image processing device, the readout processing unit may correct the distortion of the captured images if distortion-free images are required for that application.

[0171] <Example of Distance Measurement Utilization Processing 1> The distance measurement utilization processing, which processes captured images using distance measurement information, can be any type of processing. For example, as shown in the table in Figure 19, the distance measurement utilization processing may be the estimation of a depth map using sparse depth information. In that case, an RGB image with a depth map may be output as the processing result (output file) of the distance measurement utilization processing. Generally, the user needs to determine the coordinates on the image from the XYZ coordinates, but in this application, it is sufficient to know the depth of a specific pixel on the image, so the distance measurement data can be used directly. Alternatively, the distance measurement utilization processing may be SLAM (Simultaneous Localization and Mapping) processing using sparse depth information. In that case, 3D shape data of the camera path and scene may be output as the processing result (output file) of the distance measurement utilization processing. Alternatively, the distance measurement utilization processing may be photogrammetry processing that generates 3D data of the subject from multiple captured images. In that case, 3D shape data and texture information of the object may be output as the processing result (output file) of the distance measurement utilization processing. Furthermore, the distance measurement utilization process may also be a point cloud recording format conversion process. In that case, a point cloud file may be output as the processing result (output file) of the distance measurement utilization process. If the recording format can be converted, compatibility with software that does not support the uvZ method can be ensured. Also, the distance measurement utilization process may be image editing of the captured image, such as deformation or cropping. In that case, an image file with distance measurement data may be output as the processing result (output file) of the distance measurement utilization process. When editing an image, the distance measurement data can be updated at the same time to maintain the correct correspondence. This can be easily implemented by simply applying the same changes to the uv coordinates of the distance measurement data as to the uv coordinates of the image. This concept can also be used when recording distance measurement data along with the RAW image and developing it later.

[0172] Furthermore, the distance measurement processing may also be DELTAR (Depth Estimation from a Light-weight ToF Sensor and RGB Image). DELTAR is a process that combines a process that estimates high-resolution depth values ​​from the image with low-resolution measured depth information from the distance measurement sensor to obtain highly accurate high-resolution depth values. The obtained high-resolution depth information can be used, for example, in keying processes (depth maps) to extract objects or in synthesis processes. By understanding the front-to-back relationship, it becomes possible to perform depth-appropriate editing, such as inserting a new image (bottle) in a position that is behind the flowerpot but in front of the wall.

[0173] <Example of Distance Measurement Utilization Processing 2> Alternatively, the distance measurement utilization processing may involve determining focus in a predetermined area. That is, as part of the distance measurement utilization processing, a process may be performed to determine whether or not the captured image is in focus by combining the distance measurement information with the focus range (focus distance and depth of field) obtained from lens information (focus distance and aperture). For example, in an image processing device, the distance measurement utilization processing unit may compare the distance measurement information with the focus range of the captured image to determine whether or not the captured image is in focus.

[0174] For example, the captured image 301 shown in Figure 20A contains a subject 303. Among the UV distance measuring points 302 (white circles) corresponding to this captured image 301, the depth value of the UV distance measuring point within the region 304 that includes the subject 303 indicates the position of the subject 303 in the depth direction. Therefore, by comparing this depth value with the focus range, it is possible to more easily determine whether or not the subject 303 is in focus.

[0175] <Example 3 of Distance Measurement Utilization Processing> Alternatively, the distance measurement utilization processing may be image recognition. In other words, the distance measurement utilization processing may involve determining the actual size of the region of interest in the captured image based on the distance measurement information, and then performing image recognition based on that actual size. For example, in an image processing device, the distance measurement utilization processing unit may derive the actual size of the region of interest in the captured image based on the distance measurement information, and then perform image recognition of the captured image based on that derived actual size.

[0176] For example, in the captured image 311 shown in Figure 20B, a person 313, the subject of the image, is visible in the foreground. In the background, a large signboard 314 with a person depicted on it is visible. Due to perspective, the person depicted on the signboard 314 and the subject of the image, person 313, appear to be approximately the same size. Since the actual size (real dimensions) of the person on the signboard 314 and person 313 are unknown from the captured image alone, there is a risk that both may be recognized as actual people in image recognition of the captured image.

[0177] However, from the depth value of the UV distance measuring point 312 corresponding to this captured image 311, it can be seen that the depth value of region 316 is further back than the depth value of region 315. In other words, it is easy to see that the signboard 314 is located further back than the person 313. Therefore, in image recognition, it can be easily recognized that the actual size of the person depicted on the signboard 314 is extremely large compared to the person 313, and that it is not an actual person. That is, by using distance measurement information, the accuracy of image recognition can be improved.

[0178] <Combinations> Each of the methods described above may be applied in combination with any other method, as long as no contradiction arises. Three or more methods may be applied in combination. Furthermore, the combinatorial methods may include not only those shown as "methods" in the tables of Figure 2 and Figure 18, but all elements described herein. In addition, each of the methods described above may be applied in combination with other methods not described above.

[0179] For example, methods 2 and 2-1 described above may be applied to an imaging device to which method 1, etc., is applied. For example, the imaging device may further include a read processing unit that converts the format of the distance measurement information, which indicates the distance measurement position in the correspondence information stored in the content file that stores the captured image, as needed, based on at least one of the format and application of the distance measurement information acquired from the storage unit. Furthermore, if the distance measurement information is UV point cloud distance measurement data that associates the distance measurement position with a pixel in a UV plane parallel to the image plane, the read processing unit may convert the UV point cloud distance measurement data into 3D point cloud distance measurement data or a depth map that indicates the distance measurement position in 3D Cartesian coordinates. By reading and using the distance measurement information in this way, it is possible to more easily associate the distance measurement position in the distance measurement data with the pixels of the captured image in a wider variety of applications. For example, by converting the UV point cloud distance measurement data into 3D point cloud distance measurement data or a depth map as described above, the 3D point cloud distance measurement data or depth map can be used for processing the captured image.

[0180] <5. First Embodiment> <Imaging System> This technology can be applied to any device. For example, this technology can be applied to an imaging system using an imaging device that images a subject. Figure 21 is a block diagram showing an example of the configuration of an imaging system to which this technology is applied. The imaging system 400 shown in Figure 21 has an imaging device 401, a lens unit 402, and an external distance measuring sensor 403. The lens unit 402 and the external distance measuring sensor 403 are accessories of the imaging device 401 and are used by being mounted on the imaging device 401. Here, it is assumed that the lens unit 402 and the external distance measuring sensor 403 are properly mounted on predetermined mounting parts of the imaging device 401 and operate normally as devices constituting the imaging system 400.

[0181] In Figure 21, the main configuration examples of the lens unit 402 and the external distance measuring sensor 403 are shown as a block diagram. However, Figure 21 shows the main processing units and data flows of the lens unit 402 and the external distance measuring sensor 403, and does not necessarily represent everything. In other words, there may be processing units in the lens unit 402 and the external distance measuring sensor 403 that are not shown as blocks in Figure 21, and there may be processes and data flows that are not shown as arrows or other symbols in Figure 21.

[0182] <Lens Unit> As shown in Figure 21, the lens unit 402 includes an optical control unit 411, a lens barrel 412, a memory unit 413, a communication unit 414, and an image stabilization mechanism 415.

[0183] The optical control unit 411 performs processing related to the control of each processing unit of the lens unit. For example, the optical control unit 411 may control the drive of the lens barrel 412 and the image stabilization mechanism 415. For example, the optical control unit 411 may set internal parameters and use those internal parameters to control the lens barrel 412 and the image stabilization mechanism 415.

[0184] Furthermore, the optical control unit 411 may supply information to the storage unit 413 for storage, or read information stored in the storage unit 413. For example, the optical control unit 411 may read optical calibration information stored in the storage unit 413.

[0185] Furthermore, the optical control unit 411 may control the communication unit 414 to perform communication with the imaging device 401, and exchange information with the imaging device 401 via the communication unit 414. For example, the optical control unit 411 may acquire control instructions and information supplied from the imaging device 401 via the communication unit 414. The optical control unit 411 may also supply optical calibration information (optical reference position, optical reference direction, internal parameters, etc.) to the imaging device 401 via the communication unit 414.

[0186] The lens barrel section 412 has an optical system (imaging optical system) consisting of optical elements such as lenses and apertures, and optically controls the incident light to the imager of the imaging device 401 using this optical system. For example, the lens barrel section 412 may adjust the angle of view and focal length by driving the lenses and apertures. The lens barrel section 412 may perform such processing in accordance with the control of the optical control unit 411. For example, the lens barrel section 412 may drive the lenses and apertures according to internal parameters set by the optical control unit 411.

[0187] The storage unit 413 has a storage medium and performs processing related to the storage of information. The storage unit 413 may also perform processing according to the control of the optical control unit 411. For example, the storage unit 413 may acquire information supplied from the optical control unit 411 and store it in the storage medium. For example, the storage unit 413 may acquire optical calibration information (optical reference position, optical reference direction, internal parameters, etc.) supplied from the optical control unit 411 and store it in the storage medium. Alternatively, the storage unit 413 may read information stored in the storage medium in response to a request from the optical control unit 411 and supply it to the optical control unit 411. For example, the storage unit 413 may read optical calibration information (optical reference position, optical reference direction, internal parameters, etc.) from the storage medium and supply it to the optical control unit 411.

[0188] The communication unit 414 has a predetermined communication function and performs processing related to communication with the imaging device 401. This communication may be of any type. For example, it may be wired communication, wireless communication, or both. For example, the communication unit 414 may exchange information with the imaging device 401 through this communication. The communication unit 414 may also perform processing according to the control of the optical control unit 411. For example, the communication unit 414 may acquire information (instructions, data, etc.) supplied from the imaging device 401 and supply it to the optical control unit 411. Alternatively, the communication unit 414 may acquire information (instructions, data, etc.) supplied from the optical control unit 411 and supply it to the imaging device 401. For example, the communication unit 414 may acquire optical calibration information (optical reference position, optical reference direction, internal parameters, etc.) supplied from the optical control unit 411 and supply it to the imaging device 401.

[0189] The image stabilization mechanism 415 performs image stabilization processing. For example, the image stabilization mechanism 415 may correct image stabilization by driving the lens. That is, the image stabilization mechanism 415 may drive the lens of the lens barrel 412 to perform image stabilization. The image stabilization mechanism 415 may also perform processing according to the control of the optical control unit 411. For example, the image stabilization mechanism 415 may drive the lens of the lens barrel 412 according to internal parameters set by the optical control unit 411.

[0190] <External Distance Measuring Sensor> As shown in Figure 21, the external distance measuring sensor 403 includes a distance measuring control unit 421, a distance measuring unit 422, a communication unit 423, a storage unit 424, and a timing generator 425. The distance measuring unit 422 also includes a light-emitting unit 426, a projection optical system 427, a light-receiving optical system 428, and a light-receiving unit 429.

[0191] The distance measurement control unit 421 performs processing related to the control of each processing unit of the external distance measurement sensor 403. For example, the distance measurement control unit 421 may control the drive of the distance measurement unit 422. For example, the distance measurement control unit 421 may control the light-emitting unit 426 of the distance measurement unit 422 to emit laser light. For example, the distance measurement control unit 421 may cause the light-emitting unit 426 to emit laser light at a timing corresponding to the imaging timing. In addition, the distance measurement control unit 421 may control the light-receiving unit 429 of the distance measurement unit 422 to receive the laser light (reflected light from the irradiated laser light reflected at the distance measurement position) and notify the unit of this fact. In other words, the distance measurement control unit 421 may acquire notifications supplied from the light-receiving unit 429.

[0192] Furthermore, the distance measurement control unit 421 may generate distance measurement data indicating the result of the distance measurement performed by the distance measurement unit 422. For example, the distance measurement control unit 421 may acquire time information supplied from the timing generator 425. The distance measurement control unit 421 may use this time information to measure the timing at which the laser beam is emitted and the timing at which the reflected light of the laser beam is received, and derive a distance measurement value based on the time from emission to reception. If there are multiple distance measurement points, the distance measurement control unit 421 may derive a distance measurement value for each distance measurement point. The distance measurement control unit 421 may associate the derived distance measurement value with the distance measurement point ID and generate distance measurement data.

[0193] Furthermore, the distance measurement control unit 421 may control the communication unit 423 to perform communication with the imaging device 401, and exchange information with the imaging device 401 via the communication unit 423. For example, the distance measurement control unit 421 may acquire control instructions and information supplied from the imaging device 401 via the communication unit 423. The distance measurement control unit 421 may also supply distance measurement data to the imaging device 401 via the communication unit 423. The distance measurement control unit 421 may also supply distance measurement calibration information (distance measurement reference position, distance measurement reference direction, distance measurement direction, etc.) to the imaging device 401 via the communication unit 423.

[0194] Furthermore, the distance measurement control unit 421 may supply information to the storage unit 424 for storage, or read information stored in the storage unit 424. For example, the distance measurement control unit 421 may read distance measurement calibration information (distance measurement reference position, distance measurement reference direction, distance measurement direction, etc.) stored in the storage unit 424.

[0195] The distance measuring unit 422 performs processing related to distance measurement. For example, the distance measuring unit 422 may irradiate a laser beam toward the distance measurement point and receive the reflected light. For example, the distance measuring unit 422 may perform such processing in accordance with the control of the distance measuring control unit 421.

[0196] The light-emitting unit 426 performs processing related to the emission of laser light. For example, the light-emitting unit 426 may have a laser light-emitting element (also called a laser oscillator) and emit laser light by driving the light-emitting element according to the control of the distance measuring control unit 421. For example, the light-emitting unit 426 may emit laser light at a timing specified by the distance measuring control unit 421.

[0197] The projection optical system 427 is an optical system for projecting (illuminating) the laser light emitted by the light-emitting unit 426 toward the distance measurement point. The projection optical system 427 has optical elements such as lenses and optically controls the laser light emitted by the light-emitting unit 426. For example, the projection optical system 427 may control the direction of irradiation (projection direction) of the laser light. This irradiation direction may be variable. In that case, the projection optical system 427 may control its irradiation direction according to the control of the distance measurement control unit 421.

[0198] The light-receiving optical system 428 is an optical system for receiving reflected light from the laser beam emitted (projected) from the distance-measuring unit 422 at the distance-measuring position. The light-receiving optical system 428 has optical elements such as lenses and optically controls the reflected light. For example, the light-receiving optical system 428 may control the optical path of the reflected light so that it is supplied to the light-receiving unit 429.

[0199] The projection optical system 427 and the light-receiving optical system 428 may be composed of common optical elements. That is, the projection optical system 427 and the light-receiving optical system 428 may be configured as a single optical system. In that case, the irradiation position of the laser light and the reception position of the reflected light in the distance measuring unit 422 will be the same.

[0200] The light-receiving unit 429 performs processing related to the reception of laser light (reflected light). For example, the light-receiving unit 429 may have a light-receiving element that receives laser light and converts it into an electrical signal, and may drive the light-receiving element according to the control of the distance measuring control unit 421 to receive the laser light (reflected light), and supply to the distance measuring control unit 421 that the laser light (reflected light) has been received.

[0201] The communication unit 423 has a predetermined communication function and performs processing related to communication with the imaging device 401. This communication may be of any type. For example, it may be wired communication, wireless communication, or both. For example, the communication unit 423 may exchange information with the imaging device 401 through this communication. The communication unit 423 may also perform processing according to the control of the distance measurement control unit 421. For example, the communication unit 423 may acquire information (instructions, data, etc.) supplied from the imaging device 401 and supply it to the distance measurement control unit 421. Alternatively, the communication unit 423 may acquire information (instructions, data, etc.) supplied from the distance measurement control unit 421 and supply it to the imaging device 401. For example, the communication unit 423 may acquire distance measurement data supplied from the distance measurement control unit 421 and supply it to the imaging device 401. Furthermore, the communication unit 423 may acquire distance measurement calibration information (distance measurement reference position, distance measurement reference direction, distance measurement direction, etc.) supplied from the distance measurement control unit 421 and supply it to the imaging device 401.

[0202] The storage unit 424 has a storage medium and performs processing related to the storage of information. The storage unit 424 may perform processing according to the control of the distance measurement control unit 421. For example, the storage unit 424 may acquire information supplied from the distance measurement control unit 421 and store it in the storage medium. For example, the storage unit 424 may acquire distance measurement calibration information (distance measurement reference position, distance measurement reference direction, distance measurement direction, etc.) supplied from the distance measurement control unit 421 and store it in the storage medium. Alternatively, the storage unit 424 may read information stored in the storage medium in response to a request from the distance measurement control unit 421 and supply it to the distance measurement control unit 421. For example, the storage unit 424 may read distance measurement calibration information (distance measurement reference position, distance measurement reference direction, distance measurement direction, etc.) from the storage medium and supply it to the distance measurement control unit 421.

[0203] The timing generator 425 generates time information and supplies it to the distance measurement control unit 421.

[0204] <Imaging Device> Figure 22 is a block diagram showing an example of the main configuration of the imaging device 401. Note that Figure 22 shows the main processing units and data flows of the imaging device 401, and not everything shown in Figure 22 is included. In other words, there may be processing units in the imaging device 401 that are not shown as blocks in Figure 22, and there may be processes and data flows that are not shown as arrows etc. in Figure 22.

[0205] As shown in Figure 22, the imaging device 401 includes an input unit 431, a control unit 432, an imager 433, a camera shake correction mechanism 434, a RAW correction unit 435, a YC correction unit 436, an encoding unit 437, a communication unit 438, a communication unit 439, a calibration information acquisition unit 440, a mapping unit 441, a file generation unit 442, a storage unit 443, and a communication unit 444.

[0206] The input unit 431 has input devices such as switches, buttons, dials, and touch panels, and performs processing related to information input. For example, the input unit 431 may receive user instructions and information input via its input devices and supply them to the control unit 432. The input unit 431 may perform such processing in accordance with the control of the control unit 432.

[0207] The control unit 432 performs processing related to the control of each processing unit of the imaging device 401. For example, the control unit 432 may perform this control according to user instructions or the like input via the input unit 431.

[0208] The imager 433 has an image sensor having a pixel array with photoelectric conversion elements, and performs processing related to the generation of captured images using the image sensor. For example, the imager 433 may photoelectrically convert light from an incident subject via the lens unit 402 to generate a captured image (RAW image). The captured image generated by the imager 433 may be a still image or a moving image consisting of multiple frames. The imager 433 may also supply the generated captured image (RAW image) to the RAW correction unit 435. The imager 433 may also supply the generated captured image (RAW image) to the encoding unit 437. The imager 433 may also have a signal processing function and perform signal processing on the captured image, such as converting the RAW image to a YC image. The imager 433 may also supply imager calibration information (imager position, imager orientation, imager readout resolution, etc.) to the calibration information acquisition unit 440. For example, the imager 433 may detect its own position and orientation and supply the detection results to the calibration information acquisition unit 440 as imager calibration information. Alternatively, the imager 433 may supply the setting of its own (image sensor's) readout resolution to the calibration information acquisition unit 440 as imager calibration information. The imager 433 may perform such processing in accordance with the control of the control unit 432.

[0209] The image stabilization mechanism 434 performs image stabilization processing. For example, the image stabilization mechanism 434 may correct image stabilization in the captured image by driving the imager 433 (image sensor). In other words, the image stabilization mechanism 434 may drive the imager 433 (image sensor) and perform an operation for image stabilization. The image stabilization mechanism 434 may perform such processing in accordance with the control of the control unit 432.

[0210] The RAW correction unit 435 performs signal processing (RAW correction) on the captured image (RAW image). For example, the RAW correction unit 435 may acquire the captured image (RAW image) supplied from the imager 433. The RAW correction unit 435 may also perform RAW correction on the captured image (RAW image). As described above, this RAW correction can be any processing applied to a RAW image. For example, RAW correction may be cropping to cut out a part of the captured image. RAW correction may also be padding to extend the area around the captured image. RAW correction may also be electronic image stabilization that applies cropping or padding. Furthermore, RAW correction may be any other processing, or a combination of multiple processing methods may be used. The RAW correction unit 435 may supply the captured image with appropriate RAW correction applied to the YC correction unit 436. The RAW correction unit 435 may also supply the captured image with RAW correction applied to the encoding unit 437. Furthermore, the RAW correction unit 435 may supply the parameters applied to the RAW correction to the calibration information acquisition unit 440 as signal processing calibration information (for example, parameters that specify the crop area for cropping). The RAW correction unit 435 may also perform such processing in accordance with the control of the control unit 432.

[0211] The YC correction unit 436 performs signal processing (YC correction) on the captured image (YC image). For example, the YC correction unit 436 may acquire the captured image (RAW image that has been appropriately RAW corrected) supplied from the RAW correction unit 435. The YC correction unit 436 may also convert the RAW image into a YC image. The YC correction unit 436 may also perform YC correction on the captured image (YC image). As described above, this YC correction can be any processing applied to the YC image. For example, the YC correction may be lens distortion correction. The YC correction may also be focal plane distortion correction. The YC correction may also be cropping. The YC correction may also be padding. The YC correction may also be electronic image stabilization that applies cropping or padding. The YC correction may also be resolution conversion. The YC correction may also be electronic zoom (digital zoom) that applies cropping. Furthermore, YC correction may be performed using processes other than those described above, or a combination of multiple processes may be used. The YC correction unit 436 may supply the captured image with appropriate YC correction applied to the encoding unit 437. The YC correction unit 436 may also supply the parameters applied to the YC correction as signal processing calibration information to the calibration information acquisition unit 440. For example, the YC correction unit 436 may supply parameters related to rolling shutter distortion correction amount, distortion correction amount, desqueeze ratio, parameters specifying the crop area for cropping, recording magnification, etc., as signal processing calibration information to the calibration information acquisition unit 440. The RAW correction unit 435 may perform such processing in accordance with the control of the control unit 432.

[0212] The encoding unit 437 performs encoding-related processing. For example, the encoding unit 437 may acquire an image (RAW image) supplied from the imager 433. Alternatively, the encoding unit 437 may acquire an image (RAW-corrected RAW image) supplied from the RAW correction unit 435. Furthermore, the encoding unit 437 may acquire an image (YC image) supplied from the YC correction unit 436. The encoding unit 437 may encode the acquired image and generate encoded data. This encoding method can be any method. For example, it may be an encoding method for still images or an encoding method for moving images. The encoding unit 437 may supply the generated encoded data of the image to the file generation unit 442. The encoding unit 437 may perform such processing in accordance with the control of the control unit 432.

[0213] The communication unit 438 has a predetermined communication function and performs processing related to communication with the lens unit 402. This communication may be of any type. For example, it may be wired communication, wireless communication, or both. For example, the communication unit 438 may exchange information with the lens unit 402 through this communication. For example, the communication unit 438 may acquire information (instructions, data, etc.) supplied from the control unit 432 and supply it to the lens unit 402. Alternatively, the communication unit 438 may acquire optical calibration information (optical reference position, optical reference direction, internal parameters, etc.) supplied from the lens unit 402 and supply it to the calibration information acquisition unit 440. The communication unit 438 may perform such processing in accordance with the control of the control unit 432.

[0214] The communication unit 439 has a predetermined communication function and performs processing related to communication with the external distance measuring sensor 403. This communication can be of any type. For example, it may be wired communication, wireless communication, or both. For example, the communication unit 439 may exchange information with the external distance measuring sensor 403 through this communication. For example, the communication unit 439 may acquire information (instructions, data, etc.) supplied from the control unit 432 and supply it to the external distance measuring sensor 403. The communication unit 439 may also acquire distance measurement data supplied from the external distance measuring sensor 403 and supply it to the mapping unit 441. The communication unit 439 may also acquire distance measurement calibration information (distance measurement reference position, distance measurement reference direction, distance measurement direction, etc.) supplied from the external distance measuring sensor 403 and supply it to the calibration information acquisition unit 440. The communication unit 439 may perform such processing in accordance with the control of the control unit 432.

[0215] The calibration information acquisition unit 440 performs processing related to the acquisition of calibration information. For example, the calibration information acquisition unit 440 may acquire optical calibration information supplied from the lens unit 402 via the communication unit 438. Alternatively, the calibration information acquisition unit 440 may acquire distance measurement calibration information supplied from the external distance measuring sensor 403 via the communication unit 439. Furthermore, the calibration information acquisition unit 440 may acquire imager calibration information supplied from the imager 433. Additionally, the calibration information acquisition unit 440 may acquire signal processing calibration information supplied from the RAW correction unit 435 and the YC correction unit 436. The calibration information acquisition unit 440 may supply the acquired calibration information to the mapping unit 441. The calibration information acquisition unit 440 may perform such processing in accordance with the control of the control unit 432.

[0216] The mapping unit 441 performs processing related to generating mapping information that maps the distance measurement positions of the distance measurement data to the pixels of the captured image. For example, the mapping unit 441 may acquire calibration information supplied from the calibration information acquisition unit 440. Alternatively, the mapping unit 441 may acquire distance measurement data supplied from the external distance measurement sensor 403 via the communication unit 439. The mapping unit 441 may generate mapping information using the calibration information and distance measurement data. The mapping unit 441 may supply the mapping information thus generated to the file generation unit 442. The mapping unit 441 may perform such processing in accordance with the control of the control unit 432.

[0217] The file generation unit 442 performs processing related to the generation of a content file. For example, the file generation unit 442 may acquire encoded data of the captured image supplied from the encoding unit 437. The file generation unit 442 may also acquire mapping information supplied from the mapping unit 441. The file generation unit 442 may also generate a content file and store the acquired encoded data of the captured image in that content file. The file generation unit 442 may also store the acquired mapping information in the metadata area of ​​the content file. The file generation unit 442 may supply the content file containing the captured image and mapping information generated in this way to the storage unit 443. The file generation unit 442 may also supply the content file to the communication unit 444. The file generation unit 442 may perform such processing in accordance with the control of the control unit 432.

[0218] The storage unit 443 has a storage medium and performs processing related to the storage of information. For example, the storage unit 443 may acquire a content file supplied from the file generation unit 442 and store it in the storage medium. The storage unit 443 may perform such processing in accordance with the control of the control unit 432. The storage unit 443 may also have a removable medium as the storage medium.

[0219] The communication unit 444 has a predetermined communication function and performs processing related to communication with other devices other than the imaging system 400 (for example, the image processing device 500 described later). This communication may be of any type. For example, it may be wired communication, wireless communication, or both. For example, the communication unit 444 may exchange information with other devices through this communication. For example, the communication unit 444 may acquire a content file supplied from the file generation unit 442 and supply it to the image processing device 500. The communication unit 444 may perform such processing in accordance with the control of the control unit 432.

[0220] <Calibration Information Acquisition Unit> Figure 23 is a block diagram showing an example of the main configuration of the calibration information acquisition unit 440 of the imaging device 401. Note that Figure 23 shows the main processing units and data flows of the calibration information acquisition unit 440, and not everything shown in Figure 23 is included. In other words, there may be processing units in the calibration information acquisition unit 440 that are not shown as blocks in Figure 23, and there may be processes and data flows that are not shown as arrows, etc., in Figure 23.

[0221] As shown in Figure 23, the calibration information acquisition unit 440 includes an optical calibration information acquisition unit 451, an imager calibration information acquisition unit 452, a distance measurement calibration information acquisition unit 453, and a signal processing calibration information acquisition unit 454.

[0222] The optical calibration information acquisition unit 451 performs processing related to the acquisition of optical calibration information. For example, the optical calibration information acquisition unit 451 may acquire optical calibration information (optical reference position, optical reference direction, internal parameters, etc.) from the lens unit 402 via the communication unit 438. Alternatively, the optical calibration information acquisition unit 451 may supply the acquired optical calibration information to the mapping unit 441 (parameter derivation unit 461, which will be described later). The optical calibration information acquisition unit 451 may perform such processing in accordance with the control of the control unit 432.

[0223] The imager calibration information acquisition unit 452 performs processing related to the acquisition of imager calibration information. For example, the imager calibration information acquisition unit 452 may acquire imager calibration information (imager position, imager orientation, imager readout resolution, etc.) from the imager 433. The imager calibration information acquisition unit 452 may also supply the acquired imager calibration information to the mapping unit 441 (parameter derivation unit 461, which will be described later). The imager calibration information acquisition unit 452 may perform such processing in accordance with the control unit 432.

[0224] The distance measurement calibration information acquisition unit 453 performs processing related to the acquisition of distance measurement calibration information. For example, the distance measurement calibration information acquisition unit 453 may acquire distance measurement calibration information (distance measurement reference position, distance measurement reference direction, distance measurement direction, etc.) from the external distance measurement sensor 403 via the communication unit 439. Alternatively, the distance measurement calibration information acquisition unit 453 may supply the acquired distance measurement calibration information to the mapping unit 441 (parameter derivation unit 461, which will be described later). The distance measurement calibration information acquisition unit 453 may perform such processing in accordance with the control of the control unit 432.

[0225] The signal processing calibration information acquisition unit 454 performs processing related to the acquisition of signal processing calibration information. For example, the signal processing calibration information acquisition unit 454 may acquire signal processing calibration information (rolling shutter distortion correction amount, distortion correction amount, desqueeze ratio, parameters for specifying the crop area of ​​the cropping process, recording ratio, etc.) from the RAW correction unit 435 and the YC correction unit 436. The signal processing calibration information acquisition unit 454 may also supply the acquired signal processing calibration information to the mapping unit 441 (parameter derivation unit 461, which will be described later). The signal processing calibration information acquisition unit 454 may perform such processing in accordance with the control of the control unit 432.

[0226] <Matching Unit> Figure 23 also shows an example of the main configuration of the matching unit 441 of the imaging device 401. Note that Figure 23 shows the main processing units and data flows of the matching unit 441, and not everything shown in Figure 23 is included. In other words, there may be processing units in the matching unit 441 that are not shown as blocks in Figure 23, and there may be processes and data flows that are not shown as arrows etc. in Figure 23.

[0227] As shown in Figure 23, the correspondence unit 441 includes a parameter derivation unit 461 and a correspondence information generation unit 462. The correspondence information generation unit 462 includes an auxiliary information generation unit 463 and a UV point cloud distance measurement data generation unit 464.

[0228] The parameter derivation unit 461 performs processing related to the derivation of parameters applied to the process of associating the distance measurement position of the distance measurement data with the pixels of the captured image. For example, the parameter derivation unit 461 may acquire optical calibration information supplied from the optical calibration information acquisition unit 451. The parameter derivation unit 461 may also acquire imager calibration information supplied from the imager calibration information acquisition unit 452. The parameter derivation unit 461 may also acquire distance measurement calibration information supplied from the distance measurement calibration information acquisition unit 453. The parameter derivation unit 461 may also acquire signal processing calibration information supplied from the signal processing calibration information acquisition unit 454. Using the acquired configuration information, the parameter derivation unit 461 may derive parameters applied to the process of associating the distance measurement position of the distance measurement data with the pixels of the captured image. For example, the parameter derivation unit 461 may derive parameters applied to the process of converting distance measurement data (ID, R) to UV point cloud distance measurement data (ID, u', v', Z'). The parameter derivation unit 461 may supply the derived parameters to the auxiliary information generation unit 463. Alternatively, the parameter derivation unit 461 may supply the derived parameters to the UV point cloud distance measurement data generation unit 464. The parameter derivation unit 461 may perform such processing in accordance with the control of the control unit 432.

[0229] The mapping information generation unit 462 performs processing related to the generation of mapping information. For example, the mapping information generation unit 462 may acquire parameters supplied from the parameter derivation unit 461. Alternatively, the mapping information generation unit 462 may acquire distance measurement data (ID, R) from the external distance measuring sensor 403 via the communication unit 439. Furthermore, the mapping information generation unit 462 may associate the auxiliary information generated by the auxiliary information generation unit 463 with the distance measurement data (ID, R) and supply it to the file generation unit 442 as mapping information. Alternatively, the mapping information generation unit 462 may supply the UV point cloud distance measurement data generated by the UV point cloud distance measurement data generation unit 464 to the file generation unit 442 as mapping information. Note that the mapping information generation unit 462 may perform such processing in accordance with the control of the control unit 432.

[0230] The auxiliary information generation unit 463 performs processing related to the generation of auxiliary information concerning the conversion of distance measurement data to UV point cloud distance measurement data. For example, the auxiliary information generation unit 463 may acquire parameters derived by the parameter derivation unit 461, which are supplied from the parameter derivation unit 461. The auxiliary information generation unit 463 may generate auxiliary information using the acquired parameters. For example, the auxiliary information generation unit 463 may generate auxiliary information that includes the acquired parameters. The auxiliary information generation unit 463 may associate the generated auxiliary information with the distance measurement data (ID, R) and supply it to the file generation unit 442 as association information. The auxiliary information generation unit 463 may perform such processing in accordance with the control of the control unit 432.

[0231] The UV point cloud distance measurement data generation unit 464 performs processing related to the generation of UV point cloud distance measurement data. For example, the UV point cloud distance measurement data generation unit 464 may acquire parameters derived by the parameter derivation unit 461, which are supplied from the parameter derivation unit 461. Alternatively, the UV point cloud distance measurement data generation unit 464 may acquire distance measurement data (ID, R) from the external distance measurement sensor 403 via the communication unit 439. Furthermore, the UV point cloud distance measurement data generation unit 464 may use the acquired parameters to convert the acquired distance measurement data (ID, R) into UV point cloud distance measurement data (ID, u', v', Z'). The UV point cloud distance measurement data generation unit 464 may supply the UV point cloud distance measurement data thus generated to the file generation unit 442 as mapping information. Note that the UV point cloud distance measurement data generation unit 464 may perform such processing in accordance with the control of the control unit 432.

[0232] <Application of this technology> This technology may be applied to the imaging device 401 of the imaging system 400 having the configuration described above.

[0233] For example, the imaging device 401 may include a calibration information acquisition unit 440 that acquires calibration information regarding the correspondence between the coordinate systems of imaging via a lens unit and distance measurement by a distance measuring sensor, and a correspondence unit 441 that uses the acquired calibration information to associate the distance measurement position of the distance measurement data generated by the distance measurement with the pixels of the image image generated by the imaging.

[0234] For example, the imaging method performed by the imaging device 401 may include acquiring calibration information regarding the correspondence between the coordinate systems of imaging via the lens unit and distance measurement by the distance measuring sensor, and using the acquired calibration information to associate the distance measurement position of the distance measurement data generated by the distance measurement with the pixels of the image image generated by the imaging.

[0235] For example, the first program executed by the imaging device 401 may be a program that causes a computer to perform a process that includes acquiring calibration information regarding the correspondence between the coordinate systems of imaging via the lens unit and distance measurement by the distance measuring sensor, and using the acquired calibration information to associate the distance measurement positions of the distance measurement data generated by the distance measurement with the pixels of the image image generated by the imaging.

[0236] By utilizing calibration information in this way, the relative relationship between imaging and distance measurement can be determined without the need for cumbersome tasks such as capturing calibration pattern charts. Therefore, it becomes easier to associate the distance measurement position in the distance measurement data with the pixels in the captured image.

[0237] The calibration information may include optical calibration information relating to the correspondence between the lens unit and the optical system of the imaging device. This optical calibration information may include at least one of the following: optical reference position, optical reference direction, and internal parameters.

[0238] The calibration information acquisition unit 440 may acquire optical calibration information stored in the storage unit 413 of the lens unit 402. By doing so, the imaging device 401 can acquire optical calibration information more easily. Therefore, the imaging device 401 can more easily associate the distance measurement position of the distance measurement data with the pixels of the captured image.

[0239] Furthermore, the calibration information may include imager calibration information relating to the imager 433 of the imaging device 401. This imager calibration information may include at least one of the following: the position of the imager 433, the orientation of the imager 433, and the readout resolution of the imager 433.

[0240] For example, the imaging device 401 may further include an imager 433 that stores imager calibration information. The calibration information acquisition unit 440 may then acquire the imager calibration information stored in the imager 433. By doing so, the imaging device 401 can utilize the imager calibration information at a desired timing. Therefore, the imaging device 401 can more accurately associate the distance measurement position of the distance measurement data with the pixels of the captured image.

[0241] Furthermore, the calibration information may include distance calibration information relating to the correspondence between the distance measuring system of the distance measuring sensor and the optical system of the imaging device. This distance calibration information may include at least one of the following: distance reference position, distance reference direction, and distance direction.

[0242] The calibration information acquisition unit 440 may acquire distance measurement calibration information stored in the storage unit 424 of the external distance measuring sensor 403. By doing so, the imaging device 401 can acquire distance measurement calibration information more easily. Therefore, the imaging device 401 can more easily associate the distance measurement position of the distance measurement data with the pixels of the captured image.

[0243] Furthermore, the calibration information may include signal processing calibration information relating to signal processing on the captured image. This signal processing calibration information may include at least one of the following: rolling shutter distortion correction amount, distortion correction amount, desqueeze ratio, parameter specifying the crop area for cropping, and recording magnification.

[0244] The imaging device 401 may further include a RAW correction unit 435 and a YC correction unit 436 that store signal processing calibration information. The calibration information acquisition unit 440 may acquire the signal processing calibration information stored in the RAW correction unit 435 and the YC correction unit 436. In this way, the imaging device 401 can utilize the signal processing calibration information at a desired timing. Therefore, the imaging device 401 can more accurately associate the distance measurement position of the distance measurement data with the pixels of the captured image.

[0245] The correspondence unit 441 may generate UV point cloud distance measurement data that associates the distance measurement position of the distance measurement data with the pixels of the captured image in a UV plane parallel to the image plane. The distance measurement position projected onto the UV plane (distance measurement position in the UV plane) is expressed using UV coordinates (u,v) of the UV plane. These UV coordinates (u,v) can be set independently of the pixel positions of the captured image. For example, these u and v coordinates may each take values ​​with decimal precision.

[0246] The correspondence unit 441 may convert the distance measurement data into UV point cloud distance measurement data using the camera reference position, camera reference direction, and calibration information.

[0247] The correspondence unit 441 may convert the distance measurement data obtained by the external distance measuring sensor 403 into polar coordinate distance measurement data, in which the distance measurement position is indicated in polar coordinates. The correspondence unit 441 may convert the polar coordinate distance measurement data into 3D point cloud distance measurement data, in which the distance measurement position is indicated in 3D Cartesian coordinates. The correspondence unit 441 may convert the 3D point cloud distance measurement data into UV point cloud distance measurement data by projecting the distance measurement position indicated in 3D Cartesian coordinates onto the UV plane.

[0248] The correspondence unit 441 may correct the UV point cloud distance measurement data in accordance with the RAW correction applied to the captured image. By doing so, it is possible to suppress the disruption of the correspondence between the distance measurement position in the distance measurement data and the pixels in the captured image due to RAW correction. In other words, it is possible to suppress a reduction in the accuracy of the correspondence between the distance measurement position in the distance measurement data and the pixels in the captured image.

[0249] This RAW correction may include at least one of the following: cropping, padding, and electronic image stabilization that utilizes cropping and padding.

[0250] The matching unit 441 may correct the origin position of the coordinates of the captured image when RAW correction is performed on the captured image.

[0251] The correspondence unit 441 may correct the origin position using signal processing calibration information related to signal processing on the captured image.

[0252] By doing so, RAW correction-compatible correction processing can be realized, and the reduction in the accuracy of the correspondence between the distance measurement position in the distance measurement data and the pixels in the captured image can be suppressed.

[0253] The mapping unit may correct the UV point cloud distance measurement data according to the YC correction applied to the captured image. By doing so, it is possible to suppress the disruption of the correspondence between the distance measurement position in the distance measurement data and the pixels in the captured image due to YC correction. In other words, it is possible to suppress a reduction in the accuracy of the correspondence between the distance measurement position in the distance measurement data and the pixels in the captured image.

[0254] This YC correction may include at least one of the following: lens distortion correction, focal plane distortion correction, cropping / electronic image stabilization, resolution conversion, and electronic zoom.

[0255] The matching unit 441 may also apply coordinate transformations and magnifications to the captured image when YC correction is performed on the captured image.

[0256] The mapping unit 441 may use signal processing calibration information related to signal processing on the captured image to perform coordinate transformations and apply magnification to the captured image.

[0257] The correspondence unit 441 may generate auxiliary information (point cloud conversion parameters) related to the conversion of the distance measurement data into UV point cloud distance measurement data that associates the distance measurement position with pixels in a UV plane parallel to the image plane, and associate the generated auxiliary information with the distance measurement data.

[0258] This auxiliary information may include at least one of the following: camera reference position, camera reference direction, and calibration information.

[0259] The imaging device 401 may further include a storage unit 443 that stores association information, which associates distance measurement positions with pixels, in relation to the captured image.

[0260] The imaging device 401 may further include a file generation unit 442 that generates a content file for storing the captured image and stores mapping information to the generated content file. The storage unit 443 may then store the generated content file.

[0261] In this way, by storing the correspondence information and the captured image in a single file, it becomes easier to associate the correspondence information with the captured image.

[0262] The file generation unit 442 may store UV point cloud distance measurement data, which associates distance measurement positions with pixels in a UV plane parallel to the image plane, as correspondence information in a content file that stores the captured image. The storage unit 443 may then store this content file.

[0263] The file generation unit 442 may store the distance measurement data and the distance measurement data as correspondence information in a content file, along with auxiliary information relating to the conversion of the distance measurement data into UV point cloud distance measurement data that associates the distance measurement position with pixels in a UV plane parallel to the image plane. The storage unit 443 may then store the content file.

[0264] If the captured image is a moving image, the file generation unit may store the variable values ​​of the distance measurement data and auxiliary information in the real-time metadata area of ​​the content file that stores the captured image, and store the fixed values ​​of the auxiliary information in the non-real-time metadata area of ​​the same content file.

[0265] The fixed values ​​may include at least one of the following: camera reference position, camera reference direction, distance measurement reference position, distance measurement reference direction, and distance measurement direction. The variable values ​​may include at least one of the following: optical reference position, optical reference direction, internal parameters, imager position, imager attitude, rolling shutter distortion correction amount, distortion correction amount, desqueeze magnification, parameter specifying the crop area for cropping, recording magnification, and distance measurement value.

[0266] In this way, by storing the distance measurement data, auxiliary information, and captured images in a single file, it becomes easier to associate the UV point cloud distance measurement data with the captured images.

[0267] <Flow of Imaging Process> Next, an example of the flow of the imaging process performed by the imaging system 400 (imaging device 401) will be explained with reference to the flowchart in Figure 24.

[0268] When the imaging process is started, the calibration information acquisition unit 440 acquires fixed value calibration information by executing the calibration information acquisition process in step S401.

[0269] In step S402, the imager 433 captures an image of the subject and generates an image. At this time, the image stabilization mechanism 434 performs image stabilization as necessary.

[0270] In step S403, the RAW correction unit 435 performs RAW correction on the captured image as necessary. The YC correction unit 436 performs YC correction on the captured image as necessary.

[0271] In step S404, the encoding unit 437 encodes the captured image (RAW image, RAW-corrected RAW image, or YC image) and generates encoded data for the captured image.

[0272] In step S405, the control unit 432 controls the external distance measuring sensor 403 to measure distance and generate distance measurement data showing the result of the distance measurement. The correspondence unit 441 acquires the generated distance measurement data.

[0273] In step S406, the calibration information acquisition unit 440 acquires calibration information for variable values ​​by executing a calibration information acquisition process.

[0274] In step S407, the correspondence unit 441 performs a correspondence process to associate the distance measurement position of the distance measurement data with the pixels of the captured image.

[0275] In step S408, the control unit 432 determines whether or not to terminate the imaging process. If it is determined not to terminate, the process returns to step S402 and the subsequent processes are repeated. In other words, the processes from step S402 to step S408 are repeatedly executed (for each frame).

[0276] Then, if it is determined in step S408 that the imaging process should be terminated for the current frame, the process proceeds to step S409.

[0277] In step S409, the file generation unit 442 generates a content file and stores the encoded data of the captured image and the correspondence information.

[0278] In step S410, the storage unit 443 stores the generated content file.

[0279] In step S411, the communication unit 444 transmits the generated content file to another device.

[0280] When the process in step S411 is completed, the imaging process is terminated.

[0281] <Calibration Information Acquisition Process Flow 1> Next, an example of the calibration information acquisition process flow, which acquires fixed value configuration information and is performed in step S401 of Figure 24, will be explained with reference to the flowchart in Figure 25.

[0282] When the calibration information acquisition process is started, the optical calibration information acquisition unit 451 acquires fixed-value optical calibration information in step S431.

[0283] In step S432, the distance measurement calibration information acquisition unit 453 acquires distance measurement calibration information.

[0284] Once the process in step S452 is completed, the calibration information acquisition process ends, and the process returns to Figure 24.

[0285] <Calibration Information Acquisition Process Flow 2> Next, an example of the calibration information acquisition process flow, which acquires configuration information of variable values ​​and is performed in step S406 of Figure 24, will be explained with reference to the flowchart in Figure 26.

[0286] When the calibration information acquisition process is started, the optical calibration information acquisition unit 451 acquires variable value optical calibration information in step S451.

[0287] In step S452, the imager calibration information acquisition unit 452 acquires imager calibration information.

[0288] In step S453, the signal processing calibration information acquisition unit 454 acquires signal processing calibration information.

[0289] Once the process in step S453 is completed, the calibration information acquisition process ends, and the process returns to Figure 24.

[0290] <Flow of the mapping process> Next, an example of the flow of the mapping process performed in step S407 of Figure 24 will be explained with reference to the flowchart in Figure 27.

[0291] When the correspondence process is started, the parameter derivation unit 461 derives parameters used to associate the distance measurement position of the distance measurement data with the pixels of the captured image in step S471, based on the calibration information.

[0292] In step S472, the correspondence information generation unit 462 determines whether or not to generate UV point cloud distance measurement data. If it is determined that UV point cloud distance measurement data should be generated, the process proceeds to step S473.

[0293] In step S473, the UV point cloud distance measurement data generation unit 464 generates UV point cloud distance measurement data using parameters and uses it as correspondence information. When the processing in step S473 is completed, the correspondence processing is completed and the process returns to Figure 24.

[0294] Furthermore, if it is determined in step S472 that UV point cloud distance measurement data will not be generated, the process proceeds to step S474.

[0295] In step S474, the auxiliary information generation unit 463 generates auxiliary information using parameters and associates the auxiliary information with the distance measurement data. When the processing in step S474 is completed, the association process is finished and the process returns to Figure 24.

[0296] By performing each process as described above, the relative relationship between imaging and distance measurement can be determined without requiring cumbersome tasks such as capturing calibration pattern charts. Therefore, it becomes easier to associate the distance measurement position in the distance measurement data with the pixels in the captured image.

[0297] <6. Second Embodiment> <Image Processing Device> Furthermore, this technology can be applied to an image processing device that processes content files generated by the imaging system 400 described above. Figure 28 is a block diagram showing an example of the configuration of an image processing device to which this technology is applied. The image processing device 500 shown in Figure 28 acquires content files generated by the imaging system 400, acquires captured images and the like stored in the content files, and processes them.

[0298] Note that Figure 28 shows the main processing units and data flows of the image processing device 500, and does not necessarily represent everything. In other words, there may be processing units in the image processing device 500 that are not shown as blocks in Figure 28, and there may be processes and data flows that are not shown as arrows or other symbols in Figure 28.

[0299] As shown in Figure 28, the image processing device 500 includes an input unit 501, a control unit 502, a storage unit 521, a communication unit 522, a readout processing unit 523, a distance measurement utilization processing unit 524, a display unit 525, a storage unit 526, and a communication unit 527.

[0300] The input unit 501 has an input device such as a keyboard, mouse, switch, button, dial, or touch panel, and performs processing related to information input. For example, the input unit 501 may receive user instructions or information input via its input device and supply it to the control unit 502. The input unit 501 may perform such processing in accordance with the control of the control unit 502.

[0301] The control unit 502 performs processing related to the control of each processing unit of the image processing device 500. For example, the control unit 502 may perform this control according to user instructions or the like input via the input unit 501.

[0302] The storage unit 521 has a storage medium and performs processing related to the storage of information. For example, the storage unit 521 may have content files generated by the imaging system 400 stored in advance. Alternatively, the storage unit 521 may read content files stored on the storage medium in response to a request from the read processing unit 523 and supply them to the read processing unit 523 as input content files. The storage unit 521 may perform such processing in accordance with the control of the control unit 502. The storage unit 521 may also have removable media as its storage medium.

[0303] The communication unit 522 has a predetermined communication function and performs processing related to communication with other devices (for example, the imaging device 401). This communication may be of any type. For example, it may be wired communication, wireless communication, or both. For example, the communication unit 522 may exchange information with other devices through this communication. For example, the communication unit 522 may communicate with the imaging device 401, acquire a content file supplied from the imaging device 401, and supply it to the read processing unit 523 as an input content file. The communication unit 522 may perform such processing in accordance with the control unit 502.

[0304] The read processing unit 523 performs processing related to the reading of information. For example, the read processing unit 523 may request a content file stored in the storage unit 521 and acquire it as an input content file. Alternatively, the read processing unit 523 may acquire a content file supplied from the imaging device 401 via the communication unit 522 as an input content file. The read processing unit 523 may also read the captured image stored in the input content file and supply it to the distance measurement utilization processing unit 524. Furthermore, the read processing unit 523 may read the mapping information stored in the input content file and supply it to the distance measurement utilization processing unit 524. In addition, the read processing unit 523 may read other metadata stored in the input content file and supply it to the distance measurement utilization processing unit 524. The read processing unit 523 may perform such processing in accordance with the control of the control unit 502.

[0305] The distance measurement utilization processing unit 524 performs processing related to distance measurement utilization. For example, the distance measurement utilization processing unit 524 may acquire the captured image supplied from the readout processing unit 523. The distance measurement utilization processing unit 524 may also acquire correspondence information supplied from the readout processing unit 523. The distance measurement utilization processing unit 524 may also acquire other metadata supplied from the readout processing unit 523. The distance measurement utilization processing unit 524 may use this information to perform distance measurement utilization processing.

[0306] As mentioned above, the distance measurement utilization process can be any type of process. For example, the distance measurement utilization process may be depth map estimation, SLAM processing, photogrammetry processing, point cloud recording format conversion processing, image editing of captured images such as deformation and cropping, or DELTAR. Furthermore, the distance measurement utilization process may be focus determination for a predetermined area or image recognition.

[0307] The distance measurement processing unit 524 may generate a display image showing the processing result of the distance measurement processing and supply it to the display unit 525 for display. Alternatively, the distance measurement processing unit 524 may generate an output content file containing the processing result of the distance measurement processing and supply it to the storage unit 526 for storage. Alternatively, the distance measurement processing unit 524 may supply the output content file to the communication unit 527 for supply to other devices. The distance measurement processing unit 524 may perform the above processing in accordance with the control of the control unit 502.

[0308] The display unit 525 has a display device such as an LCD (Liquid Crystal Display), and performs processing related to the display of visual information using the display device. For example, the display unit 525 may acquire a display image supplied from the distance measurement utilization processing unit 524 and display it on the display device. The display unit 525 may perform such processing in accordance with the control of the control unit 502.

[0309] The storage unit 526 has a storage medium and performs processing related to the storage of information. For example, the storage unit 526 may acquire an output content file supplied from the distance measurement utilization processing unit 524 and store it in the storage medium. The storage unit 526 may perform such processing in accordance with the control of the control unit 502. The storage unit 526 may also have a removable medium as the storage medium.

[0310] The communication unit 527 has a predetermined communication function and performs processing related to communication with other devices. This communication may be of any type. For example, it may be wired communication, wireless communication, or both. For example, the communication unit 527 may exchange information with other devices through this communication. For example, the communication unit 527 may acquire an output content file supplied from the distance measurement utilization processing unit 524 and supply it to other devices. The communication unit 527 may perform such processing in accordance with the control of the control unit 502.

[0311] <Application of this technology> This technology may be applied to an image processing device 500 with the configuration described above.

[0312] For example, the image processing device 500 may include a reading processing unit 523 that reads distance measurement information relating to distance measurement data generated by distance measurement by a distance measuring sensor from a content file that stores captured images generated by imaging via a lens unit, and converts the format of the distance measurement information as necessary based on at least one of the format and application of the distance measurement information, and a distance measurement utilization processing unit 524 that performs predetermined processing using the distance measurement information whose format has been converted as necessary and the captured image.

[0313] For example, the image processing method executed by the image processing device 500 may include reading distance measurement information relating to distance measurement data generated by distance measurement by a distance measuring sensor from a content file that stores an image captured by imaging via a lens unit, converting the format of the distance measurement information as necessary based on at least one of the format and application of the distance measurement information, and performing predetermined processing using the distance measurement information whose format has been converted as necessary and the image captured.

[0314] For example, the second program executed by the image processing device 500 may be a program that causes a computer to perform a process that includes reading distance measurement information relating to distance measurement data generated by distance measurement by a distance measuring sensor from a content file that stores an image captured by imaging via a lens unit, converting the format of the distance measurement information as necessary based on at least one of the format and purpose of the distance measurement information, and performing a predetermined process using the distance measurement information whose format has been converted as necessary and the image captured.

[0315] By reading and utilizing the distance measurement information in this way, it becomes easier to associate the distance measurement position in the distance measurement data with the pixels in the captured image for a wider range of applications.

[0316] The readout processing unit 523 may omit format conversion if the distance measurement information is 3D point cloud distance measurement data or a depth map that indicates the distance measurement position of the distance measurement data in 3D Cartesian coordinates. By doing so, 3D point cloud distance measurement data and depth maps can be used for processing the captured image.

[0317] The readout processing unit 523 may omit format conversion if the distance measurement information is UV point cloud distance measurement data that associates the distance measurement position of the distance measurement data with the pixels of the captured image in a UV plane parallel to the image plane, and if the 3D orthogonal transformation coordinates of the distance measurement position are not required for that application. In this way, the UV point cloud distance measurement data can be used for processing the captured image.

[0318] The readout processing unit 523 may convert the UV point cloud distance measurement data into 3D point cloud distance measurement data that shows the distance measurement position in 3D Cartesian coordinates if the distance measurement information is UV point cloud distance measurement data that associates the distance measurement position of the distance measurement data with the pixels of the captured image in a UV plane parallel to the image plane, and if 3D Cartesian coordinates of the distance measurement position are required for that application. In this way, when 3D point cloud distance measurement data is required for processing the captured image, the UV point cloud distance measurement data can be converted to generate and use 3D point cloud distance measurement data.

[0319] The readout processing unit 523 may omit distortion correction of the captured image if a distortion-free captured image is not required for the purpose of using the distance measurement information. Alternatively, the readout processing unit 523 may correct the distortion of the captured image if a distortion-free captured image is required for the purpose of using the distance measurement information.

[0320] The distance measurement processing unit 524 may compare the distance measurement information with the focus range of the captured image to determine whether or not the captured image is in focus. By comparing the depth value with the focus range in this way, it is possible to more easily determine whether or not the image is in focus.

[0321] The distance measurement processing unit 524 may derive the actual size of the region of interest in the captured image based on the distance measurement information, and perform image recognition of the captured image based on the derived actual size. By utilizing the distance measurement information in this way, the accuracy of image recognition can be improved.

[0322] <Image Processing Flow> Next, an example of the image processing flow performed by the image processing device 500 will be explained with reference to the flowchart in Figure 29.

[0323] When image processing is started, the read processing unit 523 of the image processing device 500 acquires the input content file in step S501. For example, the read processing unit 523 may read the input content file stored in the storage unit 521. Alternatively, the read processing unit 523 may acquire the input content file from, for example, the imaging device 401 via the communication unit 522.

[0324] In step S502, the read processing unit 523 executes a read process and reads the captured image, distance measurement information, and other metadata from the input content file.

[0325] In step S503, the distance measurement processing unit 524 performs distance measurement processing and processes the captured image using the distance measurement information. At that time, the distance measurement processing unit 524 may also use other metadata.

[0326] In step S504, the display unit 525 displays a display image showing the processing result.

[0327] In step S505, the storage unit 526 stores an output content file that shows the processing result.

[0328] In step S506, the communication unit 527 transmits the output content file showing the processing result to another device.

[0329] When the process in step S506 is completed, the image processing is finished.

[0330] <Reading Process Flow> Next, an example of the reading process flow executed in step S502 of Figure 29 will be explained with reference to the flowchart in Figure 30.

[0331] When the reading process begins, the reading processing unit 523 of the image processing device 500 determines in step S521 whether or not distance measurement information is stored in the input content file. If it is determined that distance measurement information is stored in the input content file, the process proceeds to step S522.

[0332] In step S522, the readout processing unit 523 reads the captured image, distance measurement information, and other metadata from the input content file.

[0333] In step S523, the readout processing unit 523 determines whether the readout distance measurement information is 3D point cloud distance measurement data or a depth map. If it is determined that the distance measurement information is UV point cloud distance measurement data and not distance measurement data or a depth map, the process proceeds to step S524.

[0334] In step S524, the readout processing unit 523 determines whether 3D point cloud distance measurement data is required for the distance measurement utilization process. If it is determined that 3D point cloud distance measurement data is required, the process proceeds to step S525.

[0335] In step S525, the readout processing unit 523 converts the UV point cloud distance measurement data into 3D point cloud distance measurement data.

[0336] In step S526, the readout processing unit 523 determines whether or not a distortion-free captured image is required for the distance measurement utilization process. If it is determined that a distortion-free captured image is required, the process proceeds to step S527.

[0337] In step S527, the readout processing unit 523 corrects the distortion of the captured image. When the processing in step S527 is completed, the readout process ends, and the process returns to Figure 29. In other words, in this case, the distortion-corrected captured image and the 3D point cloud distance measurement data are used for the distance measurement utilization process.

[0338] Furthermore, if it is determined in step S526 that a distortion-free image is not required for the distance measurement utilization process, distortion correction of the image is omitted. In other words, the readout process ends, and the process returns to Figure 29. That is, in this case, the uncorrected image and 3D point cloud distance measurement data are used for the distance measurement utilization process.

[0339] Furthermore, if it is determined in step S524 that 3D point cloud distance measurement data is not required for the distance measurement utilization process, the conversion of UV point cloud distance measurement data is omitted. In other words, the readout process ends, and the process returns to Figure 29. That is, in this case, the captured image and UV point cloud distance measurement data are used for the distance measurement utilization process.

[0340] Furthermore, if it is determined in step S523 that the distance measurement information is distance measurement data or a depth map, the readout process ends and the process returns to Figure 29. In other words, in this case, the captured image and the distance measurement data or depth map are used for the distance measurement utilization process.

[0341] Furthermore, if it is determined in step S521 that distance measurement information is not stored in the input content file, the reading process ends and the process returns to Figure 29. In other words, in this case, since distance measurement information cannot be used, the distance measurement utilization process is canceled (omitted).

[0342] <Flow of Distance Measurement Utilization Processing 1> Next, an example of the flow of distance measurement utilization processing performed in step S503 of Figure 29 will be explained with reference to the flowchart in Figure 31. Here, we will explain the case in which focus determination is performed using distance measurement information as the distance measurement utilization processing.

[0343] When the distance measurement utilization process is started, the distance measurement utilization processing unit 524 of the image processing device 500 determines an evaluation area in the captured image for determining focus in step S541.

[0344] In step S542, the distance measurement utilization processing unit 524 acquires data of UV distance measurement points within its evaluation area.

[0345] In step S543, the distance measurement utilization processing unit 524 obtains the focus range of the lens unit from optical calibration information, etc.

[0346] In step S544, the distance measurement utilization processing unit 524 determines whether the distance Z' of the UV distance measurement point within the evaluation area is within the focus range. If it is determined to be within the focus range, the process proceeds to step S545.

[0347] In step S545, the distance measurement utilization processing unit 524 determines that it has achieved focus on a subject within the evaluation area. Once the processing in step S545 is complete, the process proceeds to step S547.

[0348] Furthermore, if it is determined in step S544 that the distance Z' of the UV distance measurement point within the evaluation area is not within the focus range, the process proceeds to step S546.

[0349] In step S546, the distance measurement utilization processing unit 524 determines that the subject within the evaluation area is not in focus. Once the processing in step S546 is completed, the process proceeds to step S547.

[0350] In step S547, the distance measurement utilization processing unit 524 generates an output content file showing the determination result. When the processing in step S547 is completed, the distance measurement utilization processing is completed, and the process returns to Figure 29.

[0351] <Flowchart of Distance Measurement Utilization Process 2> Next, another example of the flowchart of distance measurement utilization process performed in step S503 of Figure 29 will be explained with reference to the flowchart in Figure 32. Here, we will explain the case in which image recognition using distance measurement information is performed as the distance measurement utilization process.

[0352] When the distance measurement utilization process is started, the distance measurement utilization processing unit 524 of the image processing device 500 performs image recognition on the captured image in step S561 and sets candidate evaluation areas and subjects of interest.

[0353] In step S562, the distance measurement utilization processing unit 524 acquires data of UV distance measurement points within its evaluation area.

[0354] In step S563, the distance measurement processing unit 524 derives the actual size of the candidate within the evaluation region based on the depth value Z'.

[0355] In step S564, the distance measurement utilization processing unit 524 determines whether the derived actual size is valid. If it is determined that the derived actual size is valid as a candidate size, the process proceeds to step S565.

[0356] In step S565, the distance measurement processing unit 524 sets the candidate as the subject of interest. Once the processing in step S565 is complete, the process proceeds to step S567.

[0357] Furthermore, if it is determined in step S564 that the derived actual size is not a valid candidate size, the process proceeds to step S566.

[0358] In step S566, the distance measurement utilization processing unit 524 rejects the candidate. Once the processing in step S566 is complete, the process proceeds to step S567.

[0359] In step S567, the distance measurement processing unit 524 determines whether all candidates have been processed. If it is determined that there are unprocessed candidates, the process returns to step S562, and the subsequent processing is repeated for the new candidates. That is, each process from step S562 to step S567 is executed for each candidate.

[0360] Then, if it is determined in step S567 that all candidates have been processed, the process proceeds to step S568.

[0361] In step S568, the distance measurement utilization processing unit 524 generates an output content file showing the processing results described above. When the processing in step S568 is completed, the distance measurement utilization processing is finished, and the process returns to Figure 29.

[0362] By performing each process as described above, the relative relationship between imaging and distance measurement can be determined without requiring cumbersome tasks such as capturing calibration pattern charts. Therefore, it becomes easier to associate the distance measurement position in the distance measurement data with the pixels in the captured image.

[0363] <Combination with imaging device> The configuration of the image processing device 500 described above may also be implemented as the configuration of the imaging device 401. For example, the imaging device 401 shown in Figure 22 may further have the configuration of the image processing device 500 shown in Figure 28, with the storage unit 521 (Figure 28) and the storage unit 443 (Figure 22) being the same storage unit, and the read processing unit 523 (Figure 28) may acquire the content file generated by the file generation unit 442 (Figure 22) via its storage unit 443 (storage unit 521).

[0364] In that case, for example, the read processing unit 523 may, if necessary, convert the format of the distance measurement information indicating the distance measurement position in the correspondence information stored in the content file that stores the captured image, which has been acquired from the storage unit 443 (storage unit 521), based on at least one of the format and use of the distance measurement information. For example, if the distance measurement information is UV point cloud distance measurement data that associates the distance measurement position with a pixel in a UV plane parallel to the image plane, the read processing unit 523 may convert the UV point cloud distance measurement data into 3D point cloud distance measurement data that indicates the distance measurement position in 3D Cartesian coordinates or into a depth map. For example, the read processing unit 523 may perform a read process as shown in the flowchart of Figure 30.

[0365] By reading and utilizing the distance measurement information in this way, it becomes easier to associate the distance measurement position in the distance measurement data with the pixels in the captured image for a wider range of applications. For example, by converting UV point cloud distance measurement data into 3D point cloud distance measurement data or depth maps as described above, the 3D point cloud distance measurement data or depth maps can be used for processing captured images.

[0366] <7. Addendum> <Computer> The series of processes described above can be executed by hardware or by software. When the series of processes are executed by software, the programs that make up the software are installed on a computer. Here, a computer includes computers built into dedicated hardware, as well as general-purpose personal computers, for example, that can perform various functions by installing various programs.

[0367] Figure 33 is a block diagram showing an example of the hardware configuration of a computer that executes the series of processes described above using a program.

[0368] In the computer 900 shown in Figure 33, the CPU (Central Processing Unit) 901, ROM (Read-Only Memory) 902, and RAM (Random Access Memory) 903 are interconnected via a bus 904.

[0369] An input / output interface 910 is also connected to the bus 904. An input / output interface 910 is connected to an input unit 911, an output unit 912, a storage unit 913, a communication unit 914, and a drive 915.

[0370] The input unit 911 consists of, for example, a keyboard, mouse, microphone, touch panel, and input terminals. The output unit 912 consists of, for example, a display, speaker, and output terminals. The storage unit 913 consists of, for example, a hard disk, RAM disk, and non-volatile memory. The communication unit 914 consists of, for example, a network interface. The drive 915 drives removable media 921 such as a magnetic disk, optical disk, magneto-optical disk, or semiconductor memory.

[0371] In a computer configured as described above, the CPU 901 loads, for example, a program stored in the memory unit 913 into the RAM 903 via the input / output interface 910 and the bus 904, and executes it, thereby performing the series of processes described above. The RAM 903 also appropriately stores data necessary for the CPU 901 to perform various processes.

[0372] The program executed by the computer can be recorded and applied, for example, on removable media 921 such as a package medium. In this case, the program can be installed in the storage unit 913 via the input / output interface 910 by inserting the removable media 921 into the drive 915.

[0373] Furthermore, this program can also be provided via wired or wireless transmission media such as a local area network, the internet, or digital satellite broadcasting. In that case, the program can be received by the communication unit 914 and installed in the storage unit 913.

[0374] In addition, this program can be pre-installed in ROM 902 or memory unit 913.

[0375] <Applications of this technology> This technology can be applied to any configuration. For example, this technology can be applied to various electronic devices.

[0376] Furthermore, this technology can also be implemented as part of a device, such as a processor as a system LSI (Large Scale Integration) (e.g., a video processor), a module using multiple processors (e.g., a video module), a unit using multiple modules (e.g., a video unit), or a set with additional functions added to a unit (e.g., a video set). In particular, in each figure, the functional blocks of the content provision server 111 and terminal device 112 described above may each be composed of independent modules or processors, or one or more modules or processors may be configured to perform processing for multiple parts.

[0377] Furthermore, this technology can also be applied to network systems composed of multiple devices. For example, this technology may be implemented as cloud computing, where multiple devices share and collaborate on processing via a network. For example, this technology may be implemented in a cloud service that provides image (video) related services to any terminal such as computers, AV (Audio Visual) equipment, portable information processing terminals, and IoT (Internet of Things) devices.

[0378] In this specification, a system refers to a collection of multiple components (devices, modules (parts), etc.), regardless of whether all components are located in the same enclosure. Therefore, multiple devices housed in separate enclosures and connected via a network, and a single device containing multiple modules within a single enclosure, are both considered systems.

[0379] <Applicable Fields and Applications of This Technology> Systems, devices, and processing units incorporating this technology can be used in any field, such as transportation, medical care, security, agriculture, livestock farming, mining, beauty, factories, home appliances, weather, and nature monitoring. Furthermore, the applications are entirely arbitrary.

[0380] <Other> In this specification, "flag" refers to information used to identify multiple states, and includes not only information used to identify two states, true (1) or false (0), but also information capable of identifying three or more states. Therefore, the values ​​that this "flag" can take are, for example, two values, 1 / 0, or three or more values. In other words, the number of bits that constitute this "flag" is arbitrary, and can be 1 bit or multiple bits. Furthermore, identification information (including flags) is envisioned not only in the form of including the identification information itself in the bitstream, but also in the form of including difference information of the identification information relative to a certain reference information in the bitstream. Therefore, in this specification, "flag" and "identification information" include not only the information itself, but also difference information relative to the reference information.

[0381] Furthermore, various types of information (metadata, etc.) related to encoded data (bitstream) may be transmitted or recorded in any form as long as they are associated with the encoded data. Here, the term "associate" means, for example, making it possible to use (link) one data when processing the other. In other words, associated data may be combined into a single data, or they may be individual data. For example, information associated with encoded data (image) may be transmitted on a different transmission path than the encoded data (image). Also, for example, information associated with encoded data (image) may be recorded on a different recording medium (or a different recording area on the same recording medium) than the encoded data (image). Note that this "association" may not apply to the entire data, but only to a part of it. For example, an image and the information corresponding to that image may be associated with each other in any unit, such as multiple frames, one frame, or a part within a frame.

[0382] In this specification, terms such as "combine," "multiplex," "add," "integrate," "include," "store," "insert," "insert," and "place" mean combining multiple things into one, such as combining encoded data and metadata into a single data, and represent one method of "associating" as described above.

[0383] Furthermore, the embodiments of this technology are not limited to those described above, and various modifications are possible without departing from the spirit of this technology.

[0384] For example, the configuration described as a single device (or processing unit) may be divided and configured as multiple devices (or processing units). Conversely, the configurations described above as multiple devices (or processing units) may be combined and configured as a single device (or processing unit). Furthermore, it is also possible to add configurations other than those described above to the configuration of each device (or each processing unit). In addition, if the overall system configuration and operation are substantially the same, a part of the configuration of one device (or processing unit) may be included in the configuration of another device (or other processing unit).

[0385] Furthermore, for example, the program described above may be executed on any device. In that case, the device should have the necessary functions (such as functional blocks) and be able to obtain the necessary information.

[0386] Furthermore, for example, each step of a flowchart may be executed by one device, or it may be divided among multiple devices. Additionally, if a single step includes multiple processes, these processes may be executed by one device, or they may be divided among multiple devices. In other words, multiple processes included in a single step can be executed as multiple steps. Conversely, processes described as multiple steps can be combined and executed as a single step.

[0387] Furthermore, for example, a program executed by a computer may be structured so that the steps of the program are executed chronologically in the order described herein, or they may be executed in parallel or individually at necessary times, such as when a call is made. In other words, the steps may be executed in an order different from the order described above, as long as no inconsistencies arise. Moreover, the steps of this program may be executed in parallel with the processing of other programs, or in combination with the processing of other programs.

[0388] Furthermore, for example, multiple technologies relating to this technology can be implemented independently, as long as they do not create a contradiction. Of course, any multiple technologies can also be implemented in combination. For example, some or all of the technologies described in one embodiment can be implemented in combination with some or all of the technologies described in another embodiment. Also, some or all of the above-mentioned technologies can be implemented in combination with other technologies not mentioned above.

[0389] Furthermore, this technology can also be configured as follows: (1) An imaging device comprising: a calibration information acquisition unit that acquires calibration information relating to the correspondence between the coordinate systems of imaging via a lens unit and distance measurement by a distance measuring sensor; and a correspondence unit that uses the acquired calibration information to associate the distance measurement position of the distance measurement data generated by the distance measurement with the pixels of the image image generated by the imaging. (2) The imaging device according to (1), wherein the calibration information includes optical calibration information relating to the correspondence between the optical system of the lens unit and the imaging device. (3) The imaging device according to (2), wherein the optical calibration information includes at least one of the optical reference position, optical reference direction, and internal parameters. (4) The imaging device according to (2) or (3), wherein the calibration information acquisition unit acquires the optical calibration information stored in the memory unit of the lens unit. (5) The imaging device according to any one of (1) to (4), wherein the calibration information includes imager calibration information relating to the imager of the imaging device. (6) The imaging apparatus according to (5), wherein the imager calibration information includes at least one of the position of the imager, the orientation of the imager, and the readout resolution of the imager. (7) The imaging apparatus according to (5) or (6), further comprising a storage unit for storing the imager calibration information, wherein the calibration information acquisition unit acquires the imager calibration information stored in the storage unit. (8) The imaging apparatus according to any one of (1) to (7), wherein the calibration information includes distance calibration information relating to the correspondence between the distance measuring system of the distance measuring sensor and the optical system of the imaging apparatus. (9) The imaging apparatus according to (8), wherein the distance calibration information includes at least one of the distance reference position, distance reference direction, and distance direction. (10) The imaging apparatus according to (8) or (9), wherein the calibration information acquisition unit acquires the distance calibration information stored in the storage unit of the distance measuring sensor. (11) The imaging apparatus according to any one of (1) to (10), wherein the calibration information includes signal processing calibration information relating to signal processing on the captured image.(12) The imaging apparatus according to (11), wherein the signal processing calibration information includes at least one of the following: rolling shutter distortion correction amount, distortion correction amount, desqueeze magnification, a parameter for specifying the crop area of ​​the cropping process, and recording magnification. (13) The imaging apparatus according to (12), further comprising a storage unit for storing the signal processing calibration information, wherein the calibration information acquisition unit acquires the signal processing calibration information stored in the storage unit. (14) The imaging apparatus according to any one of (1) to (13), wherein the correspondence unit generates UV point cloud distance measurement data that associates the distance measurement position with the pixels in a UV plane parallel to the image plane. (15) The imaging apparatus according to (14), wherein the correspondence unit converts the distance measurement data into UV point cloud distance measurement data using the camera reference position, camera reference direction, and the calibration information. (16) The imaging apparatus according to (15), wherein the matching unit converts the distance measurement data into polar coordinate distance measurement data indicating the distance measurement position in polar coordinates, converts the polar coordinate distance measurement data into 3D point cloud distance measurement data indicating the distance measurement position in 3D Cartesian coordinates, and converts the 3D point cloud distance measurement data into UV point cloud distance measurement data by projecting the distance measurement position onto the UV plane. (17) The imaging apparatus according to any one of (14) to (16), wherein the matching unit corrects the UV point cloud distance measurement data according to the RAW correction applied to the captured image. (18) The imaging apparatus according to (17), wherein the RAW correction includes at least one of cropping, padding, and electronic image stabilization applying cropping and padding. (19) The imaging apparatus according to (18), wherein the matching unit corrects the origin position of the coordinates of the captured image when the RAW correction is applied to the captured image. (20) The imaging apparatus according to (19), wherein the correspondence unit corrects the origin position using signal processing calibration information relating to signal processing on the captured image. (21) The imaging apparatus according to any one of (14) to (20), wherein the correspondence unit corrects the UV point cloud distance measurement data according to the YC correction on the captured image.(22) The imaging apparatus according to (21), wherein the YC correction includes at least one of lens distortion correction, focal plane distortion correction, cropping / electronic image stabilization, resolution conversion, and electronic zoom. (23) The imaging apparatus according to (22), wherein the correspondence unit applies coordinate transformation and magnification to the captured image when the YC correction is performed on the captured image. (24) The imaging apparatus according to (23), wherein the correspondence unit applies coordinate transformation and magnification using signal processing calibration information relating to signal processing on the captured image. (25) The imaging apparatus according to any one of (1) to (13), wherein the correspondence unit generates auxiliary information relating to the conversion of the distance measurement data into UV point cloud distance measurement data that associates the distance measurement position with the pixels in a UV plane parallel to the image plane, and associates the generated auxiliary information with the distance measurement data. (26) The imaging apparatus according to (25), wherein the auxiliary information includes at least one of camera reference position, camera reference direction, and calibration information. (27) The imaging apparatus according to any one of (1) to (26), further comprising a storage unit that stores correspondence information relating the distance measurement position to the pixels in association with the captured image. (28) The imaging apparatus according to (27), further comprising a file generation unit that generates a content file for storing the captured image and stores the correspondence information in the generated content file, wherein the storage unit stores the generated content file. (29) The imaging apparatus according to (28), wherein the file generation unit stores UV point cloud distance measurement data relating the distance measurement position to the pixels in a UV plane parallel to the image plane as correspondence information in the content file. (30) The imaging apparatus according to (28), wherein the file generation unit stores auxiliary information relating to the conversion of the distance measurement data to UV point cloud distance measurement data relating the distance measurement position to the pixels in a UV plane parallel to the image plane, and the distance measurement data, as correspondence information in the content file.(31) The imaging apparatus according to (30), wherein the file generation unit stores the variable values ​​of the distance measurement data and the auxiliary information in the real-time metadata area of ​​the content file, and stores the fixed values ​​of the auxiliary information in the non-real-time metadata area of ​​the content file, when the captured image is a moving image. (32) The imaging apparatus according to (31), wherein the fixed values ​​include at least one of the following: camera reference position, camera reference direction, distance measurement reference position, distance measurement reference direction, and distance measurement direction. (33) The imaging apparatus according to (31) or (32), wherein the variable values ​​include at least one of the following: optical reference position, optical reference direction, internal parameter, imager position, imager orientation, rolling shutter distortion correction amount, distortion correction amount, desqueeze magnification, parameter specifying the crop area for cropping, recording magnification, and distance measurement value. (34) The imaging apparatus according to any one of (28) to (33), further comprising a read processing unit that converts the format as necessary based on at least one of the format and use of the distance measurement information indicating the distance measurement position of the correspondence information stored in the content file acquired from the storage unit. (35) The imaging apparatus according to (34), wherein the read processing unit converts the UV point cloud distance measurement data into 3D point cloud distance measurement data or a depth map indicating the distance measurement position in 3D Cartesian coordinates when the distance measurement information is UV point cloud distance measurement data that associates the distance measurement position with the pixels in a UV plane parallel to the image plane. (36) An imaging method comprising acquiring calibration information relating to the correspondence between the coordinate systems of imaging via a lens unit and distance measurement by a distance measurement sensor, and using the acquired calibration information to associate the distance measurement position of the distance measurement data generated by the distance measurement with the pixels of the image image generated by the imaging. (37) A program for causing a computer to perform a process that includes acquiring calibration information relating to the correspondence between the coordinate systems of imaging via a lens unit and distance measurement by a distance measuring sensor, and using the acquired calibration information to associate the distance measurement position of the distance measurement data generated by the distance measurement with the pixels of the image image generated by the imaging.

[0390] (41) An image processing apparatus comprising: a reading processing unit that reads distance information relating to distance measurement data generated by distance measurement by a distance measuring sensor from a content file that stores an image captured by imaging via a lens unit, and converts the format of the distance measurement information as necessary based on at least one of the format and application of the distance measurement information; and a distance utilization processing unit that performs predetermined processing using the distance measurement information whose format has been converted as necessary and the image captured. (42) The image processing apparatus according to (41), wherein the reading processing unit omits the format conversion if the distance measurement information is 3D point cloud distance measurement data or a depth map that indicates the distance measurement position of the distance measurement data in 3D orthogonal coordinates. (43) The image processing apparatus according to (41) or (42), wherein the reading processing unit omits the format conversion if the distance measurement information is UV point cloud distance measurement data that associates the distance measurement position of the distance measurement data with the pixels of the image captured in a UV plane parallel to the image plane, and 3D orthogonal transformed coordinates of the distance measurement position are not required in the application. (44) The image processing apparatus according to any one of (41) to (43), wherein the readout processing unit converts the UV point cloud distance measurement data into 3D point cloud distance measurement data that shows the distance measurement position in 3D orthogonal coordinates, if the distance measurement information is UV point cloud distance measurement data that associates the distance measurement position with the pixels of the captured image in a UV plane parallel to the image plane, and if 3D orthogonal coordinates of the distance measurement position are required in the application. (45) The image processing apparatus according to (44), wherein the readout processing unit omits distortion correction of the captured image if a distortion-free captured image is not required in the application. (46) The image processing apparatus according to (44) or (45), wherein the readout processing unit corrects distortion of the captured image if a distortion-free captured image is required in the application. (47) The image processing apparatus according to any one of (41) to (46), wherein the distance measurement utilization processing unit compares the distance measurement information with the focus range of the captured image to determine whether the captured image is in focus or not.(48) The distance measurement utilization processing unit derives the actual size of the region of interest in the captured image based on the distance measurement information, and performs image recognition of the captured image based on the derived actual size, as described in any of (41) to (47). (49) An image processing method comprising: reading distance measurement information relating to distance measurement data generated by distance measurement by a distance measurement sensor from a content file storing a captured image generated by imaging via a lens unit; converting the format of the distance measurement information as necessary based on at least one of the format and use of the distance measurement information; and performing predetermined processing using the distance measurement information whose format has been converted as necessary and the captured image. (50) A program for causing a computer to perform processing comprising: reading distance measurement information relating to distance measurement data generated by distance measurement by a distance measurement sensor from a content file storing a captured image generated by imaging via a lens unit; converting the format of the distance measurement information as necessary based on at least one of the format and use of the distance measurement information; and performing predetermined processing using the distance measurement information whose format has been converted as necessary and the captured image.

[0391] 400 Imaging system, 401 Imaging device, 402 Lens unit, 403 External distance sensor, 411 Optical control unit, 412 Lens barrel section, 413 Memory unit, 414 Communication unit, 415 Image stabilization mechanism, 421 Distance measurement control unit, 422 Distance measurement unit, 423 Communication unit, 424 Memory unit, 425 Timing generator, 426 Light emission unit, 427 Projection optical system, 428 Light receiving optical system, 429 Light receiving unit, 431 Input unit, 432 Control unit, 433 Imager, 434 Image stabilization mechanism, 435 RAW correction unit, 436 YC correction unit, 437 Encoding unit, 438 Communication unit, 439 Communication unit, 440 Calibration information acquisition unit, 441 441 Correspondence unit, 442 File generation unit, 443 Storage unit, 444 Communication unit, 451 Optical calibration information acquisition unit, 452 Imager calibration information acquisition unit, 453 Distance measurement calibration information acquisition unit, 454 Signal processing calibration information acquisition unit, 461 Parameter derivation unit, 462 Correspondence information generation unit, 463 Auxiliary information generation unit, 464 UV point cloud distance measurement data generation unit, 500 Image processing device, 501 Input unit, 502 Control unit, 521 Storage unit, 522 Communication unit, 523 Readout processing unit, 524 Distance measurement utilization processing unit, 525 Display unit, 526 Storage unit, 527 Communication unit, 900 Computer

Claims

1. An imaging device comprising: a calibration information acquisition unit that acquires calibration information relating to the correspondence between the coordinate systems of imaging via a lens unit and distance measurement by a distance measuring sensor; and a correspondence unit that uses the acquired calibration information to associate the distance measurement position of the distance measurement data generated by the distance measurement with the pixels of the image image generated by the imaging.

2. The imaging device according to claim 1, wherein the calibration information includes optical calibration information relating to the correspondence between the lens unit and the optical system of the imaging device.

3. The imaging apparatus according to claim 2, wherein the optical calibration information includes at least one of the following: an optical reference position, an optical reference direction, and internal parameters.

4. The imaging apparatus according to claim 2, wherein the calibration information acquisition unit acquires the optical calibration information stored in the memory unit of the lens unit.

5. The imaging apparatus according to claim 1, wherein the calibration information includes imager calibration information relating to the imager of the imaging apparatus.

6. The imaging apparatus according to claim 5, wherein the imager calibration information includes at least one of the position of the imager, the orientation of the imager, and the readout resolution of the imager.

7. The imaging apparatus according to claim 5, further comprising a storage unit for storing the imager calibration information, wherein the calibration information acquisition unit acquires the imager calibration information stored in the storage unit.

8. The imaging device according to claim 1, wherein the calibration information includes distance measurement calibration information relating to the correspondence between the distance measuring system of the distance measuring sensor and the optical system of the imaging device.

9. The imaging device according to claim 8, wherein the distance measurement calibration information includes at least one of the distance measurement reference position, distance measurement reference direction, and distance measurement direction.

10. The imaging device according to claim 8, wherein the calibration information acquisition unit acquires the distance measurement calibration information stored in the memory unit of the distance measurement sensor.

11. The imaging apparatus according to claim 1, wherein the calibration information includes signal processing calibration information relating to signal processing on the captured image.

12. The imaging apparatus according to claim 11, wherein the signal processing calibration information includes at least one of the following: rolling shutter distortion correction amount, distortion correction amount, desqueeze magnification, a parameter for specifying the crop area of ​​the cropping process, and recording magnification.

13. The imaging apparatus according to claim 1, wherein the correspondence unit generates UV point cloud distance measurement data that associates the distance measurement position with the pixels in a UV plane parallel to the image plane.

14. The imaging apparatus according to claim 1, wherein the correspondence unit generates auxiliary information relating to the conversion of the distance measurement data into UV point cloud distance measurement data that associates the distance measurement position with the pixels in a UV plane parallel to the image plane, and associates the generated auxiliary information with the distance measurement data.

15. The imaging apparatus according to claim 1, further comprising: a file generation unit that generates a content file and stores the captured image and correspondence information that associates the distance measurement position with the pixels in the generated content file; and a storage unit that stores the content file containing the captured image and the correspondence information.

16. The imaging apparatus according to claim 15, wherein the file generation unit stores UV point cloud distance measurement data that associates the distance measurement position with the pixels in a UV plane parallel to the image plane as the correspondence information in the content file.

17. The imaging apparatus according to claim 15, wherein the file generation unit stores the distance measurement data and the distance measurement data as the correspondence information in the content file, along with auxiliary information relating to the conversion of the distance measurement data into UV point cloud distance measurement data that associates the distance measurement position with the pixels in a UV plane parallel to the image plane.

18. The imaging apparatus according to claim 15, further comprising a read processing unit that, if necessary, converts the format based on at least one of the format and use of the distance measurement information indicating the distance measurement position of the correspondence information stored in the content file acquired from the storage unit.

19. The imaging apparatus according to claim 18, wherein the readout processing unit converts the UV point cloud distance data, which associates the distance measurement position with the pixels in a UV plane parallel to the image plane, into 3D point cloud distance data or a depth map that shows the distance measurement position in 3D Cartesian coordinates.

20. An imaging method comprising: acquiring calibration information relating to the correspondence between the coordinate systems of imaging via a lens unit and distance measurement by a distance measuring sensor; and using the acquired calibration information to associate the distance measurement position of the distance measurement data generated by the distance measurement with the pixels of the image image generated by the imaging.

Images