Image processing device and image calibration method

Abstract

Disclosed an image processing device capable of calibrating a wide-angle camera without using a large chart. The image processing device has: image processing units 20a and 20b that acquire two images captured by two cameras 10a and 10b having partially overlapping imaging regions and perform processing including correction of pixel shift in the images; a stereo parallax image generation unit 30 that generates a parallax image on the basis of the overlapping imaging regions of the two images; a road surface cross-sectional shape estimation unit 50 that generates a first distance image including distance information to three-dimensional objects for the overlapping imaging regions on the basis of the parallax image; and an image calibration unit 90 that generates the second distance image by acquiring the distance to three-dimensional objects in a monocular visual field captured only in one of the images using motion parallax obtained by movement, and generates calibration information for calibrating the amount of pixel shift correction on the basis of the first distance image and the second distance image.

Description

Image processing device and image calibration method

[0001] The present invention relates to an image processing apparatus and an image calibration method, and more particularly to an image processing apparatus and an image calibration method thereof that are suitable for use mounted on a moving object to recognize the surrounding conditions of the moving object.

[0002] Conventionally, stereo cameras have been known as devices for recognizing objects in three dimensions. Stereo cameras utilize the differences in how images are captured by multiple cameras placed at different positions to obtain parallax between cameras based on trigonometry, and use that parallax to detect the depth and position of an object, thereby accurately detecting the position of the object being measured.

[0003] In recent years, stereo cameras have been mounted on vehicles such as automobiles and applied to in-vehicle sensing technology, which detects the location of obstacles and other objects around the vehicle. In in-vehicle sensing technology, there is a demand for wider field of view to detect obstacles over a wider area and for longer-range detection to detect obstacles at greater distances, in order to accommodate a wide range of use cases. As a technology to address these demands, for example, Patent Document 1 discloses a stereo image processing device that widens the field of view and achieves a wider field of view by shifting the center positions of the sensors toward or toward the optical axis of the lenses.

[0004] Furthermore, in order to avoid the effects of dirt and other factors, in-vehicle stereo cameras are generally mounted inside the vehicle, and subjects such as obstacles are captured through the windshield. Conventionally, stereo cameras have undergone calibration work called aiming during vehicle manufacturing or inspection to correct for optical distortion and positional deviations caused by the influence of the windshield. As a method for calibrating stereo cameras that capture subjects through a transparent material such as a windshield, for example, the technology disclosed in Patent Document 2 is known. Patent Document 2 discloses obtaining a correction parameter that calibrates both absolute position deviation and relative position deviation by correcting one of the absolute position correction parameters of two cameras calculated by simulation using an image obtained by photographing a calibration chart.

[0005] Japanese Patent Publication No. 2019-32409 Japanese Patent Publication No. 2019-132855

[0006] In the technology disclosed in Patent Document 2, as described above, correction parameters are acquired using a calibration chart. However, calibrating a wide-angle image processing device, such as the one described in Patent Document 1, requires a chart large enough to cover a wide field of view, making the chart unnecessarily large. Such large charts can create problems, for example, requiring large-scale modifications to the manufacturing line where calibration work is performed at a vehicle manufacturer. Furthermore, since stereo camera calibration is also performed when the windshield is replaced, there is a risk that dealerships and other repair shops may face challenges such as securing storage space for the charts.

[0007] This invention has been made in view of the above-mentioned problems, and provides an image processing device and an image calibration method that can calibrate a stereo camera with a wide maximum field of view with high accuracy without using a large chart.

[0008] In one preferred embodiment, the image processing apparatus according to the present invention includes: an image processing unit that acquires images captured by a first camera and a second camera mounted on a moving body; a stereo parallax image generation unit that acquires parallax information relating to the parallax between the first imaging region captured by the first camera and the first imaging region captured by the second camera for a first imaging region that is captured in both images captured by the first camera and the second camera; a distance image generation unit that acquires the distance to a three-dimensional object in the first imaging region based on the parallax information and generates a first distance image including first distance information relating to said distance; and a hybrid image including the first imaging region and the second imaging region from the first distance image and an image of the second imaging region captured by either the first camera or the second camera but not captured by the other camera, and acquires the distance to a three-dimensional object captured in the second imaging region based on the first distance information. The system includes a BRID 3D object detection unit, a moving parallax image generation unit that generates a moving parallax image including moving parallax information relating to the parallax between the first image and the second image, using either the first image or the second image as a reference image, based on a first image captured at a first time including a second imaging area and a second image captured at a second time different from the first time including a second imaging area, and using either the first image or the second image as a reference image, a movement amount acquisition unit that acquires the amount of movement of a moving object between the first time and the second time, a distance conversion unit that generates a second distance image including second distance information relating to the distance to a 3D object captured in the second imaging area of ​​the reference image, based on the moving parallax image and the amount of movement, and a calibration unit that generates calibration information used to calibrate correction information for correcting distortion of an image captured by a first camera or a second camera that captured an image including a second imaging area, based on a first distance image and a second distance image generated based on an image captured at the same time as the reference image.

[0009] Furthermore, in another preferred embodiment, the image processing apparatus according to the present invention includes: an image processing unit that acquires an image from a camera mounted on a moving body; a distance calculation unit that acquires the distance to a three-dimensional object based on the position of the three-dimensional object in the image; a moving parallax image generation unit that generates a moving parallax image including moving parallax information relating to the parallax between a first image and a second image, using either the first image or the second image as a reference image, based on a first image taken at a first time and a second image taken at a second time different from the first time; a movement amount acquisition unit that acquires the amount of movement of the moving body between the first time and the second time; a distance conversion unit that generates a distance image including distance information relating to the distance to a three-dimensional object in the reference image based on the moving parallax image and the amount of movement; and a calibration unit that generates calibration information used to calibrate correction information for correcting distortion of an image captured by a camera, based on the position of the three-dimensional object on the reference image and distance information.

[0010] Furthermore, in one preferred embodiment, the image calibration method according to the present invention acquires parallax information relating to the parallax between the image captured by the first camera and the image captured by the second camera for a first imaging region that is captured in both of two images captured by a first camera and a second camera mounted on a moving object, acquires the distance to a three-dimensional object in the first imaging region based on the parallax information, and generates a first distance image including first distance information relating to said distance, and a first image captured at a first time that includes a second imaging region captured by either the first camera or the second camera but not by the other camera, and a second image captured at a different time that includes the second imaging region Based on the second image captured at the first time, a moving parallax image is generated, using either the first or second image as a reference image, and including moving parallax information relating to the parallax between the first and second images. The amount of movement of the moving object between the first and second time points is obtained. Based on the moving parallax image and the amount of movement, a second distance image is generated, including second distance information relating to the distance to the three-dimensional object captured in the second imaging area of ​​the reference image. Based on the first and second distance images generated based on the reference image, calibration information is generated, which is used to calibrate correction information for correcting distortion of images captured by the first or second camera that captured the image including the second imaging area.

[0011] According to the present invention, it is possible to calibrate cameras with a wide maximum field of view with high accuracy without using large charts. Other novel features of the present invention and the technical problems solved thereby will become apparent from the description and drawings herein.

[0012] This is a schematic block diagram showing the logical configuration of the first embodiment of the image processing apparatus according to the present invention. This is a schematic diagram showing the imaging area by the camera. This is a schematic block diagram showing an example of the configuration of the image processing unit. This is a schematic diagram showing the relationship between the windshield and the incident light to the camera on a horizontal plane viewed from above the vehicle. This is a schematic diagram showing the relationship between the windshield and the incident light to the camera on a vertical plane viewed from the side of the vehicle. This is a schematic diagram showing the change in the position of the image formed on the image sensor with and without the windshield. This is a schematic diagram showing the change in the amount of pixel shift in the vertical direction. This is a schematic block diagram showing an example of the configuration of the image calibration unit. This is a schematic diagram for explaining the distance error between the first distance image and the second distance image. This is a schematic diagram for explaining the acquisition of the distance to an object using a moving parallax image. This is a schematic diagram showing the amount of pixel shift in the horizontal field of view direction of a stereo parallax image based on two images obtained by moving the camera. This is a schematic diagram showing the magnitude of the parallax error in the horizontal field of view direction of a stereo parallax image based on two images obtained by moving the camera. This is a graph showing an example of the relationship between the field of view and the difference in the field of view to the object in the two images. This is a schematic diagram showing the movement of the camera in the direction of alignment. This is a schematic diagram showing movement in a direction different from the camera arrangement direction. This is a schematic block diagram showing the logical configuration of the image processing device in the second embodiment. This is a schematic block diagram showing the logical configuration of the image processing device in the third embodiment. This is a schematic diagram showing the imaging area in the third embodiment.

[0013] Hereinafter, representative embodiments of the present invention will be described with reference to the drawings. Note that the embodiments and drawings described below are illustrative examples for illustrating the present invention, and have been omitted or simplified as appropriate for clarity of explanation. Furthermore, please note that the positions, sizes, shapes, and extents of the components shown in the drawings may not necessarily accurately represent them, in order to facilitate understanding of the invention.

[0014] Figure 1 is a schematic block diagram showing the logical configuration in a first embodiment of the image processing apparatus according to the present invention.

[0015] The image processing device 1 is used, for example, mounted on a vehicle such as an automobile, to detect the distance to other vehicles, buildings, pedestrians, and other three-dimensional objects in the vicinity of the vehicle. In this embodiment, an image processing device mounted on a vehicle is described as an example, but it should be noted that the image processing device of this embodiment can also be applied to other moving objects and other applications.

[0016] Image processing device 1 is connected to cameras 10a and 10b, which constitute a stereo camera, to form a stereo camera system. The image processing device 1 may be housed together with cameras 10a and 10b in a single housing, or they may be housed separately in different housings. In the latter case, the image processing device 1 is connected to cameras 10a and 10b, for example, via an in-vehicle network (not shown), controls these cameras via the in-vehicle network, and acquires images captured by these cameras. In the following description, when it is not necessary to distinguish between cameras 10a and 10b individually, the alphabetical subscripts attached to their reference numbers are omitted, and they are simply referred to as camera 10.

[0017] Camera 10 is equipped with a lens and an image sensor (not shown), and the image sensor converts the light guided through the lens to the image sensor into an electrical signal, which is then acquired as image information. The image processing device 1 is configured to detect three-dimensional objects in the surroundings based on the image acquired by camera 10 and to issue alarms, etc.

[0018] Figure 2 is a schematic diagram showing the area captured by the camera 10. In Figure 2, A shows the area around the vehicle 100 on which the image processing device 1 is mounted, as seen from above. B shows the image captured by the camera 10.

[0019] In this embodiment, camera 10a is mounted on the right side of the vehicle 100 to capture the area in front of the vehicle 100. Similarly, camera 10b is mounted on the left side of the vehicle to capture the area in front of the vehicle 100. Here, "right side" and "left side" refer to the relative positions of the two cameras, and do not mean their absolute positions on the vehicle 100. Cameras 10a and 10b are positioned on the vehicle 100 such that a portion of the area they each capture (area 110) overlaps, forming a so-called stereo camera. Hereafter, the overlapping area 110 will be referred to as the stereo viewing area. In this embodiment, the image captured by camera 10a includes the stereo viewing area 110, as well as the area 111 to the front left of the vehicle, which is not captured by camera 10b. Similarly, the image captured by camera 10b includes the stereo viewing area 110, as well as the area 112 to the front right of the vehicle, which is not captured by camera 10a. The region captured by only one of camera 10a or camera 10b is referred to here as the monocular view region.

[0020] The image processing device 1 can measure the distance to a three-dimensional object, for example, the distance to a pedestrian 120 as a three-dimensional object, based on the parallax between two images captured by cameras 10a and 10b in the stereo viewing area 110. On the other hand, in the monocular viewing areas 111 and 112, the distance to a three-dimensional object outside the stereo viewing area 110, for example, the pedestrian 120, is measured by utilizing the distance measurement results based on the parallax in the stereo viewing area 110.

[0021] The distances measured in monocular areas 111 and 112 are estimated using the distance measurement results in stereo viewing area 110, and therefore their accuracy is inferior to that of the distance measurement results in stereo viewing area 110. However, by shifting the imaging areas of camera 10a and camera 10b, it becomes possible to detect three-dimensional objects over a wider range (wider angle) compared to the case where the imaging areas of camera 10a and camera 10b completely overlap and the entire area is imaged as a stereo viewing area.

[0022] Returning to Figure 1, the image processing device 1 includes an image processing unit 20a, an image processing unit 20b, a stereo disparity image generation unit 30, a monocular image generation unit 40, a road surface cross-section shape estimation unit 50, a stereoscopic object detection unit 60, a hybrid object detection unit 70, an alarm control unit 80, and an image calibration unit 90.

[0023] The image processing units 20a and 20b each function as image acquisition units that acquire images captured from the corresponding cameras 10a and 10b, and perform predetermined processing on the acquired images. The specific processing performed by the image processing units 20a and 20b will be described later. In the following description, as with the camera 10, if it is not necessary to distinguish between the image processing units 20a and 20b individually, the alphabetical subscript will be omitted and they will be referred to as the image processing unit 20.

[0024] The stereo parallax image generation unit 30 performs a matching process using the two images processed by the image processing units 20a and 20b, and generates a stereo parallax image that includes parallax information regarding the parallax between the two images for a three-dimensional object present in the stereo viewing region 110. The matching process may be performed on the entire image including the monocular viewing regions 111 and 112, or it may be performed only on the stereo viewing region 110 after extracting the stereo viewing region 110 from the two images. In the latter case, the amount of computation can be reduced and the processing time can be shortened compared to the former. Furthermore, the stereo parallax image generation unit 30 may be configured to, for example, analyze the brightness of the image based on the image acquired from the image processing unit 20a, and then provide feedback control to the exposure amount and sensitivity of the camera 10 so that the next image captured is captured with appropriate brightness and the appearance of the same subject captured by the image processing unit 20a and the image processing unit 20b is as similar as possible.

[0025] The monocular image generation unit 40 extracts images of monocular regions 111 and 112 from the two images processed by the image processing unit 20 and generates a monocular image. At this time, the monocular image generation unit 40 applies a projection transformation to each image so that the unit length along the same horizontal line of each monocular image represents an equal distance from one another.

[0026] The road surface cross-section shape estimation unit 50 estimates the distance to a three-dimensional object from which parallax information has been acquired based on the parallax information of the stereo parallax image, generates a first distance image including first distance information relating to the estimated distance, and estimates the cross-sectional shape of the road surface that the vehicle 100 is expected to travel on.

[0027] The stereoscopic object detection unit 60 detects three-dimensional objects in the stereoscopic viewing area 110 based on the stereo disparity image generated by the stereo disparity image generation unit 30, and identifies what type of three-dimensional object it is, such as a pedestrian, bicycle, vehicle, or building. If the identified three-dimensional object is a moving object such as a pedestrian, bicycle, or vehicle, the stereoscopic object detection unit 60 also estimates the position of the moving object and further tracks the moving object to estimate its direction of movement and speed.

[0028] The hybrid 3D object detection unit 70 generates a hybrid image from the first distance image generated by the road surface cross-section shape estimation unit 50 and the left and right monocular images generated by the monocular image generation unit 40, and detects 3D objects based on the hybrid image. The hybrid image is generated by placing monocular images on both sides of the stereoscopic viewing area of ​​the distance image and combining them, as shown as image B in Figure 2.

[0029] The hybrid 3D object detection unit 70 detects 3D objects present in the monocular view portion of the hybrid image and obtains the distance to those 3D objects based on first distance information included in the stereoscopic view portion. For example, if the 3D object (object to be measured) detected in the monocular view portion is a person, the distance to the person is estimated by detecting the position of the person's feet in the vertical direction of the monocular view image and obtaining first distance information for the vertical image position corresponding to the foot position in the stereoscopic view portion of the image. Similarly, if the object to be measured is a vehicle, the distance to the vehicle is estimated by detecting the contact surface between the vehicle and the ground. The hybrid 3D object detection unit 70 identifies the type of the detected 3D object in the same way as the stereoscopic 3D object detection unit 60, and estimates the position of the 3D object identified as a moving object, as well as its direction of movement and speed of movement.

[0030] The alarm control unit 80 utilizes the detection results of three-dimensional objects from the stereoscopic object detection unit 60 and the hybrid three-dimensional object detection unit 70 to perform driver assistance control to support vehicle operation by the driver, or preventive safety control such as issuing alarms and emergency braking in emergencies. For example, if the detected three-dimensional object is identified as a vehicle, the detection result will be used to perform follow control to the preceding vehicle or emergency braking control. If the detected three-dimensional object is identified as a pedestrian or bicycle, emergency braking control or alarm activation control will be performed.

[0031] The image calibration unit 90 performs calibration of correction information used to correct image distortion caused by the influence of the vehicle's windshield, which will be described later. Based on the monocular image generated by the monocular image generation unit 40 and the estimation results from the road surface cross-section shape estimation unit 50, the image calibration unit 90 detects the amount of image shift on the monocular image due to the influence of distortion and estimates a pixel shift correction amount to correct it.

[0032] In this embodiment, the pixel shift correction amount estimated by the image calibration unit 90 is supplied to each image processing unit 20 and used to calibrate the correction information for eliminating the effects of distortion in image processing in the image processing unit 20.

[0033] Physically, the image processing apparatus 1 of this embodiment is configured to include a computing device (not shown), memory, and an interface for the computing device to exchange data with the camera 10 and other control devices. The functions of each of the above-mentioned parts, excluding the camera 10, are realized when the computing device executes a control program stored in memory. The computing device can be a processing device such as a CPU (Central Processing Unit) and / or a GPU (Graphics Processing Unit). Some or all of the functions of each of the above-mentioned parts may be realized by hardware such as an ASIC (Application Specific Integrated Circuit) or FPGA (Field Programmable Gate Array).

[0034] Hereinafter, the image processing unit 20 and the image correction unit 90 will be described in detail with reference to the drawings. Specific processes performed by the stereo disparity image generation unit 30, the monocular vision image generation unit 40, the road surface cross-sectional shape estimation unit 50, the stereo vision solid object detection unit 60, and the hybrid solid object detection unit 70 are known from, for example, Japanese Patent Application Laid-Open No. 2019-106026, and detailed descriptions thereof are omitted here.

[0035] FIG. 3 is a schematic block diagram showing an example of the configuration of the image processing unit 20a. The image processing unit 20a and the image processing unit 20b have the same configuration, and the following description is common to both. Therefore, it should be noted that in FIG. 3 and the following description, the alphabetic suffixes added to the reference numerals are omitted for the purpose of distinguishing between the two.

[0036] When the image processing unit 20 acquires the image captured by the camera 10 by the image acquisition unit 210, the affine processing unit 220 performs an affine transformation process on the acquired image using the parameters held in the affine table storage unit 230. The coordinate transformation process performed by the affine transformation process may be either linear or non-linear. In the present embodiment, a fisheye lens with the widest maximum angle of view is used for the lens of the camera 10, and fsinθ in the orthographic projection of the fisheye lens is projected and transformed into the coordinate system of (ftanθx, ftanθy) by the affine transformation process. Here, f is the focal length of the fisheye lens, θ is the incident angle (angle of view) of the light ray with respect to the lens, and θx and θy are the horizontal and vertical components of the incident angle θ in the orthogonal coordinate system. Also, in the present embodiment, the displacement of the pixels in the vertical direction (referred to as pixel shift) due to the influence of the windshield is corrected during the affine transformation process in the affine processing unit 220. That is, the affine processing unit 220 also functions as a pixel shift correction processing unit that corrects the pixel shift accompanying the image distortion due to the influence of the windshield. Note that in the affine processing unit 220, other distortion transformation processes may be executed in addition to the above-described processes.

[0037] The affinity table storage unit 230 stores an affinity table used by the affinity processing unit 220. The affinity table holds parameters used when adjusting the position and angle of an image captured by the camera 10 through affinity processing. The parameters include those that are calibrated by the pixel shift correction amount estimated by the image calibration unit 90 and serve as correction information for pixel shift correction. The affinity table is actually stored using, for example, an area in a memory (not shown) provided in the image processing device 1.

[0038] The luminance correction unit 240 corrects the luminance of each pixel in the image converted by the affinity processing unit 220. In the luminance correction unit 240, for example, based on the gain of the camera 10 that captured the image, the difference in sensitivity of each pixel in the image, etc., the luminance of each pixel in the image is corrected so as to obtain a luminance suitable for processing in subsequent processing units.

[0039] The pixel interpolation unit 250 performs a demosaicking process on the image whose luminance has been corrected, and converts the RAW image into a color coordinate image.

[0040] The luminance information generation unit 260 obtains the luminance information of each pixel based on the color information of each pixel of the image converted into a color image. This luminance information is used, for example, when the stereo disparity image generation unit 30 generates a disparity image. The image processing unit 20 passes the image processed by the processing of each unit described above to the stereo disparity image generation unit 30 and the monocular vision image generation unit 40.

[0041] Hereinafter, prior to a specific description of the image calibration unit 90, the influence of the front glass on the captured image will be described.

[0042] As described above, in this embodiment, in the hybrid stereo object detection unit 70, the distance to these objects is estimated using the positions of the grounding parts such as the feet of people and vehicles in the monocular vision areas 111 and 112. Therefore, if the detection position is shifted in the vertical direction of the image due to the influence of the front glass, an error will occur in the measured distance.

[0043] Figure 4 is a schematic diagram showing the relationship between the windshield and the incident light on the camera in a horizontal plane as viewed from above the vehicle.

[0044] Light rays entering the camera 10 are refracted according to Snell's law as they pass through the windshield 200. If the inclination of the incident and exit surfaces of the light rays on the windshield 200 is the same, then the incident and exit angles of the light rays are the same, and the inclination of the light rays is the same before and after passing through the windshield 200. On the other hand, if there is a difference in the inclination of the incident and exit surfaces of the light rays on the windshield 200, the larger the difference, the larger the difference between the incident and exit angles of the light rays, and the larger the inclination of the light rays before and after passing through the windshield 200.

[0045] Generally, the windshield 200 has a curved shape, and if the angle of incidence is minimal, the inclination of both sides of the windshield 200 through which the light ray passes is equal. As the angle of incidence increases, the difference between the inclination of the surface of the windshield 200 from which the light ray enters and the inclination of the surface of the windshield from which the light ray exits becomes larger. The camera 10 that images the front of the vehicle is often installed near the center of the windshield 200 in the horizontal direction. Therefore, a light ray entering the camera 10 from the front of the vehicle passes through a point where the inclination of the outer and inner surfaces of the windshield 200 is almost the same, and the inclination of the light ray does not change much before and after passing through the windshield 200. For example, the angle of the light ray R00 entering the camera 10a from the front of the vehicle does not change much before and after passing through the windshield 200. On the other hand, a light ray from the diagonal front of the vehicle, such as a light ray R01, enters the windshield 200 at an angle, so the direction of the light ray changes as it passes through the windshield 200. Therefore, objects located to the left or right of the front of the vehicle will appear to be in a horizontally shifted position from their actual location.

[0046] Figure 5 is a schematic diagram showing the relationship between the windshield and the incident light on the camera in a vertical plane viewed from the side of the vehicle.

[0047] Generally, the windshield 200 of a vehicle is mounted at an angle to the vertical. As a result, light rays from the front of the vehicle enter the windshield 200 at an overall angle. For example, as shown in Figure 5, when viewed in the vertical direction, the direction of the light rays R10 passing near the optical axis of the camera 10a and the light rays R11 at the lower field of view change significantly before and after passing through the windshield 200. As a result, when viewing an object through the windshield 200, the position of the object appears to be shifted downwards in the vertical direction compared to its actual position.

[0048] Figure 6 is a schematic diagram showing the change in the position of the image on the image sensor of camera 10a with and without the windshield. Here, camera 10a is used as an example, but the same applies to camera 10b. Note that the image actually captured on the image sensor 300 will be inverted vertically and horizontally compared to the diagram, but for convenience, this explanation will assume that it will be the same as what a person sees. In addition, it is assumed that there is no distortion in the optical system of camera 10a other than that caused by the windshield 200.

[0049] Now, consider four objects whose images are formed in the dashed regions 330, 331, 332, and 333 on the image sensor 300 when the windshield 200 is absent. Region 330 is the region near the optical axis 310 of the camera 10a, region 331 is the wide-angle region on the left along the horizontal line 320 passing through the optical axis 310, region 332 is the wide-angle region on the lower side along the vertical line 321 passing through the optical axis 310, and region 333 is the wide-angle region on the left along the horizontal line 322 passing through region 332. When the objects whose images are formed in these regions when the windshield 200 is absent are imaged with the windshield 200 present, as described above, the path of the light rays changes when they pass through the windshield 200, so the regions in which the images of the objects are formed change to the solid regions 340, 341, 342, and 343, respectively. The arrows in each region in the figure schematically show the direction and magnitude of the change in the position of the object's image.

[0050] Generally, in a camera, images are captured by focusing light rays incident on the lens onto an image sensor 300. If the path of the light rays is shifted, the position of the image on the image sensor also changes. This shift in the path of the light rays to the image sensor is called pixel shift. The amount of pixel shift caused by the presence of the windshield varies depending on the position on the image sensor, and this appears as distortion in the captured image.

[0051] Figure 7 is a schematic diagram showing the change in the amount of vertical pixel shift that occurs when light rays pass through the windshield 200.

[0052] As described above, in this embodiment, the distance estimation of monocular areas 111 and 112 performed by the hybrid 3D object detection unit 70 is carried out by detecting the ground contact position of the object in the vertical direction (up and down direction on the image) of the monocular image, such as a person's feet. For this reason, the amount of pixel shift in the vertical direction of the image will affect the distance estimation. Furthermore, the distance estimation is performed using the distance measured in the stereoscopic area 110 near the central field of view in the horizontal direction. Consequently, if the amount of pixel shift in the monocular image changes in relation to the amount of pixel shift in the stereoscopic area 110 near the central field of view, an error will occur in the estimated distance.

[0053] In Figure 7, the vertical axis of the graph shows the change in the amount of pixel shift in the vertical direction, with the amount of pixel shift at zero horizontal field of view as the reference. The range of the horizontal field of view indicated by arrow 401 shows the change in the amount of pixel shift in the stereo viewing region 110, and the range of the horizontal field of view indicated by arrow 402 shows the change in the amount of pixel shift in the monocular viewing regions 111 and 112. As shown in the figure, the change in the amount of pixel shift increases as the horizontal field of view increases, and it is understood that the rate of change is larger in the range of the horizontal field of view that constitutes the monocular viewing region compared to the stereo viewing region. The image distortion caused by this change in the amount of pixel shift can cause errors in the estimation of distance in the monocular viewing regions 111 and 112, and can be a factor that hinders the normal operation of the image processing device 1. Therefore, in this embodiment, the amount of pixel shift is corrected in the affine processing unit 220.

[0054] The amount of pixel shift depends on the shape of the windshield 200, its tilt when mounted, and the mounting position of the camera. In particular, the amount of pixel shift increases when the radius of curvature of the windshield 200 is small or when the windshield 200 is mounted with a large tilt relative to the vertical. Furthermore, the radius of curvature and tilt of the windshield 200 vary from vehicle to vehicle on which the image processing device 1 is mounted, and the amount of pixel shift changes accordingly. Therefore, the amount of pixel shift correction needs to be calibrated for each vehicle. In this embodiment, the image calibration unit 90 acquires calibration information that is used in the affine processing unit 220 to obtain an appropriate correction amount.

[0055] Figure 8 is a schematic block diagram showing an example of the configuration of the image calibration unit 90. The image calibration unit 90 includes, for example, an image recording unit 91, a moving parallax image generation unit 92, a distance conversion unit 93, a movement amount acquisition unit 94, and a pixel shift correction amount calibration unit 95.

[0056] The image recording unit 91 records at least two images taken at different times for each of the cameras 10a and 10b. The two images are taken by the same camera 10 at positions shifted in the direction of the vehicle's movement. These images are recorded, for example, using a memory area (not shown) provided by the image processing device 1.

[0057] The moving parallax image generation unit 92 takes two images captured by the same camera 10 and recorded by the image recording unit 91, transforms them so that they become equivalent to images viewed from a direction perpendicular to the direction of vehicle movement, and performs matching processing in the same way as the stereo parallax image generation unit 30 to generate a stereo parallax image. The stereo parallax image generated here is an image that includes parallax information (moving parallax information) obtained by the matching processing, and hereafter this image will be referred to as the moving parallax image. The moving parallax image is a parallax image that includes parallax when viewed from a direction perpendicular to the direction of vehicle movement.

[0058] The distance conversion unit 93 generates a second distance image in the direction of movement based on the moving parallax image and the distance between the imaging points of the two images obtained by the movement amount acquisition unit 94. The specific method for generating the distance image based on the moving parallax image will be described later.

[0059] The movement amount acquisition unit 94 acquires the distance between two points where images used to generate the moving parallax image are captured as the movement amount of the camera 10. The distance between two points where images are captured can be acquired, for example, using distance information measured in the stereo viewing area. If the vehicle has other distance measuring devices that have the function of acquiring distance information, such as LiDAR (Light Detection and Ranging), millimeter-wave radar, or other camera devices, information on movement such as moving speed and moving distance may be obtained from these devices via the in-vehicle network. Alternatively, although less accurate, it is also possible to acquire the distance between two imaging points based on information such as the vehicle's speed and tire rotation speed obtained via the in-vehicle network, and information on the time when the images used to generate the moving parallax image were captured.

[0060] The pixel shift correction amount calibration unit 95 estimates the pixel shift amount based on the second distance information contained in the second distance image and the first distance information contained in the first distance image generated by the road surface cross-section shape estimation unit 50 based on the image used to obtain the second distance image.

[0061] Figure 9 is a schematic diagram illustrating the distance error between the first and second distance images.

[0062] Figure 9 shows an example where a plane parallel to the alignment direction of cameras 10a and 10b is imaged on the road surface at a predetermined distance from the vehicle. Arrow 500 indicates the horizontal field of view range corresponding to the stereo viewing area 110. Arrows 501 and 502 indicate the horizontal field of view ranges corresponding to the monocular viewing areas 111 and 112, respectively. The dashed line 510 shows the distance error of the first distance image, and the solid line 520 shows the distance error of the second distance image. As explained with reference to Figure 7, the first distance image has a vertical pixel shift, resulting in a distance error. On the other hand, the distance error of the second distance image is very small, as will be described later. Therefore, by calibrating the parameters used in the affine processing unit 220 using a pixel shift amount that matches the second distance image to the first distance image, and correcting the pixel shift of the image used to generate the first distance image in the affine processing unit 220, the first distance image can be made correct. This makes it possible to improve the accuracy of the distance estimated by the hybrid 3D object detection unit 70. Specifically, the pixel shift amount can be obtained by using feature points on the second depth image, such as a person's feet, the contact surface of a vehicle's tires, or a structure like a building, comparing these feature points with any point on the first depth image that has distance information indicating the same distance as the feature points, and calculating the difference in the vertical position (pixel coordinate position) between them on the image.

[0063] The above processing is performed using images captured by camera 10a and camera 10b, respectively, and the pixel shift correction amount is estimated for camera 10a and camera 10b. The pixel shift correction amount obtained in this way is sent as calibration information to image processing unit 20a and image processing unit 20b, respectively, and is used to calibrate the parameters stored in the affine table storage unit 230 of image processing unit 20a and image processing unit 20b, respectively.

[0064] Furthermore, the amount of pixel shift can be obtained using features such as those on a person's feet, the contact patch of a vehicle's tires, or structures like buildings. In addition, if parameter calibration is performed at a car manufacturer's factory or a dealer's service center, patterns such as charts displayed on the road surface or charts standing upright on the ground may be used. Even in this case, since the estimation of the amount of pixel shift utilizes the movement of the camera position, the size of the chart used does not need to be large.

[0065] Figure 10 is a schematic diagram illustrating the acquisition of distance to an object using moving parallax images performed by the distance conversion unit 93. Here, we will explain the case where the distance to an object 600 captured in two images taken by camera 10a at times t1 and t2 as the vehicle moves is acquired. For simplicity, we will use an object 600 on the right side of the vehicle as an example, but the distance to an object on the left side of the vehicle can be determined in the same way. Similarly, the distance to an object captured by camera 10b can also be determined in the same way.

[0066] Now, let's consider camera 10a at time t1 as a virtual camera 610, and camera 10a at time t2 as a virtual camera 611. In this case, if we consider images captured by cameras 610 and 611 in a direction perpendicular to the direction of movement, these two images are composed of cameras 610 and 611, and the distance traveled B ｄ This can be seen as an image captured by a stereo camera with a baseline length of . Therefore, the distance Ld from camera 611 to object 600 in a direction perpendicular to the direction of movement is,

[0067]

[0068] It can be obtained as follows: Here, d ｄ is the parallax between the two images, and f is the focal length of cameras 610 and 611, i.e., the focal length of camera 10a. In practice, since the images captured by cameras 610 and 611 are images in the direction of movement, the captured images need to be converted so that they are equivalent to images in the direction perpendicular to the direction of movement.

[0069] Then, the angle formed by the straight line connecting the camera 611 and the object 600 with respect to the direction of vehicle movement is α. １ Therefore, the distance L in the direction of movement from camera 611 to object 600 is,

[0070]

[0071] It can be obtained as follows: Here, α １ Since the projection of the camera 10's lens is known, the position of the object 600 on the image can be determined from this. In this embodiment, the position of the object in the horizontal direction on the image is determined from the optical axis position on the image sensor of the camera 611 to fsinα １ Therefore, the position will be such that α is determined based on the focal length and the distance from the optical axis position on the image to the object 600. １ This can be obtained. １ Although the windshield may cause some error, it is a negligible amount and therefore not a problem.

[0072] In this embodiment, the moving parallax image generation unit 92 converts the image captured by the camera 10 to be equivalent to an image perpendicular to the direction of movement, thereby generating a moving parallax image. Based on this moving parallax image, d ｄ This can be obtained. The distance conversion unit 93 can use the moving parallax image to obtain the distance L based on equations 1 and 2 and generate a distance image corresponding to the moving parallax image. The distance conversion unit 93 converts the distance image thus generated to become an image in the direction of movement and generates a second distance image that includes distance information in the direction of movement (second distance information). Here, the distance is obtained using the image captured at time t2 as the reference image, but the distance from the camera 610 to the object 600 may also be obtained using the image captured at time t1 as the reference image. Since the second distance image is generated based on the reference image, the first distance image that is compared with the second distance image by the distance conversion unit 93 also uses a distance image generated based on the reference image (an image captured at the same time as the reference image). The distance information contained in the second distance image generated in this way can be treated as correct for the reasons explained below.

[0073] FIG. 11 is a schematic diagram showing the pixel shift amount in the horizontal angular direction of a stereo parallax image based on two images obtained by moving the camera 10.

[0074] The graph in FIG. 11 shows the relationship between the horizontal angular position (horizontal axis) of the image captured by the camera 10 and the pixel shift amount in the horizontal angular direction at the horizontal angular position (vertical axis). The dashed-dotted line 700 indicates the pixel shift amount in the horizontal angular direction of the image g1 captured before the movement (corresponding to the image captured by the camera 610 in FIG. 10), and the dotted line 710 indicates the pixel shift amount in the horizontal angular direction of the image g2 captured after the movement (corresponding to the image captured by the camera 611 in FIG. 10). This pixel shift amount is a factor that causes a distance error in the second distance information included in the second distance image generated by the distance conversion unit 93. The parallax error between the two images can be shown as the difference between the pixel shift amount of the image g1 and the pixel shift amount of the image g2 at the same horizontal angle. The characteristics of the pixel shift amounts of the image g1 and the image g2 (the dashed-dotted line 700 and the dotted line 710) are shifted in the horizontal direction. This shift corresponds to the change in the horizontal angular angle to the object accompanying the movement of the camera 10 (the angle α １ and the angle α ０ in FIG. 9). Since the change in the horizontal angular angle is very small with respect to the maximum horizontal angular angle of the stereo camera composed of the cameras 10a and 10b, the parallax error is small over the entire horizontal angular range, as shown by the solid line 620 in FIG. 12.

[0075] FIG. 13 is a graph showing an example of the relationship between the angular range and the difference in the angular range to the object in two images. Here, B ｄ in FIG. 10 is set to 1 m, and the relationship between the angular range to the object in the horizontal direction (angle α ｄ ) and the angular range difference (angle α １ -angle α １ ) when L ０ is changed to 30, 50, and 70 m is shown. From FIG. 13, L ｄIt can be seen that using a larger object reduces the difference in horizontal field of view. Furthermore, the difference decreases as the horizontal field of view increases. As shown in Figure 11, the amount of pixel shift is greater in the wide-angle portion than in the center of the field of view, but in the wide-angle portion, the difference in horizontal field of view between the two images is small, so as shown in Figure 12, there is almost no parallax error caused by the difference in field of view.

[0076] From the above, the distance indicated by the distance image obtained based on the moving parallax image is approximately correct. Therefore, by calibrating the parameters stored in the affine table with the pixel shift correction amount acquired by the image calibration unit 90, it becomes possible to measure distances with high accuracy over a wide field of view.

[0077] In this explanation, we have used a stereo camera that images the area in front of the vehicle as an example, and therefore described the direction of movement of camera 10 as being perpendicular to the direction in which the two cameras 10 are aligned. However, the direction of movement of the camera is not limited to this. For example, in the case of a stereo camera that images the area to the side of the vehicle using camera 10, the direction of movement may be the direction in which the cameras 10 are aligned, as shown in Figure 14. Alternatively, the movement may be in a direction different from the direction in which the cameras 10 are aligned, as shown in Figure 15. In these cases as well, the same effects as described above can be obtained.

[0078] As described above, according to this embodiment, by shifting the imaging areas of the two cameras constituting a monocular stereo camera to widen the field of view, the amount of pixel shift in the monocular image captured by the stereo camera is calibrated using the camera's moving parallax, making it possible to perform highly accurate measurements over a wide field of view. Since the calibration of the pixel shift amount is performed using the camera's moving parallax, it can be done without using a large chart that covers a wide field of view for calibration.

[0079] In this embodiment, the distance the camera moves as the vehicle is driven is used as the baseline length for obtaining a moving parallax image. However, in places such as an automobile manufacturer's production site or a dealer's service center, a moving parallax image may be obtained by fixing the vehicle with a jig and moving it by a predetermined amount. In this case, the amount of vehicle movement measured externally may be provided to the movement amount acquisition unit 94 from an external device that can be connected to the in-vehicle network.

[0080] The image processing device 1 of this embodiment can be applied, for example, to an automatic emergency braking (AEB), which is one of the safety functions for preventing or mitigating traffic accidents. In an AEB, objects around the vehicle are detected, the distance to those objects is measured, their movement is tracked to estimate their speed, and control such as warnings and braking is performed. In particular, for moving objects, it is preferable to be able to detect their presence over a wider range and measure their distance and speed. The image processing device of this embodiment can accurately measure the distance of objects in a wide-angle range and can be used for more appropriate control such as warnings and braking.

[0081] Figure 16 is a schematic block diagram showing the logical configuration of the image processing device in the second embodiment. The image processing device 2 in this embodiment also basically has the same configuration as the image processing device 1 in the first embodiment. For this reason, in Figure 16, the same reference numerals as in Figure 1 are used for parts that have the same functions and configurations as in the first embodiment, and redundant explanations are omitted except where particularly necessary.

[0082] In the first embodiment, the image processing device 1 performed pixel shift correction in the image processing unit 20, but in this embodiment, the image processing device 2 performs pixel shift correction in the hybrid 3D object detection unit 71.

[0083] The image processing unit 21 of this embodiment basically has the same configuration as the image processing unit 20 shown in Figure 3. However, the affine table stored in the affine table storage unit 230 does not include parameters used to correct pixel shift. Consequently, the affine transformation process performed by the affine processing unit 220 does not correct pixel shift. In other words, the affine processing unit 220 of the image processing unit 21 does not function as a pixel shift correction processing unit that corrects pixel shift.

[0084] The hybrid 3D object detection unit 71, similar to the hybrid 3D object detection unit 70 in the first embodiment, generates a hybrid image by combining a first distance image generated by the road surface cross-section shape estimation unit 50 with left and right monocular images generated by the monocular image generation unit 40. The hybrid 3D object detection unit 71 stores a pixel shift amount as correction information used to correct the pixel shift in the monocular region described above. When the hybrid 3D object detection unit 71 acquires the distance to a 3D object in the monocular region based on the hybrid image, it uses the stored pixel shift amount to correct the pixel shift of the monocular image and calculates the distance. The pixel shift amount held by the hybrid 3D object detection unit 71 is calibrated by the pixel shift amount estimated by the image calibration unit 90. As a result, the hybrid 3D object detection unit 71 can obtain highly accurate distance information.

[0085] In the first embodiment, the pixel shift is directly corrected by the affine transformation of the image captured by the camera 10, whereas in this embodiment, as described above, the pixel shift is corrected during distance calculation to obtain the distance to a three-dimensional object based on the hybrid image. Alternatively, instead of correcting the pixel shift during distance calculation, the system may be configured to correct the pixel shift of the left and right monocular images when generating the hybrid image.

[0086] In this embodiment as well, by shifting the imaging areas of the two cameras constituting a monocular stereo camera to widen the field of view, the amount of pixel shift in the monocular image captured by the stereo camera is calibrated using the camera's moving parallax, making it possible to perform highly accurate measurements over a wide field of view.

[0087] Figure 17 is a schematic block diagram showing the logical configuration of the image processing apparatus in the third embodiment. The image processing apparatus 3 in this embodiment also basically has the same configuration as the image processing apparatus in the first and second embodiments. For this reason, in Figure 17, the same reference numerals as in Figure 1 or Figure 16 are used for parts that have the same functions and configurations as in the first or second embodiment, and redundant explanations are omitted except where particularly necessary.

[0088] In the first and second embodiments, cameras 10a and 10b with the same field of view are used to form a stereoscopic and monocular viewing area by shifting their fields of view (imaging area). However, in this embodiment, the field of view of the right camera and the left camera are different, with a wide-angle camera 11 on the right and a narrow-angle camera 12 on the left having a narrower field of view than the wide-angle camera 11. Note that the relative positions of the wide-angle camera 11 and the narrow-angle camera 12 may be reversed.

[0089] Figure 18 is a schematic diagram showing the imaging area in this embodiment.

[0090] The wide-angle camera 11 and the narrow-angle camera 12 are installed such that the field of view captured by the wide-angle camera 11 encompasses the field of view captured by the narrow-angle camera 12. In this embodiment, the area 113 that can be captured by both the wide-angle camera 11 and the narrow-angle camera 12 is used as the stereo field of view, and the area 114 that is outside the field of view captured by the narrow-angle camera 12 and is captured only by the wide-angle camera 11 is used as the monocular field of view.

[0091] The wide-angle camera image processing unit 22 is configured in the same way as the image processing unit 20 in the first embodiment. That is, the affine table stored in the affine table storage unit of the wide-angle camera image processing unit 22 includes parameters for correcting pixel shift among the parameters used in affine processing, and an image with corrected pixel shift is generated when affine transformation is performed in the affine processing unit. The parameters used for correcting pixel shift are calibrated based on the amount of pixel shift estimated by the image calibration unit 90, as in the first embodiment.

[0092] On the other hand, the narrow-angle camera image processing unit 23 is configured similarly to the image processing unit 21 in the second embodiment, and the affine processing unit of the narrow-angle camera image processing unit 23 does not function as a pixel shift correction processing unit. Basically, the imaging area of ​​the image captured by the narrow-angle camera 12 does not include the area outside the field of view of the image captured by the wide-angle camera 11, so the image processed by the narrow-angle camera image processing unit 23 does not include the monocular view area used for distance measurement. For this reason, pixel shift correction is unnecessary in the narrow-angle camera image processing unit 23. The image processed by the narrow-angle camera image processing unit 23 is sent only to the stereo disparity image generation unit 30 and not to the monocular view image generation unit 40.

[0093] The monocular image generation unit 40 receives only images processed by the wide-angle camera image processing unit 22 and processes only images captured by the wide-angle camera 11. However, the image processing itself is the same as in the first and second embodiments. The same applies to the hybrid 3D object detection unit 70.

[0094] Furthermore, the image calibration unit 90 will only estimate the pixel shift correction amount for images captured by the wide-angle camera 11, but the process of estimating the pixel shift correction amount using moving parallax is no different from that of the first and second embodiments. In this embodiment, the pixel shift amount estimated by the image calibration unit 90 is only for images captured by the wide-angle camera 11, and is sent only to the wide-angle camera image processing unit 22, where it is used to calibrate the parameters held by the wide-angle camera image processing unit 22.

[0095] This embodiment also allows for the calibration of the parameters used to correct the pixel shift of images in the monocular field of view captured by a wide-angle camera, using the amount of pixel shift estimated using the camera's moving parallax, thereby enabling highly accurate measurements over a wide field of view.

[0096] Although the present invention has been described above using representative embodiments as examples, the present invention is not limited to these embodiments. For example, it is possible to replace a part of the configuration of one embodiment with the configuration of another embodiment, or to add the configuration of another embodiment to the configuration of one embodiment. Furthermore, for each embodiment, it is possible to delete or replace components, or to add other configurations.

[0097] For example, in the third embodiment, pixel shift correction is performed in the wide-angle image processing device, but this can also be done in the hybrid 3D object detection unit, similar to the second embodiment. Also, in each of the embodiments described above, a moving parallax image is generated based on the monocular image processed by the monocular image generation unit 40, but for example, the same processing as the monocular image generation unit 40 may be performed in the brightness information generation unit 260 of the image processing unit 20.

[0098] In the embodiments described above, the correction of pixel shift due to the influence of the windshield was explained as an example, but it goes without saying that image distortion due to other factors can also be corrected. For example, image distortion due to misalignment of the lens and sensor, and image distortion due to changes in magnification due to the lens temperature can also be corrected in the same way. When the position of the lens and sensor shifts due to changes over time and temperature, the correction by the affine processing unit becomes inappropriate, and image distortion occurs. Also, when the lens temperature changes, the magnification of the image changes, and image distortion occurs. Pixel shifts caused by these can also be corrected based on the amount of pixel shift estimated by the image calibration unit. The image processing device may continuously perform calibration by the image calibration unit after being started up. Furthermore, for example, if the image processing device is configured to acquire environmental information such as the temperature of the camera and lens, and even humidity inside and outside the vehicle, it may be configured to detect when the environment has changed by a predetermined amount based on the environmental information and perform calibration by the image calibration unit.

[0099] Furthermore, in the embodiments described above, distance information in the monocular view region is obtained based on distance information in the stereo view region, but distance information obtained through stereo viewing is not necessarily required. For example, the distance to an object may be obtained based on its vertical position in the monocular view image, or based on the size of people or cars in the image. In these cases as well, pixel shift correction and calibration of the correction amount can be performed based on distance information obtained from the moving parallax image and the position and size of objects in the image. Therefore, the present invention is not limited to stereo cameras and can also be applied to monocular cameras. In addition, the direction of pixel shift is not limited to the vertical direction, but may also be applied to horizontal pixel shift.

[0100] Thus, the present invention can be implemented in various ways without departing from the spirit of the invention as described in the claims.

[0101] 1, 2, 3...Image processing device, 10a, 10b...Camera, 11...Wide-angle camera, 12...Narrow-angle camera, 20a, 20b, 21a, 21b...Image processing unit, 22...Wide-angle camera image processing unit, 23...Narrow-angle camera image processing unit, 30...Stereo disparity image generation unit, 40...Monocular image generation unit, 50...Road surface cross-section shape estimation unit, 60...Stereo 3D object detection unit, 70, 71...Hybrid 3D object detection unit, 80...Alarm control unit, 90...Image calibration unit, 91...Image recording unit, 92...Stereo disparity image generation unit, 93...Distance conversion unit, 94...Movement amount acquisition unit, 95...Pixel shift correction amount calibration unit

Claims

An image processing device mounted on a mobile body for recognizing the surrounding conditions of the mobile body, comprising: an image processing unit that acquires images captured by a first camera and a second camera mounted on the mobile body; A stereo disparity image generation unit acquires disparity information relating to the disparity between the first imaging region captured by the first camera and the first imaging region captured by the second camera, with respect to the first imaging region captured in both images taken by the first camera and the second camera, respectively. A distance image generation unit that obtains the distance to a three-dimensional object in the first imaging area based on the parallax information and generates a first distance image that includes first distance information relating to that distance, A hybrid three-dimensional object detection unit generates a hybrid image including the first imaging region and the second imaging region from the first distance image and an image of a second imaging region captured by either the first camera or the second camera but not by the other camera, and acquires the distance to the three-dimensional object captured in the second imaging region based on the first distance information. A moving parallax image generation unit generates a moving parallax image that includes moving parallax information relating to the parallax between the first image and the second image, based on a first image captured at a first time including the second imaging region and a second image captured at a second time different from the first time including the second imaging region, using either the first image or the second image as a reference image. A movement amount acquisition unit that acquires the amount of movement of the moving body between the first time and the second time, A distance conversion unit generates a second distance image which includes second distance information relating to the distance to a three-dimensional object captured in the second imaging area of ​​the reference image based on the moving parallax image and the amount of movement, A calibration unit generates calibration information used to calibrate correction information for correcting distortion of an image captured by the first camera or the second camera that captured an image including the second imaging region, based on the first distance image generated based on the reference image and the second distance image, An image processing device having 　 The image processing apparatus according to claim 1, wherein the hybrid three-dimensional object detection unit acquires the distance to the three-dimensional object based on the position of the three-dimensional object on the hybrid image. 　 The image processing apparatus according to claim 1, wherein the hybrid three-dimensional object detection unit acquires the distance to the three-dimensional object based on the size of the object in the image. 　 The image processing apparatus according to claim 1, wherein the movement amount acquisition unit acquires information regarding the movement of the moving body measured by another distance measuring device mounted on the moving body, and acquires the movement amount based on the information regarding the movement. 　 The image processing apparatus according to claim 1, wherein the movement amount acquisition unit acquires information regarding the movement of the moving body measured outside the moving body, and acquires the movement amount based on the information regarding the movement. 　 The image processing apparatus according to claim 1, wherein the processing by the moving disparity image generation unit, the distance conversion unit, and the calibration unit is configured to be performed in accordance with changes in the environment inside and outside the moving body. 　 The image processing apparatus according to claim 6, wherein the change in the environment is a change in humidity. 　 The image processing apparatus according to claim 6, wherein the change in the environment is a change in the temperature of the first camera or the second camera. 　 The image processing apparatus according to claim 1, wherein the moving disparity image generation unit converts the first image and the second image from images in the direction of movement of the moving body to images in a direction perpendicular to the direction of movement to generate the moving disparity image, and the moving disparity information is the disparity in the perpendicular direction. 　 The image processing apparatus according to claim 9, wherein the distance conversion unit obtains a distance in the perpendicular direction, which is the distance to the three-dimensional object in the perpendicular direction, based on the moving parallax information, and obtains a distance to the three-dimensional object in the moving direction based on the said distance in the perpendicular direction. 　 The image processing apparatus according to claim 10, wherein the calibration unit generates the calibration information based on the difference between the position of the first distance information, which corresponds to the second distance information, on the first distance image and the position of the three-dimensional object on the second distance image. 　 The image processing apparatus according to claim 1, wherein the image processing apparatus holds parameters including correction information that is calibrated by the calibration information, performs image conversion processing using the parameters, and corrects the distortion of the image. The image processing apparatus according to claim 1, wherein the hybrid three-dimensional object detection unit is configured with the calibration information, holds the correction information for correcting image distortion in the second imaging region included in the hybrid image, and uses the correction information to obtain the distance to the three-dimensional object captured in the second imaging region. 　 An image processing device mounted on a mobile body for recognizing the surrounding conditions of the mobile body, comprising: an image processing unit that acquires an image from a camera mounted on the mobile body; A distance calculation unit that obtains the distance to a three-dimensional object based on the position of the three-dimensional object in the image, A moving parallax image generation unit generates a moving parallax image that includes moving parallax information relating to the parallax between the first image and the second image, based on a first image captured at a first time and a second image captured at a second time different from the first time, using either the first image or the second image as a reference image. A movement amount acquisition unit that acquires the amount of movement of the moving body between the first time and the second time, A distance conversion unit generates a distance image that includes distance information relating to the distance to a three-dimensional object depicted in the reference image, based on the moving parallax image and the amount of movement. A calibration unit generates calibration information used to calibrate correction information for correcting distortion in an image captured by the camera, based on the position of the three-dimensional object on the reference image and the distance information. An image processing device having 　 An image calibration method for calibrating an image processing device mounted on a mobile body for recognizing the surrounding conditions of the mobile body, Images captured by the first camera and the second camera mounted on the mobile body are acquired. For the first imaging region captured in both of the two images captured by the first camera and the second camera, disparity information relating to the disparity between the image captured by the first camera and the image captured by the second camera is acquired. Based on the parallax information, the distance to the three-dimensional object within the first imaging area is obtained, and a first distance image is generated that includes first distance information relating to that distance. Based on a first image captured at a first time, which includes a second imaging region captured by either the first camera or the second camera but not captured by the other camera, and a second image captured at a second time different from the first time, which includes the second imaging region, a moving parallax image is generated, which includes moving parallax information relating to the parallax between the first image and the second image, with either the first image or the second image as the reference image. The amount of movement of the moving body between the first time and the second time is obtained, Based on the moving disparity image and the amount of movement, a second distance image is generated which includes second distance information relating to the distance to the three-dimensional object captured in the second imaging area of ​​the reference image. Based on the first distance image and the second distance image generated based on the aforementioned reference image, calibration information is generated that is used to calibrate correction information for correcting distortion of images captured by the first or second camera. Image calibration methods.

Images