Stereoscopic imaging method, apparatus and device, and storage medium, program product, chip and vehicle

Abstract

Provided in the present application are a stereoscopic imaging method, apparatus and device, and a storage medium, a program product, a chip and a vehicle. The method is applied to a stereoscopic imaging device comprising cameras having different focal lengths, and comprises: acquiring a first short-focal-length image and a first long-focal-length image at a first moment, and a second short-focal-length image and a second long-focal-length image at a second moment; acquiring, from the second short-focal-length image, a region that does not match the first short-focal-length image and the second long-focal-length image and matches the first long-focal-length image; for each first pixel point in the region, on the basis of a movement distance of a stereoscopic imaging device from the first moment to the second moment, a first imaging position in a short-focal-length camera that corresponds to the first pixel point, and a second imaging position in a long-focal-length camera that corresponds to a second pixel point, using a depth relational expression to determine a depth corresponding to the first pixel point; and on the basis of the depth corresponding to each first pixel point, outputting a target stereoscopic image corresponding to the second moment.

Description

Stereoscopic imaging methods, devices, equipment, storage media, software products, chips, and vehicles Cross-references to related applications This application claims priority to Chinese Patent Application No. 202510358092.9, filed with the Chinese Patent Office on March 25, 2025, the entire contents of which are incorporated herein by reference. Technical Field

[0001] This application relates to, but is not limited to, the field of visual perception technology, specifically to a stereoscopic imaging method, apparatus, device, storage medium, program product, chip, and vehicle. Background Technology

[0002] Visual perception is a key technology in the intelligent driving industry. In order for vehicles to perceive the driving environment, it is generally necessary to perform target detection and stereo imaging based on images captured by cameras. Currently, most of them use cameras with the same focal length, such as binocular cameras with two identical focal lengths. Based on the images captured by the cameras with the same focal length, close-range targets can be detected.

[0003] Compared to cameras with the same focal length, telephoto cameras (such as stereo cameras with different focal lengths) can detect targets at greater distances. Based on images acquired by cameras with different focal lengths, it is possible to detect not only close-range targets but also distant targets. Summary of the Invention

[0004] The following is an overview of the subject matter described in detail herein. This overview is not intended to limit the scope of the claims.

[0005] The main objective of this application is to provide a stereo imaging method, apparatus, device, and storage medium, which aims to expand the stereo imaging range of cameras with different focal lengths.

[0006] A first aspect of this application provides a stereo imaging method applied to a stereo imaging device, the stereo imaging device including a short-focus camera and a long-focus camera, the stereo imaging method including: acquiring a first short-focus image and a second short-focus image captured by the short-focus camera at a first time and a second time respectively, and a first long-focus image and a second long-focus image captured by the long-focus camera at the first time and the second time respectively; wherein, the first time is the previous image acquisition time of the second time; acquiring a first target region; wherein, the first target region is a region in the second short-focus image that does not match the first short-focus image and the second long-focus image, but matches the first long-focus image; for each first pixel in the first target region, based on the stereo imaging device, from the... The first moving distance from the first moment to the second moment, the first imaging position corresponding to the first pixel in the short-focus camera, and the second imaging position corresponding to the second pixel in the long-focus camera are used to determine the depth corresponding to the first pixel using a first depth relation. The second pixel is the pixel in the first long-focus image that matches the first pixel. The first depth relation is determined based on the third imaging position of the first spatial point in the long-focus camera at the third moment and the fourth imaging position of the first spatial point in the short-focus camera at the fourth moment, using the principles of similar triangles and parallax. The third moment is the previous image acquisition moment of the fourth moment. Based on the depth corresponding to each first pixel, the target stereo image corresponding to the second moment is output.

[0007] In one embodiment, before determining the depth corresponding to the first pixel point using a first depth relationship based on the first moving distance of the stereo imaging device from the first time point to the second time point, the first imaging position corresponding to the first pixel point in the short-focus camera, and the second imaging position corresponding to the second pixel point in the long-focus camera, the stereo imaging method further includes: acquiring the first spatial point, the third pixel point corresponding to the third imaging position, the fourth pixel point corresponding to the fourth imaging position, the first optical center point of the short-focus camera at the third time point, the second optical center point of the short-focus camera at the fourth time point, and the third optical center point of the long-focus camera at the third time point; determining the intersection point of the imaging plane of the short-focus camera at the fourth time point and a first straight line as a first target point; the first straight line is a straight line passing through the second optical center point and parallel to the second straight line; the second straight line is a straight line passing through the first spatial point and the third optical center point; determining the intersection point of the third straight line and the second straight line as a second target point; the third straight line is a straight line passing through the fourth pixel point and parallel to the imaging plane of the short-focus camera at the fourth time point; determining the intersection point of the fourth straight line and a fifth straight line as a second target point; the third straight line is a straight line passing through the fourth pixel point and parallel to the imaging plane of the short-focus camera at the fourth time point; the fourth straight line and the fifth straight line are then... The intersection of the lines is determined as the third target point; the fourth straight line is a straight line passing through the first optical center point and parallel to the imaging plane of the short-focal-length camera at the third moment; the fifth straight line is a straight line passing through the first spatial point and the second optical center point; the intersection of the first straight line and the fourth straight line is determined as the fourth target point; the intersection of the fourth straight line and the second straight line is determined as the fifth target point; using the principle of similar triangles, based on the third pixel point, the fourth pixel point, the third target point, the fifth target point, the first optical center point, the second optical center point, and the third optical center point, a first distance relationship is determined; the first distance relationship is used to solve for the distance between the third target point and the fifth target point; using the principle of similar triangles, based on the third pixel point, the fourth pixel point, the first optical center point, the second optical center point, the third optical center point, the first target point, the fourth target point, and the fifth target point, a second distance relationship is determined; the second distance relationship is used to solve for the distance between the fourth pixel point and the second target point; using the principle of parallax, based on the first distance relationship and the second distance relationship, a first depth relationship is determined.

[0008] In one embodiment, the first depth relation is:

[0009] Where t1 is the fourth time point, Z t1 B is the depth of the first spatial point relative to the second optical center point of the short-focus camera at the fourth time; B is the baseline length between the short-focus camera and the long-focus camera; fl f is the focal length of the telephoto camera; s S is the focal length of the short-focal-length camera; S is the second moving distance of the stereo imaging device from the third time point to the fourth time point; x1 is the lateral distance between the third pixel point corresponding to the third imaging position and the second center point of the imaging plane of the telephoto camera at the third time point; x3 is the lateral distance between the fourth pixel point corresponding to the fourth imaging position and the first center point of the imaging plane of the short-focal-length camera at the fourth time point.

[0010] In one embodiment, the step of outputting the target stereo image corresponding to the second time moment based on the depth corresponding to each first pixel includes: determining a first stereo image corresponding to the first target region at the second time moment based on the depth corresponding to each first pixel; obtaining a second target region; the second target region is a region in the second short-focus image that does not overlap with the first short-focus image but overlaps with the second long-focus image; for each fifth pixel in the second target region, determining the depth corresponding to the fifth pixel based on the fifth imaging position of the fifth pixel in the short-focus camera and the sixth imaging position of the sixth pixel in the long-focus camera using a second depth relation; the sixth pixel is a pixel in the second long-focus image that matches the fifth pixel; the second depth relation is determined based on the seventh imaging position of the second spatial point in the short-focus camera at the fifth time moment and the eighth imaging position of the second spatial point in the long-focus camera at the fifth time moment using the similar triangle principle and the parallax principle; determining a second stereo image corresponding to the second target region at the second time moment based on the depth corresponding to each fifth pixel; and outputting the target stereo image based on the first stereo image and the second stereo image.

[0011] In one embodiment, the step of outputting the target stereo image corresponding to the second time moment based on the depth corresponding to each first pixel includes: determining a first stereo image corresponding to the first target region at the second time moment based on the depth corresponding to each first pixel; obtaining a third target region; the third target region being the region in the first short-focus image that overlaps with the first long-focus image; for each seventh pixel in the third target region, determining the depth corresponding to the seventh pixel based on the ninth imaging position of the seventh pixel in the short-focus camera and the tenth imaging position of the eighth pixel in the long-focus camera using a second depth relation; the eighth pixel being a pixel in the first long-focus image that matches the seventh pixel; the second depth relation being determined based on the seventh imaging position of the second spatial point in the short-focus camera at the fifth time moment and the eighth imaging position of the second spatial point in the long-focus camera at the fifth time moment using the principle of similar triangles and the principle of parallax; determining a third stereo image corresponding to the third target region at the first time moment based on the depth corresponding to each seventh pixel; and outputting the target stereo image based on the first stereo image and the third stereo image.

[0012] In one embodiment, the step of outputting the stereo image corresponding to the second moment based on the depth corresponding to each first pixel includes: for each first pixel, using the principle of similar triangles, determining the spatial coordinates corresponding to the first pixel based on the depth corresponding to the first pixel, the first imaging position corresponding to the first pixel in the short-focus camera, and the focal length of the short-focus camera; and outputting the target stereo image based on the spatial coordinates corresponding to each first pixel.

[0013] In one embodiment, obtaining the first target region includes: obtaining the spatial target categories corresponding to pixels in the first short-focus image, the first long-focus image, the second short-focus image, and the second long-focus image, respectively; using a pixel matching algorithm, matching the pixels in the first short-focus image, the first long-focus image, the second short-focus image, and the second long-focus image according to the spatial target categories corresponding to the pixels, to obtain pixel matching results; and determining the first target region based on the pixel matching results.

[0014] In one embodiment, obtaining the first target region includes: obtaining the first field of view region of the short-focus camera and the second field of view region of the telephoto camera at the first moment, and the third field of view region of the short-focus camera and the fourth field of view region of the telephoto camera at the second moment; determining the region in the third field of view region that does not overlap with the first field of view region and the fourth field of view region, but overlaps with the second field of view region, as the fifth field of view region; and determining the region in the second short-focus image that corresponds to the fifth field of view region as the first target region.

[0015] A second aspect of this application provides a stereo imaging device, comprising a first acquisition module, a second acquisition module, a depth determination module, and a stereo imaging module; the first acquisition module is configured to acquire a first short-focus image and a second short-focus image captured by a short-focus camera at a first time and a second time, respectively, and a first telephoto image and a second telephoto image captured by a telephoto camera at the first time and the second time, respectively; wherein the first time is the previous image acquisition time of the second time; the second acquisition module is configured to acquire a first target region; wherein the first target region is a region in the second short-focus image that does not match the first short-focus image and the second telephoto image, but matches the first telephoto image; the depth determination module is configured to, for each first pixel in the first target region, based on stereo imaging... The stereo imaging device determines the depth corresponding to the first pixel by using a first depth relation based on the first moving distance from the first time point to the second time point, the first imaging position corresponding to the first pixel in the short-focus camera, and the second imaging position corresponding to the second pixel in the long-focus camera; the second pixel is the pixel in the first long-focus image that matches the first pixel; wherein, the first depth relation is determined based on the third imaging position of the first spatial point in the long-focus camera at the third time point and the fourth imaging position of the first spatial point in the short-focus camera at the fourth time point, using the principle of similar triangles and the principle of parallax; the third time point is the previous image acquisition time at the fourth time point; the stereo imaging module is configured to output the target stereo image corresponding to the second time point according to the depth corresponding to each first pixel point.

[0016] A third aspect of this application provides a stereoscopic imaging device, which includes at least one memory and at least one processor. The memory stores a computer program, and the processor executes the computer program to implement the stereoscopic imaging method described above.

[0017] A fourth aspect of this application provides a non-transitory computer-readable storage medium storing a computer program that, when executed by a processor, implements the above-described stereoscopic imaging method.

[0018] The fifth aspect of this application discloses a computer program product, including a computer program, which, when executed by at least one processor, implements the stereoscopic imaging method described above.

[0019] A sixth aspect of this application discloses a chip including a processor and a communication interface, wherein the communication interface is coupled to the processor, and the processor is configured to execute computer programs or instructions to implement the above-described stereoscopic imaging method.

[0020] A seventh aspect of this application discloses a vehicle including the stereoscopic imaging device of the second aspect, or the stereoscopic imaging apparatus of the third aspect, or the non-transitory computer-readable storage medium of the fourth aspect, or the computer program product of the fifth aspect, or the chip of the sixth aspect, or one or more processors configured to implement the stereoscopic imaging method described above.

[0021] The stereo imaging method, apparatus, device, storage medium, program product, chip, and vehicle provided in this application determine the regions in the second short-focus image that do not match the first short-focus image and the second long-focus image, but match the first long-focus image. For each first pixel in the region, based on the first moving distance of the stereo imaging device from the first moment to the second moment, the first imaging position corresponding to the first pixel in the short-focus camera, and the second imaging position corresponding to the second pixel in the long-focus camera, the depth corresponding to the first pixel is determined using a first depth relationship. Based on the depth corresponding to the first pixel, the target stereo image corresponding to the second moment is output. This enables stereo imaging of regions that do not match in the short-focus and long-focus images acquired at the same moment, thereby expanding the stereo imaging range of cameras with different focal lengths.

[0022] It should be understood that the above general description and the following detailed description are exemplary and explanatory only, and are not intended to limit this specification. Other aspects will become clear after reading and understanding the accompanying drawings and detailed description. Attached Figure Description

[0023] The accompanying drawings are used to provide a further understanding of the technical solutions of this application and constitute a part of the specification. They are used together with the embodiments of this application to explain the technical solutions of this application and do not constitute a limitation on the technical solutions of this application.

[0024] Figure 1 is a flowchart illustrating the stereoscopic imaging method provided in an embodiment of this application.

[0025] Figure 2 is a schematic diagram of the process of determining the first depth relation provided in the embodiments of this application.

[0026] Figure 3 is a schematic diagram of the process of determining the second depth relation provided in the embodiments of this application.

[0027] Figure 4 is a schematic diagram of the specific process of the stereoscopic imaging method provided in the embodiments of this application.

[0028] Figure 5 is a schematic diagram of the structure of the stereoscopic imaging device provided in the embodiment of this application.

[0029] Figure 6 is a schematic diagram of the structure of the stereoscopic imaging device provided in the embodiment of this application. Detailed Implementation

[0030] The technical solutions of the embodiments of this application will be clearly described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this application. All other embodiments obtained by those skilled in the art based on the embodiments of this application are within the scope of protection of this application.

[0031] The terms "first," "second," etc., used in the specification and claims of this application are used to distinguish similar objects and not to describe a specific order or sequence. It should be understood that such use of data can be interchanged where appropriate so that embodiments of this application can be implemented in orders other than those illustrated or described herein, and the objects distinguished by "first," "second," etc., are generally of the same class and the number of objects is not limited; for example, a first object can be one or more. Furthermore, in the specification and claims, "and / or" indicates at least one of the connected objects, and the character " / " generally indicates that the preceding and following objects are in an "or" relationship.

[0032] In intelligent driving environments, the detection range of cameras with the same focal length is limited, making it impossible to detect distant targets. Although cameras with different focal lengths can be used, the stereo imaging range of these cameras is currently limited, mostly confined to the overlapping area of ​​short-focal-length and long-focal-length images, which affects their effectiveness.

[0033] In view of the above, embodiments of this application provide a stereoscopic imaging method, apparatus, device, storage medium, program product, chip, and vehicle.

[0034] The stereoscopic imaging method provided in this application can be applied to stereoscopic imaging devices, which can be vehicles or electronic devices. In some embodiments, the electronic device can be a terminal or a server. In some embodiments, the terminal can be a smartphone, tablet, laptop, desktop computer, etc.; the server can be configured as an independent physical server, or as a server cluster or distributed system composed of multiple physical servers, or as a cloud server providing basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, CDN (Content Delivery Network), and big data and artificial intelligence platforms. Furthermore, the stereoscopic imaging method provided in this application can also be applied to the software of the stereoscopic imaging device, which can be an application that implements the stereoscopic imaging method, but is not limited to the above forms.

[0035] The stereoscopic imaging method provided in this application will be described in detail below with reference to the accompanying drawings and specific embodiments.

[0036] This application provides a stereo imaging method applied to a stereo imaging device, which includes a short-focus camera and a long-focus camera. Referring to Figure 1, the stereo imaging method provided in this application may include the following steps S101 to S104.

[0037] Step S101: Acquire the first short-focus image and the second short-focus image captured by the short-focus camera at the first time and the second time respectively, and the first telephoto image and the second telephoto image captured by the telephoto camera at the first time and the second time respectively; wherein, the first time is the previous image acquisition time of the second time.

[0038] Step S102: Obtain the first target region; wherein, the first target region is the region in the second short-focus image that does not match the first short-focus image and the second long-focus image, but matches the first long-focus image.

[0039] Step S103: For each first pixel in the first target region, based on the first moving distance of the stereo imaging device from the first time to the second time, the first imaging position corresponding to the first pixel in the short-focus camera, and the second imaging position corresponding to the second pixel in the telephoto camera, the depth corresponding to the first pixel is determined using the first depth relationship formula; the second pixel is the pixel in the first telephoto image that matches the first pixel; wherein, the first depth relationship formula is determined based on the third imaging position of the first spatial point in the telephoto camera at the third time and the fourth imaging position of the first spatial point in the short-focus camera at the fourth time, using the principle of similar triangles and the principle of parallax; the third time is the previous image acquisition time of the fourth time;

[0040] Step S104: Output the target stereo image corresponding to the second time step based on the depth corresponding to each first pixel.

[0041] This application embodiment identifies regions in the second short-focus image that do not match the first short-focus image and the second long-focus image, but match the first long-focus image. For each first pixel in these regions, based on the first moving distance of the stereo imaging device from the first moment to the second moment, the first imaging position corresponding to the first pixel in the short-focus camera, and the second imaging position corresponding to the second pixel in the long-focus camera, the depth corresponding to the first pixel is determined using a first depth relationship. Based on the depth corresponding to each first pixel, the target stereo image corresponding to the second moment is output. This enables stereo imaging of regions that do not match in the short-focus and long-focus images acquired at the same moment, thereby expanding the stereo imaging range of cameras with different focal lengths.

[0042] In this embodiment, for a first target region in the current short-focus image that cannot be matched with either the current long-focus image or the short-focus image from a historical time, the depth is accurately acquired by using the long-focus image information from a historical time and the motion information of the device itself. This results in the depth information of the region in the final output stereoscopic image of the target at the current time being more complete and accurate, significantly improving the completeness of the output stereoscopic image of the target and the depth accuracy of the target.

[0043] In some examples, the first short-focus image is an image captured by the short-focus camera at a first moment, the second short-focus image is an image captured by the short-focus camera at a second moment, the first telephoto image is an image captured by the telephoto camera at a first moment, and the second telephoto image is an image captured by the telephoto camera at a second moment.

[0044] In some examples, matching the first target region with the first telephoto image means that, for the image content in the first target region, a region in the first telephoto image with the same physical scene as it can be found.

[0045] In one embodiment, obtaining the first target region in step S102 includes: obtaining the spatial target categories corresponding to pixels in the first short-focus image, the first long-focus image, the second short-focus image, and the second long-focus image, respectively; using a pixel matching algorithm, matching the pixels in the first short-focus image, the first long-focus image, the second short-focus image, and the second long-focus image according to the spatial target categories corresponding to the pixels, to obtain pixel matching results; and determining the first target region based on the pixel matching results.

[0046] In some examples, the spatial target category is the classification of objects in three-dimensional space, such as pedestrians, vehicles, manhole covers, speed bumps, curbs, and potholes. The spatial target category corresponding to a pixel can be determined based on the pixel's features, which may include the pixel's color, texture, and shape. The pixel matching algorithm may include at least one of convolutional neural networks, multi-scale feature extraction networks, disparity-guided networks, and generative adversarial networks.

[0047] Furthermore, given the pixel matching results, the region in which the pixels in the second short-focus image do not match the pixels in the first short-focus image or the pixels in the second long-focus image, but match the pixels in the first long-focus image, can be defined as the first target region.

[0048] This application embodiment utilizes a pixel matching algorithm to match pixels in a first short-focus image, a first long-focus image, a second short-focus image, and a second long-focus image based on the spatial target category corresponding to each pixel. This can yield accurate pixel matching results and improve the accuracy of determining the first target region.

[0049] In another embodiment, obtaining the first target region in step S102 includes: obtaining the first field of view region of the short-focus camera and the second field of view region of the telephoto camera at a first moment, and the third field of view region of the short-focus camera and the fourth field of view region of the telephoto camera at a second moment; determining the region in the third field of view region that does not overlap with the first and fourth field of view regions but overlaps with the second field of view region as the fifth field of view region; and determining the region in the second short-focus image that corresponds to the fifth field of view region as the first target region.

[0050] Based on the overlap of the field of view angles of the telephoto and short-focus cameras at different times, this application embodiment can quickly and accurately determine the fifth field of view angle region in the third field of view angle region that does not overlap with the first and fourth field of view angle regions but overlaps with the second field of view angle region. Furthermore, by matching the fifth field of view angle region with the region in the second short-focus image, the first target region in the second short-focus image that does not match the first short-focus image and the second telephoto image but matches the first telephoto image can be quickly and accurately determined.

[0051] In some examples, the aforementioned first depth formula can be a formula pre-stored by the stereo imaging device or a formula derived on the fly. The aforementioned third moment and the aforementioned first moment can be the same moment or different moments, and the aforementioned fourth moment and the aforementioned second moment can be the same moment or different moments. Where the third moment and the first moment are different, and the fourth moment and the second moment are different, the aforementioned first depth formula is a formula pre-stored by the stereo imaging device, where the third moment is a historical moment earlier than the first moment, and the fourth moment is a historical moment earlier than the second moment. This can be understood as the first depth formula being a fixed formula that can be directly called when needed. Where the third moment and the first moment are the same, and the fourth moment and the second moment are the same, the aforementioned first depth formula is a formula derived on the fly by the stereo imaging device. This can be understood as the first depth formula being a non-fixed formula that can be derived on the fly when needed.

[0052] In some examples, the first depth relationship is the relationship between the depth of the first spatial point at the fourth time relative to the optical center of the short-focal-length camera, the baseline length between the short-focal-length camera and the telephoto camera, the focal length of the telephoto camera, the focal length of the short-focal-length camera, the second movement distance of the stereo imaging device from the third time to the fourth time, the lateral distance between the third pixel point corresponding to the third imaging position and the second center point of the imaging plane of the telephoto camera at the third time, and the lateral distance between the fourth pixel point corresponding to the fourth imaging position and the first center point of the imaging plane of the short-focal-length camera at the fourth time.

[0053] Furthermore, the first moving distance of the stereo imaging device from the first moment to the second moment, the lateral distance corresponding to the first imaging position, the lateral distance corresponding to the second imaging position, the baseline length between the short-focus camera and the long-focus camera, and the focal lengths of the long-focus camera and the short-focus camera can be substituted into the first depth relationship to obtain the depth corresponding to the first pixel. In some examples, the depth corresponding to the first pixel represents the depth of the spatial point corresponding to the first pixel relative to the optical center point of the short-focus camera at the second moment.

[0054] In one embodiment, before determining the depth corresponding to the first pixel point using a first depth relationship based on the first moving distance of the stereo imaging device from the first time point to the second time point, the first imaging position corresponding to the first pixel point in the short-focus camera, and the second imaging position corresponding to the second pixel point in the long-focus camera in step S103, the stereo imaging method provided in this application further includes: acquiring a first spatial point, a third pixel point corresponding to the third imaging position, a fourth pixel point corresponding to the fourth imaging position, a first optical center point of the short-focus camera at the third time point, a second optical center point of the short-focus camera at the fourth time point, and a third optical center point of the long-focus camera at the third time point; determining the intersection point of the imaging plane of the short-focus camera at the fourth time point and a first straight line as a first target point; the first straight line is a straight line passing through the second optical center point and parallel to the second straight line; the second straight line is a straight line passing through the first spatial point and the third optical center point; determining the intersection point of the third straight line and the second straight line as a second target point; the third straight line is a straight line passing through the fourth pixel point and parallel to the imaging plane of the short-focus camera at the fourth time point; The intersection of the fourth and fifth lines is defined as the third target point; the fourth line is a line passing through the first optical center and parallel to the imaging plane of the short-focus camera at the third moment; the fifth line is a line passing through the first spatial point and the second optical center. The intersection of the first and fourth lines is defined as the fourth target point; the intersection of the fourth and second lines is defined as the fifth target point. Using the principle of similar triangles, based on the third pixel, fourth pixel, third target point, fifth target point, first optical center, second optical center, and third optical center, a first distance relationship is determined; the first distance relationship is used to solve for the distance between the third target point and the fifth target point. Using the principle of similar triangles, based on the third pixel, fourth pixel, first optical center, second optical center, third optical center, first target point, fourth target point, and fifth target point, a second distance relationship is determined; the second distance relationship is used to solve for the distance between the fourth pixel and the second target point. Using the principle of parallax, based on the first and second distance relationships, a first depth relationship is determined.

[0055] Please refer to Figure 2. At the third moment, the field of view (FOV) area of ​​the short-focus camera is sector #1, and the FOV area of ​​the long-focus camera is sector #2. At the fourth moment, the FOV area of ​​the short-focus camera is sector #3, and the FOV area of ​​the long-focus camera is sector #4. As shown in Figure 2, when the stereo imaging device moves forward a second distance S from the third moment to the fourth moment, both the long-focus and short-focus cameras move forward synchronously by the second distance S. At the fourth moment, the FOV area of ​​the short-focus camera contains regions D1 and D3 that do not overlap with the FOV areas of the short-focus camera at the third moment or the long-focus camera at the fourth moment, but overlap with the FOV area of ​​the long-focus camera at the third moment. Correspondingly, the image regions corresponding to regions D1 and D3 in the image acquired by the short-focus camera at the fourth moment do not overlap with the images acquired by the short-focus camera at the third moment or the long-focus camera at the fourth moment, but overlap with the image acquired by the long-focus camera at the third moment.

[0056] Please continue referring to Figure 2. The first target spatial point a can be any spatial point in region D1 or region D3. The fourth pixel point c, the second optical center point e, and the first center point g of the imaging plane of the short-focal-length camera at the fourth moment form triangle △ecg. The second optical center point e, the first optical center point d, and the third target point 3 form triangle △ed3. Triangles △ecg and △ed3 are similar triangles.

[0057] The intersection point n of the fourth and sixth lines, the third optical center point f, and the fifth target point 5 form triangle △nf5. The third pixel point b, the second center point m of the telephoto camera's imaging plane at the third moment, and the third optical center point f form triangle △bmf. Triangles △nf5 and △bmf are similar triangles. The sixth line is a line passing through the second center point m and the third optical center point f.

[0058] The first distance relationship can be determined by using the principle of similar triangles and combining formulas (1) to (3): A = B - x6 + x5 (3) where x1 is the lateral distance between the third pixel b and the second center point m, x3 is the lateral distance between the fourth pixel c and the first center point g, x5 is the lateral distance between the third target point 3 and the first optical center point d, x6 is the lateral distance between the fifth target point 5 and the intersection point n, f l f is the focal length of a telephoto camera. s S is the focal length of the short-focus camera, S is the second moving distance of the stereo imaging device from the third time point to the fourth time point, B is the baseline distance between the short-focus camera and the long-focus camera, and A is the lateral distance between the third target point 3 and the fifth target point 5.

[0059] Please continue to refer to Figure 2. The first target point 1, the first center point g, and the second optical center point e form a triangle △ge1. △ge1 and △bmf are similar triangles. The second optical center point e, the first optical center point d, and the fourth target point 4 form a triangle △ed4. △ed4 and △ge1 are similar triangles. The first target point 1 and the second target point 2 form line segment L12. The fourth target point 4 and the fifth target point 5 form line segment L45. The length of line segment L12 is equal to the length of line segment L45.

[0060] The second distance relationship can be determined by using the principle of similar triangles and combining formulas (1), (4) to (8): x7=B-x2-x6 (6) D=x4+x3 (7) x8=x7-D (8) Where, x2 is the lateral distance between the first optical center point d and the fourth target point 4, x4 is the lateral distance between the first target point 1 and the first center point g, x7 is the lateral distance between the fourth target point 4 and the fifth target point 5, D is the parallax of the telephoto camera at the third moment and the short-focus camera at the fourth moment relative to the first spatial point a, and x8 is the lateral distance between the fourth pixel point c and the second target point 2.

[0061] Please continue to refer to Figure 2. The first spatial point a, the fourth pixel point c, and the second target point 2 form a triangle △ac2. The first spatial point a, the third target point 3, and the fifth target point 5 form a triangle △a35. Triangles △ac2 and △a35 are similar triangles.

[0062] The first depth relationship can be determined by using the parallax principle and combining formulas (1) to (10): Z t1 =Z t0 -S (10) where t0 is the third time step, Z t0 Z represents the depth of the first spatial point relative to the first optical center point of the short-focus camera at the third time step, t1 represents the fourth time step, and Z represents the depth of the first spatial point relative to the first optical center point of the short-focus camera at the third time step. t1 The depth of the first spatial point relative to the second optical center point of the short-focus camera at the fourth moment is denoted as .

[0063] This application embodiment, based on a first spatial point, a third pixel point corresponding to a third imaging position, a fourth pixel point corresponding to a fourth imaging position, a first optical center point of a short-focus camera at a third moment, a second optical center point of a short-focus camera at a fourth moment, and a third optical center point of a telephoto camera at a third moment, uses the principle of similar triangles and the principle of parallax to determine a first depth relationship. This ensures the effectiveness of the first depth relationship and the accuracy of determining the depth corresponding to the first pixel point using the first depth relationship.

[0064] In one embodiment, the first depth relationship described above is formula (11): Where t1 is the fourth time step, Z t1 B is the depth of the first spatial point relative to the second optical center point of the short-focus camera at the fourth time step; B is the baseline length between the short-focus camera and the long-focus camera; f l f is the focal length of a telephoto camera. s denoted as the focal length of the short-focus camera; S is the second moving distance of the stereo imaging device from the third moment to the fourth moment; x1 is the lateral distance between the third pixel corresponding to the third imaging position and the second center point of the imaging plane of the telephoto camera at the third moment; and x3 is the lateral distance between the fourth pixel corresponding to the fourth imaging position and the first center point of the imaging plane of the short-focus camera at the fourth moment.

[0065] In some examples, the second movement distance of the stereo imaging device from the third time point to the fourth time point is determined by formula (12): Where n is the total number of vehicle speed data collections during the time period t0 to t1, Δt is the time interval between two adjacent vehicle speed data collection times, and v(ti) is the vehicle speed collected at time i.

[0066] By utilizing the first depth relationship described above, the efficiency and accuracy of determining the depth corresponding to the first pixel point can be improved in this embodiment. In addition, the vehicle speed collected from the first moment to the second moment can be substituted into formula (12), and the first moving distance of the stereo imaging device from the first moment to the second moment can be accurately determined by integration, thereby further improving the accuracy of determining the depth corresponding to the first pixel point using the first depth relationship described above.

[0067] In one embodiment, step S103 above, which outputs the stereo image corresponding to the second time step based on the depth corresponding to each first pixel, includes: for each first pixel, using the principle of similar triangles, determining the spatial coordinates corresponding to the first pixel based on the depth corresponding to the first pixel, the first imaging position corresponding to the first pixel in the short-focus camera, and the focal length of the short-focus camera; and outputting the target stereo image corresponding to the second time step based on the spatial coordinates corresponding to each first pixel.

[0068] In some examples, the spatial coordinates of the fourth pixel are determined by using the principle of similar triangles and combining formulas (11), (13), and (14):

[0069] The spatial coordinates corresponding to the fourth pixel are (X... t1 Y t1 Z t1 ), X t1 Y is the lateral distance between the first spatial point and the imaging plane of the short-focal-length camera at the first center point at the fourth time step. t1x3 is the longitudinal distance between the first spatial point and the first center point of the imaging plane of the short-focal-length camera at the fourth time step; x4 is the lateral distance between the fourth pixel point corresponding to the fourth imaging position and the first center point of the imaging plane of the short-focal-length camera at the fourth time step; and y3 is the longitudinal distance corresponding to the fourth imaging position. In some examples, the lateral and longitudinal directions mentioned in the embodiments of this application are two mutually perpendicular directions within the imaging plane of the short-focal-length camera or the telephoto camera.

[0070] Furthermore, the depth corresponding to the first pixel, the horizontal and vertical distances corresponding to the first imaging position, and the focal length of the short-focal-length camera can be substituted into the above formulas (13) and (14) to obtain the spatial coordinates corresponding to the first pixel.

[0071] In some examples, the target stereo image corresponding to the second time moment can be the first stereo image of the first target region corresponding to the second time moment. Specifically, based on the spatial target corresponding to each first pixel in the first target region, the spatial point where the imaging position of the second time moment is located in the first target region can be restored according to a preset ratio to obtain the first stereo image of the first target region at the second time moment, and the first stereo image is output as the target stereo image corresponding to the second time moment. Referring to Figure 2, the target stereo image corresponding to the second time moment can include the stereo image of region D1 and the stereo image of region D3.

[0072] This application embodiment utilizes the principle of similar triangles. Based on the depth corresponding to the first pixel, the first imaging position corresponding to the first pixel in the short-focus camera, and the focal length of the short-focus camera, the spatial coordinates corresponding to the first pixel can be quickly and accurately determined. Furthermore, based on the spatial coordinates corresponding to each first pixel, the target stereo image corresponding to the second time moment is output, which can improve the efficiency and accuracy of stereo imaging of the first target area.

[0073] In one embodiment, step S103 above, which outputs the target stereo image corresponding to the second time step based on the depth corresponding to each first pixel, includes: determining the first stereo image corresponding to the first target region at the second time step based on the depth corresponding to each first pixel; obtaining the second target region; the second target region is the region in the second short-focus image that does not match the first short-focus image but matches the second long-focus image; for each fifth pixel in the second target region, determining the depth corresponding to the fifth pixel based on the fifth imaging position of the fifth pixel in the short-focus camera and the sixth imaging position of the sixth pixel in the long-focus camera using a second depth relation; the sixth pixel is the pixel in the second long-focus image that matches the fifth pixel; the second depth relation is determined based on the seventh imaging position of the second spatial point in the short-focus camera and the eighth imaging position of the second spatial point in the long-focus camera at the fifth time step, using the principle of similar triangles and the principle of parallax; determining the second stereo image corresponding to the second target region at the second time step based on the depth corresponding to each fifth pixel; and outputting the target stereo image corresponding to the second time step based on the first stereo image and the second stereo image.

[0074] In some examples, based on the pixel matching results described above, the region in the second short-focus image that does not match the pixels in the first short-focus image but matches the pixels in the second long-focus image can be identified as the second target region. Alternatively, the region in the second short-focus image that corresponds to the sixth field of view region can be identified as the second target region. The sixth field of view region is the region in the third field of view region that does not overlap with the first field of view region but overlaps with the fourth field of view region.

[0075] In some examples, the second depth relation can be a pre-stored relation of the stereo imaging device or a relation derived on the fly. The fifth moment and the aforementioned second moment can be the same moment or different moments. When the fifth moment and the second moment are different moments, the fifth moment is a historical moment earlier than the second moment, and when using the second depth relation to determine the depth corresponding to the fifth pixel, the second depth relation is a pre-stored relation of the stereo imaging device. It can be understood that in this case, the second depth relation is a fixed formula that can be directly called when needed. When the fifth moment and the second moment are the same moment, when using the second depth relation to determine the depth corresponding to the fifth pixel, the aforementioned second depth relation is a relation derived on the fly by the stereo imaging device. It can be understood that in this case, the second depth relation is a non-fixed formula that can be derived on the fly when needed.

[0076] In some examples, the second depth relationship is the relationship between the lateral distance corresponding to the seventh imaging position, the lateral distance corresponding to the eighth imaging position, the baseline distance between the short-focus camera and the long-focus camera, and the focal length of the short-focus camera and the focal length of the long-focus camera.

[0077] Furthermore, the lateral distance corresponding to the fifth imaging position, the lateral distance corresponding to the sixth imaging position, the baseline distance between the short-focus camera and the long-focus camera, and the focal length of the short-focus camera and the long-focus camera can be substituted into the second depth relationship to obtain the depth corresponding to the fifth pixel.

[0078] Furthermore, the principle of similar triangles can be used to determine the spatial target corresponding to the fifth pixel point based on the depth corresponding to the fifth pixel point, combined with formulas (13) and (14). Based on the spatial target corresponding to each fifth pixel point in the second target area, the spatial point of the imaging position at the second moment in the second target area can be restored according to a preset ratio to obtain the second stereo image of the second target area at the second moment.

[0079] In some examples, the target stereo image corresponding to the second time moment can be a stereo image obtained by stitching together the first stereo image of the first target region corresponding to the second time moment and the second stereo image of the second target region corresponding to the second time moment. Referring to Figure 3, the target stereo image corresponding to the second time moment can be a stitched result of the stereo images of region D1, region D2, and region D3.

[0080] This embodiment of the application identifies regions in the second short-focus image that do not match the first short-focus image but match the second long-focus image. For each fifth pixel in this region, based on the fifth imaging position of the fifth pixel in the short-focus camera and the sixth imaging position of the sixth pixel in the long-focus camera, the depth corresponding to the fifth pixel is determined using a second depth relationship. Based on the depth corresponding to each fifth pixel, the second stereoscopic image corresponding to the second target region at the second time is determined. Finally, by stitching the first stereoscopic image and the second stereoscopic image, the target stereoscopic image corresponding to the second time is determined and output. This enables stereoscopic imaging not only of matching regions in the short-focus and long-focus images acquired at the same time, but also of mismatched regions in the short-focus and long-focus images acquired at the same time, thereby expanding the stereoscopic imaging range of cameras with different focal lengths.

[0081] In this embodiment, in addition to the depth information of the first target region, the depth information of the second target region is also determined. By complementing the depth information of these two types of regions, the information completeness of the target stereo image in a camera-moving scene is significantly improved. The second target region corresponds to the part of the scene that newly enters the field of view of the short-focal-length camera due to device movement and is simultaneously covered by the telephoto camera. By acquiring the depth of this region, the part of the scene newly entering the field of view due to field-of-view updates can obtain complete depth information in the target stereo image.

[0082] In one embodiment, step S103 above, which outputs the target stereo image corresponding to the second time step based on the depth corresponding to each first pixel, includes: determining the first stereo image corresponding to the first target region at the second time step based on the depth corresponding to each first pixel; obtaining the third target region; the third target region is the region in the first short-focus image that matches the first long-focus image; for each seventh pixel in the third target region, determining the depth corresponding to the seventh pixel based on the ninth imaging position of the seventh pixel in the short-focus camera and the tenth imaging position of the eighth pixel in the long-focus camera using a second depth relation; the eighth pixel is the pixel in the first long-focus image that matches the seventh pixel; the second depth relation is determined based on the seventh imaging position of the second spatial point in the short-focus camera at the fifth time step and the eighth imaging position of the second spatial point in the long-focus camera at the fifth time step, using the principle of similar triangles and the principle of parallax; determining the third stereo image corresponding to the third target region at the first time step based on the depth corresponding to each seventh pixel; and outputting the target stereo image corresponding to the second time step based on the first stereo image and the third stereo image.

[0083] In some examples, based on the pixel matching results described above, the region in which the pixels in the first short focal length image match the pixels in the first long focal length image can be determined as the third target region. Alternatively, the region in the first short focal length image that corresponds to the seventh field of view region can be determined as the third target region, wherein the seventh field of view region is the region in the first field of view region that overlaps with the second field of view region.

[0084] In some examples, the fifth moment and the first moment mentioned above can be the same moment or different moments. When the fifth moment and the first moment are different moments, the fifth moment is a historical moment earlier than the first moment. When using the second depth relation to determine the depth corresponding to the seventh pixel, the second depth relation is a relation pre-stored by the stereo imaging device. It can be understood that in this case, the second depth relation is a fixed formula that can be directly called when needed. When the fifth moment and the first moment are the same moment, when using the second depth relation to determine the depth corresponding to the seventh pixel, the second depth relation is a relation derived by the stereo imaging device in real time. It can be understood that in this case, the second depth relation is a non-fixed formula that can be derived in real time when needed.

[0085] In some examples, the lateral distance corresponding to the ninth imaging position, the lateral distance corresponding to the tenth imaging position, the baseline distance between the short-focus camera and the long-focus camera, and the focal length of the short-focus camera and the long-focus camera can be substituted into the second depth relationship to obtain the depth corresponding to the seventh pixel.

[0086] Furthermore, the principle of similar triangles can be used to determine the spatial target corresponding to the seventh pixel point based on the depth corresponding to the seventh pixel point, combined with formulas (13) and (14). Based on the spatial target corresponding to each seventh pixel point in the third target area, the spatial point of the imaging position at the first moment in the third target area can be restored according to a preset ratio to obtain the third stereo image of the third target area at the first moment.

[0087] In some examples, the target stereo image corresponding to the second time step can be a stereo image obtained by stitching together the first stereo image of the first target region at the second time step and the third stereo image of the third target region at the first time step. Referring to Figure 3, the target stereo image corresponding to the second time step can be a stitched result of the stereo images of region D1, region D3, and region A.

[0088] In some examples, the target stereo image corresponding to the second time moment can also be a stereo image obtained by stitching together the first stereo image of the first target region at the second time moment, the second stereo image of the second target region at the second time moment, and the third stereo image of the third target region at the first time moment. Referring to Figure 3, the target stereo image corresponding to the second time moment can be a stitched result of the stereo images of regions D1, D2, D3, and A.

[0089] In some examples, before stitching the third stereo image corresponding to the third target region at the first moment to obtain the target stereo image corresponding to the second moment, the third stereo image can be cropped, and only the stereo image corresponding to the space in front of the stereo imaging device in the third stereo image can be used for stitching. In this way, only the stereo image of the space in front of the stereo imaging device can be presented in the target stereo image corresponding to the second moment, thereby improving the stereo imaging visual effect.

[0090] This application embodiment determines the region in the first short-focus image that matches the first long-focus image, and for each seventh pixel in that region, based on the ninth imaging position of the seventh pixel corresponding to the short-focus camera and the tenth imaging position of the eighth pixel corresponding to the long-focus camera, uses a second depth relationship to determine the depth corresponding to the seventh pixel, and based on the depth corresponding to the seventh pixel, determines the third stereo image corresponding to the third target region at the first moment. Finally, by stitching the first stereo image and the third stereo image, the target stereo image corresponding to the second moment is determined and output, so that the stereo images corresponding to the matching regions in the short-focus image and the long-focus image acquired at the previous same moment, as well as the stereo images corresponding to the mismatched regions in the short-focus image and the long-focus image acquired at the next same moment, can be presented simultaneously, thereby expanding the stereo imaging range of cameras with different focal lengths.

[0091] In one embodiment, before determining the depth corresponding to the fifth pixel using the second depth relationship based on the fifth imaging position of the fifth pixel in the short-focus camera and the sixth imaging position of the sixth pixel in the long-focus camera, or before determining the depth corresponding to the seventh pixel using the second depth relationship based on the ninth imaging position of the seventh pixel in the short-focus camera and the tenth imaging position of the eighth pixel in the long-focus camera, the stereo imaging method provided in this application further includes: acquiring a second spatial point, a ninth pixel corresponding to the seventh imaging position, a tenth pixel corresponding to the eighth imaging position, a fourth optical center point of the short-focus camera at the fifth moment, and a fifth optical center point of the long-focus camera at the fifth moment; determining the intersection of the imaging plane of the short-focus camera at the fifth moment and the seventh straight line as the eighth target point; the seventh straight line is a straight line passing through the fourth optical center point and parallel to the eighth straight line; the eighth straight line is a straight line passing through the second spatial point and the fifth optical center point; determining the intersection of the ninth straight line and the eighth straight line as the ninth target point; the ninth straight line is a straight line passing through the fourth optical center point and parallel to the imaging plane of the short-focus camera at the fifth moment. Using the principle of similar triangles, a third distance relationship is determined based on the tenth pixel, the ninth target point, the fourth optical center point, and the fifth optical center point. This third distance relationship is used to calculate the distance between the fourth optical center point and the ninth target point. Using the principle of similar triangles, a fourth distance relationship is determined based on the eighth target point, the ninth pixel, the tenth pixel, the fourth optical center point, the fifth optical center point, and the third distance relationship. This fourth distance relationship is used to calculate the distance between the tenth pixel and the ninth pixel. Using the principle of parallax, a second depth relationship is determined based on the third and fourth distance relationships.

[0092] Referring to Figure 3, the Field of View (FOV) area of ​​the short-focus camera at the fifth moment is sector 5 or sector 6, while the FOV area of ​​the long-focus camera at the fifth moment is sector 7 or sector 8. As shown in Figure 3, there is an overlapping region D2 or region A within the FOV area of ​​the short-focus camera at the fifth moment with the FOV area of ​​the long-focus camera at the fifth moment. Correspondingly, the image region corresponding to region D2 or region A in the image acquired by the short-focus camera at the fifth moment matches the image acquired by the long-focus camera at the fourth moment.

[0093] Please continue referring to Figure 3. The second target spatial point a' can be any spatial point in region D2 or region A. The intersection point t of the ninth and tenth lines, the ninth target point 9, and the fifth optical center point s form triangle △ts9. The fifth optical center point s, the tenth pixel point q, and the fourth center point u of the telephoto camera's imaging plane at the fifth moment form triangle △squ. Triangles △ts9 and △squ are similar triangles. The tenth line is a line passing through the fifth optical center point s and the fourth center point u.

[0094] The third distance relationship can be determined by using the principle of similar triangles and combining formulas (15) and (16): A'=B-x'5 (16)

[0095] Where x′1 is the lateral distance between the tenth pixel q and the fourth center point u, which is the lateral distance corresponding to the eighth imaging position, x′5 is the lateral distance between the ninth target point 9 and the intersection point t, and A′ is the lateral distance between the fourth optical center point r and the ninth target point 9.

[0096] Please continue to refer to Figure 3. At the fifth moment, the imaging plane of the short-focal-length camera forms a triangle △rv8 with the fifth center point v, the eighth target point 8, and the fourth optical center point r. Triangle △rv8 and triangle △squ are similar triangles. The eighth target point 8 and the tenth pixel point q form a line segment Lq8. The fourth optical center point r and the ninth target point 9 form a line segment Lr9. The length of line segment Lq8 is equal to the length of line segment Lr9.

[0097] The fourth distance relationship can be determined by using the principle of similar triangles and combining formulas (15) to (19): D'=x'2+x'3 (18) x'4=A'-D' (19)

[0098] Where x′3 is the lateral distance between the eighth target point 8 and the fifth center point v, x′2 is the lateral distance between the ninth pixel point p and the fifth center point v, that is, the lateral distance corresponding to the seventh imaging position, D' is the parallax between the telephoto camera and the short-focus camera at the fifth moment for the second spatial point a', and x′4 is the lateral distance between the ninth pixel point p and the tenth pixel point q.

[0099] Please continue to refer to Figure 3. The second spatial point a', the ninth pixel point p, and the tenth pixel point q form a triangle △a'pq. The second spatial point a', the fourth optical center point r, and the ninth target point 9 form a triangle △a'r9. Triangles △a'pq and △a'r9 are similar triangles.

[0100] The parallax principle can be used, combined with formulas (15) to (20), to determine the second depth relationship (21):

[0101] Where t2 is the fifth time step, Z t2 The depth of the second spatial point relative to the fourth optical center point of the short-focus camera at the fifth moment.

[0102] This application embodiment is based on the second spatial point, the ninth pixel corresponding to the seventh imaging position, the tenth pixel corresponding to the eighth imaging position, the fourth optical center point of the short-focus camera at the fifth moment, and the fifth optical center point of the telephoto camera at the fifth moment. It uses the principle of similar triangles and the principle of parallax to determine the second depth relationship, which can ensure the effectiveness of the second depth relationship and the accuracy of determining the depth corresponding to the fifth pixel point or the seventh pixel point using the second depth relationship.

[0103] Please refer to Figure 4. An embodiment of this application provides a stereoscopic imaging method, which may include the following steps S201 to S207.

[0104] Step S201: Perform intrinsic and extrinsic parameter calibration on the short-focus camera and the long-focus camera to obtain the focal length of the short-focus camera, the focal length of the long-focus camera, and the baseline distance between the short-focus camera and the long-focus camera.

[0105] Step S202: Acquire the first short-focus image and the second short-focus image captured by the short-focus camera at the first time and the second time respectively, and the first long-focus image and the second long-focus image captured by the long-focus camera at the first time and the second time respectively.

[0106] Step S203: Match the pixels in the first short-focus image, the first long-focus image, the second short-focus image, and the second long-focus image to obtain the pixel matching result.

[0107] Step S204: Determine the target region based on the pixel matching result; the target region includes a first target region in the second short-focus image that does not match the first short-focus image and the second long-focus image, but matches the first long-focus image, a second target region in the second short-focus image that does not match the first short-focus image but matches the second long-focus image, and at least one of the third target regions in the first short-focus image that matches the first long-focus image.

[0108] Step S205: For each pixel in the target region, determine the depth corresponding to each pixel in the target region using the corresponding depth formula.

[0109] Step S206: Using the principle of similar triangles, determine the spatial coordinates of each pixel in the target area based on the depth corresponding to each pixel in the target area.

[0110] Step S207: Based on the spatial coordinates of each pixel in the target region, output the target stereo image corresponding to the second time step.

[0111] It is worth mentioning that, during the pixel matching process in step S203 of the first short-focus image, the first long-focus image, the second short-focus image, and the second long-focus image, if any of the first short-focus image, the first long-focus image, the second short-focus image, and the second long-focus image has been cropped or scaled, then the focal length of the short-focus camera, the focal length of the long-focus camera, the baseline distance between the short-focus camera and the long-focus camera, the position of the center point of the imaging plane of the short-focus camera, and the position of the center point of the imaging plane of the long-focus camera, which were calibrated in step S201, need to be updated so as to accurately determine the depth corresponding to each pixel in the target area.

[0112] In addition, the specific implementation process of steps S203 to S206 is the same as that in the above embodiment, and will not be repeated here.

[0113] Through the above steps S201 to S207, the embodiments of this application can output a stereoscopic image of the first target region as the target stereoscopic image at the second moment, or output a stitched image of the stereoscopic image of the first target region and the stereoscopic image of the second target region as the target stereoscopic image at the second moment, or output a stitched image of the stereoscopic image of the first target region and the stereoscopic image of the third target region as the target stereoscopic image at the second moment, or output a stitched image of the stereoscopic image of the first target region, the stereoscopic image of the second target region and the stereoscopic image of the third target region as the target stereoscopic image at the second moment, thereby expanding the range and flexibility of stereoscopic imaging with cameras of different focal lengths.

[0114] It is worth mentioning that when the stereo imaging device is a vehicle, the vehicle can detect distant targets through a telephoto camera, which can meet the usage requirements of intelligent driving functions. Furthermore, by implementing the stereo imaging method provided in this application embodiment, the stereo imaging range of cameras with different focal lengths can be expanded, which can meet the usage requirements of the chassis magic carpet function. In this way, the intelligent driving function and the chassis magic carpet function can be balanced, and the usage effect of the intelligent driving function and the chassis magic carpet function can be improved.

[0115] Please refer to Figure 5. This application embodiment also provides a stereo imaging device 500, which includes a first acquisition module 501, a second acquisition module 502, a depth determination module 503, and a stereo imaging module 504.

[0116] The first acquisition module 501 is configured to acquire a first short-focus image and a second short-focus image captured by a short-focus camera at a first time and a second time, respectively, and a first telephoto image and a second telephoto image captured by a telephoto camera at a first time and a second time, respectively; wherein the first time is the previous image acquisition time of the second time.

[0117] The second acquisition module 502 is configured to acquire a first target region; wherein, the first target region is a region in the second short-focus image that does not match the first short-focus image and the second long-focus image, but matches the first long-focus image;

[0118] The depth determination module 503 is configured to determine the depth corresponding to each first pixel in the first target region based on the first moving distance of the stereo imaging device from the first moment to the second moment, the first imaging position corresponding to the first pixel in the short-focus camera, and the second imaging position corresponding to the second pixel in the long-focus camera, using a first depth relationship; the second pixel is the pixel in the first long-focus image that matches the first pixel; wherein, the first depth relationship is determined based on the third imaging position of the first spatial point in the long-focus camera at the third moment and the fourth imaging position of the first spatial point in the short-focus camera at the fourth moment, using the principle of similar triangles and the principle of parallax; the third moment is the previous image acquisition moment of the fourth moment.

[0119] The stereo imaging module 504 is configured to output the target stereo image at the second time step based on the depth corresponding to each first pixel.

[0120] The stereo imaging device provided in this application embodiment can implement all the steps of the stereo imaging method embodiment described above and achieve the same technical effect. To avoid repetition, it will not be described again here.

[0121] This application also provides a stereo imaging device, including at least one processor and a memory. The at least one memory stores a program or instructions that can run on the processor. When the program or instructions are executed by the processor, they implement the various steps of the stereo imaging method embodiments described above and achieve the same technical effect. To avoid repetition, they will not be described again here.

[0122] Figure 6 is a schematic diagram of the hardware structure of a stereoscopic imaging device that implements an embodiment of this application. The stereoscopic imaging device includes the following components.

[0123] At least one processor 601 may be implemented using a general-purpose central processing unit (CPU), microprocessor, application-specific integrated circuit (ASIC), or one or more integrated circuits, and is configured to execute relevant programs to implement the technical solutions provided in the embodiments of this application.

[0124] At least one memory 602 may be implemented as a read-only memory (ROM), static storage device, dynamic storage device, or random access memory (RAM). The memory 602 may store the operating system and other application programs. When the technical solutions provided in the embodiments of this specification are implemented through software or firmware, the relevant program code is stored in the memory 602 and is called and executed by the processor 601 to execute the stereoscopic imaging method of the embodiments of this application.

[0125] Input / output interface 603 is configured to enable information input and output.

[0126] Communication interface 604 is configured to enable communication and interaction between this device and other devices. Communication can be achieved via wired means (such as USB, Ethernet cable, etc.) or wireless means (such as mobile network, WIFI, Bluetooth, etc.).

[0127] Bus 605 is configured to transmit information between various components of the device, such as processor 601, memory 602, input / output interface 603, and communication interface 604.

[0128] The processor 601, memory 602, input / output interface 603, and communication interface 604 are connected to each other within the device via bus 605.

[0129] The stereo imaging device provided in this application embodiment can implement all the steps of the above stereo imaging method embodiment and achieve the same technical effect. To avoid repetition, it will not be described again here.

[0130] This application also provides a non-transitory computer-readable storage medium storing a program or instructions. When the program or instructions are executed by at least one processor, they implement the various steps of the above-described stereoscopic imaging method embodiments and achieve the same technical effect. To avoid repetition, they will not be described again here.

[0131] The processor is the processor in the stereoscopic imaging device described in the above embodiments. The computer-readable storage medium includes computer-readable storage media such as computer read-only memory (ROM), random access memory (RAM), magnetic disk, or optical disk.

[0132] This application also provides a computer program product, including a computer program that, when executed by at least one processor, implements the stereoscopic imaging method described in the above embodiments.

[0133] This application also provides a vehicle including the stereoscopic imaging device described above, or the stereoscopic imaging equipment described above, or the non-transitory computer-readable storage medium described above, or the computer program product described above, or one or more processors described above, wherein the one or more processors are configured to implement the stereoscopic imaging method described above.

[0134] This application also provides a chip, which includes a processor and a communication interface. The communication interface and the processor are coupled. The processor is configured to run programs or instructions to implement the various steps of the above-described stereoscopic imaging method embodiments and achieve the same technical effect. To avoid repetition, it will not be described again here.

[0135] It should be understood that the chip mentioned in the embodiments of this application may also be referred to as a system-on-a-chip, system chip, chip system, or system-on-a-chip, etc.

[0136] This application provides a computer program product, which is stored in a storage medium and executed by at least one processor to implement the various steps of the stereoscopic imaging method embodiment described above, and can achieve the same technical effect. To avoid repetition, it will not be described again here.

[0137] It should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not delete other identical elements present in the process, method, article, or apparatus that includes that element. Furthermore, it should be noted that the scope of the methods and apparatuses in the embodiments of this application is not limited to performing functions in the order shown or discussed, but may also include performing functions substantially simultaneously or in the reverse order, depending on the functions involved. For example, the described methods may be performed in a different order than described, and various steps may be added, omitted, or combined. Additionally, features described with reference to certain examples may be combined in other examples.

[0138] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods of the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of this application, in essence, or the part that makes technical contributions, can be embodied in the form of a computer software product. This computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk) and includes several instructions to cause a terminal (which may be a mobile phone, computer, server, or network device, etc.) to execute the methods described in the various embodiments of this application.

[0139] The embodiments of this application have been described above with reference to the accompanying drawings. However, this application is not limited to the specific embodiments described above. The specific embodiments described above are merely illustrative and not restrictive. Those skilled in the art can make many other forms under the guidance of this application without departing from the spirit and scope of the claims, and all of these forms are within the protection scope of this application.

Claims

1. A stereo imaging method applied to a stereo imaging device, the stereo imaging device comprising a short-focus camera and a long-focus camera, the stereo imaging method comprising: Acquire a first short-focus image and a second short-focus image captured by the short-focus camera at a first time and a second time, respectively, and a first telephoto image and a second telephoto image captured by the telephoto camera at the first time and the second time, respectively; wherein, the first time is the previous image acquisition time of the second time; Obtain a first target region; wherein, the first target region is a region in the second short-focus image that does not match the first short-focus image and the second long-focus image, but matches the first long-focus image; For each first pixel in the first target region, the depth corresponding to the first pixel is determined using a first depth relation based on the first moving distance of the stereo imaging device from the first time point to the second time point, the first imaging position corresponding to the first pixel in the short-focus camera, and the second imaging position corresponding to the second pixel in the long-focus camera; the second pixel is the pixel in the first long-focus image that matches the first pixel; wherein, the first depth relation is determined using the principle of similar triangles and the principle of parallax based on the third imaging position of the first spatial point in the long-focus camera at the third time point and the fourth imaging position of the first spatial point in the short-focus camera at the fourth time point; the third time point is the previous image acquisition time at the fourth time point. Based on the depth corresponding to each of the first pixels, output the target stereo image at the second time point.

2. The stereoscopic imaging method of claim 1, wherein, Before determining the depth corresponding to the first pixel using a first depth relationship based on the first moving distance of the stereo imaging device from the first time point to the second time point, the first imaging position corresponding to the first pixel in the short-focus camera, and the second imaging position corresponding to the second pixel in the long-focus camera, the stereo imaging method further includes: The first spatial point, the third pixel point corresponding to the third imaging position, the fourth pixel point corresponding to the fourth imaging position, the first optical center point of the short-focus camera at the third moment, the second optical center point of the short-focus camera at the fourth moment, and the third optical center point of the telephoto camera at the third moment are obtained. The intersection of the imaging plane of the short-focal-length camera at the fourth moment and the first straight line is determined as the first target point; the first straight line is a straight line that passes through the second optical center point and is parallel to the second straight line; the second straight line is a straight line that passes through the first spatial point and the third optical center point; The intersection of the third line and the second line is determined as the second target point; the third line is a line that passes through the fourth pixel and is parallel to the imaging plane of the short-focus camera at the fourth moment. The intersection of the fourth and fifth lines is determined as the third target point; the fourth line is a line that passes through the first optical center point and is parallel to the imaging plane of the short-focus camera at the third moment; the fifth line is a line that passes through the first spatial point and the second optical center point. The intersection of the first line and the fourth line is determined as the fourth target point; The intersection of the fourth line and the second line is determined as the fifth target point; Using the principle of similar triangles, a first distance relationship is determined based on the third pixel, the fourth pixel, the third target point, the fifth target point, the first optical center point, the second optical center point, and the third optical center point; the first distance relationship is used to solve for the distance between the third target point and the fifth target point. Using the principle of similar triangles, a second distance relationship is determined based on the third pixel, the fourth pixel, the first optical center, the second optical center, the third optical center, the first target point, the fourth target point, and the fifth target point; the second distance relationship is used to solve for the distance between the fourth pixel and the second target point. Using the parallax principle, the first depth relationship is determined based on the first distance relationship and the second distance relationship.

3. The stereoscopic imaging method of claim 1 or 2, wherein, The first depth relation is: wherein t1 is the fourth time, Z t1 is a depth of the first spatial point relative to a second optical center point of the short-focus camera at the fourth time; B is a baseline length between the short-focus camera and the long-focus camera; f l is a focal length of the long-focus camera; f s is a focal length of the short-focus camera; S is a second moving distance of the stereo imaging device from the third time to the fourth time; x1 is a horizontal distance between a third pixel point corresponding to the third imaging position and a second center point of the imaging plane of the long-focus camera at the third time, and x3 is a horizontal distance between a fourth pixel point corresponding to the fourth imaging position and a first center point of the imaging plane of the short-focus camera at the fourth time.

4. The stereoscopic imaging method of any one of claims 1 to 3, wherein, The step of outputting the target stereo image at the second time step based on the depth corresponding to each of the first pixels includes: Based on the depth corresponding to each of the first pixels, determine the first stereo image corresponding to the first target region at the second time. Obtain the second target region; the second target region is the region in the second short-focus image that does not match the first short-focus image but matches the second long-focus image; For each fifth pixel in the second target region, the depth corresponding to the fifth pixel is determined using a second depth relation based on the fifth imaging position of the fifth pixel in the short-focus camera and the sixth imaging position of the sixth pixel in the long-focus camera; the sixth pixel is the pixel in the second long-focus image that matches the fifth pixel; the second depth relation is determined using the principle of similar triangles and the principle of parallax based on the seventh imaging position of the second spatial point in the short-focus camera at the fifth time and the eighth imaging position of the second spatial point in the long-focus camera at the fifth time. Based on the depth corresponding to each fifth pixel, determine the second stereoscopic image corresponding to the second target region at the second time. The target stereo image is output based on the first stereo image and the second stereo image.

5. The stereoscopic imaging method of any one of claims 1 to 3, wherein, The step of outputting the target stereo image at the second time step based on the depth corresponding to each of the first pixels includes: Based on the depth corresponding to each of the first pixels, determine the first stereo image corresponding to the first target region at the second time. Obtain a third target region; the third target region is the region in the first short-focus image that matches the first long-focus image; For each seventh pixel in the third target region, the depth corresponding to the seventh pixel is determined using a second depth relation based on the ninth imaging position of the seventh pixel in the short-focus camera and the tenth imaging position of the eighth pixel in the long-focus camera; the eighth pixel is the pixel in the first long-focus image that matches the seventh pixel; the second depth relation is determined using the principle of similar triangles and the principle of parallax based on the seventh imaging position of the second spatial point in the short-focus camera at the fifth time and the eighth imaging position of the second spatial point in the long-focus camera at the fifth time. Based on the depth corresponding to each of the seventh pixels, the third stereoscopic image corresponding to the third target region at the first moment is determined; The target stereo image is output based on the first stereo image and the third stereo image.

6. The stereoscopic imaging method of any one of claims 1 to 5, wherein, The step of outputting the stereo image corresponding to the second time step based on the depth corresponding to each of the first pixels includes: For each first pixel, the spatial coordinates of the first pixel are determined based on the similar triangle principle, the depth corresponding to the first pixel, the first imaging position of the first pixel in the short-focus camera, and the focal length of the short-focus camera. The target stereo image is output based on the spatial coordinates corresponding to each of the first pixels.

7. The stereoscopic imaging method of any one of claims 1 to 6, wherein, The acquisition of the first target region includes: The spatial target categories corresponding to the pixels in the first short-focus image, the first long-focus image, the second short-focus image, and the second long-focus image are obtained respectively; Using a pixel matching algorithm, pixels in the first short-focus image, the first long-focus image, the second short-focus image, and the second long-focus image are matched according to the spatial target category corresponding to the pixel to obtain pixel matching results; The first target region is determined based on the pixel matching results.

8. The stereoscopic imaging method of any one of claims 1 to 6, wherein, The acquisition of the first target region includes: Obtain the first field of view region of the short-focus camera and the second field of view region of the long-focus camera at the first moment, and the third field of view region of the short-focus camera and the fourth field of view region of the long-focus camera at the second moment. The region in the third visual field region that does not overlap with the first visual field region and the fourth visual field region, but overlaps with the second visual field region, is defined as the fifth visual field region. The region in the second short-focus image that corresponds to the fifth field of view region is determined as the first target region.

9. A stereoscopic imaging device, characterized by The stereo imaging device includes a first acquisition module, a second acquisition module, a depth determination module, and a stereo imaging module; The first acquisition module is configured to acquire a first short-focus image and a second short-focus image captured by a short-focus camera at a first time and a second time, respectively, and a first telephoto image and a second telephoto image captured by a telephoto camera at the first time and the second time, respectively; wherein, the first time is the previous image acquisition time of the second time; The second acquisition module is configured to acquire a first target region; wherein, the first target region is a region in the second short-focus image that does not match the first short-focus image and the second long-focus image, but matches the first long-focus image; The depth determination module is configured to, for each first pixel in the first target region, determine the depth corresponding to the first pixel based on the first moving distance of the stereo imaging device from the first time point to the second time point, the first imaging position corresponding to the first pixel in the short-focus camera, and the second imaging position corresponding to the second pixel in the long-focus camera, using a first depth relationship formula; the second pixel is the pixel in the first long-focus image that matches the first pixel; wherein, the first depth relationship formula is determined based on the third imaging position of the first spatial point in the long-focus camera at the third time point and the fourth imaging position of the first spatial point in the short-focus camera at the fourth time point, using the principle of similar triangles and the principle of parallax; the third time point is the previous image acquisition time at the fourth time point; The stereo imaging module is configured to output the target stereo image at the second time point based on the depth corresponding to each of the first pixels.

10. A stereoscopic imaging device, characterized by The stereo imaging device includes a memory and a processor. The memory stores a computer program, and the processor executes the computer program to implement the stereo imaging method as described in any one of claims 1 to 8. 11.A non-transitory computer-readable storage medium storing a computer program, wherein, When the computer program is executed by at least one processor, it implements the stereoscopic imaging method as described in any one of claims 1 to 8.

12. A computer program product comprising a computer program that, when executed by at least one processor, implements the stereoscopic imaging method according to any one of claims 1 to 8.

13. A chip comprising a processor and a communication interface, the communication interface being coupled to the processor, the processor being configured to implement the stereoscopic imaging method as described in any one of claims 1 to 8 when executing a computer program or instructions.

14. A vehicle comprising: The stereoscopic imaging device according to claim 9, or The stereoscopic imaging device according to claim 10, or The non-transitory computer-readable storage medium according to claim 11, or The computer program product according to claim 12, or The chip according to claim 13, or One or more processors configured to implement the stereo imaging method of any one of claims 1 to 8.

Images