Panoramic depth map generation method, model training method, electronic device and unmanned aerial vehicle

Abstract

The present disclosure can be applied in the technical field of image processing. Provided is a panoramic depth map generation method. The panoramic depth map generation method comprises: for at least two image groups which are obtained by means of grouping target images corresponding to different orientations in a target scene, performing feature combination on target image feature volumes of each image group, so as to obtain at least two target panoramic feature volumes, wherein the target images included in each image group cover a panoramic field of view of the target scene, and the target image feature volumes represent three-dimensional features of a target image; performing correlation processing on every two target panoramic features in the at least two target panoramic feature volumes, so as to obtain at least one target correlation volume; and on the basis of an initial depth map and the at least one target correlation volume, performing panoramic depth estimation, so as to obtain a target panoramic depth map for the target scene. Further provided in the present disclosure are a model training method, an electronic device and an unmanned aerial vehicle.

Description

Panoramic depth map generation methods, model training methods, electronic devices and drones Technical Field

[0001] This disclosure relates to the field of image processing technology, specifically to a panoramic depth map generation method, a model training method, an electronic device, and a drone. Background Technology

[0002] Panoramic depth estimation based on a surround-view camera array is a 3D reconstruction method that can acquire the complete structure of the surrounding scene. Panoramic depth estimation is a fundamental technology for applications such as autonomous mobile robots and mixed reality.

[0003] In related technologies, binocular stereo matching is generally used for panoramic stereo matching, but this method has low matching speed and accuracy. Summary of the Invention

[0004] In view of the above problems, this disclosure provides a panoramic depth map generation method, a model training method, an electronic device, and a drone.

[0005] According to a first aspect of this disclosure, a method for generating a panoramic depth map is provided, comprising: for at least two image groups obtained by grouping target images involving different orientations in a target scene, performing feature combination on the target image feature body of each image group to obtain at least two target panoramic feature bodies, wherein the target images contained in each image group cover the panoramic field of view of the target scene, and the target image feature body characterizes the three-dimensional stereoscopic features of the target image.

[0006] Correlation processing is performed on every two target panoramic features in at least two target panoramic feature volumes to obtain at least one target correlation volume; and

[0007] A panoramic depth map of the target scene is obtained by performing panoramic depth estimation based on the initial depth map and at least one target-related object.

[0008] According to embodiments of this disclosure, each target image feature body has its own corresponding target weight, and feature combination is performed on the target image feature bodies of each image group to obtain at least two target panoramic feature bodies, including:

[0009] By utilizing the target weights of each target image feature body, the target image feature bodies of each image group are weighted and summed to obtain at least two target panoramic feature bodies.

[0010] According to embodiments of this disclosure, the target weight of each target image feature volume is determined using the following operation:

[0011] For each image group, the target image feature volume in the image group is concatenated to obtain the first concatenated target image feature volume;

[0012] Multi-layer perceptual processing is performed on the target image feature volume after the first concatenation to obtain the target weights of each target image feature volume in the image group.

[0013] According to embodiments of this disclosure, the target weight of each target image feature volume is determined using the following operation:

[0014] For each pixel in the feature body of the target image, determine the distance between the pixel and the target pixel, where the target pixel corresponds to the center of the camera;

[0015] The positional weight of a pixel is determined based on the distance.

[0016] The target weight of the target image feature body is determined based on the position weight of each pixel.

[0017] According to embodiments of this disclosure, determining the positional weight for a pixel based on distance includes:

[0018] If the distance is less than a preset distance threshold, the position weight of the pixel is determined to be the first value;

[0019] If the distance is greater than or equal to a preset distance threshold, the position weight of the pixel is determined to be a second value, wherein the first value is different from the second value.

[0020] According to embodiments of this disclosure, the target weight of each target image feature volume is determined using the following operation:

[0021] The target image feature volumes of target images from different orientations in the target scene are concatenated to obtain the second concatenated target image feature volume.

[0022] Multi-layer perceptual processing is performed on the target image feature volume after the second concatenation to obtain the target weight of each target image feature volume.

[0023] According to embodiments of this disclosure, the image group is divided using the following operation:

[0024] For target images in a target scene that involve different orientations, two target images facing away from each other are divided into an image group, resulting in at least two image groups. The two target images facing away from each other represent that the orientations of the two target images are opposite.

[0025] According to embodiments of this disclosure, correlation processing is performed on every two target panoramic feature bodies in at least two target panoramic feature bodies to obtain at least one target correlation body, including:

[0026] Calculate the inner product of every two target panoramic feature volumes in at least two target panoramic feature volumes to obtain at least one target related volume.

[0027] According to embodiments of this disclosure, panoramic depth estimation is performed based on an initial depth map and at least one target-related object to obtain a target panoramic depth map for the target scene, including:

[0028] A panoramic depth estimation is performed based on an initial depth map, a preset target context feature body, and at least one target related body to obtain a target panoramic depth map for the target scene. The target context feature body is determined based on at least one target panoramic feature body.

[0029] According to embodiments of this disclosure, panoramic depth estimation is performed based on an initial depth map, a preset target context feature body, and at least one target related body to obtain a target panoramic depth map for the target scene, including:

[0030] Using the initial depth map as the current estimated depth map, panoramic depth estimation is performed based on the current estimated depth map, target context feature volume, and at least one target related object to obtain the depth estimation increment;

[0031] The current depth estimation map is updated based on the depth estimation increment to obtain the updated estimated depth map;

[0032] Using the updated estimated depth as the current estimated depth map, repeat the above operation until the number of iterations reaches the first preset iteration threshold to obtain the target panoramic depth map for the target scene.

[0033] According to embodiments of this disclosure, panoramic depth estimation is performed based on the current estimated depth map, target context features, and at least one target-related object to obtain the depth estimation increment, which includes:

[0034] Using the current estimated depth map and preset sampling neighbor values, sample the target context feature body and at least one target related body to obtain the current context feature map and the current related feature map;

[0035] Panoramic depth estimation is performed using the current context feature map and the current relevant feature map to obtain the depth estimation increment.

[0036] According to embodiments of this disclosure, the method further includes:

[0037] For each target image in the target scene involving different orientations, feature extraction is performed on the target image to obtain a target feature map;

[0038] The target feature map is processed by spherical scanning to obtain the target image feature volume.

[0039] According to embodiments of this disclosure, the target images in the target scene involving different orientations include target images from at least four orientations.

[0040] According to embodiments of this disclosure, target images in different orientations within a target scene are acquired by fisheye lenses in different orientations within the target scene.

[0041] According to embodiments of this disclosure, the method further includes:

[0042] Obstacles are identified based on the panoramic depth map, enabling drones to perform obstacle avoidance maneuvers accordingly.

[0043] The second aspect of this disclosure provides a method for training a depth estimation model, comprising:

[0044] For at least two sample image groups obtained by grouping sample images from different orientations in a sample scene based on training samples, feature combination is performed on the sample image feature volume of each sample image group to obtain at least two sample panoramic feature volumes. Each sample image group contains sample images that cover the panoramic field of view of the sample scene, and the sample image feature volume represents the three-dimensional stereo features of the sample image.

[0045] Correlation processing is performed on every two panoramic features in at least two sample panoramic feature volumes to obtain at least one sample correlation volume; and

[0046] Based on the initial depth map and at least one sample related body, panoramic depth estimation is performed to obtain a predicted panoramic depth map for the sample scene.

[0047] The loss information of the depth estimation model is determined based on the predicted panoramic depth map, and the network parameters of the depth estimation model are iteratively adjusted based on the loss information until the loss information meets the iteration stopping condition. The network parameters obtained when the iteration stopping condition is met are used as the trained depth estimation model.

[0048] According to embodiments of this disclosure, each sample image feature body has its own sample weight, and feature combination is performed on the sample image feature bodies of each sample image to obtain at least two sample panoramic feature bodies, including:

[0049] By utilizing the sample weights of each sample image feature body, the sample image feature bodies of each sample image group are weighted and summed to obtain at least two sample panoramic feature bodies.

[0050] According to embodiments of this disclosure, each sample image feature volume has its own sample weight, which is determined by the following operation:

[0051] For each group of sample images, the sample image feature bodies in the sample image group are concatenated to obtain the first concatenated sample image feature bodies;

[0052] Multi-layer perceptron processing is performed on the feature volumes of the first concatenated sample images to obtain the sample weights of each feature volume in the sample image group.

[0053] According to embodiments of this disclosure, each sample image feature volume has its own sample weight, which is determined by the following operation:

[0054] For each sample pixel in the feature body of the sample image, determine the distance between the sample pixel and the target sample pixel;

[0055] The positional weights for sample pixels are determined based on the distance.

[0056] The sample weights of the feature bodies of the sample image are determined based on the positional weights of each sample pixel.

[0057] According to embodiments of this disclosure, determining the positional weight for a sample pixel based on distance includes:

[0058] If the distance is less than a preset distance threshold, the position weight of the sample pixel is determined to be the first value;

[0059] If the distance is greater than or equal to a preset distance threshold, the position weight of the sample pixel is determined to be a second value, wherein the first value is different from the second value.

[0060] According to embodiments of this disclosure, each sample image feature volume has its own sample weight, which is determined by the following operation:

[0061] The sample image feature volumes of sample images from different orientations in the sample scene are concatenated to obtain the second concatenated sample image feature volume.

[0062] Multi-layer perceptual processing is performed on the feature volume of the second concatenated sample image to obtain the sample weight of each sample image feature.

[0063] According to embodiments of this disclosure, the sample image group is divided using the following operation:

[0064] For sample images involving different orientations in a sample scene, two sample images facing away from each other are divided into a sample image group, resulting in at least two sample image groups. The two sample images facing away from each other represent that the orientations of the two sample images are opposite.

[0065] According to embodiments of this disclosure, correlation processing is performed on every two sample panoramic feature bodies in at least two sample panoramic feature bodies to obtain at least one sample correlation body, including:

[0066] Calculate the inner product of every two panoramic feature volumes in at least two sample panoramic feature volumes to obtain at least one sample correlation volume.

[0067] According to embodiments of this disclosure, panoramic depth estimation is performed based on an initial depth map and at least one sample related body to obtain a sample panoramic depth map for the sample scene, including:

[0068] Panoramic depth estimation is performed based on the initial depth map, the preset sample context feature volume, and at least one sample related volume to obtain a sample panoramic depth map for the sample scene. The sample context feature volume is determined based on at least one sample panoramic feature volume.

[0069] According to embodiments of this disclosure, panoramic depth estimation is performed based on an initial depth map, preset sample context features, and at least one sample related object to obtain a sample panoramic depth map for the sample scene, including:

[0070] Using the initial depth map as the current sample estimated depth map, panoramic depth estimation is performed based on the current sample estimated depth map, sample context feature volume, and at least one sample related volume to obtain the sample depth estimation increment;

[0071] The current sample depth estimation map is updated based on the sample depth estimation increment, resulting in the updated sample depth estimation map;

[0072] Using the updated sample estimated depth as the current sample estimated depth map, repeat the above operation until the number of iterations reaches the second preset iteration threshold to obtain a sample panoramic depth map for the sample scene.

[0073] According to embodiments of this disclosure, panoramic depth estimation is performed based on the current estimated depth map, sample context features, and at least one sample related body to obtain the sample depth estimation increment, which includes:

[0074] Using the current sample to estimate the depth map and the preset sampling neighbor value, sample the sample context feature body and at least one sample related body respectively to obtain the current sample context feature map and the current sample related feature map;

[0075] The panoramic depth estimation is performed using the current sample context feature map and the current sample related feature map to obtain the sample depth estimation increment.

[0076] According to embodiments of this disclosure, the method further includes:

[0077] For each sample image in the sample scene involving different orientations, feature extraction is performed on the sample image to obtain a sample feature map;

[0078] The sample feature map is processed by spherical scanning to obtain the sample image feature volume.

[0079] According to embodiments of this disclosure, the sample images in the sample scene involving different orientations include sample images from at least four orientations.

[0080] According to embodiments of this disclosure, sample images involving different orientations in a sample scene are acquired by fisheye lenses involving different orientations in the sample scene.

[0081] A third aspect of this disclosure provides an electronic device comprising: one or more processors; and a memory for storing one or more computer programs, wherein the one or more processors execute the one or more computer programs to implement the steps of the method.

[0082] A fourth aspect of this disclosure provides a drone including the aforementioned electronic equipment.

[0083] The fifth aspect of this disclosure also provides a computer-readable storage medium having a computer program or instructions stored thereon, which, when executed by a processor, implement the steps of the above-described method.

[0084] A sixth aspect of this disclosure also provides a computer program product, including a computer program or instructions that, when executed by a processor, implement the steps of the above-described method. Attached Figure Description

[0085] The foregoing contents, as well as other objects, features, and advantages of this disclosure, will become clearer from the following description of embodiments with reference to the accompanying drawings, in which:

[0086] Figure 1 schematically illustrates a scene diagram of a method for generating a panoramic depth map according to an embodiment of the present disclosure;

[0087] Figure 2 schematically illustrates a flowchart of a method for generating a panoramic depth map according to an embodiment of the present disclosure;

[0088] Figure 3 schematically illustrates a flowchart of a method for determining target image features according to an embodiment of the present disclosure;

[0089] Figure 4 schematically illustrates a method for determining target related entities according to an embodiment of the present disclosure;

[0090] Figure 5 schematically illustrates a diagram of determining the target weight using a complementary adaptive weighting method according to an embodiment of the present disclosure;

[0091] Figure 6 schematically illustrates a diagram of determining the target weight using a fully weighted method according to an embodiment of the present disclosure;

[0092] Figure 7 schematically illustrates the target weight volume determined using different methods;

[0093] Figure 8 is a flowchart schematically illustrating a method for generating a panoramic depth map according to another embodiment of the present disclosure;

[0094] Figure 9 schematically illustrates a method for obtaining a panoramic depth map by processing an image using the RomniStero model;

[0095] Figure 10 schematically illustrates a flowchart of a training method for a depth estimation model according to an embodiment of the present disclosure;

[0096] Figure 11 schematically illustrates a structural block diagram of a panoramic depth map generation apparatus according to an embodiment of the present disclosure;

[0097] Figure 12 schematically illustrates a structural block diagram of a training apparatus for a depth estimation model according to an embodiment of the present disclosure; and

[0098] Figure 13 schematically illustrates a block diagram of an electronic device suitable for implementing the above-described method according to an embodiment of the present disclosure. Detailed Implementation

[0099] The embodiments of the present disclosure will now be described with reference to the accompanying drawings. However, it should be understood that these descriptions are exemplary only and are not intended to limit the scope of the disclosure. In the following detailed description, numerous specific details are set forth to provide a thorough understanding of the embodiments of the present disclosure for ease of explanation. However, it will be apparent that one or more embodiments may be practiced without these specific details. Furthermore, descriptions of well-known structures and techniques are omitted in the following description to avoid unnecessarily obscuring the concepts of the present disclosure.

[0100] The terminology used herein is for the purpose of describing particular embodiments only and is not intended to limit this disclosure. The terms “comprising,” “including,” etc., as used herein indicate the presence of the stated features, steps, operations, and / or components, but do not exclude the presence or addition of one or more other features, steps, operations, or components.

[0101] All terms used herein (including technical and scientific terms) have the meanings commonly understood by those skilled in the art, unless otherwise defined. It should be noted that the terms used herein are to be interpreted in a manner consistent with the context of this specification, and not in an idealized or overly rigid way.

[0102] When using expressions such as "at least one of A, B, and C", they should generally be interpreted in accordance with the meaning that is commonly understood by a person skilled in the art (e.g., "a system having at least one of A, B, and C" should include, but is not limited to, a system having A alone, a system having B alone, a system having C alone, a system having A and B, a system having A and C, a system having B and C, and / or a system having A, B, and C, etc.).

[0103] Panoramic stereo matching depth estimation based on surround-view camera arrays is a reliable 3D reconstruction method that can acquire the complete structure of the surrounding scene, serving as a fundamental technology for applications in autonomous mobile robots and mixed reality. Existing technologies mainly introduce binocular stereo matching methods to panoramic stereo matching, but their speed and accuracy are not high.

[0104] For example, the SweepNet model uses spherical sweeping to construct the cost volume of the panoramic space, enabling the application of semi-global matching (SGM) to panoramic stereo matching. The SweepNet model first performs spherical sweeping on the input fisheye image to construct a 3D image space (three dimensions: length, width, and discrete sampling depth). Then, a local convolutional neural network processes each local 2D image patch in the 3D image space, constructing the cost volume based on the extracted 3D volume. SweepNet trains the parameters of the aforementioned local convolutional neural network by constructing a loss function on the cost volume using the true panoramic depth of the dataset. This convolutional neural network processes the 3D image space patch by patch, which is slow due to the large number of patches processed. Furthermore, SweepNet uses a semi-global matching method to process the cost volume; since it is not trained end-to-end, the overall model accuracy is relatively low.

[0105] For example, the OmniMVS model fully utilizes deep learning for panoramic stereo matching. It employs a 17-layer pure convolutional deep neural network to extract features from four surround-view fisheye images. Then, it obtains the corresponding 3D panoramic feature volumes for the four surround-view fisheye images through spherical scanning, further concatenates them to form the cost volume, and finally uses an encoder-decoder block based on 3D convolution, similar to the PSMNet algorithm, to aggregate the cost volumes and obtain the probabilities of each preset discrete depth. The expectation of each preset discrete depth on this probability prediction is used as the final estimated depth. Because OmniMVS uses a deep neural network for the entire model, it can be optimized end-to-end, and both speed and accuracy are significantly improved compared to SweepNet. However, the high complexity of the 3D convolutional encoder-decoder structure still significantly limits the speed and accuracy of OmniMVS. Furthermore, since the cost volume of OmniMVS is derived from the feature volumes generated by concatenating various cameras, an increase in the number of cameras results in a larger cost volume, further increasing the complexity of the subsequent 3D convolutional encoder-decoder structure. Therefore, the OmniMVS model has poor scalability.

[0106] In the field of stereo binocular matching for pinhole images, a method based on Recurrent All-Pairs Field Transforms (RAFT) has been proposed. RAFT was first applied to optical flow estimation, and then RAFTStereo was extended to stereo matching. RAFT-Stereo achieves higher accuracy than methods based on 3D convolutional encoder-decoder cost volume aggregation, while also having lower memory footprint and faster computation time. RAFT-Stereo starts with a zero disparity map and continuously estimates disparity residuals through a 2D convolutional gated recurrent unit (GRU) to obtain the final disparity map. The input to the 2D GRU is a correlation feature map obtained by sampling a series of neighboring values ​​of the currently estimated disparity. The correlation volume is calculated by computing the correlation in the disparity dimension using the feature maps of the reference and target images.

[0107] However, in surround view multi-view panoramic stereo matching, there is no physical panoramic reference image and panoramic target image, making it difficult to use the RAFT architecture for surround view multi-view panoramic stereo matching.

[0108] To address the aforementioned technical issues, this disclosure introduces the RAFT framework into multi-view panoramic stereo matching tasks, constructing a flexible, efficient, and high-precision cyclic omnidirectional stereo matching model (RomniStero). RomniStereo can construct virtual reference panoramic feature volumes and target panoramic feature volumes based on a given surround-view camera structure. It then obtains panoramic correlation volumes based on these features, and subsequently samples relevant feature maps from these correlation volumes. Finally, it iteratively estimates the panoramic depth map using gated cyclic units.

[0109] Figure 1 schematically illustrates a scene diagram of a method for generating a panoramic depth map according to an embodiment of the present disclosure.

[0110] The panoramic depth map generation method of this embodiment can be applied to a panoramic camera system. As shown in Figure 1, the panoramic camera system can be an orthogonal panoramic four-eye fisheye camera, which includes four outward fisheye cameras located at the four corners of a square on the same plane, namely fisheye cameras 101, 102, 103 and 104. The orientation of each fisheye camera is 220° to ensure that each orientation in the space is covered by more than two cameras.

[0111] This embodiment of the disclosure can acquire images, such as images 105, 106, 107 and 108, through a panoramic camera system as shown in FIG1. ​​Then, images 105, 106, 107 and 108 are input into the RomniStereo model 109 provided in this disclosure for image processing, thereby obtaining a panoramic depth map 110.

[0112] It should be noted that, in this embodiment of the disclosure, the panoramic camera system is not limited to four orthogonally panoramic fisheye lenses, but rather to at least four fisheye lenses, and the images captured by the at least four fisheye lenses contain at least two image groups, wherein each image group contains images that can cover the panoramic field of view. For example, the panoramic camera system can be equipped with more than four fisheye lenses, such as six fisheye lenses, which can be divided into at least two image groups, such that each image group contains images that can cover the panoramic field of view.

[0113] It should be understood that a panoramic depth map can distinguish the distance of objects within a region, and therefore, obstacles can be identified from the panoramic depth map.

[0114] As one application scenario of this disclosure, the panoramic camera system can be the fuselage of a drone or an external device attached to the drone. The drone can be either an unmanned aerial vehicle or an unmanned robot. In this embodiment, the application of the panoramic camera system on a drone can provide the drone with a panoramic depth map to perceive the surrounding environment, and detect obstacles based on the panoramic depth map, thereby enabling the drone to perform obstacle avoidance or path planning.

[0115] Figure 2 schematically illustrates a flowchart of a method for generating a panoramic depth map according to an embodiment of the present disclosure.

[0116] As shown in Figure 2, the method for generating the panoramic depth map in this embodiment 200 includes operations S110 to S130. This method can be executed by a server or by a terminal device. The terminal device can be a camera or a drone.

[0117] In operation S210, for at least two image groups obtained by grouping target images from different orientations in the target scene, feature combinations are performed on the target image feature volumes of each image group to obtain at least two target panoramic feature volumes. Each image group contains target images covering the panoramic field of view of the target scene, and the target image feature volumes characterize the three-dimensional features of the target images.

[0118] According to embodiments of this disclosure, target images involving different orientations in a target scene may include target images from at least four orientations. For example, target images involving different orientations in a target scene may be four target images involving four different orientations, or six target images involving six different orientations.

[0119] According to embodiments of this disclosure, target images involving different orientations in a target scene can be acquired by fisheye lenses involving different orientations in the target scene.

[0120] For example, target images from different orientations could be four target images from different orientations taken by an orthogonal panoramic four-eye fisheye camera.

[0121] According to embodiments of this disclosure, target images in different orientations within a target scene can be acquired by ordinary lenses in different orientations within the target scene.

[0122] According to embodiments of this disclosure, the image group is divided as follows: for target images involving different orientations in the target scene, two back-to-back target images are divided into an image group to obtain at least two image groups, wherein the two back-to-back target images represent that the orientations of the two target images are opposite.

[0123] For example, as shown in Figure 1, with four fisheye lenses 101, 102, 103, and 104, fisheye lenses 101 and 103 are two back-to-back lenses, and fisheye lenses 102 and 104 are two back-to-back lenses. The image taken by fisheye lens 101 (e.g., image 105) and the image taken by fisheye lens 103 (e.g., image 107) are two back-to-back target images, and images 105 and 107 can be grouped into one image group. Similarly, images 106 and 108 can be grouped into another image group. It should be noted that images 105 and 107 in the image group can cover the panoramic field of view of the target scene, and images 106 and 108 in the image group can also cover the panoramic field of view of the target scene.

[0124] By dividing two back-to-back target images into an image group, it is possible to ensure that the target images contained in each image group cover the panoramic field of view, while calculating the target panoramic feature volume using the fewest number of target images, thereby reducing the amount of computation.

[0125] The target feature volume can be obtained by extracting features from the target image to obtain a feature map, and then performing a spherical scan on the feature map. The target feature volume can include the panoramic image size of the target scene and a preset number of discrete depths. Each target image feature volume is not complete in two-dimensional space, and each has blank areas, so it is not suitable as a reference frame and target frame in the calculation of the correlation volume.

[0126] Figure 3 schematically illustrates a flowchart of a method for determining target image features according to an embodiment of the present disclosure.

[0127] As shown in Figure 3, the method for determining the target image feature body in this embodiment 300 includes operations S310 to S320.

[0128] In operation S310, for each target image in the target scene involving different orientations, feature extraction is performed on the target image to obtain the target feature map.

[0129] Feature extraction of target images can be performed using the conventional operations in OminiMVS, employing a 2D convolutional neural network to extract features from each target image. The size of the extracted target feature map is H. f ×W f This corresponds to the size of the target image.

[0130] In operation S320, spherical scanning processing is performed on the target feature map to obtain the target image feature volume.

[0131] Spherical scanning maps the features of a target image onto a series of spheres centered on reference points. By performing spherical scanning on the target feature map, the three-dimensional features of each target image are obtained, i.e., the target image feature volume. The size of each target image feature volume is Hp×Wp×D, corresponding to the size of the panoramic image of the target scene and the preset number of discrete depths, respectively.

[0132] Each of the at least two target panoramic feature bodies must cover the entire field of view of the target scene, and there must be differences between the at least two target panoramic feature bodies.

[0133] In one example, image group 1 and image group 2 are included. Image group 1 includes target image feature body a1 and target image feature body b1, and image group 2 includes target image feature body a2 and target image feature body b2. Feature combination is performed on target image feature body a1 and target image feature body b1 in image group 1 to obtain target panoramic feature body c1; feature combination is performed on target image feature body a2 and target image feature body b2 in image group 2 to obtain target panoramic feature body c2. Target panoramic feature body c1 and target panoramic feature body c2 are obtained using different combinations of target images; therefore, target panoramic feature body c1 and target panoramic feature body c2 are essentially different.

[0134] In operation S220, correlation processing is performed on every two target panoramic features in at least two target panoramic feature volumes to obtain at least one target correlation volume.

[0135] According to embodiments of this disclosure, performing correlation processing on every two target panoramic feature bodies among at least two target panoramic feature bodies to obtain at least one target correlation body includes: calculating the inner product of every two target panoramic feature bodies among at least two target panoramic feature bodies to obtain at least one target correlation body.

[0136] In one example, there are three target panoramic feature bodies, namely target panoramic feature body a, target panoramic feature body b, and target panoramic feature body c. Calculating the inner product of each pair of target panoramic feature bodies to obtain at least one target related body may include: calculating the inner product of target panoramic feature body a and target panoramic feature body b to obtain target related body ab; calculating the inner product of target panoramic feature body a and target panoramic feature body c to obtain target related body ac; and calculating the inner product of target panoramic feature body b and target panoramic feature body c to obtain target related body bc.

[0137] In operation S230, panoramic depth estimation is performed based on the initial depth map and at least one target-related object to obtain a target panoramic depth map for the target scene.

[0138] According to embodiments of this disclosure, a technical solution is proposed to obtain at least two target panoramic feature bodies by using at least two image groups obtained from grouping target images involving different orientations in a target scene, combining the feature bodies of the target images in each image group, and then performing correlation processing on every two target panoramic features in the at least two target panoramic feature bodies to obtain at least one target correlation body. Subsequently, panoramic depth estimation is performed based on the initial depth map and at least one target correlation body to obtain a target panoramic depth map for the target scene. This constructs a virtual reference panoramic feature body and a target panoramic feature body, enabling the target correlation body to be constructed. This allows for iterative estimation of the panoramic depth map by sampling relevant feature maps on the target correlation body, thereby realizing the application of the RAFT architecture to surround-view panoramic stereo matching and improving the accuracy of panoramic image depth estimation.

[0139] Figure 4 schematically illustrates a method for determining target related entities according to an embodiment of the present disclosure.

[0140] As shown in Figure 4, Embodiment 400 may include four target images taken from four directions of the target scene using an orthogonal panoramic four-eye fisheye camera, namely target image A410, target image B420, target image C430, and target image D440. The method for determining the target related entities may include the following operations: First, by performing feature extraction on target image A410, target image B420, target image C430, and target image D440 respectively using the above operations S310 and S320, target image feature bodies a411 for target image A410, b421 for target image B420, c431 for target image C430, and d441 for target image D440 are obtained. Then, target images A410, B420, C430, and D440 are segmented to obtain image group 450 and image group 460. Image group 450 may include target images A410 and C430, which can be captured by two fisheye lenses facing away from each other. Image group 460 may include target images B420 and D440, which can also be captured by two fisheye lenses facing away from each other. Next, for image group 450, target image feature bodies a411 and c431 are combined to obtain target panoramic feature body 470; for image group 460, target image feature bodies b421 and d441 are combined to obtain target panoramic feature body 480. Then, the inner product is calculated based on the target panoramic feature volume 470 and the target panoramic feature volume 480 to obtain the target related volume 490.

[0141] According to embodiments of this disclosure, each target image feature has its own corresponding target weight. Combining the target image feature of each image group to obtain at least two target panoramic feature includes: using the target weight of each target image feature, performing a weighted summation process on the target image feature of each image group to obtain at least two target panoramic feature.

[0142] By performing weighted summation based on the target weights of each target image feature volume to obtain the target panoramic feature volume, the obtained target panoramic feature volume can be closer to the actual scene, thereby helping to improve the accuracy of depth estimation.

[0143] According to embodiments of this disclosure, the target weight of each target image feature is determined by the following operation: for each image group, the target image features in the image group are cascaded to obtain the first cascaded target image features; the first cascaded target image features are subjected to multi-layer perception processing to obtain the target weight of each target image feature in the image group.

[0144] To ensure the stability of the weights, the sum of the weights of two target image feature bodies, such as target image feature body a and target image feature body b, should be 1 when inputting into a multilayer perceptron. To achieve this goal while adapting the weights to the target image feature bodies, an Opposite Adaptive Weighting method is provided. This method uses the concatenation of the two target image feature bodies to predict the weight of one of the target image feature bodies, and then subtracts the predicted weight from 1 to obtain the weight of the other target image feature body.

[0145] Figure 5 schematically illustrates a diagram of determining the target weight using a complementary adaptive weighting method according to an embodiment of the present disclosure.

[0146] As shown in Figure 5, Embodiment 500 includes target image A510, target image B520, target image C530, and target image D550. The method for determining the target weight of each target image feature body may include the following operations: First, by using the above operations S310 and S320 to extract features from target image A510, target image B520, target image C530, and target image D550 respectively, target image feature body a511 for target image A510, target image feature body b521 for target image B520, target image feature body c531 for target image C530, and target image feature body d551 for target image D550 are obtained. Then, the target images A510, B520, C530, and D550 are segmented to obtain image group 550 and image group 560. Image group 550 may include target image A510 and target image C530; image group 560 may include target image B520 and target image D550. Next, for image group 550, the target image feature body a511 for target image A510 and the target image feature body c531 for target image C530 are concatenated to obtain the first concatenated target image feature body 570. The target image feature body b5211 for target image B520 and the target image feature body d551 for target image D550 are concatenated to obtain the first concatenated target image feature body 580. Then, a multilayer perceptron is used to estimate the weights of the target image feature body 570 and the target image feature body 580 after the first concatenation, respectively, to obtain the target weights 512 of target image feature body a, 522 of target image feature body b, 532 of target image feature body c, and 542 of target image feature body d.

[0147] According to embodiments of this disclosure, an Opposite Interleving method is also provided to determine the target weights for each target image feature volume.

[0148] For example, for each pixel in the target image feature body, determine the distance between the pixel and the target pixel, where the target pixel corresponds to the camera center; determine the position weight of the pixel based on the distance; and determine the target weight of the target image feature body based on the position weight of each pixel.

[0149] The camera center can be the direction of the line of sight, and the target pixel corresponding to the camera center can be a pixel in the target image feature body located in the direction of the line of sight.

[0150] For example, when the target image is a fisheye image taken with a fisheye lens, the target pixel can be the center of the fisheye lens.

[0151] According to embodiments of this disclosure, determining the position weight of a pixel based on distance includes: determining the position weight of the pixel as a first value when the distance is less than a preset distance threshold; and determining the position weight of the pixel as a second value when the distance is greater than or equal to the preset distance threshold, wherein the first value and the second value are different.

[0152] The first and second values ​​can be any values, as long as they are different. For example, the first value can be any non-zero value, and the second value can be zero.

[0153] The preset threshold can be determined according to actual needs, and this disclosure does not limit the specific value of the preset threshold.

[0154] By employing a complementary interleaving method, binary weights are obtained directly based on the distance from the pixel to the camera center, and these weights are the same for different sampling depths.

[0155] According to embodiments of this disclosure, an all-weighting method is also provided for determining the target weights of each target image feature volume.

[0156] For example, target image feature volumes involving different orientations in a target scene are cascaded to obtain a second cascaded target image feature volume; multi-layer perception processing is then performed on the second cascaded target image feature volume to obtain the target weight of each target image feature volume.

[0157] Figure 6 schematically illustrates a diagram of determining the target weight using a fully weighted method according to an embodiment of the present disclosure.

[0158] As shown in Figure 6, Embodiment 600 includes target image A610, target image B620, target image C630, and target image D640. The method for determining the target weight of each target image feature body may include: First, by performing feature extraction on target image A610, target image B620, target image C630, and target image D640 respectively using the above-mentioned operations S310 and S320, target image feature body a611 for target image A610, target image feature body b621 for target image B620, target image feature body c631 for target image C630, and target image feature body d641 for target image D640 are obtained. Then, target image feature bodies a611, b621, c631, and d641 are concatenated to obtain a second concatenated target image feature body 650. Then, the target image feature volume 650 after the second concatenation is input into the multilayer perceptron to estimate the weight volume, and the target weight 612 of target image feature volume a, the target weight 622 of target image feature volume b, the target weight 632 of target image feature volume c, and the target weight 642 of target image feature volume d are obtained.

[0159] Figure 7 schematically illustrates the target weights determined using different methods.

[0160] As shown in Figure 7, the weight maps corresponding to the farthest (d0), middle (dN / 2), and nearest (dN-1) sampling depths of the target image feature volume determined by the above three methods (Opposite Interleving, Opposite Adaptive Weighting, and All Weighting) are listed respectively, for example, weight map 710, weight map 720, and weight map 730.

[0161] The weight map 710 obtained using the Opposite Interleving method shown in the first column has binary weights, so the corresponding weight map 710 will vary for different d values. n It is fixed. The weight map 720 obtained using the Opposite Adaptive Weighting method shown in the second column can change with d. n It adapts and changes adaptively with the scene structure. The weight map 730 obtained using the All Weighting method shown in the third column has discontinuous staggered boundaries.

[0162] According to embodiments of this disclosure, performing panoramic depth estimation based on an initial depth map and at least one target related object to obtain a target panoramic depth map for a target scene includes: performing panoramic depth estimation based on an initial depth map, a preset target context feature object, and at least one target related object to obtain a target panoramic depth map for a target scene, wherein the target context feature object is determined based on at least one target panoramic feature object.

[0163] The initial depth map can be the furthest depth. In this embodiment of the disclosure, inverse depth is used, with the furthest point being zero; therefore, the initial depth map can be a zero matrix. The preset target context feature volume can be one of at least one target panoramic feature volume.

[0164] According to embodiments of this disclosure, performing panoramic depth estimation based on an initial depth map, a preset target context feature, and at least one target related object to obtain a target panoramic depth map for a target scene includes: using the initial depth map as the current estimated depth map, performing panoramic depth estimation based on the current estimated depth map, the target context feature, and at least one target related object to obtain a depth estimation increment; updating the current estimated depth map according to the depth estimation increment to obtain an updated estimated depth map; using the updated estimated depth as the current estimated depth map, repeatedly performing the above operations until the number of iterations reaches a first preset iteration threshold to obtain a target panoramic depth map for the target scene.

[0165] Updating the current depth estimation map based on the depth estimation increment can result in an updated estimated depth map by adding the depth estimation increment to the current depth estimation map.

[0166] The first preset cycle threshold can be pre-set, for example, it can be 10 times, 12 times, etc.

[0167] It should be noted that during the panoramic depth estimation process, for the input target image, there is first an initial depth map D_0 (e.g., all depths are 0), and then a series of estimated depth sequences D_1, D_2, ..., D_n are obtained through iterative loops. The last image of these estimated depth sequences is then used as the target panoramic depth map.

[0168] According to embodiments of this disclosure, panoramic depth estimation is performed based on the current estimated depth map, target context features, and at least one target related object to obtain the depth estimation increment. This includes: sampling the target context features and at least one target related object using the current estimated depth map and preset sampling neighbor values ​​respectively to obtain the current context feature map and the current related feature map; and performing panoramic depth estimation using the current context feature map and the current related feature map to obtain the depth estimation increment.

[0169] The process of sampling the target context feature body and at least one target related body using the current estimated depth map and preset sampling neighbor values ​​respectively to obtain the current context feature map and the current related feature map includes: sampling the target context feature body to obtain the current context feature map using the current estimated depth map and preset sampling neighbor values; and sampling the target related body to obtain the current related feature map using the current estimated depth map and preset sampling neighbor values.

[0170] When the number of target related entities is greater than one, the following method can be used to determine the current related feature map.

[0171] In one example, information from multiple target related entities is combined to obtain a joint related entity. A current related feature map is then obtained by sampling from the joint related entity using the current estimated depth map and preset sampling neighbor values. Combining the information from multiple target related entities may include concatenating the features of the multiple target related entities by channel.

[0172] For example, when determining the current relevant feature map for target related body 1, target related body 2, and target related body 3, the information of target related body 1, target related body 2, and target related body 3 can be combined to obtain a joint related body. Then, the current relevant feature map can be obtained by sampling from the joint related body using the current estimated depth map and the preset sampling neighbor value.

[0173] In another example, multiple target related objects are sampled using the current estimated depth map and preset sampling neighbor values, resulting in multiple sampling results. The information from these multiple sampling results is then combined to obtain the current related object feature map. For example, for target related objects 1, 2, and 3, when determining the current related feature map, sampling result 1 can be obtained from target related object 1 using the current estimated depth map and preset sampling neighbor values; sampling result 2 can be obtained from target related object 2 using the current estimated depth map and preset sampling neighbor values; and sampling result 3 can be obtained from target related object 3 using the current estimated depth map and preset sampling neighbor values. Then, the information from sampling result 1, sampling result 2, and sampling result 3 is combined to obtain the current related feature map.

[0174] Panoramic depth estimation can be performed using the current context feature map and the current relevant feature map. The depth estimation increment can be obtained by inputting the current context feature map and the current relevant feature map into the GRU module of the RomniStereo model, performing depth estimation, and outputting the depth estimation increment.

[0175] Figure 8 is a flowchart schematically illustrating a method for generating a panoramic depth map according to another embodiment of the present disclosure.

[0176] As shown in Figure 8, the panoramic depth map generation method of this embodiment 800 includes operations S210 and S220 as well as operations S810 to S870.

[0177] When operating S810, the initial depth map is used as the current depth estimation map.

[0178] In operation S820, the current estimated depth map and the preset sampling neighbor value are used to sample the preset target context feature body and at least one target related body respectively to obtain the current context feature map and the current related feature map.

[0179] When operating the S830, the current context feature map and the current relevant feature map are input into the GRU module to perform panoramic depth estimation, and the depth estimation increment is obtained.

[0180] When operating the S840, the current depth estimation map is updated based on the depth estimation increment to obtain the updated estimated depth map.

[0181] In operation S850, it is determined whether the number of iterations has reached a first preset threshold. If it has, operation S870 is executed; otherwise, operation S860 is executed.

[0182] In operation S860, the updated estimated depth is used as the current estimated depth map, and then operation S820 is executed.

[0183] When operating the S870, the updated estimated depth is used as the target panoramic depth map.

[0184] Figure 9 schematically illustrates a method for obtaining a panoramic depth map by processing an image using the RomniStero model.

[0185] As shown in Figure 9, the surround-view camera in embodiment 900 includes four fisheye cameras, which respectively acquire images from four directions of the target scene. The panoramic depth map generation method in this embodiment includes a first stage 910, a second stage 920, and a third stage 930.

[0186] In the first stage 910, firstly, feature extraction and spherical scanning are performed on the fisheye images 911, 912, 913, and 914 acquired by the surround-view camera to obtain target image feature bodies 915 for fisheye image 911, 916 for fisheye image 912, 917 for fisheye image 913, and 918 for fisheye image 914. Among them, fisheye images 911 and 913 are two back-to-back images, and fisheye images 912 and 914 are two back-to-back images. Therefore, fisheye images 911 and 913 are divided into the first image group, and fisheye images 912 and 914 are divided into the second image group.

[0187] In the second stage 920, for fisheye images 911 and 913 in the first image group, the target image feature body 915 for fisheye image 911 and the target image feature body 917 for fisheye image 913 are combined to obtain the target panoramic feature body 921; for fisheye images 912 and 914 in the second image group, the target image feature body 916 for fisheye image 912 and the target image feature body 918 for fisheye image 914 are combined to obtain the target panoramic feature body 922; then, correlation calculation is performed based on the target panoramic feature body 921 and the target panoramic feature body 922 to obtain the target correlation body 924; the context feature body is initialized based on the target panoramic feature body 921 to obtain the target context feature body 923.

[0188] In the third stage 930, the initial depth map is first used as the current depth estimation map. Based on the current estimated depth 931, the current context feature map 932 is obtained by sampling from the target context feature volume 923, and the current related feature map 933 is obtained by sampling from the target related volume 924 based on the current estimated depth 931. Then, the current context feature map 932 and the current related feature map 933 are input into the GRU module for depth estimation, and the depth estimation increment is output. The current depth estimation map is updated based on the depth estimation increment to obtain the updated depth estimation map 936. If the loop threshold is not reached at this time, the updated depth estimation map 936 is used as the current depth estimation map 931 for loop iteration. If the loop threshold is reached at this time, the updated depth estimation map 936 is output to obtain the target panoramic depth map 937. The scene is reconstructed based on the target panoramic depth map 937 to obtain the reconstructed image 938 of the target scene.

[0189] According to embodiments of this disclosure, an efficient model for surround-view panoramic depth estimation, namely the RomniStero model, is proposed, extending the RAFT framework to surround-view multi-view stereo matching tasks. To bridge the gap between OSM and traditional pinhole image matching, this disclosure utilizes camera structure to construct target-related volumes before adaptively combining relative views for subsequent iterative processing. Furthermore, this disclosure introduces two beneficial techniques into the RomniStereo model: mesh embedding, such as multilayer perceptron embedding, and adaptive context feature generation, such as automatically generating a context feature volume using one of the target-related volumes. Extensive experiments demonstrate the effectiveness and efficiency of this method.

[0190] According to embodiments of this disclosure, the panoramic depth estimation model (RomniStereo model) of this disclosure and the panoramic depth estimation models (S-OmiNVS model and OmiNVS model) of related technologies are evaluated using the datasets OmniThingsOT, OminiHouse(OH), Sunny(Sn), Cloudy(Cd) and Sunset(Ss), respectively. The results are shown in Tables 1 and 2.

[0191] As shown in Tables 1 and 2, the RomniStereo model provided in this disclosure is twice as fast as the original OmniMVS model, and exhibits small depth estimation errors in numerous model configurations and test datasets. Specifically, the best model configuration of the surround-view depth estimation model in this disclosure reduces the mean error (MAE) by an average of 40.7% across five datasets compared to the best model configuration of the OmniMVS model.

[0192] Table 1

[0193] Table 2

[0194] Figure 10 schematically illustrates a flowchart of a training method for a depth estimation model according to an embodiment of the present disclosure.

[0195] As shown in Figure 10, the training method of the depth estimation model in this embodiment includes operations S1010 to S1040.

[0196] In operation S1010, for at least two sample image groups obtained by grouping sample images from different orientations in the sample scene based on training samples, the feature volume of the sample image in each sample image group is combined to obtain at least two sample panoramic feature volumes. The sample images contained in each sample image group cover the panoramic field of view of the sample scene, and the sample image feature volume represents the three-dimensional stereo features of the sample image.

[0197] In operation S1020, correlation processing is performed on every two sample panoramic features in at least two sample panoramic feature volumes to obtain at least one sample correlation volume.

[0198] In operation S1030, panoramic depth estimation is performed based on the initial depth map and at least one sample related body to obtain a predicted panoramic depth map for the sample scene.

[0199] In operation S1040, the loss information of the depth estimation model is determined based on the predicted panoramic depth map, and the network parameters of the depth estimation model are iteratively adjusted based on the loss information until the loss information meets the iteration stopping condition. The network parameters obtained when the iteration stopping condition is met are used as the trained depth estimation model.

[0200] According to embodiments of this disclosure, it may also include acquiring a sample panoramic depth map for the sample scene.

[0201] The iteration stopping condition can include a preset number of iterations, or minimizing the error between the predicted panoramic depth map and the sample panoramic depth map.

[0202] For example, determining the loss information of the depth estimation model based on the predicted panoramic depth map may include determining the loss value of the predicted panoramic depth map and the sample panoramic depth map based on a preset loss function, and stopping the iteration when the loss value is less than a preset threshold.

[0203] According to embodiments of this disclosure, each sample image feature body has its own sample weight, and feature combination is performed on the sample image feature bodies of each sample image to obtain at least two sample panoramic feature bodies, including:

[0204] By utilizing the sample weights of each sample image feature body, the sample image feature bodies of each sample image group are weighted and summed to obtain at least two sample panoramic feature bodies.

[0205] According to embodiments of this disclosure, each sample image feature volume has its own sample weight, which is determined by the following operation:

[0206] For each group of sample images, the sample image feature bodies in the sample image group are concatenated to obtain the first concatenated sample image feature bodies;

[0207] Multi-layer perceptron processing is performed on the feature volumes of the first concatenated sample images to obtain the sample weights of each feature volume in the sample image group.

[0208] According to embodiments of this disclosure, each sample image feature volume has its own sample weight, which is determined by the following operation:

[0209] For each sample pixel in the feature body of the sample image, determine the distance between the sample pixel and the target sample pixel;

[0210] The positional weights for sample pixels are determined based on the distance.

[0211] The sample weights of the feature bodies of the sample image are determined based on the positional weights of each sample pixel.

[0212] According to embodiments of this disclosure, determining the positional weight for a sample pixel based on distance includes:

[0213] If the distance is less than a preset distance threshold, the position weight of the sample pixel is determined to be the first value;

[0214] If the distance is greater than or equal to a preset distance threshold, the position weight of the sample pixel is determined to be a second value, wherein the first value is different from the second value.

[0215] According to embodiments of this disclosure, each sample image feature volume has its own sample weight, which is determined by the following operation:

[0216] The sample image feature volumes of sample images from different orientations in the sample scene are concatenated to obtain the second concatenated sample image feature volume.

[0217] Multi-layer perceptual processing is performed on the feature volume of the second concatenated sample image to obtain the sample weight of each sample image feature.

[0218] According to embodiments of this disclosure, the sample image group is divided using the following operation:

[0219] For sample images involving different orientations in a sample scene, two sample images facing away from each other are divided into a sample image group, resulting in at least two sample image groups. The two sample images facing away from each other represent that the orientations of the two sample images are opposite.

[0220] According to embodiments of this disclosure, correlation processing is performed on every two sample panoramic feature bodies in at least two sample panoramic feature bodies to obtain at least one sample correlation body, including:

[0221] Calculate the inner product of every two panoramic feature volumes in at least two sample panoramic feature volumes to obtain at least one sample correlation volume.

[0222] According to embodiments of this disclosure, panoramic depth estimation is performed based on an initial depth map and at least one sample related body to obtain a sample panoramic depth map for the sample scene, including:

[0223] Panoramic depth estimation is performed based on the initial depth map, the preset sample context feature volume, and at least one sample related volume to obtain a sample panoramic depth map for the sample scene. The sample context feature volume is determined based on at least one sample panoramic feature volume.

[0224] According to embodiments of this disclosure, panoramic depth estimation is performed based on an initial depth map, preset sample context features, and at least one sample related object to obtain a sample panoramic depth map for the sample scene, including:

[0225] Using the initial depth map as the current sample estimated depth map, panoramic depth estimation is performed based on the current sample estimated depth map, sample context feature volume, and at least one sample related volume to obtain the sample depth estimation increment;

[0226] The current sample depth estimation map is updated based on the sample depth estimation increment, resulting in the updated sample depth estimation map;

[0227] Using the updated sample estimated depth as the current sample estimated depth map, repeat the above operation until the number of iterations reaches the second preset iteration threshold to obtain a sample panoramic depth map for the sample scene.

[0228] According to embodiments of this disclosure, panoramic depth estimation is performed based on the current estimated depth map, sample context features, and at least one sample related body to obtain the sample depth estimation increment, which includes:

[0229] Using the current sample to estimate the depth map and the preset sampling neighbor value, sample the sample context feature body and at least one sample related body respectively to obtain the current sample context feature map and the current sample related feature map;

[0230] The panoramic depth estimation is performed using the current sample context feature map and the current sample related feature map to obtain the sample depth estimation increment.

[0231] According to embodiments of this disclosure, the method further includes:

[0232] For each sample image in the sample scene involving different orientations, feature extraction is performed on the sample image to obtain a sample feature map;

[0233] The sample feature map is processed by spherical scanning to obtain the sample image feature volume.

[0234] According to embodiments of this disclosure, the sample images in the sample scene involving different orientations include sample images from at least four orientations.

[0235] According to embodiments of this disclosure, sample images involving different orientations in a sample scene are acquired by fisheye lenses involving different orientations in the sample scene.

[0236] According to embodiments of this disclosure, the training method for the depth estimation model is the same as the method for generating panoramic depth maps described above, and will not be repeated here.

[0237] Based on the above-described method for generating panoramic depth maps, this disclosure also provides an apparatus for generating panoramic depth maps. The apparatus will be described in detail below with reference to Figure 11.

[0238] Figure 11 schematically shows a structural block diagram of a panoramic depth map generation apparatus according to an embodiment of the present disclosure.

[0239] As shown in Figure 11, the panoramic depth map generation device 1100 of this embodiment includes a first feature combination module 1110, a first correlation processing module 1120, and a first depth estimation module 1130.

[0240] The first feature combination module 1110 is used to combine the feature volumes of the target images in each of at least two image groups obtained by grouping target images from different orientations in the target scene, to obtain at least two panoramic feature volumes. Each image group contains target images that cover the panoramic field of view of the target scene, and the target image feature volumes characterize the three-dimensional features of the target images. In one embodiment, the first feature combination module 1110 can be used to perform the operation S210 described above, which will not be repeated here.

[0241] The first correlation processing module 1120 is used to perform correlation processing on every two target panoramic features in at least two target panoramic feature volumes to obtain at least one target correlation volume. In one embodiment, the first correlation processing module 1120 can be used to perform the operation S220 described above, which will not be repeated here.

[0242] The first depth estimation module 1130 is used to perform panoramic depth estimation based on an initial depth map and at least one target-related object to obtain a target panoramic depth map for the target scene. In one embodiment, the first depth estimation module 1130 can be used to perform the operation S230 described above, which will not be repeated here.

[0243] According to embodiments of this disclosure, each target image feature has its own corresponding target weight.

[0244] According to an embodiment of this disclosure, the first feature combination module includes: a first weighted summation processing submodule.

[0245] The first weighted summation processing submodule is used to perform weighted summation processing on the target image feature bodies of each image group using the target weights of each target image feature body, so as to obtain at least two target panoramic feature bodies.

[0246] According to embodiments of this disclosure, the panoramic depth map generation apparatus further includes: a first cascaded module and a first multilayer sensing module.

[0247] The first cascade module is used to perform cascade processing on the target image feature bodies in each image group to obtain the target image feature bodies after the first cascade.

[0248] The first multi-layer perception module is used to perform multi-layer perception processing on the target image feature body after the first cascade to obtain the target weights of the target image feature bodies in the image group.

[0249] According to embodiments of this disclosure, the panoramic depth map generation apparatus further includes: a first determining module, a second determining module, and a third determining module.

[0250] The first determining module is used to determine the distance between each pixel in the feature body of the target image and the target pixel, wherein the target pixel corresponds to the center of the camera.

[0251] The second determining module is used to determine the position weight of a pixel based on the distance.

[0252] The third determining module is used to determine the target weight of the target image feature body based on the position weight of each pixel.

[0253] According to embodiments of this disclosure, the second determining module includes: a first determining submodule and a second determining submodule.

[0254] The first determination submodule is used to determine the position weight of a pixel as a first value when the determined distance is less than a preset distance threshold.

[0255] The second determining submodule is used to determine the position weight of a pixel as a second value when the determined distance is greater than or equal to a preset distance threshold, wherein the first value is different from the second value.

[0256] According to embodiments of this disclosure, the panoramic depth map generation apparatus further includes: a second cascade module and a second multilayer sensing module.

[0257] The second cascade module is used to cascade the target image feature bodies of target images from different orientations in the target scene to obtain the target image feature bodies after the second cascade.

[0258] The second multi-layer perception module is used to perform multi-layer perception processing on the target image feature volume after the second concatenation to obtain the target weight of each target image feature volume.

[0259] According to embodiments of this disclosure, the panoramic depth map generation apparatus further includes a first segmentation module.

[0260] The first segmentation module is used to divide two back-to-back target images in the target scene into an image group for target images involving different orientations, thereby obtaining at least two image groups, wherein the two back-to-back target images represent that the orientations of the two target images are opposite.

[0261] According to an embodiment of this disclosure, the first correlation processing module includes: a first inner product calculation submodule.

[0262] The first inner product calculation submodule is used to perform inner product calculation on every two target panoramic feature volumes in at least two target panoramic feature volumes to obtain at least one target related volume.

[0263] According to an embodiment of this disclosure, the first depth estimation module includes: a first depth estimation submodule.

[0264] The first depth estimation submodule is used to perform panoramic depth estimation based on an initial depth map, a preset target context feature body, and at least one target related body to obtain a target panoramic depth map for the target scene, wherein the target context feature body is determined based on at least one target panoramic feature body.

[0265] According to embodiments of this disclosure, the first depth estimation submodule includes: a first depth estimation unit, a first update unit, and a first loop unit.

[0266] The first depth estimation unit is used to perform panoramic depth estimation based on the initial depth map as the current estimated depth map, the target context feature body, and at least one target related body, to obtain the depth estimation increment.

[0267] The first update unit is used to update the current depth estimation map based on the depth estimation increment, so as to obtain the updated estimated depth map.

[0268] The first loop unit is used to repeatedly perform the above operation with the updated estimated depth as the current estimated depth map until the number of loops reaches the first preset loop threshold, so as to obtain the target panoramic depth map for the target scene.

[0269] According to embodiments of this disclosure, the first depth estimation unit includes: a first sampling subunit and a first depth estimation subunit.

[0270] The first sampling subunit is used to sample the target context feature body and at least one target related body using the current estimated depth map and the preset sampling neighbor value, respectively, to obtain the current context feature map and the current related feature map.

[0271] The first depth estimation subunit is used to perform panoramic depth estimation using the current context feature map and the current relevant feature map to obtain the depth estimation increment.

[0272] According to embodiments of this disclosure, the panoramic depth map generation apparatus further includes: a first feature extraction module and a first spherical scanning module.

[0273] The first feature extraction module is used to extract features from each target image in the target scene that involves different orientations, and obtain a target feature map.

[0274] The first spherical scanning module is used to perform spherical scanning processing on the target feature map to obtain the target image feature volume.

[0275] According to embodiments of this disclosure, the target images in the target scene involving different orientations include target images from at least four orientations.

[0276] According to embodiments of this disclosure, target images in different orientations within a target scene are acquired by fisheye lenses in different orientations within the target scene.

[0277] According to embodiments of this disclosure, the depth map generation apparatus further includes an obstacle determination module.

[0278] The obstacle determination module is used to determine obstacles based on the panoramic depth map, so that the drone can perform obstacle avoidance.

[0279] According to embodiments of this disclosure, any plurality of modules among the first feature combination module 1110, the first correlation processing module 1120, and the first depth estimation module 1130 may be combined into one module, or any one of these modules may be split into multiple modules. Alternatively, at least a portion of the functionality of one or more of these modules may be combined with at least a portion of the functionality of other modules and implemented in one module. According to embodiments of this disclosure, at least one of the first feature combination module 1110, the first correlation processing module 1120, and the first depth estimation module 1130 may be at least partially implemented as hardware circuitry, such as a field-programmable gate array (FPGA), a programmable logic array (PLA), a system-on-a-chip, a system-on-a-substrate, a system-on-package, an application-specific integrated circuit (ASIC), or implemented in hardware or firmware by any other reasonable means of integrating or packaging the circuitry, or implemented in any one of software, hardware, and firmware methods, or in a suitable combination of any of these methods. Alternatively, at least one of the first feature combination module 1110, the first correlation processing module 1120, and the first depth estimation module 1130 may be at least partially implemented as a computer program module, which can perform corresponding functions when the computer program module is run.

[0280] Based on the training method of the depth estimation model described above, this disclosure also provides a training device for the depth estimation model. The device will be described in detail below with reference to Figure 12.

[0281] Figure 12 schematically illustrates a structural block diagram of a training apparatus for a depth estimation model according to an embodiment of the present disclosure.

[0282] As shown in Figure 12, the panoramic depth map generation device 1200 of this embodiment includes a second feature combination module 1210, a second correlation processing module 1220, a second depth estimation module 1230, and an iterative adjustment module 1240.

[0283] The second feature combination module 1210 is used to combine the features of the sample image feature bodies of each sample image group obtained by grouping sample images involving different orientations in a sample scene based on training samples, to obtain at least two sample panoramic feature bodies. The sample images contained in each sample image group cover the panoramic field of view of the sample scene, and the sample image feature bodies represent the three-dimensional stereo features of the sample images.

[0284] The second correlation processing module 1220 is used to perform correlation processing on every two sample panoramic features in at least two sample panoramic feature bodies to obtain at least one sample correlation body.

[0285] The second depth estimation module 1230 is used to perform panoramic depth estimation based on the initial depth map and at least one sample related body to obtain a predicted panoramic depth map for the sample scene.

[0286] The iterative adjustment module 1240 is used to determine the loss information of the depth estimation model based on the predicted panoramic depth map, and iteratively adjust the network parameters of the depth estimation model based on the loss information until the loss information meets the iteration stopping condition. The network parameters obtained when the iteration stopping condition is met are used as the trained depth estimation model.

[0287] According to embodiments of this disclosure, each sample image feature has its own sample weight.

[0288] According to embodiments of this disclosure, the second feature combination module includes: a second weighted summation processing submodule.

[0289] The second weighted summation processing submodule is used to perform weighted summation processing on the sample image feature bodies of each sample image group using the sample weights of each sample image feature body, so as to obtain at least two sample panoramic feature bodies.

[0290] According to embodiments of this disclosure, the training device further includes a third cascade module and a third multilayer sensing module.

[0291] The third cascade module is used to perform cascade processing on the sample image feature bodies in each sample image group to obtain the sample image feature bodies after the first cascade.

[0292] The third multi-layer perception module is used to perform multi-layer perception processing on the feature bodies of the sample images after the first concatenation to obtain the sample weights of the feature bodies of the sample images in the sample image group.

[0293] According to embodiments of this disclosure, the training device further includes a fourth determining module, a fifth determining module, and a sixth determining module.

[0294] The fourth determination module is used to determine the distance between the sample pixel and the target sample pixel for each sample pixel in the feature body of the sample image.

[0295] The fifth determination module is used to determine the position weight of the sample pixel based on the distance.

[0296] The sixth determination module is used to determine the sample weight of the sample image feature body based on the position weight of each sample pixel.

[0297] According to embodiments of this disclosure, the fifth determining module includes a third determining submodule and a fourth determining submodule.

[0298] The third determination submodule is used to determine the position weight of the sample pixel as the first value when the determined distance is less than the preset distance threshold.

[0299] The fourth determination submodule is used to determine the position weight of the sample pixel as a second value when the determined distance is greater than or equal to a preset distance threshold, wherein the first value is different from the second value.

[0300] According to embodiments of this disclosure, the training device further includes a fourth cascade module and a fourth multilayer sensing module.

[0301] The fourth concatenation module is used to concatenate the sample image feature bodies of sample images from different orientations in the sample scene to obtain the sample image feature bodies after the second concatenation.

[0302] The fourth multi-layer perception module is used to perform multi-layer perception processing on the feature volume of the sample image after the second concatenation to obtain the sample weight of each sample image feature.

[0303] According to embodiments of this disclosure, the training device further includes a second partitioning module.

[0304] The second segmentation module is used to divide two back-to-back sample images into a sample image group for sample images involving different orientations in the sample scene, so as to obtain at least two sample image groups, wherein the two back-to-back sample images represent that the orientations of the two sample images are opposite.

[0305] According to embodiments of this disclosure, the second correlation processing module includes a second inner product calculation submodule.

[0306] The second inner product calculation submodule is used to perform inner product calculation on every two sample panoramic feature bodies in at least two sample panoramic feature bodies to obtain at least one sample correlation body.

[0307] According to embodiments of this disclosure, the second depth estimation module includes a second depth estimation submodule.

[0308] The second depth estimation submodule is used to perform panoramic depth estimation based on the initial depth map, the preset sample context feature volume and at least one sample related volume, to obtain a sample panoramic depth map for the sample scene, wherein the sample context feature volume is determined based on at least one sample panoramic feature volume.

[0309] According to embodiments of this disclosure, the second depth estimation submodule includes: a second depth estimation unit, a second update unit, and a second loop unit.

[0310] The second depth estimation unit is used to estimate the depth map of the current sample using the initial depth map as the current sample depth map, and to perform panoramic depth estimation based on the current sample estimated depth map, sample context feature volume and at least one sample related volume to obtain the sample depth estimation increment.

[0311] The second update unit is used to update the current sample depth estimation map based on the sample depth estimation increment, so as to obtain the updated sample depth estimation map.

[0312] The second loop unit is used to repeatedly execute the above operation with the updated sample estimated depth as the current sample estimated depth map until the number of loops reaches the second preset loop threshold, so as to obtain a sample panoramic depth map for the sample scene.

[0313] According to embodiments of this disclosure, the second depth estimation unit includes a second depth subunit and a sampling subunit.

[0314] The second depth estimation subunit is used to estimate the depth map of the current sample and sample the sample context feature body and at least one sample correlation body respectively using the current sample estimated depth map and the current sample correlation feature map.

[0315] The second sampling subunit is used to perform panoramic depth estimation using the current sample context feature map and the current sample related feature map to obtain the sample depth estimation increment.

[0316] According to embodiments of this disclosure, the training device further includes: a second feature extraction module and a second spherical scanning module.

[0317] The second feature extraction module is used to extract features from each sample image in the sample scene that involves different orientations, and obtain a sample feature map.

[0318] The second spherical scanning module is used to perform spherical scanning processing on the sample feature map to obtain the sample image feature volume.

[0319] According to embodiments of this disclosure, the sample images in the sample scene involving different orientations include sample images from at least four orientations.

[0320] According to embodiments of this disclosure, sample images involving different orientations in a sample scene are acquired by fisheye lenses involving different orientations in the sample scene.

[0321] According to embodiments of this disclosure, any plurality of modules among the second feature combination module 1210, the second correlation processing module 1220, the second depth estimation module 1230, and the iterative adjustment module 1240 can be combined into one module, or any one of these modules can be split into multiple modules. Alternatively, at least part of the functionality of one or more of these modules can be combined with at least part of the functionality of other modules and implemented in one module. According to embodiments of this disclosure, at least one of the second feature combination module 1210, the second correlation processing module 1220, the second depth estimation module 1230, and the iterative adjustment module 1240 can be at least partially implemented as hardware circuitry, such as a field-programmable gate array (FPGA), a programmable logic array (PLA), a system-on-a-chip, a system-on-a-substrate, a system-on-package, an application-specific integrated circuit (ASIC), or implemented in hardware or firmware by any other reasonable means of integrating or packaging the circuitry, or implemented in software, hardware, or firmware, or in any appropriate combination of any of these three implementation methods. Alternatively, at least one of the second feature combination module 1210, the second correlation processing module 1220, the second depth estimation module 1230, and the iterative adjustment module 1240 can be at least partially implemented as a computer program module, which can perform corresponding functions when the computer program module is run.

[0322] Embodiments of this disclosure also provide an electronic device, including: one or more processors; and a memory for storing one or more computer programs, wherein the one or more processors execute the one or more computer programs to implement the steps of the method.

[0323] Embodiments of this disclosure also provide a drone, including the aforementioned electronic equipment.

[0324] Embodiments of this disclosure also provide a computer-readable storage medium having a computer program or instructions stored thereon, which, when executed by a processor, implement the steps of the above-described method.

[0325] Embodiments of this disclosure also provide a computer program product, including a computer program or instructions that, when executed by a processor, implement the steps of the above-described method.

[0326] Figure 13 schematically illustrates a block diagram of an electronic device suitable for implementing the above-described method according to an embodiment of the present disclosure.

[0327] As shown in FIG13, an electronic device 1300 according to an embodiment of the present disclosure includes a processor 1301, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 1302 or a program loaded from a storage portion 1308 into a random access memory (RAM) 1303. The processor 1301 may include, for example, a general-purpose microprocessor (e.g., a CPU), an instruction set processor and / or an associated chipset and / or a special-purpose microprocessor (e.g., an application-specific integrated circuit (ASIC)), etc. The processor 1301 may also include onboard memory for caching purposes. The processor 1301 may include a single processing unit or multiple processing units for performing different actions of the method flow according to an embodiment of the present disclosure.

[0328] RAM 1303 stores various programs and data required for the operation of electronic device 1300. Processor 1301, ROM 1302, and RAM 1303 are interconnected via bus 1304. Processor 1301 performs various operations of the method flow according to embodiments of the present disclosure by executing programs in ROM 1302 and / or RAM 1303. It should be noted that the programs may also be stored in one or more memories other than ROM 1302 and RAM 1303. Processor 1301 may also perform various operations of the method flow according to embodiments of the present disclosure by executing programs stored in said one or more memories.

[0329] According to embodiments of this disclosure, the electronic device 1300 may further include an input / output (I / O) interface 1305, which is also connected to a bus 1304. The electronic device 1300 may also include one or more of the following components connected to the input / output (I / O) interface 1305: an input section 1306 including a keyboard, mouse, etc.; an output section 1307 including a cathode ray tube (CRT), liquid crystal display (LCD), etc., and a speaker, etc.; a storage section 1308 including a hard disk, etc.; and a communication section 1309 including a network interface card such as a LAN card, modem, etc. The communication section 1309 performs communication processing via a network such as the Internet. A drive 1310 is also connected to the input / output (I / O) interface 1305 as needed. A removable medium 1311, such as a disk, optical disk, magneto-optical disk, semiconductor memory, etc., is installed on the drive 1310 as needed so that computer programs read from it can be installed into the storage section 1308 as needed.

[0330] This disclosure also provides a computer-readable storage medium, which may be included in the device / apparatus / system described in the above embodiments; or it may exist independently and not assembled into the device / apparatus / system. The computer-readable storage medium carries one or more programs that, when executed, implement the method according to the embodiments of this disclosure.

[0331] According to embodiments of this disclosure, the computer-readable storage medium may be a non-volatile computer-readable storage medium, such as including, but not limited to: portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof. In this disclosure, the computer-readable storage medium may be any tangible medium that contains or stores a program that can be used by or in conjunction with an instruction execution system, apparatus, or device. For example, according to embodiments of this disclosure, the computer-readable storage medium may include ROM 1302 and / or RAM 1303 and / or one or more memories other than ROM 1302 and RAM 1303 described above.

[0332] Embodiments of this disclosure also include a computer program product comprising a computer program containing program code for performing the methods shown in the flowchart. When the computer program product is run on a computer system, the program code is used to cause the computer system to implement the methods provided in the embodiments of this disclosure.

[0333] When the computer program is executed by the processor 1301, it performs the functions defined in the system / apparatus of this disclosure embodiments. According to embodiments of this disclosure, the systems, apparatuses, modules, units, etc., described above can be implemented by computer program modules.

[0334] In one embodiment, the computer program may rely on a tangible storage medium such as an optical storage device or a magnetic storage device. In another embodiment, the computer program may also be transmitted and distributed in the form of signals over a network medium, and may be downloaded and installed via the communication section 1309, and / or installed from the removable medium 1311. The program code contained in the computer program can be transmitted using any suitable network medium, including but not limited to: wireless, wired, etc., or any suitable combination thereof.

[0335] In such an embodiment, the computer program can be downloaded and installed from a network via the communication section 1309, and / or installed from the removable medium 1311. When the computer program is executed by the processor 1301, it performs the functions defined in the system of this disclosure embodiment. According to embodiments of this disclosure, the systems, devices, apparatuses, modules, units, etc., described above can be implemented by computer program modules.

[0336] According to embodiments of this disclosure, program code for executing the computer programs provided in embodiments of this disclosure can be written in any combination of one or more programming languages. Specifically, these computational programs can be implemented using high-level procedural and / or object-oriented programming languages, and / or assembly / machine languages. Programming languages ​​include, but are not limited to, languages ​​such as Java, C++, Python, "C", or similar programming languages. The program code can execute entirely on the user's computing device, partially on the user's device, partially on a remote computing device, or entirely on a remote computing device or server. In cases involving remote computing devices, the remote computing device can be connected to the user's computing device via any type of network, including a local area network (LAN) or a wide area network (WAN), or it can be connected to an external computing device (e.g., via the Internet using an Internet service provider).

[0337] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this disclosure. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in a block diagram or flowchart, and combinations of blocks in a block diagram or flowchart, may be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0338] Those skilled in the art will understand that the features described in the various embodiments of this disclosure can be combined and / or combined in various ways, even if such combinations or combinations are not explicitly described in this disclosure. In particular, the features described in the various embodiments of this disclosure can be combined and / or combined in various ways without departing from the spirit and teachings of this disclosure. All such combinations and / or combinations fall within the scope of this disclosure.

[0339] The embodiments of this disclosure have been described above. However, these embodiments are for illustrative purposes only and are not intended to limit the scope of this disclosure. Although various embodiments have been described above, this does not mean that the measures in the various embodiments cannot be used advantageously in combination. Various substitutions and modifications can be made by those skilled in the art without departing from the scope of this disclosure, and all such substitutions and modifications should fall within the scope of this disclosure.

Claims

1. A method for generating a panoramic depth map, comprising: For at least two image groups obtained by grouping target images from different orientations in a target scene, feature combination is performed on the target image feature body of each image group to obtain at least two target panoramic feature bodies, wherein the target images contained in each image group cover the panoramic field of view of the target scene, and the target image feature body characterizes the three-dimensional stereoscopic features of the target image; Correlation processing is performed on every two target panoramic features in at least two target panoramic feature volumes to obtain at least one target correlation volume; and A panoramic depth map for the target scene is obtained by performing panoramic depth estimation based on the initial depth map and at least one of the target related objects.

2. The method according to claim 1, wherein, Each of the target image feature bodies has its own corresponding target weight, and the step of combining the target image feature bodies of each image group to obtain at least two target panoramic feature bodies includes: Using the target weights of each target image feature body, the target image feature bodies of each image group are weighted and summed to obtain at least two target panoramic feature bodies.

3. The method according to claim 2, wherein, The target weight of each of the target image feature volumes is determined using the following operation: For each image group, the target image feature bodies in the image group are cascaded to obtain the first cascaded target image feature bodies; Multi-layer perceptual processing is performed on the target image feature body after the first concatenation to obtain the target weights of each target image feature body in the image group.

4. The method according to claim 2, wherein, The target weight of each of the target image feature volumes is determined using the following operation: For each pixel in the target image feature body, determine the distance between the pixel and the target pixel, wherein the target pixel corresponds to the camera center; The position weight for the pixel is determined based on the distance; The target weight of the target image feature body is determined based on the position weight of each pixel.

5. The method according to claim 4, wherein, Determining the position weight for the pixel based on the distance includes: If the distance is determined to be less than a preset distance threshold, the position weight of the pixel is determined to be a first value; If the distance is determined to be greater than or equal to the preset distance threshold, the position weight of the pixel is determined to be a second value, wherein the first value is different from the second value.

6. The method according to claim 2, wherein, The target weight of each of the target image feature volumes is determined using the following operation: The target image feature bodies of target images involving different orientations in the target scene are concatenated to obtain the second concatenated target image feature body. Multi-layer perceptual processing is performed on the second concatenated target image feature volume to obtain the target weight of each target image feature volume.

7. The method according to claim 1, wherein, The image group is divided using the following operation: For target images in a target scene that involve different orientations, two target images facing away from each other are divided into an image group to obtain at least two image groups, wherein the two target images facing away from each other represent that the orientations of the two target images are opposite.

8. The method according to claim 1, wherein, The step of performing correlation processing on every two target panoramic feature bodies from at least two target panoramic feature bodies to obtain at least one target correlation body includes: The inner product of each pair of target panoramic feature bodies in at least two target panoramic feature bodies is calculated to obtain at least one target related body.

9. The method according to claim 1, wherein, The step of performing panoramic depth estimation based on the initial depth map and at least one of the target related objects to obtain a target panoramic depth map for the target scene includes: A panoramic depth estimation is performed based on an initial depth map, a preset target context feature body, and at least one of the target related features to obtain a target panoramic depth map for the target scene, wherein the target context feature body is determined based on at least one of the target panoramic features.

10. The method according to claim 9, wherein, The step of performing panoramic depth estimation based on the initial depth map, a preset target context feature body, and at least one target related body to obtain a target panoramic depth map for the target scene includes: Using the initial depth map as the current estimated depth map, panoramic depth estimation is performed based on the current estimated depth map, the target context feature body, and at least one target related body to obtain the depth estimation increment; The current depth estimation map is updated based on the depth estimation increment to obtain the updated estimated depth map; Using the updated estimated depth as the current estimated depth map, the above operation is repeated until the number of iterations reaches the first preset iteration threshold, thus obtaining the target panoramic depth map for the target scene.

11. The method according to claim 10, wherein, The panoramic depth estimation based on the current estimated depth map, the target context feature body, and at least one target related body, to obtain the depth estimation increment, includes: The target context feature body and the at least one target related body are sampled using the current estimated depth map and the preset sampling neighbor value, respectively, to obtain the current context feature map and the current related feature map; The current context feature map and the current relevant feature map are used to perform panoramic depth estimation to obtain the depth estimation increment.

12. The method according to claim 1, further comprising: For each target image in the target scene involving different orientations, feature extraction is performed on the target image to obtain a target feature map; The target feature map is subjected to spherical scanning processing to obtain the target image feature volume.

13. The method according to claim 1, wherein, The target scene includes target images from different directions, including target images from at least four directions.

14. The method according to any one of claims 1 to 13, wherein, The target images in the target scene, which involve different orientations, are acquired by fisheye lenses in the target scene, which involve different orientations.

15. The method according to claim 1, further comprising: Obstacles are identified based on the panoramic depth map, enabling the drone to perform obstacle avoidance maneuvers accordingly.

16. A method for training a depth estimation model, comprising: For at least two sample image groups obtained by grouping sample images from different orientations in a sample scene based on training samples, feature combination is performed on the sample image feature volume of each sample image group to obtain at least two sample panoramic feature volumes, wherein the sample images contained in each sample image group cover the panoramic field of view of the sample scene, and the sample image feature volume characterizes the three-dimensional stereo features of the sample image. Correlation processing is performed on every two panoramic features in at least two sample panoramic feature bodies to obtain at least one sample correlation body; and Based on the initial depth map and at least one of the sample related objects, a panoramic depth estimation is performed to obtain a predicted panoramic depth map for the sample scene. The loss information of the depth estimation model is determined based on the predicted panoramic depth map, and the network parameters of the depth estimation model are iteratively adjusted based on the loss information until the loss information meets the iteration stopping condition. The network parameters obtained when the iteration stopping condition is met are used as the trained depth estimation model.

17. The method according to claim 16, wherein, Each of the sample image feature bodies has its own sample weight, and the feature combination of each of the sample image feature bodies to obtain at least two sample panoramic feature bodies includes: Using the sample weights of each of the sample image feature bodies, the sample image feature bodies of each sample image group are weighted and summed to obtain at least two sample panoramic feature bodies.

18. The method according to claim 17, wherein, Each of the sample image feature bodies has its own sample weight, which is determined by the following operation: For each group of sample images, the sample image feature bodies in the group of sample images are cascaded to obtain the first cascaded sample image feature bodies. Multi-layer perceptron processing is performed on the first concatenated sample image feature volume to obtain the sample weights of each sample image feature volume in the sample image group.

19. The method according to claim 17, wherein, Each of the sample image feature bodies has its own sample weight, which is determined by the following operation: For each sample pixel in the feature body of the sample image, determine the distance between the sample pixel and the target sample pixel; The positional weights for the sample pixels are determined based on the distance. The sample weights of the sample image feature bodies are determined based on the positional weights of each sample pixel.

20. The method according to claim 19, wherein, Determining the position weight for the sample pixel based on the distance includes: If the distance is determined to be less than a preset distance threshold, the position weight of the sample pixel is determined to be a first value; If the distance is determined to be greater than or equal to the preset distance threshold, the position weight of the sample pixel is determined to be a second value, wherein the first value is different from the second value.

21. The method according to claim 17, wherein, Each of the sample image feature bodies has its own sample weight, which is determined by the following operation: The sample image feature bodies of sample images involving different orientations in the sample scene are concatenated to obtain the second concatenated sample image feature body. Multi-layer perceptual processing is performed on the second concatenated sample image feature volume to obtain the sample weight of each sample image feature.

22. The method according to claim 16, wherein, The sample image group is divided using the following operation: For sample images involving different orientations in a sample scene, two sample images facing away from each other are divided into a sample image group to obtain at least two sample image groups, wherein the two sample images facing away from each other represent that the orientations of the two sample images are opposite.

23. The method according to claim 16, wherein, The step of performing correlation processing on every two panoramic feature bodies from at least two sample panoramic feature bodies to obtain at least one sample correlation body includes: The inner product is calculated for every two panoramic feature bodies in at least two of the sample panoramic feature bodies to obtain at least one sample correlation body.

24. The method of claim 16, wherein, The step of performing panoramic depth estimation based on the initial depth map and at least one of the sample related objects to obtain a sample panoramic depth map for the sample scene includes: A panoramic depth estimation is performed based on an initial depth map, a preset sample context feature body, and at least one of the sample related bodies to obtain a sample panoramic depth map for the sample scene, wherein the sample context feature body is determined based on at least one of the sample panoramic feature bodies.

25. The method according to claim 24, wherein, The process of performing panoramic depth estimation based on the initial depth map, preset sample context features, and at least one of the sample related objects to obtain a sample panoramic depth map for the sample scene includes: Using the initial depth map as the current sample estimated depth map, panoramic depth estimation is performed based on the current sample estimated depth map, the sample context feature body, and at least one of the sample related bodies to obtain the sample depth estimation increment; The current sample depth estimation map is updated based on the sample depth estimation increment to obtain the updated sample depth estimation map; Using the updated sample estimated depth as the current sample estimated depth map, the above operation is repeated until the number of iterations reaches the second preset iteration threshold, thus obtaining a sample panoramic depth map for the sample scene.

26. The method according to claim 25, wherein, The panoramic depth estimation based on the current estimated depth map, the sample context feature body, and at least one of the sample related bodies, to obtain the sample depth estimation increment, includes: The current sample context feature map and the at least one sample correlation feature map are obtained by sampling the current sample context feature map and the current sample correlation feature map using the current sample estimated depth map and the preset sampling neighbor value, respectively. The current sample context feature map and the current sample related feature map are used to perform panoramic depth estimation to obtain the sample depth estimation increment.

27. The method of claim 16, further comprising: For each sample image in the sample scene involving different orientations, feature extraction is performed on the sample image to obtain a sample feature map; The sample feature map is subjected to spherical scanning processing to obtain the sample image feature volume.

28. The method according to claim 16, wherein, The sample scene involves sample images from different orientations, including sample images from at least four orientations.

29. The method according to any one of claims 16 to 28, wherein, The sample images in the sample scene, which involve different orientations, were acquired by fisheye lenses in the sample scene, which involve different orientations.

30. An electronic device comprising: One or more processors; Memory, used to store one or more programs. Wherein, when the one or more programs are executed by the one or more processors, the one or more processors implement the method of any one of claims 1 to 29.

31. A drone, comprising the electronic device according to claim 30.

Images