Structured light-based monocular depth recovery method and apparatus, and electronic device

Abstract

The present disclosure provides a structured light-based monocular depth recovery method and apparatus, and an electronic device. The method comprises: using a monocular acquisition apparatus to capture multiple target scene images of a target scene; on the basis of a system calibration parameter, constructing acquisition rays corresponding to M target pixel positions in each target scene image to obtain M target acquisition rays; on the basis of the system calibration parameter, a preset projection image corresponding to each target scene image, and K three-dimensional points on each target acquisition ray, obtaining K reference gray-scale values; on the basis of a preset voxel grid and the K three-dimensional points, obtaining K volume densities; on the basis of the K reference gray-scale values and K volume densities corresponding to each target pixel position, obtaining rendering gray-scale values; and on the basis of M rendering gray-scale values and M scene image gray-scale values respectively corresponding to the multiple target scene images, performing iterative optimization on the preset voxel grid, and performing depth recovery on the target scene on the basis of the optimized preset voxel grid.

Description

Structured light-based monocular depth recovery method, device, and electronic equipment Technical Field

[0001] This disclosure relates to the field of computer vision technology, and more specifically, to a monocular depth recovery method, apparatus, and electronic device based on structured light. Background Technology

[0002] Depth information plays a crucial role in understanding the positional relationships between objects within a 3D scene. For example, for robots or autonomous vehicles within a 3D scene, depth information allows them to determine the distances of surrounding objects, helping them avoid obstacles and adjust their next actions accordingly.

[0003] With the development of various structured light technologies in depth restoration, structured light-based depth restoration systems have become a powerful solution for measuring depth information in outdoor scenes. However, traditional structured light depth restoration methods rely on the accuracy of the matching algorithm between the projected pattern and the captured image. Any errors in the matching process, such as blurring or occlusion, will introduce significant errors into the final depth map. Therefore, how to significantly improve the accuracy of existing structured light-based depth restoration algorithms while maintaining high computational speed is a pressing problem that needs to be solved. Summary of the Invention

[0004] In view of this, this disclosure provides a method, apparatus and electronic device for monocular depth recovery based on structured light.

[0005] According to one aspect of this disclosure, a monocular depth recovery method based on structured light is provided, comprising: acquiring a target scene using a monocular acquisition device to obtain multiple target scene images, wherein each target scene image includes a preset projection image projected onto the target scene by a projector, and the preset projection images included in the multiple target scene images are different from each other; based on the multiple target scene images, iteratively performing the following operations for a preset number of rounds, wherein in the i-th round, for each target scene image, based on system calibration parameters, acquisition rays corresponding to M target pixel positions in the target scene image are constructed to obtain M target acquisition rays, wherein M is greater than or equal to 1 and less than or equal to the maximum resolution of the monocular acquisition device, and i is a positive integer; and based on the system calibration parameters, the preset projection images corresponding to the target scene images, and each target ray... K 3D points on the ray are acquired to obtain K reference grayscale values, where K is a positive integer. Based on the preset voxel grid corresponding to the target scene and the K 3D points, K voxel densities are obtained. Based on the K reference grayscale values ​​and K voxel densities corresponding to each target pixel position, the rendered grayscale value corresponding to that target pixel position is obtained. Based on the M rendered grayscale values ​​corresponding to each of the multiple target scene images and the M scene image grayscale values, the preset voxel grid is iteratively optimized. If i is less than a preset number of iterations, i is incremented, and the operation of constructing the target acquisition ray based on system calibration parameters is returned. The scene image grayscale value represents the grayscale value at the target pixel position in the target scene image. If i is greater than or equal to the preset number of iterations, depth recovery of the target scene is performed based on the optimized preset voxel grid.

[0006] For example, obtaining the rendering grayscale value corresponding to the target pixel position based on the K reference grayscale values ​​and K volume densities corresponding to each target pixel position includes: determining the k-1 target 3D points arranged before the k-th 3D point according to the arrangement order of the K 3D points along the ray projection direction of the target acquisition light, where k is an integer greater than 1 and less than or equal to K; determining the transmittance corresponding to the k-th 3D point according to the volume densities corresponding to the k-1 target 3D points respectively; and obtaining the rendering grayscale value by weighted summation of the transmittance corresponding to the K 3D points and the K reference grayscale values ​​respectively.

[0007] For example, obtaining K reference grayscale values ​​based on the system calibration parameters, the preset projection image corresponding to the target scene image, and K three-dimensional points on each target acquisition ray includes: for each three-dimensional point, based on the system calibration parameters, projecting the three-dimensional point onto the first projection plane of the projector to obtain the projection pixel position corresponding to the three-dimensional point, wherein the first projection plane is the image plane of the projector where the preset projection image is located; determining multiple neighboring pixel positions in the preset projection image that are adjacent to the projection pixel position; interpolating the grayscale values ​​at the multiple neighboring pixel positions to obtain the initial grayscale value at the projection pixel position; and determining the reference grayscale value corresponding to the three-dimensional point based on the multiple target scene images and the initial grayscale value.

[0008] For example, determining the reference gray value corresponding to the three-dimensional point based on the multiple target scene images and the initial gray value includes: determining the minimum gray value and edge contrast at the target pixel position corresponding to the three-dimensional point based on the multiple target scene images; and determining the reference gray value corresponding to the three-dimensional point based on the initial gray value, the minimum gray value, and the edge contrast.

[0009] For example, obtaining the K volume density based on the preset voxel grid corresponding to the target scene and the K three-dimensional points includes: for each three-dimensional point, determining multiple grid vertices in the preset voxel grid adjacent to the three-dimensional point; and performing trilinear interpolation on the volume density at the multiple grid vertices to obtain the volume density corresponding to the three-dimensional point.

[0010] For example, the aforementioned depth restoration of the target scene based on the optimized preset voxel grid includes: constructing acquisition rays corresponding to each pixel position on the second projection plane of the monocular acquisition device based on the system calibration parameters to obtain multiple target reconstruction rays, wherein the second projection acquisition plane represents the image plane of the monocular acquisition device where the target scene image is located; obtaining multiple reconstructed volume densities based on the multiple target reconstruction rays and the optimized preset voxel grid; and performing depth restoration of the target scene according to the multiple reconstructed volume densities and the preset volume rendering formula.

[0011] For example, the iterative optimization of the preset voxel grid based on the M rendered grayscale values ​​and M scene image grayscale values ​​corresponding to each of the multiple target scene images includes: for each target scene image, calculating the difference between the rendered grayscale value corresponding to each target pixel in the target scene image and the scene image grayscale value to obtain the color deviation corresponding to each target pixel; constructing an image loss function based on the M color deviations corresponding one-to-one with the M target pixels in the target scene image; and optimizing the preset voxel grid based on the image loss function.

[0012] For example, each target scene image includes multiple first patterns and multiple second patterns, wherein the multiple first patterns and the multiple second patterns are of the same size and are all square, the pixel value at each pixel position of the multiple first patterns is a first gray value, and the pixel value at each pixel position of the multiple second patterns is a second gray value.

[0013] According to another aspect of this disclosure, a structured light-based monocular depth recovery device is provided, comprising:

[0014] The first acquisition module is used to acquire the target scene using a monocular acquisition device to obtain multiple target scene images. Each target scene image includes a preset projection image projected onto the target scene by a projector, and the preset projection images included in the multiple target scene images are different from each other.

[0015] The second obtaining module is used to iteratively perform the following operations for a preset number of rounds based on the above multiple target scene images. In the i-th round, for each target scene image, based on the system calibration parameters, the acquisition rays corresponding to the M target pixel positions in the above target scene image are constructed to obtain M target acquisition rays, where M is greater than or equal to 1 and less than or equal to the maximum resolution of the above monocular acquisition device, and i is a positive integer.

[0016] The third module is used to obtain K reference grayscale values ​​based on the above system calibration parameters, the preset projection image corresponding to the above target scene image, and K three-dimensional points on each target acquisition ray, where K is a positive integer;

[0017] The fourth module is used to obtain the K body density based on the preset voxel mesh corresponding to the target scene and the K three-dimensional points mentioned above;

[0018] The fifth module is used to obtain the rendering grayscale value corresponding to the target pixel position based on the above K reference grayscale values ​​and the above K individual densities corresponding to each target pixel position.

[0019] The sixth module is used to iteratively optimize the preset voxel grid based on the M rendered gray values ​​and M scene image gray values ​​corresponding to each of the multiple target scene images, and increment i when i is less than the preset number of rounds, and return the operation of constructing target acquisition rays based on system calibration parameters, wherein the scene image gray values ​​represent the gray values ​​at the target pixel positions in the target scene images.

[0020] The reconstruction module is used to perform depth restoration of the target scene based on the optimized preset voxel mesh when i is greater than or equal to the preset number of rounds.

[0021] According to another aspect of this disclosure, an electronic device is provided, comprising: one or more processors; and a memory for storing one or more computer programs, characterized in that the one or more processors execute the one or more computer programs to implement the steps of the method described above.

[0022] According to embodiments of this disclosure, after acquiring multiple target scene images using a monocular acquisition device, each target scene image includes a preset projection image projected onto the target scene by a projector. Then, based on the multiple target scene images, the following operations are iteratively performed for a preset number of rounds: In the i-th round, for each target scene image, based on system calibration parameters, acquisition rays corresponding to M target pixel positions in the target scene image are constructed to obtain M target acquisition rays. Based on the system calibration parameters, the preset projection image corresponding to the target scene image, and K three-dimensional points on each target acquisition ray, K reference grayscale values ​​are obtained. Based on the preset voxel grid corresponding to the target scene and the K three-dimensional points, K voxel densities are obtained. Based on the K reference grayscale values ​​and K voxel densities corresponding to each target pixel position, the values ​​corresponding to the target pixel position are obtained. The method involves rendering grayscale values ​​and iteratively optimizing a preset voxel mesh based on M rendered grayscale values ​​corresponding to multiple target scene images and M scene image grayscale values. If i is less than a preset number of iterations, i is incremented, and the operation of constructing target acquisition rays based on system calibration parameters is returned. If i is greater than or equal to the preset number of iterations, depth recovery of the target scene is performed based on the optimized preset voxel mesh. This technique utilizes grayscale information from preset projected images as a known grayscale field to optimize scene geometry independently, accelerating convergence and improving depth estimation accuracy. Furthermore, since the depth recovery process in volume rendering does not rely on any image matching algorithm, it fundamentally overcomes the shortcomings of traditional structured light depth recovery methods in handling edges and occluded areas. Therefore, it achieves more accurate depth recovery while maintaining high computational speed, demonstrating significant practical potential. Attached Figure Description

[0023] The above and other objects, features and advantages of this disclosure will become clearer from the following description of embodiments with reference to the accompanying drawings, in which:

[0024] Figure 1 schematically illustrates a flowchart of a monocular depth recovery method based on structured light according to an embodiment of the present disclosure;

[0025] Figure 2 schematically illustrates an exemplary system architecture to which a structured light-based monocular depth recovery method can be applied according to embodiments of the present disclosure;

[0026] Figure 3 schematically illustrates a structural block diagram of a structured light-based monocular depth recovery device according to an embodiment of the present disclosure; and

[0027] Figure 4 schematically illustrates a block diagram of an electronic device suitable for implementing the structured light-based monocular depth recovery method described above, according to an embodiment of the present disclosure. Detailed Implementation

[0028] The embodiments of the present disclosure will now be described with reference to the accompanying drawings. However, it should be understood that these descriptions are exemplary only and are not intended to limit the scope of the disclosure. In the following detailed description, numerous specific details are set forth to provide a thorough understanding of the embodiments of the present disclosure for ease of explanation. However, it will be apparent that one or more embodiments may be practiced without these specific details. Furthermore, descriptions of well-known structures and techniques are omitted in the following description to avoid unnecessarily obscuring the concepts of the present disclosure.

[0029] The terminology used herein is for the purpose of describing particular embodiments only and is not intended to limit this disclosure. The terms “comprising,” “including,” etc., as used herein indicate the presence of the stated features, steps, operations, and / or components, but do not exclude the presence or addition of one or more other features, steps, operations, or components.

[0030] All terms used herein (including technical and scientific terms) have the meanings commonly understood by those skilled in the art, unless otherwise defined. It should be noted that the terms used herein are to be interpreted in a manner consistent with the context of this specification, and not in an idealized or overly rigid way.

[0031] When using expressions such as "at least one of A, B, and C", they should generally be interpreted in accordance with the meaning that is commonly understood by a person skilled in the art (e.g., "a system having at least one of A, B, and C" should include, but is not limited to, a system having A alone, a system having B alone, a system having C alone, a system having A and B, a system having A and C, a system having B and C, and / or a system having A, B, and C, etc.).

[0032] In the embodiments disclosed herein, the collection, updating, analysis, processing, use, transmission, provision, disclosure, and storage of data (e.g., including but not limited to user personal information) comply with relevant laws and regulations, are used for legitimate purposes, and do not violate public order and good morals. In particular, necessary measures have been taken to prevent unauthorized access to user personal information data and to safeguard user personal information security and network security.

[0033] Typically, a structured light-based monocular depth recovery system consists of a camera and a projector, with both devices' intrinsic and extrinsic parameters pre-calibrated. The system projects random or artificially designed patterns into three-dimensional space and extracts depth information by analyzing the deformation of these patterns in the captured image. Classical structured light algorithms aim to establish robust correspondences between multiple projected patterns.

[0034] Depth recovery in structured light systems typically involves a trade-off between scanning accuracy and the number of projected patterns or captured images. Increasing the number of captured images can improve the accuracy of correspondence matching. However, more projected patterns also lead to longer acquisition times, limiting the applicability of structured light systems in general scenarios, especially those with moving objects or short exposure times. To address this issue, techniques can be employed to embed richer information into a limited set of patterns. These methods use complex patterns to encode temporal or spatial features to reduce matching uncertainty. By decoding these features in the captured images, the structured light system can determine the correspondences required for depth estimation. Despite these advances, designing features that generate accurate, high-density depth maps while being robust to environmental influences remains a significant challenge.

[0035] Traditional structured light depth restoration methods rely on the accuracy of the matching algorithm between the projected pattern and the captured image. Any errors in the matching process, such as blurring or occlusion, can introduce significant errors into the final depth map. Related technologies have addressed this uncertainty by utilizing neural networks used in deep learning. However, most of these methods directly obtain the predicted depth by training a general model. While these models can generate dense depth maps, they require large training datasets, and the quality of the data in the training dataset significantly impacts network performance. In structured light-based depth restoration systems, constructing such training datasets is challenging due to the diversity of equipment and pattern configurations. Furthermore, training a general model from the acquired training dataset requires substantial time. Consequently, both traditional structured light depth restoration algorithms and deep learning-based depth restoration algorithms suffer from slow computation and low depth restoration accuracy due to inaccurate image matching.

[0036] Therefore, how to significantly improve the accuracy of existing structured light-based depth recovery algorithms while maintaining high computational speed is an urgent problem to be solved.

[0037] In view of this, embodiments of the present disclosure provide a monocular depth recovery method, apparatus, and electronic device based on structured light, which can be applied to the field of computer vision technology.

[0038] This disclosure provides a monocular depth recovery method based on structured light, comprising: acquiring multiple target scene images using a monocular acquisition device, wherein each target scene image includes a preset projection image projected onto the target scene by a projector, and the preset projection images included in the multiple target scene images are different from each other; based on the multiple target scene images, iteratively performing the following operations for a preset number of rounds, in the i-th round, for each target scene image, based on system calibration parameters, constructing acquisition rays corresponding to M target pixel positions in the target scene image respectively, to obtain M target acquisition rays, wherein M is greater than or equal to 1 and less than or equal to the maximum resolution of the monocular acquisition device, and i is a positive integer; according to the system calibration parameters, the preset projection image corresponding to the target scene image and each entry The system collects K 3D points on the ray to obtain K reference grayscale values, where K is a positive integer. Based on the preset voxel grid corresponding to the target scene and the K 3D points, it obtains K voxel densities. Based on the K reference grayscale values ​​and K voxel densities corresponding to each target pixel position, it obtains the rendered grayscale value corresponding to the target pixel position. Based on the M rendered grayscale values ​​corresponding to multiple target scene images and the M scene image grayscale values, the preset voxel grid is iteratively optimized. If i is less than the preset number of iterations, i is incremented, and the system returns to the operation of constructing the target ray based on the system calibration parameters. Here, the scene image grayscale value represents the grayscale value at the target pixel position in the target scene image. If i is greater than or equal to the preset number of iterations, depth recovery of the target scene is performed based on the optimized preset voxel grid.

[0039] Figure 1 schematically illustrates a flowchart of a structured light-based monocular depth recovery method according to an embodiment of the present disclosure.

[0040] As shown in Figure 1, the monocular depth recovery method based on structured light includes operations S101 to S107.

[0041] In operation S101, the target scene is acquired using a monocular acquisition device, resulting in multiple images of the target scene. Each target scene image includes a preset projection image projected onto the target scene by a projector, and the preset projection images included in the multiple target scene images are different from each other.

[0042] For example, before performing operation S101, a structured light-based monocular depth recovery system is first constructed. This system can include a monocular acquisition device and a projector. The monocular acquisition device can be a monocular structured light camera. Both the monocular structured light camera and the projector have been pre-calibrated and calibrated.

[0043] For example, in the process of using a monocular structured light camera to capture an image of the target scene to be measured, the pattern set of the preset projected image projected by the projector onto the target scene can be a black and white coded pattern composed of unit squares of a fixed scale, with the color of each square randomly set to black or white.

[0044] For example, multiple preset projection images, each corresponding one-to-one with multiple target scene images, have the same size. Each preset projection image may include a black-and-white coded pattern composed of unit squares of a fixed scale.

[0045] For example, the number of multiple target scene images can be 6. The number of multiple preset projection images is also 6. Two sets of patterns can be created using squares with unit lengths of 20 pixels, 10 pixels, and 5 pixels respectively, resulting in 6 preset projection images.

[0046] According to embodiments of this disclosure, the specific design of the preset projection image does not significantly affect the final performance of the structured light-based monocular depth recovery method provided in these embodiments. Therefore, any projection pattern that satisfies the above paradigm can be used in image capture of the target scene to be measured in these embodiments.

[0047] In operation S102, based on multiple target scene images, the following operations are iteratively performed for a preset number of rounds. In the i-th round, for each target scene image, based on the system calibration parameters, acquisition rays corresponding to the positions of M target pixels in the target scene image are constructed to obtain M target acquisition rays. Here, M is greater than or equal to 1 and less than or equal to the maximum resolution of the monocular acquisition device, and i is a positive integer.

[0048] According to embodiments of this disclosure, M can be selected according to actual conditions, and is not limited herein. For example, M can be 3000, 4096, 4100, or 4500, etc.

[0049] According to embodiments of this disclosure, the preset number of rounds can be selected according to actual conditions and is not limited thereto. For example, the preset number of rounds can be 1000, 2000, 3500, or 4000, etc.

[0050] According to embodiments of this disclosure, in the same round, the positions of M target pixels corresponding to each of multiple target scene images can be completely identical, partially identical, or completely different. In different rounds, the positions of M target pixels corresponding to the same target scene image can be completely identical, partially identical, or completely different.

[0051] For example, M pixel positions on the second projection plane of the monocular acquisition device can be selected first, and these M pixel positions can be defined as M target pixel positions in the target scene image. Then, based on the known system calibration parameters of the structured light-based monocular depth recovery system, the positions and directions of the light rays in three-dimensional space corresponding to these target pixel positions can be calculated. The target acquisition light rays are then represented using the positions and directions of the light rays in three-dimensional space corresponding to the target pixel positions. Here, the second projection acquisition plane represents the image plane of the monocular acquisition device where the target scene image is located.

[0052] In operation S103, K reference grayscale values ​​are obtained based on the system calibration parameters, the preset projection image corresponding to the target scene image, and K three-dimensional points on each target acquisition ray. Here, K is a positive integer.

[0053] According to embodiments of this disclosure, K can be selected according to actual conditions and is not limited thereto. For example, K can be 100, 128, 150, or 300, etc.

[0054] For example, a series of three-dimensional points in three-dimensional space can be sampled at different distances along each target acquisition ray, which can then be used in the subsequent volume rendering process.

[0055] For example, based on the external parameters characterizing the monocular acquisition device and the projector in the system calibration parameters, K 3D points can be projected onto the image plane of the projector. The grayscale values ​​corresponding to the K 3D points are obtained by querying a preset projection image based on the position of the projected pixels, resulting in K reference grayscale values ​​for subsequent volume rendering.

[0056] According to embodiments of this disclosure, by using a technique of obtaining K reference grayscale values ​​based on system calibration parameters, a preset projection image corresponding to the target scene image, and K three-dimensional points on each target acquisition ray, the grayscale values ​​of these sampled three-dimensional points can be interpolated using the known pixel grayscale values ​​of the preset projection image for subsequent volume rendering.

[0057] In operation S104, the K-unit density is obtained based on the preset voxel grid and K three-dimensional points corresponding to the target scene.

[0058] According to embodiments of this disclosure, before performing operation S104, a preset voxel grid for storing volume density can be constructed. The preset voxel grid is a dense three-dimensional grid. The resolution of the preset voxel grid can be 256×256×256.

[0059] For example, the positions of K 3D points can be input into a preset voxel mesh to be optimized, and the corresponding volume density can be interpolated to obtain the K volume densities. The K volume densities can then be used in the subsequent volume rendering process.

[0060] In operation S105, the rendering grayscale value corresponding to the target pixel position is obtained based on the K reference grayscale values ​​and K individual densities corresponding to each target pixel position.

[0061] For example, volume rendering is performed on the target acquisition rays corresponding to each target pixel position based on K reference gray values ​​and K individual densities. Based on the volume-rendered target acquisition rays, the rendered gray value corresponding to the target pixel position is obtained.

[0062] For example, in the process of calculating the rendering grayscale value corresponding to the target pixel position, the integral in the rendering equation can be discretized and transformed into a sum based on the transmittance corresponding to the sampled 3D point and the baseline grayscale. The transmittance is calculated from the volume density corresponding to the sampled 3D point and the volume densities of all sampled 3D points preceding the current 3D point, arranged according to the direction of the target light rays.

[0063] In operation S106, based on the M rendered grayscale values ​​corresponding to each of the multiple target scene images and the M scene image grayscale values, the preset voxel mesh is iteratively optimized. If i is less than the preset number of iterations, i is incremented, and the operation of constructing the target acquisition ray based on the system calibration parameters is returned. Here, the scene image grayscale value represents the grayscale value at the target pixel location in the target scene image.

[0064] For example, in the first round, the preset voxel mesh can be optimized using the M rendered grayscale values ​​and M scene image grayscale values ​​corresponding to the first target scene image among multiple target scene images. Then, the preset voxel mesh optimized based on the first target scene image can be iteratively optimized using the M rendered grayscale values ​​and M scene image grayscale values ​​corresponding to the second target scene image among multiple target scene images, and so on. The preset voxel mesh optimized based on the previous multiple target scene images can be iteratively optimized using the M rendered grayscale values ​​and M scene image grayscale values ​​corresponding to the last target scene image among multiple target scene images, resulting in the preset voxel mesh optimized in the first round. In the second round, the preset voxel mesh optimized in the first round is iteratively optimized, and so on, until the preset number of rounds is reached. The preset voxel mesh optimized in the previous preset number of rounds (-1) is iteratively optimized to obtain the final optimized preset voxel mesh.

[0065] According to embodiments of this disclosure, by using an explicit pre-defined voxel mesh to represent the target scene geometry, the speed of each iteration in the optimization process can be accelerated.

[0066] According to embodiments of this disclosure, by using projection transformation to optimize the preset voxel mesh with the grayscale information of the preset projected image as a known grayscale field, it helps to accelerate the convergence speed and improve the performance of the final target scene geometry.

[0067] According to embodiments of this disclosure, once the image matching in traditional structured light depth restoration methods is calculated, the impact of incorrect matching on the depth restoration results cannot be eliminated. In contrast, the structured light-based monocular depth restoration method provided in this disclosure allows for the gradual correction of scene geometry errors through voxel mesh optimization. Therefore, the structured light-based monocular depth restoration method provided in this disclosure overcomes the shortcomings of matching-based methods, thereby producing better results in edge or occluded areas.

[0068] In operation S107, when i is greater than or equal to the preset number of rounds, the target scene is depth restored based on the optimized preset voxel mesh.

[0069] According to embodiments of this disclosure, after acquiring multiple target scene images using a monocular acquisition device, each target scene image includes a preset projection image projected onto the target scene by a projector. Then, based on the multiple target scene images, the following operations are iteratively performed for a preset number of rounds: In the i-th round, for each target scene image, based on system calibration parameters, acquisition rays corresponding to M target pixel positions in the target scene image are constructed to obtain M target acquisition rays. Based on the system calibration parameters, the preset projection image corresponding to the target scene image, and K three-dimensional points on each target acquisition ray, K reference grayscale values ​​are obtained. Based on the preset voxel grid corresponding to the target scene and the K three-dimensional points, K voxel densities are obtained. Based on the K reference grayscale values ​​and K voxel densities corresponding to each target pixel position, the values ​​corresponding to the target pixel position are obtained. The method involves rendering grayscale values ​​and iteratively optimizing a preset voxel mesh based on M rendered grayscale values ​​corresponding to multiple target scene images and M scene image grayscale values. If i is less than a preset number of iterations, i is incremented, and the operation of constructing target acquisition rays based on system calibration parameters is returned. If i is greater than or equal to the preset number of iterations, depth recovery of the target scene is performed based on the optimized preset voxel mesh. This technique utilizes grayscale information from preset projected images as a known grayscale field to optimize scene geometry independently, accelerating convergence and improving depth estimation accuracy. Furthermore, since the depth recovery process in volume rendering does not rely on any image matching algorithm, it fundamentally overcomes the shortcomings of traditional structured light depth recovery methods in handling edges and occluded areas. Therefore, it achieves more accurate depth recovery while maintaining high computational speed, demonstrating significant practical potential.

[0070] According to embodiments of this disclosure, each target scene image includes multiple first patterns and multiple second patterns, wherein the multiple first patterns and multiple second patterns are of the same size and are all square, the pixel value at each pixel position of the multiple first patterns is a first grayscale value, and the pixel value at each pixel position of the multiple second patterns is a second grayscale value.

[0071] For example, the size of the multiple first patterns and multiple second patterns can all be 20 pixels, 10 pixels, or 5 pixels. The first grayscale value can be 0, and the second grayscale value can be 255. When the first grayscale value is 0, the first pattern is black. When the second grayscale value is 255, the second pattern is white.

[0072] According to embodiments of this disclosure, for operation S103 as shown in FIG1, K reference grayscale values ​​are obtained based on system calibration parameters, a preset projection image corresponding to the target scene image, and K three-dimensional points on each target acquisition ray. This may include the following operations:

[0073] For each 3D point, based on the system calibration parameters, the 3D point is projected onto the first projection plane of the projector to obtain the projection pixel position corresponding to the 3D point. The first projection plane is the image plane of the projector where the preset projection image is located.

[0074] Determine the positions of multiple neighboring pixels in the preset projected image;

[0075] The initial gray value at the projected pixel location is obtained by interpolation based on the gray values ​​at multiple neighboring pixel locations.

[0076] Based on multiple target scene images and initial grayscale values, determine the reference grayscale value corresponding to the 3D point.

[0077] According to embodiments of this disclosure, for each 3D point, based on system calibration parameters, the 3D point is projected onto the first projection plane of the projector to obtain the projected pixel position corresponding to the 3D point. The first projection plane is the image plane of the projector of the preset projected image. Multiple neighboring pixel positions of the neighboring projected pixel positions in the preset image are determined. Based on the gray values ​​at the multiple neighboring pixel positions, the initial gray value at the projected pixel position is obtained by interpolation. Based on multiple target scene images and the initial gray value, the reference gray value corresponding to the 3D point is determined. This realizes the direct calculation of the gray value of the sampled 3D point through the known projected preset projection image, avoiding the image feature matching process that must be used in the traditional structured light depth recovery method, and also avoiding the optimization of the color field in the traditional volume rendering-based depth recovery task.

[0078] According to embodiments of this disclosure, determining the reference grayscale value corresponding to a 3D point based on multiple target scene images and initial grayscale values ​​includes:

[0079] Based on multiple target scene images, determine the minimum gray value and edge contrast at the target pixel position corresponding to the 3D point;

[0080] The reference gray value corresponding to the 3D point is determined based on the initial gray value, the minimum gray value, and the edge contrast.

[0081] According to embodiments of this disclosure, the minimum grayscale value characterizes the background light level of the target acquisition light.

[0082] For example, the minimum pixel value at the target pixel position r corresponding to the 3D point can be obtained by finding the minimum grayscale value at the target pixel position r corresponding to the 3D point from multiple target scene images. The maximum pixel value at the target pixel position r corresponding to the 3D point can be obtained by finding the maximum grayscale value at the target pixel position r corresponding to the 3D point from multiple target scene images. Subtracting the minimum grayscale value from the maximum grayscale value yields the edge contrast at the target pixel position r corresponding to the 3D point.

[0083] For example, formula (1) can be used to determine the reference gray value corresponding to a 3D point based on the initial gray value, the minimum gray value, and the edge contrast. k =B(r)+F(r)P j (π(x k )) (1)

[0084] Among them, c k Let B(r) be the reference gray value of the k-th 3D point, B(r) be the minimum gray value at the target pixel position r corresponding to the k-th 3D point, determined based on multiple target scene images, and F(r) be the edge contrast at the target pixel position r corresponding to the k-th 3D point, determined based on multiple target scene images. j (∏(x k For the j-th preset projection image P, at the k-th 3D point x k The corresponding projected pixel position π(x) k The initial gray value at (), where j is a positive integer.

[0085] According to embodiments of this disclosure, the parameters of minimum grayscale value and edge contrast can handle occluded areas. Because in occluded areas, both edge contrast and minimum grayscale value approach zero, these areas contribute almost nothing to the volume rendering process.

[0086] According to embodiments of this disclosure, by determining the minimum grayscale value and edge contrast at the target pixel position corresponding to the three-dimensional point based on multiple target scene images, and determining the reference grayscale value corresponding to the three-dimensional point based on the initial grayscale value, the minimum grayscale value, and the edge contrast, the initial grayscale value is corrected based on the minimum grayscale value and the edge contrast, thereby obtaining the reference grayscale value of the three-dimensional point sampled for subsequent ray volume rendering process.

[0087] According to embodiments of this disclosure, for operation S104 as shown in FIG1, obtaining K voxel densities based on a preset voxel mesh corresponding to the target scene and K three-dimensional points may include the following operations:

[0088] For each 3D point, determine multiple mesh vertices in the preset voxel mesh that are adjacent to the 3D point;

[0089] Trilinear interpolation is performed on the volume density at multiple mesh vertices to obtain the volume density corresponding to the 3D point.

[0090] For example, for each 3D point, the position of the 3D point can be input into the preset voxel mesh to be optimized, and multiple mesh vertices of the neighboring 3D points can be obtained from the preset voxel mesh.

[0091] According to embodiments of this disclosure, for operation S105 as shown in FIG1, obtaining the rendering grayscale value corresponding to the target pixel position based on K reference grayscale values ​​and K individual densities corresponding to each target pixel position may include the following operations:

[0092] Based on the arrangement order of K three-dimensional points along the ray projection direction of the target acquisition light, determine the k-1 target three-dimensional points arranged before the k-th three-dimensional point, where k is an integer greater than 1 and less than or equal to K;

[0093] Based on the volume density corresponding to each of the k-1 target 3D points, determine the transmittance corresponding to the k-th 3D point;

[0094] The rendered grayscale value is obtained by weighted summation of the transmittance corresponding to K three-dimensional points and K reference grayscale values.

[0095] For example, according to formulas (2) and (3), the operation of obtaining the rendering gray value corresponding to the target pixel position can be realized based on the K reference gray values ​​and K individual densities corresponding to each target pixel position.

[0096] Where, σ k σ is the volume density corresponding to the k-th 3D point. s T is the volume density corresponding to the s-th 3D point. k δ represents the transmittance corresponding to the k-th three-dimensional point. k and δ s t represents the distance between adjacent 3D points. s+1 Let t be the 3D coordinate value of the (s+1)th 3D point. s Let be the three-dimensional coordinates of the s-th point.

[0097] According to embodiments of this disclosure, for operation S106 as shown in FIG1, iterative optimization of the preset voxel grid based on M rendering grayscale values ​​corresponding to each of the multiple target scene images and M scene image grayscale values ​​may include the following operations:

[0098] For each target scene image, the difference between the rendered grayscale value corresponding to each target pixel in the target scene image and the grayscale value of the scene image is calculated to obtain the color deviation corresponding to each target pixel;

[0099] An image loss function is constructed based on the M color deviations that correspond one-to-one with the M target pixels in the target scene image.

[0100] The preset voxel grid is optimized based on the image loss function.

[0101] For example, a volume rendering process similar to the aforementioned process can be performed on the volume density corresponding to the sampled 3D points to obtain the intersection points of the light rays and the target scene surface under the current optimization. By projecting these intersection points onto the image plane of the projector and querying the grayscale values, a rendered image based on the target scene surface is obtained. This image is then compared with the corresponding captured target scene image to construct a surface loss function.

[0102] Furthermore, based on the fact that the volume density along each acquisition ray should be a single-peak distribution, distortion loss can be constructed according to the size and volume density of the interval formed by the adjacent three-dimensional points sampled along the acquisition ray, minimizing the weighted distance between point pairs in all intervals, as well as the weighted size of each interval.

[0103] The optimization of the voxel mesh can be implemented using an open-source deep learning framework for machine learning and deep learning. The optimizer can be Adam (Adaptive Moment Estimation). The resolution of the constructed voxel mesh can be set to 256×256×256. During the optimization process, 4096 pixels can be randomly sampled in each iteration to form 4096 acquisition rays, and each acquisition ray upsamples 128 3D spatial points. The initial loss function can use only image loss and distortion loss, where the image loss has a weight of 1 and the distortion loss has a weight of 0.001. After 1000 iterations, a surface loss can be added with a weight of 1.

[0104] It should be noted that the parameter values ​​mentioned above are for illustrative purposes only and are not intended to be limiting. In practical applications, the content of the training dataset and the values ​​of each parameter can be adjusted based on existing technology.

[0105] According to embodiments of this disclosure, after the voxel mesh to be optimized is completed, for the target scene to be measured, the corresponding acquisition light rays of each pixel are constructed from the perspective of the structured light camera, the volume density is obtained by querying the optimized preset voxel mesh, and a dense depth map is calculated by the volume rendering formula, thereby completing the depth recovery.

[0106] According to embodiments of this disclosure, for operation S107 as shown in FIG1, performing depth restoration of the target scene based on the optimized preset voxel mesh may include the following operations:

[0107] Based on the system calibration parameters, the acquisition rays corresponding to each pixel position on the second projection plane of the monocular acquisition device are constructed to obtain multiple target reconstruction rays. The second projection acquisition plane represents the image plane of the monocular acquisition device where the target scene image is located.

[0108] Multiple reconstructed volume densities are obtained based on reconstructed rays from multiple targets and optimized preset voxel meshes.

[0109] Based on multiple reconstructed volume densities and preset volume rendering formulas, the target scene is deeply restored.

[0110] Figure 2 schematically illustrates an exemplary system architecture for applying a structured light-based monocular depth recovery method according to embodiments of the present disclosure.

[0111] As shown in Figure 2, the system architecture 200 includes a monocular structured light camera 210, a projector 220, and a processor.

[0112] During the process of capturing an image of the target scene to be measured using a monocular structured light camera 210, a projector 220 projects a preset projection image 201 onto the target scene. The pattern set of the preset projection image 201 can be a black and white coded pattern composed of unit squares of a fixed scale, with the color of each square randomly set to black or white. The surface of the target scene is 230.

[0113] After acquiring multiple target scene images 202 using a monocular structured light camera 210, the processor can construct M target acquisition rays Lt for each target scene image 202, based on system calibration parameters and corresponding to the positions of M target pixels in the target scene image 202. This results in M ​​target acquisition rays Lt. Based on the system calibration parameters, the preset projection image 201 corresponding to the target scene image 202, and K three-dimensional points x on each target acquisition ray Lt... k K baseline grayscale values ​​are obtained. Among them, I(r) j Let I be the j-th target scene image at point x in the three-dimensional plane. k The grayscale value at the corresponding target pixel position r. Based on the preset voxel mesh 203 and K 3D points x corresponding to the target scene. k K individual density values ​​are obtained. Based on the K baseline gray values ​​corresponding to each target pixel position and the K individual density values, the rendering gray value corresponding to the target pixel position is obtained, resulting in rendering image 204. Based on the M rendering gray values ​​corresponding to each of the multiple target scene images 202 and the M scene image gray values, the preset voxel mesh 203 is iteratively optimized. The depth of the target scene is then restored based on the optimized preset voxel mesh.

[0114] As can be seen from the structured light-based monocular depth recovery method shown in Figure 1 and the exemplary system architecture applying the structured light-based monocular depth recovery method shown in Figure 2, the structured light-based monocular depth recovery method provided in this disclosure has the following advantages compared with traditional structured light depth recovery methods and existing learning methods:

[0115] 1) Optimizing scene geometry by constructing a loss function through volume rendering can produce sharper object edges and better preserve the overall structure of objects; 2) Using color information from the preset projection image as a known color field to optimize scene geometry separately can accelerate convergence and improve depth estimation accuracy; 3) Using an explicit voxel mesh to represent scene geometry can accelerate the speed of each iteration in the optimization process, such as being 3 times faster than the SDF-based structured light depth recovery algorithm; 4) Since the proposed depth recovery process based on volume rendering does not rely on any image matching algorithm, it fundamentally overcomes the shortcomings of traditional methods in handling edges and occluded areas, resulting in more accurate depth recovery, such as 30%-60% smaller average depth prediction error than traditional methods such as GC on test data; 5) Since it does not require pre-training with a dataset and has no high requirements for the design of the preset projection image, it has good application capabilities in real-world scenes.

[0116] It should be noted that, unless it is explicitly stated that there is a sequential order of execution between different operations, or that there is a sequential order of execution between different operations in terms of technical implementation, the execution order between multiple operations may not be significant, and multiple operations may be executed simultaneously.

[0117] Based on the above-described structured light-based monocular depth recovery method, this disclosure provides a structured light-based monocular depth recovery device.

[0118] Figure 3 schematically illustrates a structural block diagram of a monocular depth recovery device based on structured light according to an embodiment of the present disclosure.

[0119] As shown in Figure 3, the monocular depth recovery device 300 based on structured light includes a first acquisition module 310, a second acquisition module 320, a third acquisition module 330, a fourth acquisition module 340, a fifth acquisition module 350, a sixth acquisition module 360, and a reconstruction module 370.

[0120] The first acquisition module 310 is used to acquire multiple target scene images using a monocular acquisition device. Each target scene image includes a preset projection image projected onto the target scene by a projector, and the preset projection images included in the multiple target scene images are different from each other. In one embodiment, the first acquisition module 310 can be used to perform the operation S101 described above, which will not be repeated here.

[0121] The second obtaining module 320 is used to iteratively perform the following operation for a preset number of rounds based on multiple target scene images. In the i-th round, for each target scene image, based on the system calibration parameters, it constructs acquisition rays corresponding to the positions of M target pixels in the target scene image, thereby obtaining M target acquisition rays, where M is greater than or equal to 1 and less than or equal to the maximum resolution of the monocular acquisition device, and i is a positive integer. In one embodiment, the second obtaining module 320 can be used to execute the operation S102 described above, which will not be repeated here.

[0122] The third obtaining module 330 is used to obtain K reference grayscale values ​​based on system calibration parameters, a preset projection image corresponding to the target scene image, and K three-dimensional points on each target acquisition ray, where K is a positive integer. In one embodiment, the third obtaining module 330 can be used to perform the operation S103 described above, which will not be repeated here.

[0123] The fourth obtaining module 340 is used to obtain the K-unit density based on the preset voxel mesh corresponding to the target scene and K three-dimensional points. In one embodiment, the fourth obtaining module 340 can be used to perform the operation S104 described above, which will not be repeated here.

[0124] The fifth obtaining module 350 is used to obtain the rendering grayscale value corresponding to the target pixel position based on the K reference grayscale values ​​and K individual density values ​​corresponding to each target pixel position. In one embodiment, the fifth obtaining module 350 can be used to perform the operation S105 described above, which will not be repeated here.

[0125] The sixth obtaining module 360 ​​is used to iteratively optimize a preset voxel mesh based on the M rendered grayscale values ​​corresponding to each of the multiple target scene images and the M scene image grayscale values. If i is less than a preset number of rounds, i is incremented, and the operation of constructing target acquisition rays based on system calibration parameters is returned. Here, the scene image grayscale value represents the grayscale value at the target pixel location in the target scene image. In one embodiment, the fifth obtaining module 360 ​​can be used to execute the operation S106 described above, which will not be repeated here.

[0126] The reconstruction module 370 is used to perform depth restoration of the target scene based on the optimized preset voxel mesh when i is greater than or equal to a preset number of rounds. In one embodiment, the reconstruction module 370 can be used to perform the operation S107 described above, which will not be repeated here.

[0127] Any one or more of the modules, submodules, units, and subunits according to embodiments of the present disclosure, or at least part of the functions of any one or more of them, can be implemented in one module. Any one or more of the modules, submodules, units, and subunits according to embodiments of the present disclosure can be implemented by dividing them into multiple modules. Any one or more of the modules, submodules, units, and subunits according to embodiments of the present disclosure can be at least partially implemented as hardware circuitry, such as Field Programmable Gate Arrays (FPGAs), Programmable Logic Arrays (PLAs), Systems-on-Chip, Systems-on-Substrate, Systems-on-Package, Application-Specific Integrated Circuits (ASICs), or implemented in hardware or firmware by any other reasonable means of integrating or packaging circuitry, or implemented in software, hardware, or firmware, or in any suitable combination of any of these three implementation methods. Alternatively, one or more of the modules, submodules, units, and subunits according to embodiments of the present disclosure can be at least partially implemented as computer program modules, which, when run, can perform corresponding functions.

[0128] For example, any and more of the first obtaining module 310, the second obtaining module 320, the third obtaining module 330, the fourth obtaining module 340, the fifth obtaining module 350, the sixth obtaining module 360, and the reconstruction module 370 can be combined into one module / unit / subunit, or any one of these modules / units / subunits can be split into multiple modules / units / subunits. Alternatively, at least some of the functionality of one or more of these modules / units / subunits can be combined with at least some of the functionality of other modules / units / subunits and implemented in one module / unit / subunit. According to embodiments of this disclosure, at least one of the first obtaining module 310, the second obtaining module 320, the third obtaining module 330, the fourth obtaining module 340, the fifth obtaining module 350, the sixth obtaining module 360, and the reconstruction module 370 can be at least partially implemented as hardware circuitry, such as a field-programmable gate array (FPGA), a programmable logic array (PLA), a system-on-a-chip, a system-on-a-substrate, a system-on-package, an application-specific integrated circuit (ASIC), or any other reasonable method of integrating or packaging the circuitry, or implemented in software, hardware, or firmware, or in any suitable combination of any of these three implementation methods. Alternatively, at least one of the first obtaining module 310, the second obtaining module 320, the third obtaining module 330, the fourth obtaining module 340, the fifth obtaining module 350, the sixth obtaining module 360, and the reconstruction module 370 can be at least partially implemented as a computer program module, which, when run, can perform corresponding functions.

[0129] It should be noted that the structured light-based monocular depth recovery device part in the embodiments of this disclosure corresponds to the structured light-based monocular depth recovery method part in the embodiments of this disclosure. For a specific description of the structured light-based monocular depth recovery device part, please refer to the structured light-based monocular depth recovery method part, which will not be repeated here.

[0130] Figure 4 schematically illustrates a block diagram of an electronic device suitable for implementing the structured light-based monocular depth recovery method described above, according to an embodiment of the present disclosure. The electronic device shown in Figure 4 is merely an example and should not be construed as limiting the functionality and scope of the embodiments of the present disclosure.

[0131] As shown in FIG. 4, an electronic device 400 according to an embodiment of the present disclosure includes a processor 401, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 402 or a program loaded from a storage portion 408 into a random access memory (RAM) 403. The processor 401 may include, for example, a general-purpose microprocessor (e.g., a CPU), an instruction set processor and / or an associated chipset and / or a special-purpose microprocessor (e.g., an application-specific integrated circuit (ASIC)), etc. The processor 401 may also include onboard memory for caching purposes. The processor 401 may include a single processing unit or multiple processing units for performing different actions of the method flow according to an embodiment of the present disclosure.

[0132] RAM 403 stores various programs and data required for the operation of electronic device 400. Processor 401, ROM 402, and RAM 403 are interconnected via bus 404. Processor 401 performs various operations of the method flow according to embodiments of the present disclosure by executing programs in ROM 402 and / or RAM 403. It should be noted that the programs may also be stored in one or more memories other than ROM 402 and RAM 403. Processor 401 may also perform various operations of the method flow according to embodiments of the present disclosure by executing programs stored in said one or more memories.

[0133] According to embodiments of this disclosure, the electronic device 400 may further include an input / output (I / O) interface 405, which is also connected to a bus 404. The electronic device 400 may also include one or more of the following components connected to the input / output (I / O) interface 405: an input section 406 including a keyboard, mouse, etc.; an output section 407 including a cathode ray tube (CRT), liquid crystal display (LCD), etc., and a speaker, etc.; a storage section 408 including a hard disk, etc.; and a communication section 409 including a network interface card such as a LAN card, modem, etc. The communication section 409 performs communication processing via a network such as the Internet. A drive 410 is also connected to the input / output (I / O) interface 405 as needed. A removable medium 411, such as a disk, optical disk, magneto-optical disk, semiconductor memory, etc., is installed on the drive 410 as needed so that computer programs read from it can be installed into the storage section 408 as needed.

[0134] According to embodiments of this disclosure, the method flow according to embodiments of this disclosure can be implemented as a computer software program. For example, embodiments of this disclosure include a computer program product comprising a computer program carried on a computer-readable storage medium, the computer program containing program code for performing the methods shown in the flowchart. In such embodiments, the computer program can be downloaded and installed from a network via communication section 409, and / or installed from removable medium 411. When the computer program is executed by processor 401, it performs the functions defined in the system of embodiments of this disclosure. According to embodiments of this disclosure, the systems, devices, apparatuses, modules, units, etc., described above can be implemented by computer program modules.

[0135] This disclosure also provides a computer-readable storage medium, which may be included in the device / apparatus / system described in the above embodiments; or it may exist independently and not assembled into the device / apparatus / system. The computer-readable storage medium carries one or more programs that, when executed, implement the method according to the embodiments of this disclosure.

[0136] According to embodiments of this disclosure, the computer-readable storage medium can be a non-volatile computer-readable storage medium. Examples include, but are not limited to: portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof. In this disclosure, the computer-readable storage medium can be any tangible medium that contains or stores a program that can be used by or in conjunction with an instruction execution system, apparatus, or device.

[0137] For example, according to embodiments of this disclosure, a computer-readable storage medium may include the ROM 402 and / or RAM 403 described above and / or one or more memories other than ROM 402 and RAM 403.

[0138] Embodiments of this disclosure also include a computer program product comprising a computer program containing program code for performing the methods provided in the embodiments of this disclosure. When the computer program product is run on an electronic device, the program code is used to enable the electronic device to implement the structured light-based monocular depth recovery method provided in the embodiments of this disclosure.

[0139] When the computer program is executed by the processor 401, it performs the functions defined in the system / apparatus of this disclosure embodiments. According to embodiments of this disclosure, the systems, apparatuses, modules, units, etc., described above can be implemented by computer program modules.

[0140] In one embodiment, the computer program may rely on a tangible storage medium such as an optical storage device or a magnetic storage device. In another embodiment, the computer program may also be transmitted and distributed in the form of signals over a network medium, and downloaded and installed via communication section 409, and / or installed from removable medium 411. The program code contained in the computer program can be transmitted using any suitable network medium, including but not limited to: wireless, wired, etc., or any suitable combination thereof.

[0141] According to embodiments of this disclosure, program code for executing the computer programs provided in embodiments of this disclosure can be written in any combination of one or more programming languages. Specifically, these computational programs can be implemented using high-level procedural and / or object-oriented programming languages, and / or assembly / machine languages. Programming languages ​​include, but are not limited to, languages ​​such as Java, C++, Python, "C", or similar programming languages. The program code can execute entirely on the user's computing device, partially on the user's device, partially on a remote computing device, or entirely on a remote computing device or server. In cases involving remote computing devices, the remote computing device can be connected to the user's computing device via any type of network, including a local area network (LAN) or a wide area network (WAN), or it can be connected to an external computing device (e.g., via the Internet using an Internet service provider).

[0142] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of the present disclosure. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in a block diagram or flowchart, and combinations of blocks in a block diagram or flowchart, may be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions. Those skilled in the art will understand that the features recited in the various embodiments and / or claims of this disclosure can be combined and / or combined in various ways, even if such combinations or combinations are not expressly described in this disclosure. In particular, the features described in the various embodiments and / or claims of this disclosure may be combined and / or combined in various ways without departing from the spirit and teachings of this disclosure. All such combinations and / or combinations fall within the scope of this disclosure.

[0143] The embodiments of this disclosure have been described above. However, these embodiments are for illustrative purposes only and are not intended to limit the scope of this disclosure. Although various embodiments have been described above, this does not mean that the measures in the various embodiments cannot be used advantageously in combination. The scope of this disclosure is defined by the appended claims and their equivalents. Various substitutions and modifications can be made by those skilled in the art without departing from the scope of this disclosure, and all such substitutions and modifications should fall within the scope of this disclosure.

Claims

A monocular depth recovery method based on structured light includes: The target scene is acquired using a monocular acquisition device to obtain multiple target scene images. Each target scene image includes a preset projection image projected onto the target scene by a projector, and the preset projection images included in the multiple target scene images are different from each other. Based on the multiple target scene images, the following operations are performed iteratively for a preset number of rounds. In the i-th round, for each target scene image, based on the system calibration parameters, the acquisition rays corresponding to the positions of M target pixels in the target scene image are constructed to obtain M target acquisition rays, where M is greater than or equal to 1 and less than or equal to the maximum resolution of the monocular acquisition device, and i is a positive integer. Based on the system calibration parameters, the preset projection image corresponding to the target scene image, and K three-dimensional points on each target acquisition ray, K reference grayscale values ​​are obtained, where K is a positive integer; Based on the preset voxel grid corresponding to the target scene and the K three-dimensional points, the K body density is obtained; Based on the K reference grayscale values ​​and the K individual densities corresponding to each target pixel position, the rendering grayscale value corresponding to the target pixel position is obtained; Based on the M rendered grayscale values ​​and M scene image grayscale values ​​corresponding to each of the multiple target scene images, the preset voxel grid is iteratively optimized, and if i is less than the preset number of rounds, i is incremented, and the operation of constructing target acquisition rays based on system calibration parameters is returned. The scene image grayscale value represents the grayscale value at the target pixel position in the target scene image. When i is greater than or equal to the preset number of rounds, the target scene is depth restored based on the optimized preset voxel grid. According to claim 1, the monocular depth recovery method, wherein, The step of obtaining the rendering grayscale value corresponding to the target pixel position based on the K reference grayscale values ​​and the K individual densities corresponding to each target pixel position includes: Based on the arrangement order of the K three-dimensional points along the light projection direction of the target acquisition light, determine the k-1 target three-dimensional points arranged before the k-th three-dimensional point, where k is an integer greater than 1 and less than or equal to K; Based on the volume density corresponding to the k-1 target three-dimensional points, determine the transmittance corresponding to the k-th three-dimensional point; The rendered grayscale value is obtained by weighted summation of the transmittance corresponding to the K three-dimensional points and the K reference grayscale values. The monocular depth recovery method according to claim 1 or 2, wherein, The step of obtaining K reference grayscale values ​​based on the system calibration parameters, the preset projection image corresponding to the target scene image, and K three-dimensional points on each target acquisition ray includes: For each three-dimensional point, based on the system calibration parameters, the three-dimensional point is projected onto the first projection plane of the projector to obtain the projection pixel position corresponding to the three-dimensional point, wherein the first projection plane is the image plane of the projector where the preset projection image is located; Determine the positions of multiple neighboring pixels in the preset projection image that are adjacent to the position of the projected pixel; Based on the gray values ​​at the multiple neighboring pixel locations, the initial gray value at the projected pixel location is obtained by interpolation. Based on the multiple target scene images and the initial grayscale value, a reference grayscale value corresponding to the three-dimensional point is determined. According to claim 3, the monocular depth recovery method, wherein, The step of determining the reference gray value corresponding to the 3D point based on the multiple target scene images and the initial gray value includes: Based on the multiple target scene images, determine the minimum grayscale value and edge contrast at the target pixel position corresponding to the three-dimensional point; Based on the initial gray value, the minimum gray value, and the edge contrast, a reference gray value corresponding to the three-dimensional point is determined. The monocular depth recovery method according to claim 1 or 2, wherein, The step of obtaining the K voxel density based on the preset voxel mesh corresponding to the target scene and the K three-dimensional points includes: For each 3D point, determine multiple mesh vertices in the preset voxel mesh that are adjacent to the 3D point; The volume density at the multiple mesh vertices is interpolated using trilinear interpolation to obtain the volume density corresponding to the three-dimensional point. The monocular depth recovery method according to claim 1 or 2, wherein, The step of performing depth reconstruction of the target scene based on the optimized preset voxel grid includes: Based on the system calibration parameters, the acquisition rays corresponding to each pixel position on the second projection acquisition plane of the monocular acquisition device are constructed to obtain multiple target reconstruction rays, wherein the second projection acquisition plane represents the image plane of the monocular acquisition device where the target scene image is located; Based on the reconstructed rays from the multiple targets and the optimized preset voxel mesh, multiple reconstructed volume densities are obtained; The target scene is depth restored based on the reconstructed volume density and preset volume rendering formula. The monocular depth recovery method according to claim 1 or 2, wherein, The step of iteratively optimizing the preset voxel grid based on M rendered grayscale values ​​corresponding to each of the multiple target scene images and M scene image grayscale values ​​includes: For each target scene image, the difference between the rendered grayscale value corresponding to each target pixel in the target scene image and the grayscale value of the scene image is calculated to obtain the color deviation corresponding to each target pixel; An image loss function is constructed based on the M color deviations that correspond one-to-one with the M target pixels in the target scene image. The preset voxel grid is optimized based on the image loss function. The monocular depth recovery method according to claim 1 or 2, wherein, Each target scene image includes multiple first patterns and multiple second patterns, wherein the multiple first patterns and the multiple second patterns are of the same size and are all square, the pixel value at each pixel position of the multiple first patterns is a first grayscale value, and the pixel value at each pixel position of the multiple second patterns is a second grayscale value. A monocular depth recovery device based on structured light, comprising: The first acquisition module is used to acquire the target scene using a monocular acquisition device to obtain multiple target scene images. Each target scene image includes a preset projection image projected onto the target scene by a projector, and the preset projection images included in the multiple target scene images are different from each other. The second obtaining module is used to iteratively perform the following operations for a preset number of rounds based on the multiple target scene images. In the i-th round, for each target scene image, based on the system calibration parameters, the acquisition rays corresponding to the positions of M target pixels in the target scene image are constructed to obtain M target acquisition rays, where M is greater than or equal to 1 and less than or equal to the maximum resolution of the monocular acquisition device, and i is a positive integer. The third module is used to obtain K reference grayscale values ​​based on the system calibration parameters, the preset projection image corresponding to the target scene image, and K three-dimensional points on each target acquisition ray, where K is a positive integer; The fourth module is used to obtain the K body density based on the preset voxel mesh corresponding to the target scene and the K three-dimensional points; The fifth module is used to obtain the rendering grayscale value corresponding to the target pixel position based on the K reference grayscale values ​​and the K individual densities corresponding to each target pixel position; The sixth module is used to iteratively optimize the preset voxel grid based on the M rendered gray values ​​and M scene image gray values ​​corresponding to each of the multiple target scene images, and increment i when i is less than the preset number of rounds, and return the operation of constructing target acquisition rays based on system calibration parameters, wherein the scene image gray value represents the gray value at the target pixel position in the target scene image; The reconstruction module is used to perform depth restoration of the target scene based on the optimized preset voxel mesh when i is greater than or equal to the preset number of rounds. An electronic device, comprising: One or more processors; Memory, used to store one or more computer programs. The characteristic feature is that the one or more processors execute the one or more computer programs to implement the steps of the method according to any one of claims 1 to 8.

Images