A device and method for generating 3D image data

Abstract

A method of training an AI model to generate three-dimensional, 3D, training data from captured two- dimensional images. The method comprises receiving image data of a plurality of two-dimensional training images; generating the 3D training data comprising a first model and a second model based on the image data; and concurrently optimising a first volumetric rendering based on the first model with sampling guided by the second model and optimising a second volumetric rendering based on the second model with sampling guided by the first model. There is also provided a device for generating the 3D training data, a device for training an AI according to the generated data, and a device configured to implement an AI model trained using the generated data.

Description

[0001] A DEVICE AND METHOD FOR GENERATING 3D IMAGE DATA

[0002] FIELD OF THE INVENTION

[0003] This invention relates to generating three-dimensional (3D) training data from captured two-dimensional images of a scene. The 3D data can then be used for the purposes of training models which require this type of data to be trained.

[0004] BACKGROUND

[0005] Accurate 3D reconstruction of real urban scenes from images acquired by a vehicle is an important task to scale up development of autonomous driving cars. Indeed, the reconstructed environment can be used in a simulator to fine tune and train the algorithm deployed on the real car.

[0006] To generate an accurate 3D simulation environment, reconstruction techniques (like NeRF or 3DGS) are used on real data to reduce the domain gap between synthetic and real data as much as possible. Such reconstruction algorithms provide high quality reconstructions but are not as efficient as standard 3D representation for rendering and manipulation, for example, such as triangle meshes. NeRF-based methods provide a good representation for view synthesis in the photometric domain but are poor at reconstructing the exact 3D geometry of the scene.

[0007] Other methods based on the Signed Distance Field (SDF) are more suitable from which to extract a mesh of the scene and to represent accurately the underlying 3D scene geometry. However, SDF based reconstruction methods are more costly to train and are likely to fail on complex scenes such as those recorded by sensors embedded on a car. This is because multi-view stereo, like NeRF, or SDF based reconstruction algorithms from scene centric acquisitions are ill-posed problems. This is because the overlap between observations is too small and some parts of the scene are unobserved. A NeRF is a neural radiance field which represents a scene with a volumetric formulation. A signed distance field (SDF) is a scalar field that associates for each 3D position in the scene a 1 D value that represents the distance to the closest surface. A NeuS is a neural field representing a scene with an SDF. A 3D Mesh is an explicit representation of the geometry of a scene or an object using connected 3D points that form triangles.

[0008] To overcome the aforementioned limitations, one can employ more expensive depth sensors, for example, LiDAR or depth cameras. However, such an approach is not scalable to a whole fleet of autonomous vehicles as this is very expensive. Another approach consists of improving the existing reconstruction algorithm, for example by relying on more prior knowledge from large perception models or to use better initialization of the scene geometry.

[0009] There are multiple existing methods for outdoor urban scene reconstruction from images. One existing method is using a radiance field. Popularized by NeRF, a radiance field implicitly represents a 3D scene in a continuous manner. In each 3D location of the scene a colour and a density value are learned by a neural network. By accumulating the local information along rays defined by a camera using volume rendering, the final colour and geometry of the scene is generated. Radiance fields are powerful representations for Novel View Synthesis (NVS) as their model within the 3D representation can show some light effects such as secularity. However, Radiance fields remain challenging to train on a large unbounded area and are costly to use for rendering.

[0010] Another existing method is using NeuS, which is a SDF-based solution. NeuS is the pioneering work that combines volumetric rendering with a Signed Distance Field (SDF) as the underlying representation of the scene rather than a pure volumetric model. An SDF is a scalar field that associates each 3D position in the scene with a 1 D value that represents the distance to the closest surface. If the 3D point is lying inside a structure, the value of the scalar field will be negative, while if the point is outside a surface, it will be positive. A mesh that represents the surface can be extracted at the 0 level-set of the SDF, for example by using a marching cube algorithm. Additionally, the spatial derivative of the SDF gives the unit vector direction to the closest surface, or in other words, the opposite of the normal of the closest surface. To integrate and train an SDF using volumetric rendering from NeRF, NeuS proposes a formula to convert the SDF value to a density value that can be used in the volumetric rendering formulation.

[0011] Some improvements of the above methods consist of improving the initialization of the SDF to facilitate convergence. In one such improvement, the ground plan is approximated according to the pose of the observation, and this approximation is then used to initialize the SDF. This method also relies on multiple representations to describe different areas of the scene according to a preliminary scene partitioning, using NeuS for the close range and NeRF for the long range. Another such improvement proposes to initialize the SDF with a density estimator trained from a NeRF. The proposed approach thus smoothly transitions from a pure volumetric representation of the scene to an SDF.

[0012] Although improvements have been made, the aforementioned approaches are limited by the use of prior knowledge, e.g. either from a NeRF or a road plan estimation, that bias the reconstruction toward this prior knowledge. The main disadvantage of this approach is that small structures present in the scene are likely to be erased from the final reconstruction.

[0013] It is desirable to develop an approach which is fast, flexible, and still maintains details in the reconstructed scene.

[0014] SUMMARY OF THE INVENTION

[0015] According to one aspect there is provided a method of training an Al model to generate three-dimensional, 3D, training data from captured two-dimensional images, the method comprises: receiving image data of a plurality of two-dimensional training images; generating the 3D training data comprising a first model and a second model based on the image data; and concurrently optimising a first volumetric rendering based on the first model with sampling guided by the second model and optimising a second volumetric rendering based on the second model with sampling guided by the first model. This joint optimization enables a better representation of the 3D scene.

[0016] In an embodiment, the first model may be a volumetric model and the second model may be a signed distance field, SDF, model. In an embodiment, the image data may comprise one or more camera poses and camera parameters. The volumetric model enables a fast convergence while the SDF model ensures precise geometric accuracy of the reconstructed scene.

[0017] In an embodiment, the method may comprise generating a set of one or more rendered images from each optimised volumetric rendering; and determining an accuracy of the rendered images compared to the two-dimensional training images based on a loss function. As a result, no prior 3D information is needed for scene optimization and therefore the method can be applied on large scale data such as images collected by a fleet of low-cost autonomous cars.

[0018] In an embodiment, the sampling may be a ray sampling step of the volumetric model guided by a representation learnt by the SDF model and a ray sampling step of the SDF model guided by a representation learnt by the volumetric model respectively. This guidance enables the training samples to focus close to the scene surfaces, providing faster convergence and better geometric details reconstructions.

[0019] In an embodiment, the ray sampling step may comprise: generating rays based on the image data; computing a ray uncertainty of each ray; and sampling the ray depending on whether the uncertainty value exceeds a predefined threshold value.

[0020] In an embodiment, the ray may be sampled according to the geometry of the guiding model if said ray has an uncertainty below the threshold value. Leveraging the certainty of a reference model makes the other model converge faster and to a better overall representation.

[0021] In an embodiment, the ray may be sampled according to a standard sampling method if said ray has an uncertainty at or above the threshold value. Such a technique permits to filter artefacts or errors that may come from one model to be spread to the other model.

[0022] In an embodiment, the uncertainty value may be based on geometric criteria and photometric criteria obtained by comparing the guiding model with ground truth data or with a representation of another model. Mixing geometric and photometric criteria makes the filtering algorithm more robust.

[0023] In an embodiment, the threshold may be adapted to maintain a pre-set ratio of rays with uncertainty at or above the threshold value to rays with uncertainty below the threshold value. Using a ratio rather than a hand-crafted threshold value enables generalisation of the method to a difference scene without requiring parameter re-tuning.

[0024] In an embodiment, after guided ray sampling, the method may comprise performing geometric refinement on the second volumetric model. The second step of refinement improves the geometric quality of the reconstructed scene.

[0025] In an embodiment, the method may comprise training the second volumetric model with monocular geometric clues. Monocular clue does not rely on expensive 3D sensors.

[0026] In an embodiment, the monocular geometric clues may be obtained by implementing a pretrained perception model on the training images. Off the shelf perception models are highly efficient and generalize well on urban data.

[0027] In an embodiment, determining the accuracy may comprise including a cost function for monocular geometric clues in the loss function. Monocular clue regularises the geometry of the scene on untextured or low texture regions.

[0028] In an embodiment, the method may comprise generating rendered geometry based on the refined SDF model for including in the determining of the accuracy. The SDF model directly represents the underlying geometry of the scene and is more geometrically accurate compared to pure density models such as NeRF.

[0029] In an embodiment, the method may comprise generating a 3D mesh from the optimised SDF model. A 3D mesh is easily extracted from a SDF and can be used efficiently for rendering.

[0030] In an embodiment, the 3D mesh may be generated using a marching cube. The marching cube algorithm extracts a precise 3D mesh from a SDF.

[0031] According to another aspect there is provided a device configured to generate three-dimensional training data using an Al model trained according to the method of claim 1 . Data augmentation provided by this method does not suffer from a domain gap as they are generated from real images and thanks to the precise geometry obtained as described below, lots of data from different viewpoints can be generated, as well as 3D representations.

[0032] According to another aspect there is provided a device comprising an Al model trained on 3D training data created by the device of claim 17. In an embodiment, the device may be a self-driving vehicle. The aforementioned method can be used to extract background scene to be used in a car simulator for end-to-end driving policy training. BRIEF DESCRIPTION OF THE FIGURES

[0033] The present invention will now be described by way of example with reference to the accompanying drawings. In the drawings:

[0034] Figure 1 shows an overview of the main aspects of the proposed approach.

[0035] Figure 2 shows an overview of the first step 110 of the proposed method.

[0036] Figure 3 shows the guided ray sampling in greater detail.

[0037] Figure 4 shows an overview of the second training step.

[0038] Figure 5 shows an overview of an example embodiment of the proposed approach.

[0039] Figure 6 shows a flow diagram illustrating the first step of the proposed method.

[0040] Figure 7 shows an example of a device configured to implement the trained Al model and a system for training the Al model according to the proposed method.

[0041] DETAILED DESCRIPTION OF THE INVENTION

[0042] The proposed approach is a reconstruction method which leverages booth implicit signed distance function and volumetric representation by implementing a joint concurrent training strategy. Accordingly, the benefit of fast convergence provided by volumetric implicit representation is exploited while maintaining the geometric accuracy through the regularisation imposed by the Signed Distance Field 3D representation.

[0043] Figure 1 shows an overview of the main aspects of the proposed approach. The proposed approach uses calibrated training images 102 and image data 1 18 to generate a volumetric representation 104, a Signed distance field of the scene 106, and a 3D mesh 108. The method is composed of two steps. The first step 110 jointly optimizes the volumetric and SDF representation and with a sampling method of the volumetric model guided by the SDF model, and vice versa. The guidance is controlled via uncertainty estimation based on geometric and photometric criteria. The second step 112 refines the model using monocular geometric clues 114 obtained from a pretrained model 116 given the training images 102.

[0044] The proposed approach takes as input a set of calibrated images 102, which comprises images with image data 118 corresponding to six degree of freedom poses and camera parameters, and generates each of an implicit volumetric radiance field, an implicit SDF function, and a 3D mesh extracted from the SDF.

[0045] Thus, there is proposed a reconstruction method tailored for outdoor urban scene reconstruction that uses calibrated images and generates a volumetric representation, a Signed distance field of the scene, and a 3D mesh. The method is decomposed into two main steps. The first step jointly optimizes the volumetric and SDF representation and with a cross-sampling method where the volumetric model is guided by the SDF model, and vice versa. The guidance is controlled via uncertainty estimation based on geometric and photometric criteria. Thus, there is proposed herein a method of training an Al model to generate three-dimensional, 3D, training data from captured two-dimensional images. The proposed method comprises receiving image data of a plurality of two-dimensional training images. The method further comprises generating the 3D training data comprising a first model and a second model based on the image data. The method then comprises concurrently optimising a first volumetric rendering based on the first model with sampling guided by the second model and optimising a second volumetric rendering based on the second model with sampling guided by the first model.

[0046] The first model may be a volumetric model and the second model may be a signed distance field, SDF, model. The image data may comprise one or more camera poses and camera parameters.

[0047] Figure 2 shows an overview of the first step 110 of the proposed method. The first step comprises the initial training stage 110 and consists of a joint optimization of the volumetric 104 and SDF 106 models. For each of the models, a guided ray sampling 202 is performed. This is based on training image data 118 which comprises the current training camera pose and camera parameters. The training image data 118 is used to perform a standard volumetric rendering 204 to generate the image at the target location. These rendered images 206 are compared to the ground truth (GT) images 102 in order to compute the weight update using standard error minimization via gradient descent. That is, the comparison of the training images 102 with the rendered images 206 is used to minimise a loss function 208. During this initial step 110, the ray sampling step 202 of the volumetric model 104 is guided by the representation learnt by the SDF model 106 and ray sampling step of the SDF model 106 is guided by the representation learnt by the volumetric model 104.

[0048] The proposed method may therefore further comprise generating a set of one or more rendered images from each optimised volumetric rendering and determining an accuracy of the rendered images compared to the two-dimensional training images based on the loss function.

[0049] Figure 3 shows the guided ray sampling 202 in greater detail. In order for each representation to benefit from each other, the guided ray sampling strategy presented in figure 3 is introduced. In the guided sampling the objective is to compute sample points for the volumetric rendering of the current model, guided by a reference model. That is, the ray sampling step of the volumetric model is guided by the representation learnt by the SDF model and the ray sampling step of the SDF model is guided by the representation learnt by the volumetric model.

[0050] The approach may be thought of as comprising four steps including ray generation 302, per-ray uncertainty computation 304, ray classification 306, and ray sampling 308.

[0051] For ray generation 302, a ray centre and direction are computed according to the image data including the camera pose, also known as extrinsic parameters, and the camera characteristics, also known as intrinsic parameters. For uncertainty computation 304, the guided sampling uses a per-ray uncertainty estimation. The uncertainty values are computed according to geometric and photometric criteria obtained by comparing a reference model 310 with the GT data or the representation of another model. The reference model refers to either the volumetric model or the SDF model, depending on which one is being used to guide the current ray sampling. Photometric uncertainty criteria can be quantified by computing the difference, for example the Mean Square Error (MSE), the Robust MSE or the Structural Similarity (SSIM), between the GT colour of the observation and the colour obtained by the reference model.

[0052] A low uncertainty may indicate that the reference model has converged to a good estimate and thus can be trust for ray sampling. Geometric uncertainty can be computed by comparing the expected ray termination of the reference model to the current model, where the ray termination is the distance from the ray centre to the closest surface that intersect this ray. If the uncertainty is high, it is likely that one of the models may not have converged to a precise geometry.

[0053] For ray classification 306 and sampling 308, the uncertainty value is then used to select rays according to a predefined threshold. Uncertain rays with an uncertainty value at or greater than the threshold are sampled to the standard sampling method 308a. Certain rays with an uncertainty value lower than the threshold are sampled 308b according to the reference model geometry.

[0054] The ray classification step 306 of the guided ray sampling requires a threshold to distinguish between certain and uncertain rays to have been selected. Selecting such a threshold empirically can be complicated as for different scenes and reconstruction, this value may change. There is therefore proposed a simple yet effective adaptative threshold selection to overcome this. Starting from an initial threshold value, the ratio between the number of certain and uncertain rays is computed at regular intervals during the reconstruction process. A scheduling policy may be defined that is designed to control this ratio along the whole of the training process and modify the threshold accordingly to meet the ratio requirement. For example, if at a certain step of the reconstruction process, the scheduling policy imposes a ratio of 20% of certain rays and 80% of uncertain rays and the current ratio is 30% of certain rays and 70% of uncertain rays, the threshold value may be lowered to reduce the proportion of certain rays so as to fulfil the ratio.

[0055] As described above, the described guided sampling uses per-ray uncertainty estimation. The uncertainty values are computed according to geometric and photometric criteria obtained by comparing a reference model (i.e. volumetric or SDF) with the GT data or the representation of another model. The uncertainty value may then be used to select rays according to a predefined threshold. Certain rays, with an uncertainty value lower than the threshold, may be sampled according to reference model geometry. Uncertain rays, with an uncertainty value less than the threshold, may be sampled to the standard sampling method. Therefore, the sampling may be a ray sampling step of the volumetric model guided by a representation learnt by the SDF model and a ray sampling step of the SDF model guided by a representation learnt by the volumetric model respectively.

[0056] The ray sampling step may comprise generating rays based on the image data, computing a ray uncertainty of each ray, and sampling the ray depending on whether the uncertainty value exceeds a predefined threshold value. The ray may be sampled according to the geometry of the guiding model if said ray has an uncertainty below the threshold value. The ray may be sampled according to a standard sampling method if said ray has an uncertainty at or above the threshold value.

[0057] The uncertainty value may be based on geometric criteria and photometric criteria obtained by comparing the guiding model with ground truth data or with a representation of another model. The guiding model is whichever model is providing the guiding influence on the sampling. For example, it could be the volumetric model or the SDF model. The threshold may be adapted to maintain a pre-set ratio of rays with uncertainty at or above the threshold value to rays with uncertainty below the threshold value. It should be understood that values at the threshold may be chosen to be included in the group which lies below the threshold with minimal effect on the outcome.

[0058] Figure 4 shows an overview of the second training step 112. The second training step 112 is designed to refine the scene geometry by improving the SDF estimation. The guided sampling of the first step 110 may be replaced to obtain better geometric refinement 406 of the SDF representation and add monocular geometric clues 402 to train the SDF representation. These monocular clues 402, such as predicted depth, semantic, normal, etc. are obtained by using a pretrained perception model 404 on the training images 110. Additional cost functions are added to the loss function 408 to constrain the model according to the monocular clues.

[0059] The second step refines the whole model using monocular geometric clues. In this step the guided sampling of the first step is replaced for geometric refinement of the SDF representation. Monocular geometric clues are added to train the SDF representation.

[0060] Thus, after the guided ray sampling of the initial step of training, the second step may comprise performing geometric refinement on the second volumetric model. The training may comprise training the second volumetric model with monocular geometric clues. The monocular geometric clues may be obtained by implementing a pretrained perception model on the training images.

[0061] Determining the accuracy may comprise including a cost function for monocular geometric clues in the loss function. The training may comprise generating a 3D mesh from the optimised SDF model.

[0062] Figure 5 shows an overview of an example embodiment of the proposed approach 500. In this example, a 3D colourised mesh 504 may be extracted from the SDF representation 106 with a marching cube algorithm 506 and may be used to guide the first sampling stage 110 of the volumetric representation. This mesh is updated frequently during the reconstruction and does not need to be computed at each training iteration, in order to speed up the whole process.

[0063] In this example, expected depth 502 from the volumetric representation is used to guide the sampling of the SDF model. In implementation, the forward pass of the volumetric representation is done before the forward pass of the SDF model so that the rendered depth does not need to be computed an additional time in order to optimize the computation.

[0064] For this example embodiment, the preferred implementation of each sampling step is as follows.

[0065] For the guided sampling of the volumetric representation, the per-ray uncertainty may be estimated according to the photometric consistency between colourised 3D mesh 504 obtained from the SDF model 106 and the GT image colour. Points of the certain rays are sampled near the mesh surface. While points of the uncertain rays are sampled according to the density distribution estimated by an auxiliary network trained in a self-supervised manner by the volumetric representation. The motivation behind this is that if the colour on the mesh surface is close to the GT colour, it is likely that the mesh has converged to the real scene surface. In this case, the SDF representation can be trusted to accurately train the volumetric representation.

[0066] For the guided sampling of the SDF representation, the per-ray uncertainty may be estimated accordingly to the geometric consistency between the 3D mesh surface and the depth value computed by the volumetric representation. Sampling of the certain rays may be done near the mesh surface (e.g. with proposal guidance). Sampling of the uncertain rays may be done near the expected depth obtained by the volumetric representation (e.g. with proposal guidance). The motivation behind this is that if the distance to the nearest surface is different according to the 3D mesh and the volumetric representation, it is likely that the mesh is missing a fine detail or has generated an occlusion that should not be there. In this situation, there is more confidence in the geometry of the volumetric representation given the fact such a representation is more accurate, able to represent fine structure, and converges faster.

[0067] For the second step of the method, the guided sampling of the SDF representation may be modified by replacing the guided ray sampling by self-supervised probability density estimation by the SDF model itself. By doing that, the geometric details omitted because of the coarsened of the proposal model can be omitted. It is possible to use mono-normal clues as an additional supervision signal and directly constrain the derivative of the SDF to match the predicted normal. This additional supervision improves the scene geometry, in particular for untextured planar surfaces like roads and sidewalks.

[0068] In the example embodiment, in the first step the sampling of the volumetric representation comprises the following details. For the uncertainty estimation, the photometric consistency between colourised mesh obtained from the SDF and the GT image colour. For example, if the colour on the mesh surface is close to the GT colour, it is likely that the mesh has converged to close to the real scene surface. The SDF representation may then be trusted to train the volumetric representation. Sampling of certain rays comprises sampling near the mesh surface. Sampling of uncertain rays comprises proposal sampling. The proposal sampling is the ‘standard’ sampling strategy for the volumetric representation. It relies on an additional network that is trained by the volumetric representation to estimate the density along the ray. With an additional network this is referred to as the ‘proposal network’, hence the term ‘proposal sampling’.

[0069] In the example embodiment, the sampling of the SDF representation comprises the following details. For the uncertainty estimation, the geometric consistency between the mesh surface and the depth value is computed by the volumetric representation. If the distance to the nearest surface is different according to the mesh and the volumetric representation, it is likely that the mesh is missing a fine detail or has generated an occlusion that should not be there. In this situation, it is possible to be more confident in the geometry of the volumetric representation. Sampling of certain rays comprises sampling near the mesh surface, e.g. with the proposed density estimation. Sampling of uncertain rays comprises sampling near the volumetric representation surface.

[0070] For the second step of the proposed example embodiment, the sampling refinement comprises replacing the proposal sampling method by coarse to fine sampling which is self-supervised by the SDF. It also comprises adding monocular clues and may use mono normal clues as a supervision signal.

[0071] Therefore, in an embodiment, the second training step may comprises generating rendered geometry based on the refined SDF model for including in the determining of the accuracy. In an example embodiment, the 3D mesh may be generated using a marching cube.

[0072] Figure 6 shows a flow diagram illustrating the first step of the proposed method 600 as described above in reference to figure 2 for training an Al model to generate three-dimensional, 3D, training data from captured two-dimensional images. The method 600 comprises an initial step 602 of receiving image data of a plurality of two-dimensional training images. The method comprises a further step 604 of generating the 3D training data comprising a first model and a second model based on the image data. The method comprises another step 606 of concurrently optimising a first volumetric rendering based on the first model with sampling guided by the second model and optimising a second volumetric rendering based on the second model with sampling guided by the first model. As described in reference to figure 2, the first model may be a volumetric model and the second model may be an SDF model. Further features of the method 600 result directly from the different embodiments described above and below.

[0073] The proposed method uses a combination of two implicit 3D representations, volumetric and SDF, during the reconstruction process. In doing so, the method combines the benefit of fast convergence provided by the volumetric implicit representation while being geometrically accurate through the regularisation imposed by the Signed Distance Field 3D representation.

[0074] The proposed method trains jointly the two implicit representations, with a mutual guidance acting on the ray sampling step based on geometric and photometric uncertainty quantification. Thus, the method enables the filtering of artifacts from the volumetric or SDF representation which may harm the other representation thanks to the ray filtering step based on uncertainty quantification.

[0075] Figure 7 shows an example of a device configured to implement the trained Al model to process two-dimensional images taken by an image sensor 1102 in the device 1101. Such a device 1101 typically includes some onboard processing capability. This could be provided by the processor 1104. The processor 1104 could also be used for the essential functions of the device. The device typically also comprises memory a 1103.

[0076] The transceiver 1105 is capable of communicating over a network with other entities 1110, 1111. Those entities may be physically remote from the device 1101. The network may be a publicly accessible network such as the internet. The entities 1 110, 1111 may be based in the cloud. In one example, entity 11 10 is a computing entity and entity 1 11 1 is a command and control entity. These entities are logical entities. In practice they may each be provided by one or more physical devices such as servers and datastores, and the functions of two or more of the entities may be provided by a single physical device. Each physical device implementing an entity comprises a processor and a memory. The devices may also comprise a transceiver for transmitting and receiving data to and from the transceiver 1105 of device 1101. The memory stores in a non-transient way code that is executable by the processor to implement the respective entity in the manner described herein.

[0077] Alternatively, or additionally, the command and control entity 1111 may train the artificial intelligence models used in each module of the system. This is typically a computationally intensive task, even though the resulting model may be efficiently described, so it may be efficient for the development of the algorithm to be performed in the cloud, where it can be anticipated that significant energy and computing resource is available. It can be anticipated that this is more efficient than forming such a model at a typical device.

[0078] In one implementation, once the Al model has been developed in the cloud, the command and control entity can automatically form a corresponding model and cause it to be transmitted to the relevant device. In this example, the system is implemented at the device 1101 by processor 1104.

[0079] In another possible implementation, a plurality of two-dimensional images may be captured by the device sensor 1102 and the image data may be sent by the transceiver 1105 to the cloud for processing in the system. The resulting 3D training data could then be sent back to the device 1101 , as shown at 1112 in Figure 7. Therefore, the method may be deployed in multiple ways, for example in the cloud, on the device, or alternatively in dedicated hardware. As indicated above, the cloud facility could perform training to develop new algorithms or refine existing ones. Depending on the compute capability near to the data corpus, the training could either be undertaken close to the source data, or could be undertaken in the cloud, e.g. using an inference engine. The training system may also be implemented at the device, in a dedicated piece of hardware, or in the cloud.

[0080] Accordingly, there is provided herein a device configured to generate three-dimensional training data using an Al model trained according to the method described above. Specifically, by receiving image data of a plurality of two-dimensional training images; generating the 3D training data comprising a first model and a second model based on the image data; and concurrently optimising a first volumetric rendering based on the first model with sampling guided by the second model and optimising a second volumetric rendering based on the second model with sampling guided by the first mode.

[0081] There is further proposed a device comprising an Al model trained on 3D training data created by the device described above. For example, the device could be a self-driving vehicle.

[0082] The applicant hereby discloses in isolation each individual feature described herein and any combination of two or more such features, to the extent that such features or combinations are capable of being carried out based on the present specification as a whole in the light of the common general knowledge of a person skilled in the art, irrespective of whether such features or combinations of features solve any problems disclosed herein, and without limitation to the scope of the claims. The applicant indicates that aspects of the present invention may consist of any such individual feature or combination of features. In view of the foregoing description it will be evident to a person skilled in the art that various modifications may be made within the scope of the invention.

Claims

CLAIMS1 . A method of training an Al model to generate three-dimensional, 3D, training data from captured two-dimensional images, the method comprises:receiving image data of a plurality of two-dimensional training images;generating the 3D training data comprising a first model and a second model based on the image data; andconcurrently optimising a first volumetric rendering based on the first model with sampling guided by the second model and optimising a second volumetric rendering based on the second model with sampling guided by the first model.

2. The method according to claim 1 , wherein the first model is a volumetric model and the second model is an signed distance field, SDF, model.

3. The method according to claim 1 or 2, wherein the image data comprises one or more camera poses and camera parameters.

4. The method according to any preceding claim, wherein the method comprises:generating a set of one or more rendered images from each optimised volumetric rendering; anddetermining an accuracy of the rendered images compared to the two-dimensional training images based on a loss function.

5. The method according to any previous claim, wherein the sampling is a ray sampling step of the volumetric model guided by a representation learnt by the SDF model and a ray sampling step of the SDF model guided by a representation learnt by the volumetric model respectively.

6. The method according to claim 5, wherein the ray sampling step comprises:generating rays based on the image data;computing a ray uncertainty of each ray; andsampling the ray depending on whether the uncertainty value exceeds a predefined threshold value.

7. The method according to claim 6, wherein the ray is sampled according to the geometry of the guiding model if said ray has an uncertainty below the threshold value.

8. The method according to claim 6, wherein the ray is sampled according to a standard sampling method if said ray has an uncertainty at or above the threshold value.

9. The method according to any of claims 6 to 8, wherein the uncertainty value is based on geometric criteria and photometric criteria obtained by comparing the guiding model with ground truth data or with a representation of another model.

10. The method according to any of claims 6 to 9, wherein the threshold is adapted to maintain a preset ratio of rays with uncertainty at or above the threshold value to rays with uncertainty below the threshold value.

11. The method according to any preceding claim, wherein after guided ray sampling, the method comprises performing geometric refinement on the second volumetric model.

12. The method according to any preceding claim, wherein the method comprises training the second volumetric model with monocular geometric clues.

13. The method according to claim 11 , wherein the monocular geometric clues are obtained by implementing a pretrained perception model on the training images.

14. The method of any of claims 4 to 13, wherein determining the accuracy comprises including a cost function for monocular geometric clues in the loss function.

15. The method according to any of claims 12 to 14, wherein the method comprises generating rendered geometry based on the refined SDF model for including in the determining of the accuracy.

16. The method of any preceding claim, wherein the method comprises generating a 3D mesh from the optimised SDF model.

17. The method according to claim 16, wherein the 3D mesh is generated using a marching cube.

18. A device configured to generate three-dimensional training data using an Al model trained according to the method of claim 1 .

19. A device comprising an Al model trained on 3D training data created by the device of claim 17.

20. The device according to claim 19, wherein the device is a self-driving vehicle.

Images