Conditioned generative model for consistent novel views synthesis

Abstract

Described is a computer apparatus (1400) configured to: obtain a reference image (202) of a 3D scene from a reference viewpoint; obtain a reference point map (203) corresponding to the reference image (202), the reference point map (203) comprising the coordinates of a plurality of pixels of the reference image (202); obtain a target point map (204) corresponding to the rendered image (206), the target point map (204) comprising the coordinates of a plurality of pixels of the rendered image (206); and generate the rendered image (206) in dependence on the reference image (202), the reference point map (203) and the target point map (204). In this way, the rendering process may be guided by the positions in the point maps.

Description

[0001] CONDITIONED GENERATIVE MODEL FOR CONSISTENT NOVEL VIEWS SYNTHESIS

[0002] FIELD OF THE INVENTION

[0003] This invention relates to an apparatus and method for generating a rendered image of a 3D scene.

[0004] BACKGROUND

[0005] Photorealistic simulation can be essential for ensuring the safety and reliability of autonomous driving systems by faithfully recreating both every-day and rare, challenging scenarios under a variety of conditions. This may allow for precise fine-tuning and optimization of the vehicle’s onboard algorithms. Moreover, high-fidelity simulation offers a scalable, efficient way to produce the vast amounts of data required to train the learning algorithms embedded in the vehicles, making it a critical tool for their development and testing.

[0006] A fair number of autonomous driving simulators often rely on advanced computer graphics techniques to reconstruct medium-scale synthetic scenarios, which, while effective, are time-consuming to develop, costly to scale to large amounts of data, and still contain considerable gap to real data. In recent years, neural rendering techniques for Novel View Synthesis (NVS), such as Neural Radiance Fields (NeRFs) and 3D Gaussian Splatting (3DGS), can enable the ability to synthetize photo-realistic street views from vehicle acquired data, playing a critical role on autonomous driving simulation.

[0007] Despite the remarkable progress achieved with these techniques, handling driving data remains a challenge. While these methods excel at reproducing environments from viewpoints similar to those encountered during training, they struggle to maintain rendering quality when the ego-vehicle deviates significantly from these perspectives. This limitation restricts the freedom of movement within reconstructed scenarios. Additionally, the reconstruction process is confined to areas observed by the sparse set of views captured by the vehicle's onboard sensors, lacking the ability to fill in or extrapolate unobserved regions. As a result, there is a need for advancements that allow for greater flexibility and completeness in simulated environment.

[0008] In recent years, the emergence of Stable Diffusion (SD) models trained on large-scale datasets has shown great promise in enhancing the realism of simulated environments. These models leverage vast amounts of data to recreate lifelike views across various scenarios, potentially addressing the limitations of existing neural rendering techniques. By augmenting and regularizing the training processes of these models, diffusion methods can improve the representation of observed regions and facilitate the extrapolation of data beyond training views.

[0009] However, a significant challenge remains in conditioning diffusion models on both the geometric and photometric aspects of a given scene. Achieving the necessary consistency to effectively augment neural rendering methods is still an unresolved issue. Current approaches to diffusion-augmented neural rendering tend to focus predominantly on object-centric data, often struggling to deliver reliable outcomes in complex driving scenarios. This gap highlights the need for further research to develop more robust solutions that can handle the intricacies of dynamic environments typical in autonomous driving.

[0010] Below are defined some key concepts within the context of the present system:

[0011] Radiance field: popularized by NeRF, a radiance field implicitly represents a 3D scene in a continuous manner by learning color and density values at each 3D point through a neural network. This process involves accumulating local information along rays defined by a camera using volume rendering techniques, resulting in the final color and geometry of the scene. Radiance fields are particularly powerful for Novel View Synthesis (NVS) as they can model various light effects, including secularity, thereby enhancing the realism of generated images. However, training radiance fields on large, unbounded areas remains a significant challenge.

[0012] 3D Gaussian Splats: Introduced in 2023 at “3D Gaussian Splatting” (3DGS) scientific article, this technique represents a scene using a set of trainable 3D Gaussians, where each Gaussian is associated with trainable parameters for pose, color, and density. This approach employs a training strategy similar to that of Neural Radiance Fields (NeRFs) but offers significantly improved rendering efficiency, making it suitable for real-time applications.

[0013] Rasterization: Concerns the process of converting 3D primitives (such as triangles derived from a mesh) into 2D rendering space. This technique is heavily optimized by modem hardware, particularly Graphics Processing Units (GPUs), which allow for parallel processing of each pixel in the rendered frame.

[0014] Diffusion Model: Diffusion models are a class of generative models that generate data by gradually transforming random noise into coherent outputs (potentially conditioned on some guidance signal). Initially, a diffusion process introduces noise into data, and the model is trained to reverse this process, effectively learning to denoise and generate new samples. This iterative refinement allows for the generation of high-quality images and complex data distributions.

[0015] Neural Rendering: Also referred to as deferred shading, neural rendering is not a traditional method for converting 3D scenes to 2D images. Instead, it serves as a decoding step from a 2D buffer frame obtained by one of the aforementioned methods to the final rendered frame. This approach enables the enhancement of image quality and spatial resolution at a reduced computational cost. By leveraging neural networks, neural rendering can also introduce specific visual effects or stylistic modifications to the rendered output.

[0016] Image Warping: Transformative process that alters the geometry of a source image by mapping all positions on such image plane into a second plane to create a new representation. This technique involves modifying the pixel arrangement based on a defined mapping, which can be achieved through various algorithms, such as affine transformations, perspective transformations, or non-linear warping methods.

[0017] In order to achieve a photorealistic simulator with free-vehicle control (to some extent), prior art methods may rely on different solutions:

[0018] Design of a synthetic autonomous driving simulator [CARLA: An Open Urban Driving Simulator, AirSim: High-Fidelity Visual and Physical Simulation for Autonomous Vehicles, Nvidia drive end-to-end platform for software-defined vehicles]: These simulators are designed to replicate urban driving environments with realistic physics and visuals to aid in the development, training, and validation of autonomous systems. However, they are time-consuming to develop, difficult to scale, and often exhibit a gap between synthetic and real-world data, limiting their effectiveness in certain cases. Further available simulators are disclosed [Choose Your Simulator Wisely: A Review on Open-source Simulators for Autonomous Driving],

[0019] 3D reconstruction algorithms to reconstruct scenes from real acquired data [Ghost Gym: A Neural Simulator for Autonomous Driving, Neural Scene Graphs for Dynamic Scenes, Panoptic Neural Fields: A Semantic Object-Aware Neural Scene Representation, HUGS: Holistic Urban 3D Scene Understanding via Gaussian Splatting, G3R: Gradient Guided Generalizable Reconstruction]: Recent advances using NeRF and 3D Gaussian Splatting (3DGS) allow partial 3D scene reconstruction from real- world data, offering limited control to extend simulation views beyond the training data. However, Structure-from-Motion (SfM) and Multi-View Stereo (MVS) methods remain ill-posed for vehicle-based scene reconstruction. Current state-of-the-art algorithms can only reconstruct small parts of a scene, often usable only as replay buffers in simulations, with novel viewpoints limited to interpolation between recorded observations. Additionally, reconstructing content outside of the acquired data is still not feasible.

[0020] Diffusion-based techniques to augment simulation environment [SGD: Street View Synthesis with Gaussian Splatting and Diffusion Prior, Graphic Rendering Using Diffusion Models, Image and Object Inpainting with Diffusion Models]: Recent approaches use diffusion models to enhance neural rendering, generating missing data and improving reconstruction quality at viewpoints far from the training data. However, conditioning these models on visual and geometric information to maintain consistency remains a significant challenge. While progress has been made in object-centric or indoor reconstruction [MultiDiff: Consistent Novel View Synthesis from a Single Image, GenWarp: Single Image to Novel Views with Semantic-Preserving Generative Warping, ViewCrafter: Taming Video Diffusion Models for High-fidelity Novel View Synthesis, RGBD2: Generative Scene Synthesis via Incremental View Inpainting using RGBD Diffusion Models, Three-dimensional Diffusion Models], applying diffusion models effectively to complex driving scenarios is still unresolved.

[0021] The aforementioned solutions may suffer from the following limitations:

[0022] Lack of Realism in computer-graphics simulators. Autonomous driving simulators often suffer from a "reality gap," where synthetic environments fail to capture the complex visual, physical, and behavioural details of real- world scenarios. This gap may lead to suboptimal deep learning model training, reducing their performance and generalizability in real- world conditions.

[0023] Limited view extrapolation for neural rendering-based simulators. Neural rendering-based simulators face significant constraints in extrapolating viewpoints beyond the training data, which restricts the ego-vehicle’s free movement within the reconstructed scene.

[0024] Visual inconsistency for diffusion-based methods. Diffusion models used in simulation often produce visually inconsistent outputs that lack coherence with the reference images or scene data. This inconsistency undermines the realism and reliability of the generated views.

[0025] Limitation to object-centric data. While diffusion models have shown success in reconstructing 3D objects from limited images, these methods remain insufficient for complex driving scenarios.

[0026] High Computational Demands: Fine-tuning diffusion models conditioned on visual and geometric signals to ensure the necessary consistency across 3D scenes requires substantial computational resources.

[0027] Common solution for NVS based simulator is presented in Figure 1. Figure 1 shows a pipeline 100 for NVS-method based simulation. The NVS model 103 could be neural rendering methods (NeRF or 3DGS) or a condition-based diffusion model. Visual conditioning 101 and geometric conditioning 102 from acquired data is used to generate new views from unobserved poses. This may provide an output image 104.

[0028] It is desirable to develop an apparatus and method that overcomes the above problems.

[0029] SUMMARY

[0030] According to a first aspect, there is provided a computer apparatus for generating a rendered image, the computer apparatus comprising one or more processors and a memory storing in non-transient form data defining program code executable by the one or more processors, wherein the program code is executable by the one or more processors so that the computer apparatus is configured to: obtain a reference image of a 3D scene from a reference viewpoint; obtain a reference point map corresponding to the reference image, the reference point map comprising the coordinates of a plurality of pixels of the reference image; obtain a target point map corresponding to the rendered image, the target point map comprising the coordinates of a plurality of pixels of the rendered image; and generate the rendered image in dependence on the reference image, the reference point map and the target point map. In this way, the rendering process may be guided by the positions in the point maps.

[0031] In some implementations, the apparatus may be configured to generate the reference point map from the reference viewpoint and / or the 3D scene; and / or generate the target point map from a target viewpoint of the 3D scene and / or the 3D scene. In this way, the apparatus may be able to generate the reference point map internally. In some implementations, the apparatus may be configured to generate the reference point map and / or the target point map in dependence on a 3D scene structure. In this way, the apparatus may be able to use the 3D scene to generate the point maps.

[0032] In some implementations, the apparatus may be configured to: obtain a plurality of reference images of the 3D scene, each reference image being from a different reference viewpoint; obtain a plurality of reference point maps corresponding to a respective plurality of reference images; and generate the rendered image in dependence on the plurality of reference images, the plurality of reference point maps and the target point map. In this way, the apparatus may be able to generate the rendered image based on a plurality of viewpoints, which may give more information for generating a more accurate rendered image.

[0033] In some implementations, the apparatus may be configured to: obtain a further target point map corresponding to a further rendered image, the further target point map comprising the coordinates of a plurality of pixels of the further rendered image; and generate the further rendered image in dependence on the rendered image, the target point map and the further target point map. In this way, the apparatus may be able to generate a rendered image from a viewpoint that is even further from the reference image.

[0034] In some implementations, the apparatus may be configured to generate the rendered image by means of a generative model. In this way, the apparatus may be trained to generate improved rendered images.

[0035] According to a second aspect, there is provided a computer apparatus for generating a rendered image, the computer apparatus comprising one or more processors and a memory storing in non-transient form data defining program code executable by the one or more processors, wherein the program code is executable by the one or more processors so that the computer apparatus is configured to: obtain a reference image of a 3D scene from a reference viewpoint; obtain a target viewpoint of the 3D scene corresponding to the rendered image; and generate the rendered image in dependence on the reference image and the target viewpoint by means of a generative model; wherein the generative model comprises: a target UNet comprising a cross attention component; and a reference UNet configured to provide cross attention guidance to the cross attention component. In this way, the rendering process may be guided by the cross attention guidance.

[0036] In some implementations, the target UNet may comprise in the following order: a first layer, the cross attention component, a zero convolution layer and a second layer; and

[0037] the reference UNet may comprise in the following order: a first layer and a second layer; wherein the cross attention guidance may be provided to the cross attention component after the first layer of the reference UNet. In this way, the cross attention guidance may be provided to the target UNet for use during the production of the rendered image.

[0038] In some implementations, the reference UNet may be configured to: receive a reference feature image, the reference feature image corresponding to the reference image; receive reference guidance from a first ControlNet, the reference guidance corresponding to the reference viewpoint; and generate the cross attention guidance in dependence on the reference feature image and the reference guidance. In this way, the generation of the cross attention guidance may be guided by the reference viewpoint features.

[0039] In some implementations, the target UNet may be configured to: receive input noise, the input noise corresponding to the target viewpoint; receive target guidance from a second ControlNet, the target guidance corresponding to the target viewpoint; and generate a denoised feature image in dependence on the input noise, the target guidance, and the cross attention guidance. In this way, the generation of the rendered image may be guided by the target viewpoint features. In some implementations, the apparatus may be configured to generate the rendered image by rendering the denoised feature image. In this way, denoised image produced by the target UNet may be adapted into a higher accuracy rendered image.

[0040] In some implementations, the apparatus may be configured to: obtain a target image; compare the rendered image with the target image to calculate a loss; and update the generative model in dependence on the loss. In this way, the generative model of the apparatus may be trained to produce better quality rendered images.

[0041] According to a third aspect, there is provided a method for generating a rendered image, carried out by a computer apparatus, comprising the steps of: obtaining a reference image of a 3D scene from a reference viewpoint; obtaining a reference point map corresponding to the reference image, the reference point map comprising the coordinates of a plurality of pixels of the reference image; obtaining a target point map corresponding to the rendered image, the target point map comprising the coordinates of a plurality of pixels of the rendered image; and generating the rendered image in dependence on the reference image, the reference point map and the target point map. In this way, the rendering process may be guided by the positions in the point maps.

[0042] According to a fourth aspect, there is provided a method for generating a rendered image, carried out by a computer apparatus, comprising the steps of: obtaining a reference image of a 3D scene from a reference viewpoint; obtaining a target viewpoint of the 3D scene corresponding to the rendered image; and generating the rendered image in dependence on the reference image and the target viewpoint by means of a generative model; wherein the generative model comprises: a target UNet comprising a cross attention component; and a reference UNet configured to provide cross attention guidance to the cross attention component. In this way, the rendering process may be guided by the cross attention guidance.

[0043] BRIEF DESCRIPTION OF THE FIGURES

[0044] The present invention will now be described by way of example with reference to the accompanying drawings. In the drawings: Figure 1 is a schematic illustration of the procedure of a prior art system.

[0045] Figure 2 is a schematic illustration of a top level view of the procedure of the present system.

[0046] Figure 3 is a schematic illustration of a point map.

[0047] Figure 4 is a schematic illustration of a detailed view of the procedure of the present system.

[0048] Figure 5a is a schematic illustration of a top level view of the generative model. Figure 5b is a schematic illustration of a cross attention component of the generative model.

[0049] Figure 6 is a schematic illustration of the procedure of the present system during training.

[0050] Figure 7 is a schematic illustration of the procedure of the present system during inference.

[0051] Figure 8 is a schematic illustration of the procedure of the present system during inference for multiple reference images. Figure 9a is a schematic illustration of a detailed view of the generative model for multiple reference signals / images. Figure 9b is a schematic illustration of cross attention guidance for multiple reference signals / images.

[0052] Figure 10 is a schematic illustration of the procedure of the present system during inference for auto-regressive generation. Figure 11 is a schematic illustration of an example implementation of the present system.

[0053] Figure 12 shows an example of a first computer implemented method of the procedure of the present system.

[0054] Figure 13 shows an example of a second computer implemented method of the procedure of the present system.

[0055] Figure 14 shows an example of a computer apparatus configured to perform the methods described herein.

[0056] DETAILED DESCRIPTION

[0057] The apparatus and method described herein concern generating a rendered image of a 3D scene. Embodiments of the present invention may tackle one or more of the problems previously mentioned by: generating the rendered image in dependence on the reference image, the reference point map and the target point map. In this way, the rendering process may be guided by the positions in the point maps.

[0058] Embodiments of the present invention may tackle one or more of the problems previously mentioned by: the reference UNet being configured to provide cross attention guidance to the cross attention component. In this way, the rendering process may be guided by the cross attention guidance.

[0059] The following acronyms are used herein:

[0060] Neural Radiance Field - NeRF

[0061] 3D Gaussian Splatting - 3DGS

[0062] Degree of Freedom - DoF

[0063] Structure from Motion - SIM

[0064] Multi- View Stereo - MVS

[0065] Stable Diffusion - SD

[0066] The following key terms are used herein:

[0067] Rendering: (2D rendering): the process of creating a 2D representation of a 3D environment from a specific location and point of view.

[0068] Pose: 6 DoF position and orientation of a camera or an object.

[0069] Image Frame: 2D grid, where on each position on the grid a punctual information is stored (such as color for an image). Diffusion: Generative process that iteratively transforms random noise into coherent data by reversing a noise-adding process.

[0070] The present system may leverage a generative model conditioned on both the visual and geometric properties of the scene to generate novel views that remain consistent with observed regions while plausibly filling in unseen areas with coherent hallucinated content. By maintaining coherence with the original scene, the model may enable enhanced neural rendering reconstructions, supporting broader view extrapolation. Produced images may be used to enhance or augment neural rendering reconstruction to enable larger view extrapolation and free vehicle displacement within reconstructed scene.

[0071] The present system may propose a new strategy to train diffusion models that requires a fewer number of parameters to train when compared to SoTA solutions while improving consistency on output images. The present system may introduce a new geometric signal encoding (referred to as PointMap, or point map) that is expressed in a unified reference frame which provides explicit matching information and enables multi- frame conditioning without additional training.

[0072] The present system may produce 2D RGB images at novel viewpoints that are consistent to the visual and geometric content of the original 3D scene. This may be done by leveraging 2D diffusion model guided by reference visual and geometric conditioning of observed frames and target geometry of target frame. The present system model may comprise a novel multibranch diffusion architecture for feedforward consistent NVS.

[0073] The present system may rely on a reference image for photometric guidance. The present system may rely on the usage of PointMaps for geometric guidance which may provide explicit matching information. All PointMaps may be expressed on a target reference frame, which enables the use of many reference images without additional training burden. Furthermore, in contrast to existing solutions, the present system may not need to fine-tune the UNet branches as the input to the model remains unchanged, permitting the exploitation of up-to-date generative models in a modular fashion. The general overview of the present system is presented in Figure 2 and a detailed architecture is shown in Figure 4. Referring to Figure 2, a top level pipeline 200 is shown. The pipeline 200 may receive a number of inputs, 201, 202, 203, 204. The pipeline 200 may output a rendered image 206.

[0074] The pipeline 200 may receive reference signals 201. The reference signals 201 may include a reference image 202. The reference image 202 may be an RGB image 202. The reference image 202 may be an image of a 3D scene from a reference viewpoint. The reference signals 201 may include a reference point map 203. The reference point map 203 may correspond to the reference image 203. The reference point map 203 may be a geometric point map 203 encoded in the target reference frame.

[0075] The pipeline 200 may receive target signals. The target signals may include a target point map 204. The target point map 204 may correspond to the rendered image 206. The target point map 204 may be a geometric point map 204 encoded on the projection (if 3D is known) or the warp of the source geometric information of the 3D scene referenced from the target viewpoint 701.

[0076] The pipeline 200 may output a rendered output image 206. The pipeline 200 generate the rendered image 206 in dependence on the reference image 202, the reference point map 203 and the target point map 204. The reference image 202, the reference point map 203 and the target point map 204 may form the inputs and the rendered image 206 may form the output. The pipeline 200 may comprise various components to generate the rendered image 206 from the inputs. The pipeline 200 may comprise a generative model 205. The generative model 205 may generate the rendered image 206.

[0077] Figure 3 illustrates the point map structure. PointMap: Given the geometry of a 3D scene, a PointMap 302 comprises a rasterization of the 3D point coordinates (xyz) 304 of a given point 305 projected into the 2D image space as a function of the extrinsic and intrinsic parameters of the camera within the 3D scene. Put another way, the frame 301 comprises points 303, which may be pixels. The point map 302 comprises points 304. Each of the points 304 comprise coordinates of the point 305 representing the pixel 303. Put simply, the reference point map 203 may comprise the coordinates of a plurality of pixels 303 of the reference image 202, and the target point map 204 may comprise the coordinates of a plurality of pixels 303 of the rendered image 206.

[0078] The use of the PointMaps 203, 204 may provide the following advantages:

[0079] • PointMap as explicit matching information. Considering two images A & B, with their corresponding PointMaps, where both PointMaps are expressed in the frame of image A; computing the nearest neighbor (NN) search in the 3D PointMap space can give correspondences between pixels of two images A and B.

[0080] • Rasterization to directly encode positional features into 2D image space. PointMap representation may enable direct projection of the 3D coordinates of a point into pixel space through rasterization. Since PointMaps are normalized, the method may be robust to different geometric signals such as pointclouds or depth maps (metric and non-metric) although they may present different densities or scales.

[0081] • Provide geometric information of the scene. In contrast to methods that warp RGB information according to the geometric signal, the PointMap may directly encode 3D geometry of the scene. This may make it particularly useful for incorporating 3D awareness into diffusion models without compromising their 2D semantic quality.

[0082] The advantage of PointMap representation is that it may provide a stronger geometric signal compared to RGB warping or 2D coordinates projection, providing explicit 3D to 3D correspondences through PointMap conditioning. Therefore, the present use of PointMaps 203, 204 can recover from inaccuracies in the scene geometry or create specular effect in the generate view as it is not constraint to map directly reference pixels to target pixels. Analogous to works that perform image warping of features according to geometric signal, the present system may project all geometric information into the target reference frame which additionally enables to condition on multiple reference frames without computational burden.

[0083] Referring to Figure 4, a detailed pipeline 200 is shown. The pipeline 200 may receive a number of inputs, 201, 202, 203, 204. The pipeline 200 may output a rendered image 206. Figure 4 illustrates the details of the generative model 205.

[0084] The architecture of the present system may comprise two frozen UNet branches, such as Stable Diffusion (SD) UNet branches, reference 403 and target 402, guided by a shared ControlNet 401, 402 conditioned on PointMap 203, 204 to learn pixel correspondences. The target UNet 402 may generate the novel view, geometrically conditioned with the target PointMap 204, while features from the reference UNet 403 may be injected via cross-attention layers 505 at multiple scales. This architecture may be compatible with 3D reconstruction methods, enabling few-shot capabilities and scene extrapolation.

[0085] Unlike existing methods that fine-tune the UNets to adapt to the guidance signal, the target UNet 402 may only take the original latent noise input 407, avoiding the need to fine-tune the whole architecture. Instead, the pipeline 200 may introduce the geometric guidance signal 410, 411 of the PointMaps 203, 204 through features produced by two shared-weight ControlNets 401 , 402, which may serve as conditioning networks to steer the image generation. This approach may “control” the generative process to achieve the desired output characteristics.

[0086] The generative model 205 may comprise a reference UNet 403. The reference UNet 403 may be configured to provide cross attention guidance 412 to the cross attention component 505 (see Figure 5b). The reference UNet 403 may receive a reference feature image 408. The reference feature image 408 may correspond to the reference image 202. The reference feature image 408 may be a latent image 408. The latent image 408 may be generated from the reference image 202 by a first video auto encoder (VAE) component 405. The reference UNet 403 may receive reference guidance 411 from a first ControlNet 404. The reference guidance 411 may correspond to the reference viewpoint and / or the reference point map 203. The reference point map 203 may be input in the first ControlNet 404. The first ControlNet 404 may determine the reference guidance 411 from the reference point map 203. The reference UNet 403 may generate the cross attention guidance 412 in dependence on the reference feature image 408 and the reference guidance 411.

[0087] The generative model 205 may comprise a target UNet 402. The target UNet 402 may comprise a cross attention component 505 (see Figure 5b). The target UNet 402 may be configured to generate a denoised feature image 409. The target UNet 402 may receive input noise 407. The input noise 407 may corresponding to the target viewpoint 701. The input noise 407 may be generated from the target point map 204. The target UNet 402 may receive target guidance 410 from a second ControlNet 401. The target guidance 410 may corresponding to the target viewpoint 701 and / or the target point map 204. The target point map 204 may be input in the second ControlNet 401. The second ControlNet 401 may determine the target guidance 410 from the target point map 204. The target UNet 402 may generate the denoised feature image 409 in dependence on the input noise 407, the target guidance 410, and the cross attention guidance 412. The denoised feature image 409 may be a latent image 409.

[0088] The pipeline 200 may comprise a second VAE component 406. The second VAE component 406 may generate the rendered image 206 by decoding the denoised feature image 409.

[0089] This VAE components 405, 406 may be specific to a Stable diffusion model: the Unet 402, 403 may take as input a latent image representation 408 and also generate a latent image representation 409. This latent image 408, 409 may be decoded (or encoded) using a VAE 405, 406. This may enable faster and more stable training because the latent image 408, 409 may be smaller compared to the original images. Other generative models may be used which may not require the use of a VAE to convert from image space to latent space. Figure 5a is a top level view of the target UNet 402 and the reference UNet 403 of the generative model 205. Figure 5b is a detailed view of the target UNet 402 and the reference UNet 403 of the generative model 205, including their interaction through the cross attention module 505.

[0090] To enable guidance flow from the reference signals (RGB frames 202 and corresponding PointMaps 203, 204) to the target UNet 402, the pipeline 200 may inject multi-scale, features from the reference UNet 403. This may be achieved through crossattention 505 and zero-convolution layers 507 added at each resolution of the target UNet 402 decoder. This setup may facilitate stable initial guidance by preserving the initial UNet output during early iterations, thanks to the zero-convolution layers 507. In this setup, the only trainable parameters may be the shared- weight ControlNet 401, 402, along with the multi-resolution cross-attention 505 and zero-convolution layers 507. This approach may reduce the trainable parameters by over 50% compared to fully fine-tuning the SD UNet model, resulting in faster training.

[0091] The target UNet 402 may comprise in the following order: a first layer 501, the cross attention component 505, a zero convolution layer 507 and a second layer 502. The reference UNet 403 may comprise in the following order: a first layer 503 and a second layer 504. The cross attention guidance 412 may be provided to the cross attention component 505 after the first layer 503 of the reference UNet 403. The cross attention guidance 412 may be provided to the cross attention component 505 before the second layer 504 of the reference UNet 403.

[0092] Figure 6 illustrates a training pipeline 600 for the present system. The training pipeline 600 may include all the features of the pipeline 200. The pipeline 200 may be an inference pipeline 200. The training pipeline 600 may operate in the same way at the inference pipeline 200.

[0093] The training pipeline 600 may compute reference and target PointMaps 203, 204 for different sequences of major autonomous driving datasets. PointMaps 203, 204 may be obtained from 3D geometric cues provided or computed for each dataset (e.g. pointclouds, depth maps) or calculated 3D scene structure (e.g. mesh) from different means (e.g. neural radiance field, MVS). The pipeline 600 may be trained on a classical diffusion fashion by teaching the network to predict the amount Gaussian noise added to the output target image at different steps for progressive denoising conditioned on the target and reference PointMaps 203, 204. The model may learn to reverse this process by predicting and removing noise at each step, denoising an image back to its original form from pure noise.

[0094] The training pipeline 600 may be further configured to obtain a target image 601. The target image 601 may be the image on which the pipeline 600, specifically the generative model 205, may be trained on. The target image 601 may be the image the pipeline 600 is trying to output. The training pipeline 600 may generate the rendered image 206 in the same way as the inference pipeline 200. The rendered image 206 may be compared with the target image 601 to calculate a loss 605. The loss 605 may be a quantitative difference between the target image 601 and the rendered image 206. The generative model 205 may be updated based on the loss 605. The generative model 205 may be updated so that future outputted rendered images 206 are closer to the target image 601.

[0095] Figure 7 illustrates the inference pipeline 200. At inference time, the target viewpoint 701 may be used to generate PointMaps 203 in target reference frame and the output image 206 may be predicted from pure noise conditioned on target and reference PointMaps 203, 204 along with reference RGB images 202.

[0096] The reference point map 203 may be received by the pipeline 200. Alternatively, the pipeline 200 may generate the reference point map 203. Figure 7 illustrates the generation of the reference point map 203. The pipeline 200 may receive the reference viewpoint and / or the 3D scene. The pipeline 200 may generate the reference point map 203 from the reference viewpoint and / or the 3D scene. For example, a monocular depth estimation may be used. In this case, the reference point map 203 could be obtained from the reference image 202, without need of prior 3D knowledge. A point map computation unit 602 may be used. Optionally, the 3D scene structure 603 may be used to generate the reference point map 203.

[0097] The target point map 204 may be received by the pipeline 200. Alternatively, the pipeline 200 may generate the target point map 204. Figure 7 illustrates the generation of the target point map 204. The pipeline 200 may receive the target viewpoint 701 and / or the 3D scene. The pipeline 200 may generate the target point map 204 from the target viewpoint 701 and / or the 3D scene. For example, a monocular depth estimation may be used. In this case, the target point map 204 could be obtained from the target viewpoint 701 and reference image 202, without need of prior 3D knowledge. A point map computation unit 602 may be used. Optionally, the 3D scene structure 603 may be used to generate the target point map 204.

[0098] Figure 8 illustrates a pipeline 800 outputting the rendered image 206 based on a plurality of reference images 202.

[0099] The pipeline 800 may include all the features of the pipeline 200. The pipeline 800 may receive a plurality of reference images 202 of the 3D scene. Each reference image 202 may be from a different reference viewpoint. F or example, each of the reference images 202 may have been captured from different camera position and / or angles.

[0100] The pipeline 800 may also receive a plurality of reference point maps 203. Each reference point map 203 may correspond to a respective reference image 202. The reference point maps 203 may have been received by the pipeline 200. Alternatively, as described with reference to Figure 7 (and as shown in Figure 8), the reference point maps 203 may be determined by the pipeline 200 (specifically by the point map computation unit 602).

[0101] The pipeline 800 may generate the rendered image 206 in dependence on the plurality of reference images 202, the plurality of reference point maps 203 and the target point map 204. The generative model 205 may generate the rendered image 206. Each of the reference images 202 and reference point maps 203 may provide different information to the generative model 205, such as further views and scene information. Based on this different information, the generative model 205 may be able to produce a more accurate image. This is because the generative model 205 can be confident on a higher proportion of the output image 206, with less of the imagine being hallucinated.

[0102] Figure 9a illustrates a detailed view of the generative model 205 with a plurality of reference UNets 403. Figure 9b illustrates how the cross attention guidance 412 is generated.

[0103] A key advantage of the present system may be the ability to use multiple reference frames 202. This is because the pipeline 200 may encode all reference and target geometry into target reference frame. This may contrast from existing methods that use the reference frame instead. Multi-reference conditioning may be achieved by passing each frame through the reference UNet 403 and fusing multi-scale features of all reference features through a cross-attention operation 412. This operation adjusts the features to the appropriate dimensions for injection into the cross-attention layer 505 of the target UNet 402, regardless of the number of reference frames 202 used. Furthermore, the pipeline 200 may be designed to be trained with single reference images 202, yet can leverage multiple reference 202 frames during inference without additional training, thanks to the flexibility of the cross-attention mechanism 412.

[0104] The cross attention guidance 901 may be produced as shown in Figure 9b. A cross attention matrix 901 may be formed from features of the target frame 902 and features of the reference images 903, 904. The dot product of this matrix 901 may produce features of the reference image 1 905 and features of the reference image N 906. Figure 10 illustrates a pipeline 1000 for generating a further rendered image 1002 based on the rendered image 206. Essentially, the pipeline 200 may be run to generate a rendered image 206, and the pipeline 200 may be run for a second time to generate a further rendered image 1002. This process may be repeated a plurality of times. This process may allow the rendered images 206, 1002 to progressively move further away from the reference image 202. This may enable significantly different views to be produced. This may allow auto-regressive generation to be carried out.

[0105] The auto-regressive generation may be carried out by using output diffusion images 206 as reference 202 and conditioning for following target viewpoints. The pipeline 1000 may be capable of generating output images 1002 on a target viewpoint that are geometrically and photometrically consistent with the 3D scene content. This permits to use the output images of the diffusion model to further improve neural rendering models (NeRF or 3DGS) to result in larger scene extrapolation and vehicle free control on reconstructed 3D scenes for photorealistic simulation.

[0106] The first pipeline 200 may generate the rendered image 206 as described herein from the reference image 202 and the target point map 204.

[0107] The pipeline 1000 may be further configured to obtain a further target point map corresponding to a further rendered image 1002. The further target point map may comprise the coordinates of a plurality of pixels of the further rendered image 1002. The rendered image 206 may also be inputted as an input image.

[0108] The second pipeline 200 may generate the further rendered image 1002 in dependence on the rendered image 206, the target point map 204 and the further target point map. The target point map 204 may essentially function as a reference point map 204 in the second running of the pipeline 200.

[0109] Figure 11 illustrates an example embodiment of the present system.

[0110] The proposed embodiment is composed by the generation of new frames at target viewpoint that are consistent with reference geometry and photometric guidance and are aligned with target geometry enabling large Novel View Synthesis (NVS) with larger capabilities than existing neural rendering techniques. Resulting images are then used to augment and improve neural rendering methods such as NeRFs or 3DGS to improve rendering quality and unlock larger extrapolation capabilities and free vehicle control.

[0111] The initial setup may be done as follows:

[0112] • Consider a new / unobserved frame viewpoint with some overlap to acquired data from the reference RGB frame or frames 202.

[0113] • Consider 3D geometry information of the scene computed with a pair of images or using an external 3D representation of the scene or even using depth information from monocular solution.

[0114] • Calculate PointMaps 203, 204 from given 3D information 603 and project reference and target PointMaps 203, 204 to target reference frame.

[0115] • Perform inference using the network 205 to generate the novel viewpoint from reference and target conditioning information and use this image 206 along with pose information to fine-tune neural rendering model.

[0116] • This process can be done iteratively to produce many viewpoints and improve 3D scene reconstruction, enabling free control photorealistic simulation. The architecture may be trained faster than other methods thanks to the drastically lower number of parameters compared to training UNet branches.

[0117] The proposed implementation may render new frames that remain consistent with reference 3D scene thanks to the design choices and reference RGB frame and reference / target PointMap consisting on geometric features as conditioning signal for a ControlNet model. The method uses a two branch UNet architecture where information from the reference frame is shared to a zero-conv initialized cross attention layer.

[0118] The principal innovative part of the present system may be the architecture for the guided consistent diffusion for photorealistic simulation, in this section secondary additional embodiments are described:

[0119] Direct geometric guidance through PointMap: using PointMap as geometric guidance improves geometric consistency with 3D scene compared to other methods.

[0120] Multi-reference conditioning: The architecture enables multi-reference conditioning thanks to the warping of all reference and target geometric information into target reference frame.

[0121] Scene extrapolation (with or without autoregressive approach): As the present solution relies on 2D diffusion models trained on internet-scale data, the method may be capable to hallucinate plausible content outside scene observed region and permit larger extrapolation in auto-regressive fashion.

[0122] 3D asset generation: Consider few shots from object to be reconstructed, sampling cameras around the object and using the model can enable 3D asset generation by using such information for neural rendering methods.

[0123] 3D scene modification: The content of the scene can be willingly modified to some limited extent by modifying directly the target PointMap which enables for instance to displace / remove / add objects within the scene by controlling PointMap information.

[0124] The following may provide the main innovative points of the present system:

[0125] ControlNet neural network used to condition the output on the target and reference PointMaps and cross-attention layers added to target UNet to process reference features: This solution enables to reduce the training parameters of the network by more than 50% compared to existing solutions and reduce training time while producing more consistent outputs in urban scenarios.

[0126] PointMaps used as conditioning signal for geometric guidance: The PointMaps directly provide 3D guidance information which improves the quality of the final output and provides direct 3D to 3D matching correspondences. Furthermore, PointMaps present a flexible solution as can be dense or sparse, computed with a pair of images or using an external 3D representation of the scene or even using depth information from monocular solution.

[0127] PointMaps are encoded into the target reference frame providing a common frame for multiple reference signal per target: Given the encoding of the PointMaps of the target reference frame and the cross-attention dot product operation we can use multiple reference frames without further training the network. Hence the network can be trained with single reference conditioning and many frames can be used for inference.

[0128] Figure 12 summarises an example of a first method 1200. At step 1201, the method 1200 comprises obtaining a reference image of a 3D scene from a reference viewpoint. At step 1202, the method 1200 comprises obtaining a reference point map corresponding to the reference image, the reference point map comprising the coordinates of a plurality of pixels of the reference image. At step 1203, the method 1200 comprises obtaining a target point map corresponding to the rendered image, the target point map comprising the coordinates of a plurality of pixels of the rendered image. At step 1204, the method 1200 comprises generating the rendered image in dependence on the reference image, the reference point map and the target point map.

[0129] Figure 13 summarises an example of a second method 1300. At step 1301, the method 1300 comprises obtaining a reference image of a 3D scene from a reference viewpoint. At step 1302, the method 1300 comprises obtaining a target viewpoint of the 3D scene corresponding to the rendered image. At step 1303, the method 1300 comprises generating the rendered image in dependence on the reference image and the target viewpoint by means of a generative model. The generative model comprises: a target UNet comprising a cross attention component; and a reference UNet configured to provide cross attention guidance to the cross attention component.

[0130] An example of computer apparatus 1400 configured to implement the first method 1200 and / or the second method 1300 is schematically illustrated in Figure 14. The computer apparatus 1400 may carry out the procedures 300, 300 and 600 illustrated in Figures 2 to 11. The computer apparatus 1400 may be implemented on an electronic device, such as a computer, a computer for a vehicle, laptop, tablet, or smart phone. The computer apparatus 1400 may be connected to the internet.

[0131] The computer apparatus 1400 comprises a processor 1401 configured to process the datasets in the manner described herein. For example, the processor 1401 may be implemented as a computer program running on a programmable device such as a Central Processing Unit (CPU). The computer apparatus 1400 comprises a memory 1402 which is arranged to communicate with the processor 1401. Memory 1402 may be a non-volatile memory. The processor 1401 may also comprise a cache (not shown in Figure 14), which may be used to temporarily store data from memory 1402. The computer apparatus 1400 may comprise more than one processor 1401 and more than one memory 1402. The memory 1402 may store data that is executable by the processor 1401. The processor 1401 may be configured to operate in accordance with a computer program stored in non-transitory form on a machine-readable storage medium. The computer program may store instructions for causing the processor to perform its methods in the manner described herein. The method steps described herein may be carried out by a computer-readable storage medium. The method steps described herein may be carried out by a computer program product.

[0132] The applicant hereby discloses in isolation each individual feature described herein and any combination of two or more such features, to the extent that such features or combinations are capable of being carried out based on the present specification as a whole in the light of the common general knowledge of a person skilled in the art, irrespective of whether such features or combinations of features solve any problems disclosed herein, and without limitation to the scope of the claims. The applicant indicates that aspects of the present invention may consist of any such individual feature or combination of features. In view of the foregoing description, it will be evident to a person skilled in the art that various modifications may be made within the scope of the invention.

Claims

CLAIMS1. A computer apparatus (1400) for generating a rendered image (206), the computer apparatus (1400) comprising one or more processors (1401) and a memory (1402) storing in non-transient form data defining program code executable by the one or more processors (1401), wherein the program code is executable by the one or more processors (1401) so that the computer apparatus (1400) is configured to:obtain a reference image (202) of a 3D scene from a reference viewpoint;obtain a reference point map (203) corresponding to the reference image (202), the reference point map (203) comprising the coordinates of a plurality of pixels of the reference image (202);obtain a target point map (204) corresponding to the rendered image (206), the target point map (204) comprising the coordinates of a plurality of pixels of the rendered image (206); andgenerate the rendered image (206) in dependence on the reference image (202), the reference point map (203) and the target point map (204).

2. A computer apparatus (1400) according to claim 1, wherein the apparatus (1400) is configured to:generate the reference point map (203) from the reference viewpoint and / or the 3D scene; and / orgenerate the target point map (204) from a target viewpoint (701 ) of the 3D scene and / or the 3D scene.

3. A computer apparatus (1400) according to claim 2, wherein the apparatus (1400) is configured to generate the reference point map (203) and / or the target point map (204) in dependence on a 3D scene structure (603).

4. A computer apparatus (1400) according to any preceding claim, wherein the apparatus (1400) is configured to: obtain a plurality of reference images (202) of the 3D scene, each reference image (202) being from a different reference viewpoint;obtain a plurality of reference point maps (203) corresponding to a respective plurality of reference images (202); andgenerate the rendered image (206) in dependence on the plurality of reference images (202), the plurality of reference point maps (203) and the target point map (204).

5. A computer apparatus (1400) according to any preceding claim, wherein the apparatus (1400) is configured to: obtain a further target point map corresponding to a further rendered image (1002), the further target point map comprising the coordinates of a plurality of pixels of the further rendered image (1002); andgenerate the further rendered image (1002) in dependence on the rendered image (206), the target point map (204) and the further target point map.

6. A computer apparatus (1400) according to any preceding claim, wherein the apparatus (1400) is configured to generate the rendered image (206) by means of a generative model (205).

7. A computer apparatus (1400) for generating a rendered image (206), the computer apparatus (1400) comprising one or more processors (1401) and a memory (1402) storing in non-transient form data defining program code executable by the one or more processors (1401), wherein the program code is executable by the one or more processors (1401) so that the computer apparatus (1400) is configured to:obtain a reference image (202) of a 3D scene from a reference viewpoint;obtain a target viewpoint (701) of the 3D scene corresponding to the rendered image (206); andgenerate the rendered image (206) in dependence on the reference image (202) and the target viewpoint (701) by means of a generative model (205);wherein the generative model (205) comprises:a target UNet (402) comprising a cross attention component (505); anda reference UNet (403) configured to provide cross attention guidance (412) to the cross attention component (505).

8. A computer apparatus (1400) according to claim 7, wherein:the target UNet (402) comprises in the following order: a first layer (501), the cross attention component (505), a zero convolution layer (507) and a second layer (502); andthe reference UNet (403) comprises in the following order: a first layer (503) and a second layer (504); wherein the cross attention guidance (412) is provided to the cross attention component (505) after the first layer (503) of the reference UNet (403).

9. A computer apparatus (1400) according to claim 7 or 8, wherein the reference UNet (403) is configured to:receive a reference feature image (408), the reference feature image (408) corresponding to the reference image (202);receive reference guidance (411) from a first ControlNet (404), the reference guidance (411) corresponding to the reference viewpoint; andgenerate the cross attention guidance (412) in dependence on the reference feature image (408) and the reference guidance (411).

10. A computer apparatus (1400) according to any of claims 7 to 9, wherein the target UNet (402) is configured to:receive input noise (407), the input noise (407) corresponding to the target viewpoint (701); receive target guidance (410) from a second ControlNet (401), the target guidance (410) corresponding to the target viewpoint (701); andgenerate a denoised feature image (409) in dependence on the input noise (407), the target guidance (410), and the cross attention guidance (412).

11. A computer apparatus (1400) according to claim 10, wherein the apparatus (1400) is configured to generate the rendered image (206) by rendering the denoised feature image (409).

12. A computer apparatus (1400) according to claim 6, comprising the generative model (205) according to any of claims 7 to 11.

13. A computer apparatus (1400) according to any of claims 6 to 12, wherein the apparatus (1400) is further configured to:obtain a target image (601);compare the rendered image (206) with the target image (206) to calculate a loss (605); and update the generative model (205) in dependence on the loss (605).

14. A method (1200) for generating a rendered image, carried out by a computer apparatus, comprising the steps of:obtaining a reference image of a 3D scene from a reference viewpoint (1201);obtaining a reference point map corresponding to the reference image, the reference point map comprising the coordinates of a plurality of pixels of the reference image (1202);obtaining a target point map corresponding to the rendered image, the target point map comprising the coordinates of a plurality of pixels of the rendered image (1203); andgenerating the rendered image in dependence on the reference image, the reference point map and the target point map (1204).

15. A method (1300) for generating arendered image, carried out by a computer apparatus, comprising the steps of:obtaining a reference image of a 3D scene from a reference viewpoint (1301);obtaining a target viewpoint of the 3D scene corresponding to the rendered image (1302); andgenerating the rendered image in dependence on the reference image and the target viewpoint by means of a generative model (1303);wherein the generative model comprises:a target UNet comprising a cross attention component; anda reference UNet configured to provide cross attention guidance to the cross attention component.16

Images