Harmonization of appearance of multi-view images for 3D scene reconstruction

The method uses a neural network to predict patch-specific bilateral grids and confidence scores for harmonizing multi-view images, addressing photometric inconsistencies in 3D scene reconstruction, enhancing reconstruction quality and stability by decoupling appearance harmonization from geometric optimization.

WO2026238250A1PCT designated stage Publication Date: 2026-11-19HUAWEI TECH CO LTD +1
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
PCT/EP2025/063495
Authority / Receiving Office
WO · WO
Patent Type
Applications
Current Assignee / Owner
Filing Date
2025-05-16
Publication Date
2026-11-19

AI Technical Summary

Technical Problem

Existing 3D scene reconstruction techniques like NeRF and 3DGS face challenges due to photometric inconsistencies arising from varying camera settings and scene illumination, leading to degraded reconstruction quality and artifacts, particularly when appearance embedding is optimized jointly with scene reconstruction, resulting in prolonged training times and susceptibility to local minima.

Method used

A method involving a neural network model, such as a transformer, predicts patch-specific bilateral grids and confidence scores to harmonize image appearance independently, followed by a decoupled 3D reconstruction process, using global and cross-attention mechanisms to ensure multi-view consistency and stability.

Benefits of technology

This approach reduces processing time and complexity while improving the quality and robustness of 3D reconstructions by decoupling appearance harmonization from geometric optimization, effectively handling photometric inconsistencies and reducing artifacts.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure EP2025063495_19112026_PF_FP_ABST
    Figure EP2025063495_19112026_PF_FP_ABST
Patent Text Reader

Abstract

In some examples, methods and apparatus for harmonizing appearance of multi-view images for 3D scene reconstruction is provided.
Need to check novelty before this filing date? Find Prior Art

Description

[0001] HARMONIZATION OF APPEARANCE OF MULTI- VIEW IMAGES FOR 3D SCENE RECONSTRUCTION

[0002] TECHNICAL FIELD

[0003] The present disclosure relates, in general, to three-dimensional scene reconstruction and inverse rendering. More specifically, although not exclusively, it relates to methods and systems for performing fast and generalizable appearance embedding for scenes captured under varying appearance and lighting conditions.

[0004] BACKGROUND

[0005] Three-dimensional (3D) scene reconstruction techniques, such as Neural Radiance Fields (NeRF) and 3D Gaussian Splatting (3DGS), often rely on multi-view images of a scene. A critical requirement for achieving high-quality reconstructions with these methods is photometric consistency across the input images. However, in real-world scenarios, photometric inconsistencies frequently arise. These inconsistencies can be caused by various factors, including, but not limited to, changes in camera internal signal processor (ISP) settings (e.g., exposure, white balance), variations in scene illumination (e.g., changing lighting conditions, shadows), and the presence of dynamic occluders (e.g., moving objects within the scene).

[0006] When 3D reconstruction techniques like 3DGS are optimized using images with varying exposure conditions and other photometric inconsistencies, the quality of the resulting reconstruction can be significantly degraded. Artifacts such as floaters or unstable colour values may appear, diminishing the accuracy and visual fidelity of the 3D model.

[0007] Appearance embedding aims to harmonize the appearance of input images. Many of these prior efforts have concentrated on scene-specific appearance embedding. In such methods, colour changes are learned on a per- view basis concurrently with the optimization of Gaussian primitives or radiance fields. While these scene-specific methods can achieve some success in modelling appearance variations, they suffer from several drawbacks. They are often slow during the training phase due to the joint optimization, and they are susceptible to converging to local minima, especially when appearance information is difficult to infer. This makes them inefficient for applications where rapid scene reconstruction is paramount.

[0008] SUMMARY

[0009] An objective of the present disclosure is to provide mechanisms for harmonizing or canonicalizing the appearance of multiview images, particularly for use in 3D scene reconstruction processes.

[0010] The foregoing and other objectives are achieved by the features of the independent claims.

[0011] Further implementation forms are apparent from the dependent claims, the description and the Figures.

[0012] A first aspect of the present disclosure provides a method for harmonizing appearance of multi-view images for 3D scene reconstruction, the method comprising receiving a plurality of input images of a scene, wherein at least one input image exhibits a photometric inconsistency relative to other input images in the plurality, receiving a reference frame, the reference frame defining a set of target photometric parameters for harmonizing the plurality of input images, for each of at least a subset of the input images, referred to as source frames dividing the source frame into a plurality of image patches, for each image patch of the plurality of image patches, predicting, using a neural network model a respective patch-specific bilateral grid, the respective patch-specific bilateral grid encoding an appearance transformation function to align an appearance of the image patch with the set of target photometric parameters defined by the reference frame, and a respective patch-specific confidence score, the respective patch-specific confidence score indicating a reliability of the predicted appearance transformation for the image patch, applying the plurality of predicted patch- specific bilateral grids to the source frame to generate a corrected source frame having an appearance harmonized according to the set of target photometric parameters, wherein each predicted patch-specific bilateral grid transforms its corresponding image patch within the source frame, and providing the corrected source frame and a confidence map derived from the plurality of predicted patch-specific confidence scores for use in a 3D scene reconstruction process.

[0013] Processing images on a patch-wise basis to predict patch- specific bilateral grids and confidence scores allows for localized appearance transformations and reliability assessments, leading to more nuanced and accurate harmonization of images with varying photometric inconsistencies before their use in 3D reconstruction.

[0014] The separation of the appearance harmonization process as a distinct, feed-forward step, prior to or concurrently with but decoupled from the 3D scene reconstruction optimization, can significantly reduce overall processing time and complexity compared to approaches that jointly optimize appearance and geometric parameters, while also potentially improving the stability of the geometric reconstruction by preventing appearance-related optimization challenges from negatively influencing it.

[0015] Generating and providing a confidence map alongside the corrected target frame enables the subsequent 3D scene reconstruction process to intelligently account for varying reliability in the harmonized appearance across different image regions, leading to more robust and higher-quality 3D models with potentially fewer artifacts, particularly when input images contain severely degraded or challenging areas.

[0016] In an implementation of the first aspect, the set of target photometric parameters can comprise at least one of white balance, exposure level, or colour grading.

[0017] Defining target photometric parameters to include specific aspects like white balance, exposure level, or colour grading provides precise control over the desired appearance of the harmonized images, ensuring consistency across these key visual characteristics.

[0018] The neural network model can comprise a transformer model. Employing a transformer model for predicting bilateral grids and confidence scores leverages its strong capabilities in capturing long-range dependencies and contextual information within and across images, enhancing the accuracy and generalizability of the appearance harmonization.

[0019] Predicting the respective patch- specific bilateral grid can comprise using a multilayer perceptron, MLP, utilizing a Tanh activation function in at least one of its terminal layers.

[0020] Utilizing a Tanh activation function in the terminal layers of a multilayer perceptron (MLP) responsible for predicting the patch-specific bilateral grid can help in appropriately bounding the output values of the grid, contributing to stable and effective appearance transformations.

[0021] The transformer model can incorporate Layer Normalization within its attention blocks. Incorporating Layer Normalization within the attention blocks of the transformer model helps stabilize the training process and improve the model’s generalization performance by normalizing the inputs to activation functions.

[0022] An encoder of the transformer model can use an alternating self-attention and global attention framework, and wherein the transformer model comprises a single output head configured to predict both the respective patch-specific bilateral grid and the respective patch-specific confidence score.An encoder architecture that utilizes an alternating self-attention and global attention framework, combined with a single output head for predicting both the bilateral grid and confidence score, allows for efficient processing and shared feature learning, leading to a more compact and effective model.

[0023] The method can further comprise using the neural network model, applying global attention across a plurality of the input images simultaneously during processing. Applying global attention across a plurality of input images simultaneously during processing enables the neural network model to achieve multi- view consistency by considering information from all available views when harmonizing a particular image.

[0024] The method can further comprise using a decoder of the transformer model, implementing cross-attention between features derived from the reference frame and features derived from the source frame, the cross-attention replacing global attention within the decoder.

[0025] Implementing cross-attention in the decoder between features from the reference frame and the target frame allows the model to effectively transfer the desired appearance characteristics from the reference to the target, replacing the need for global attention within the decoder for this specific style transfer task.

[0026] The method can further comprise deriving key and value inputs for the cross-attention from the reference frame, and a query input for the cross-attention from the source frame.

[0027] Deriving key and value inputs for cross-attention from the reference frame and the query input from the target frame provides a structured way for the model to learn and apply the appearance transformation defined by the reference.

[0028] The method can further comprise deriving the confidence map from the plurality of predicted patch-specific confidence scores using a slicing or sampling approach.

[0029] A slicing or sampling approach for deriving the confidence map from patch-specific confidence scores offers a practical way to generate a usable map that reflects the overall reliability of the harmonization process for different regions of an image.

[0030] The method can further comprise utilizing the confidence map to weight a photometric loss function during the 3D scene reconstruction process, wherein the loss function is calculated as a product of the confidence map and a photometric difference between a rendered image and a ground truth image.

[0031] Utilizing the confidence map to weight a photometric loss function during the 3D scene reconstruction process allows the reconstruction to prioritize information from more reliably harmonized image regions, leading to improved reconstruction quality, especially in areas with challenging initial exposures.

[0032] The method can further comprise pretraining the neural network model using a loss function comprising an LI loss between a corrected image and a ground truth consistent image, wherein the LI loss is weighted by a confidence map, and applying a total variation loss to the plurality of predicted patch-specific bilateral grids.

[0033] Pretraining the neural network model with a loss function that includes a confidence- weighted LI loss and a total variation loss on the bilateral grids encourages the model to learn accurate and smooth appearance transformations, improving its generalizability and the quality of corrected images.Dividing the source frame into the plurality of image patches can comprise dividing the source frame into a plurality of nonoverlapping image patches. Dividing the target frame into non-overlapping image patches ensures that each part of the image is processed distinctly, preventing redundant computations and simplifying the aggregation of patch- wise results.

[0034] Image processing is performed in the sRGB colour space. Performing image processing in the sRGB colour space ensures consistency with common image formats and display standards, simplifying integration with existing imaging pipelines.

[0035] The reference frame can comprise a first frame of a sequence of the input images or a user-selected image. Allowing the reference frame to be a first frame of a sequence or a user-selected image provides flexibility in defining the target appearance, accommodating various capture scenarios and user preferences for the final harmonized look.

[0036] A second aspect of the present disclosure provides an apparatus for harmonizing appearance of multi- view images for 3D scene reconstruction, the apparatus configured to receive a plurality of input images of a scene, wherein at least one input image exhibits a photometric inconsistency relative to other input images in the plurality, and to receive a reference frame, the reference frame defining a set of target photometric parameters for harmonizing the plurality of input images, the apparatus comprising at least one processor, and at least one memory storing instructions that, when executed by the at least one processor, cause the apparatus to perform operations comprising for each of at least a subset of the input images, referred to as source frames dividing the source frame into a plurality of image patches, for each image patch of the plurality of image patches, predicting, using a neural network model a respective patch-specific bilateral grid, the respective patch-specific bilateral grid encoding an appearance transformation function to align an appearance of the image patch with the set of target photometric parameters defined by the reference frame, and a respective patch- specific confidence score, the respective patch-specific confidence score indicating a reliability of the predicted appearance transformation for the image patch, applying the plurality of predicted patch- specific bilateral grids to the source frame, wherein each predicted patch-specific bilateral grid transforms its corresponding image patch within the source frame, to generate a corrected source frame having an appearance harmonized according to the set of target photometric parameters, and providing the corrected source frame and a confidence map derived from the plurality of predicted patch-specific confidence scores for use in a 3D scene reconstruction process.

[0037] In an implementation of the second aspect, the set of target photometric parameters can comprise at least one of white balance, exposure level, or colour grading. An apparatus that specifically handles target photometric parameters like white balance, exposure, or colour grading is equipped to address common and critical types of appearance inconsistencies, leading to visually coherent 3D models.

[0038] The neural network model can be a transformer model. The transformer model can comprise 3 encoder layers and 3 decoder layers, and utilizes a multi-head attention mechanism comprising 8 attention heads. A transformer model having a defined number of encoder / decoder layers and attention heads is optimized for a balance of performance and computational efficiency in the appearance harmonization task.

[0039] An encoder of the transformer model can use an alternating self-attention and global attention framework, and wherein the transformer model comprises a single output head configured to predict both the respective patch-specific bilateral grid and the respective patch-specific confidence score.

[0040] A decoder of the transformer model can implement cross-attention between features derived from the reference frame and features derived from the source frame, wherein key and value inputs for the cross-attention are derived from the reference frame, and a query input for the cross-attention is derived from the source frame.The instructions can further cause the apparatus to utilize the confidence map to weight a photometric loss function during a 3D scene reconstruction process performed by a 3D scene reconstruction module, the loss function calculated as a product of the confidence map and a photometric difference between a rendered image and a ground truth image.

[0041] Dividing the source frame into the plurality of image patches can comprise dividing the source frame into a plurality of nonoverlapping image patches.

[0042] A third aspect of the present disclosure provides a non-transitory computer-readable medium storing instructions that, when executed by at least one processor, cause an apparatus to perform a method for harmonizing appearance of multi- view images for 3D scene reconstruction as provided according to the first aspect.

[0043] These and other aspects of the invention will be apparent from the embodiments) described below.

[0044] BRIEF DESCRIPTION OF THE DRAWINGS

[0045] In order that the present disclosure may be more readily understood, embodiments will now be described, by way of example, with reference to the accompanying drawings, in which:

[0046] Figure 1 is a schematic representation of a generalizable bilateral grid approach for appearance embedding, according to an example;

[0047] Figure 2 is a schematic representation of method, according to an example.;

[0048] Figure 3 is a schematic representation of method, according to an example; and

[0049] Figure 4 is a schematic representation of a machine, according to an example.

[0050] DETAILED DESCRIPTION

[0051] Example embodiments are described below in sufficient detail to enable those of ordinary skill in the art to embody and implement the systems and processes herein described. It is important to understand that embodiments can be provided in many alternate forms and should not be construed as limited to the examples set forth herein.

[0052] Accordingly, while embodiments can be modified in various ways and take on various alternative forms, specific embodiments thereof are shown in the drawings and described in detail below as examples. There is no intent to limit to the particular forms disclosed. On the contrary, all modifications, equivalents, and alternatives falling within the scope of the appended claims should be included. Elements of the example embodiments are consistently denoted by the same reference numerals throughout the drawings and detailed description where appropriate.

[0053] The terminology used herein to describe embodiments is not intended to limit the scope. The articles “a,” “an,” and “the” are singular in that they have a single referent, however the use of the singular form in the present document should not preclude the presence of more than one referent. In other words, elements referred to in the singular can number one or more, unless the context clearly indicates otherwise. It will be further understood that the terms “comprises,” “comprising,” “includes,” and / or “including,” when used herein, specify the presence of stated features, items, steps, operations, elements, and / or components, but do not preclude the presence or addition of one or more other features, items, steps, operations, elements, components, and / or groups thereof. The term “and / or” is only an association relationship for describing associated objects and represents that three relationships may exist such that A and / or B may indicate that A exists alone, A and B exist at the same time, or B exists alone. The character “ / ” generally represents that the associated objects are in an “or” relationship.Unless otherwise defined, all terms (including technical and scientific terms) used herein are to be interpreted as is customary in the art. It will be further understood that terms in common usage should also be interpreted as is customary in the relevant art and not in an idealized or overly formal sense unless expressly so defined herein.

[0054] The following contains specific information related to implementations of the present disclosure. The drawings and their accompanying detailed disclosure are merely directed to implementations. However, the present disclosure is not limited to these implementations. Other variations and implementations of the present disclosure will be obvious to those skilled in the art.

[0055] The phrases “in one implementation,” or “in some implementations,” may each refer to one or more of the same or different implementations. The term “coupled” is defined as connected whether directly or indirectly through intervening components and is not necessarily limited to physical connections. The expression “at least one of A, B and C” or “at least one of the following: A, B and C” means “only A, or only B, or only C, or any combination of A, B and C.”

[0056] The terms “system” and “network” may be used interchangeably.

[0057] For the purposes of explanation and non-limitation, specific details such as functional entities, techniques, protocols, and standards are set forth for providing an understanding of the present disclosure. In other examples, detailed disclosure of well-known methods, technologies, systems, and architectures are omitted so as not to obscure the present disclosure with unnecessary details.

[0058] Persons skilled in the art will immediately recognize that any network functions) or algorithm(s) disclosed may be implemented by hardware, software or a combination of software and hardware. Disclosed functions may correspond to modules which may be software, hardware, firmware, or any combination thereof.

[0059] A software implementation may include machine- and / or computer- readable and / or executable instructions stored on a machine- and / or computer-readable medium such as memory or other types of storage devices. One or more microprocessors or general-purpose computers with communication processing capability may be programmed with corresponding executable instructions and perform the disclosed network functions) or algorithm(s).

[0060] The microprocessors or general-purpose computers may include Applications Specific Integrated Circuitry (ASIC), programmable logic arrays, and / or using one or more Digital Signal Processor (DSPs). Although some of the disclosed implementations are oriented to software installed and executing on computer hardware, alternative implementations implemented as firmware or as hardware or as a combination of hardware and software are well within the scope of the present disclosure. The computer readable medium includes but is not limited to Random Access Memory (RAM), Read Only Memory (ROM), Erasable Programmable Read-Only Memory (EPROM), Electrically Erasable Programmable Read-Only Memory (EEPROM), flash memory, Compact Disc Read-Only Memory (CD-ROM), magnetic cassettes, magnetic tape, magnetic disk storage, or any other equivalent medium capable of storing computer-readable instructions.

[0061] Some "in the Wild" variants of NeRF and 3DGS employ Multilayer Perceptrons (MLPs) to predict parameters of an affine transformation for adjusting view-dependent colour. These models may also predict uncertainty to handle dynamic objects. However, their scene-dependent nature means that appearance embedding is optimized alongside scene reconstruction, leading to prolonged training times. Another approach, Luminance-GS, predicts tone curves per view during scene reconstruction to generate pseudo-enhanced views. However, its applicability is limited to variations that can be captured by its tone curve model, restricting its flexibility in handling a broader range of real- world appearance shifts.A related method, Bilarf, jointly predicts bilateral grids and radiance fields to capture ISP and other appearance changes. While this approach shows promise, it also relies on scene- specific optimization, coupling appearance embedding with the scene reconstruction process. This not only increases training time but also introduces the risk of overfitting to specific scenes, thereby limiting the model's ability to generalize to new or unseen environments. The tight coupling in these methods between appearance embedding and scene reconstruction generally results in longer training times and an increased risk of converging to suboptimal local minima.

[0062] Systems and methods are described herein for harmonizing the appearance of multi-view images, particularly for use in 3D scene reconstruction processes. The described approaches aim to address photometric inconsistencies that commonly occur in casually captured image sets due to factors such as varying camera settings (e.g., exposure, white balance) and changing scene illumination. Unlike scene-specific methods that optimize appearance embedding jointly with 3D reconstruction, the techniques described herein provide a generalizable, feed-forward model for image harmonization.

[0063] Certain terms and acronyms as used herein, and descriptions for specific terminology and elements relevant to the disclosed methods and systems, to aid in understanding and implementation are provided:

[0064] Appearance Embedding (AE): Refers to the process of modelling and adjusting the visual appearance (e.g., colour, exposure, white balance, style) of images. In the context of this disclosure, it involves harmonizing the appearance of multi- view images to a common target appearance or set of photometric parameters, often as a pre-processing step or integrated feed- forward step for 3D scene reconstruction.

[0065] Bilateral Grid: A data structure, often multi-dimensional (e.g., 3D or higher), used to represent and apply complex, spatially-varying image transformations. In this disclosure, a patch-specific bilateral grid is predicted by a neural network for each image patch. This grid encodes an appearance transformation function that, when sampled or applied to the image patch, adjusts its pixel values (e.g., colour, intensity) to align with target photometric parameters. It allows for non-linear transformations that depend on both spatial location and pixel intensity / colour, enabling sophisticated appearance adjustments.

[0066] Patch- wise Processing: An approach where an image is divided into smaller segments or "patches," and processing is performed individually on each patch. In this disclosure, source frames are divided into a plurality of image patches, and the neural network model predicts a bilateral grid and a confidence score for each patch independently. This allows the model to adapt to local variations in appearance and apply tailored corrections.

[0067] Confidence Map / Confidence Score: A patch- specific confidence score is a value predicted by a neural network model for each image patch, indicating the model's reliability or certainty in the accuracy of the predicted appearance transformation (bilateral grid) for that patch. A confidence map is a 2D map derived from these patch-specific confidence scores, representing the spatial distribution of confidence across the entire corrected target frame. This map can be used in downstream processes, such as weighting the loss function in 3D reconstruction, to give less influence to less reliably corrected regions (e.g., over / under-exposed areas).

[0068] Multi-view Consistency: Refers to the state where the appearance (e.g., colour, lighting, exposure) of objects and surfaces is consistent across multiple images of the same scene taken from different viewpoints and / or at different times. The disclosed methods aim to enhance multi- view consistency by harmonizing all input or source images to a common reference appearance, often using global attention mechanisms in the neural network to consider information across frames.Photometric Inconsistency: Variations in the recorded appearance of a scene across different images and / or times that are not due to changes in the scene's underlying geometry or material properties. These inconsistencies can arise from factors like changing camera ISP settings (exposure, white balance), varying scene illumination (lighting, shadows), or dynamic occluders. Such inconsistencies can degrade the quality of 3D reconstructions.

[0069] Target Photometric Parameters: A set of specific visual characteristics (e.g., a particular white balance setting, exposure level, colour grading style, overall brightness, contrast) defined by a reference frame or explicitly specified, to which the input images are to be harmonized. The goal is to make all processed images appear as if they were captured under these target conditions.

[0070] 3D Gaussian Splatting (3DGS): A 3D scene reconstruction and rendering technique that represents a scene as a collection of 3D Gaussian primitives. Each Gaussian has parameters for its 3D position, orientation (e.g., as a quaternion), scale (anisotropic), colour (often view-dependent, represented by Spherical Harmonics), and opacity. 3DGS allows for high-quality, real-time rendering of photorealistic scenes.

[0071] Neural Radiance Fields (NeRF): A technique for synthesizing novel views of a 3D scene by representing the scene as a continuous volumetric radiance field, typically parameterized by a multilayer perceptron (MLP). The MLP maps 3D coordinates and viewing directions to colour and density.

[0072] Transformer Model: A type of neural network architecture that relies on attention mechanisms, particularly self-attention and cross-attention, to process sequential or set-based data. A transformer model can be used to predict patch-specific bilateral grids and confidence scores. It processes a sequence of input image patches and a reference frame, leveraging attention to capture relationships within and between images to achieve multi- view consistent appearance harmonization.

[0073] Global Attention: An attention mechanism in a neural network (e.g., a transformer) where each element in a sequence can attend to all other elements in that sequence or across multiple related sequences (e.g., across all input frames in a multi-view setup). This allows the model to capture long-range dependencies and context from the entire input, facilitating multi-view consistency in this disclosure.

[0074] Cross-Attention: An attention mechanism where elements from one sequence (e.g., queries from a source frame) attend to elements from another sequence (e.g., keys and values from a reference frame). This can be used in the decoder of a transformer model to transfer the style or appearance characteristics from the reference frame to the target frames.

[0075] Feed- forward Model: A type of neural network or processing pipeline where information flows in one direction, from input to output, without loops or recurrent connections (in the main processing path for a single inference). The appearance harmonization model is described as feed-forward, meaning that once trained, it can process input images and a reference frame to produce corrected images in a single pass without iterative optimization per scene, making it fast for inference.

[0076] Canonicalize (appearance): To transform or normalize the appearance of images to a standard or reference state. In this context, it means harmonizing the photometric properties of multi- view images to match those of a given reference frame or a set of target photometric parameters.

[0077] Spherical Harmonics (SH): A set of orthogonal functions defined on the surface of a sphere, used in computer graphics to represent functions that vary with direction, such as view-dependent colour or incident illumination. In 3D Gaussian Splatting, SH are often used to model the view-dependent colour of the Gaussian primitives.Affine Transformation (for colour): A linear transformation followed by a translation, which can be used to modify colour values. MLPs can output parameters of an affine transformation applied to Gaussian view-dependent colours for appearance modelling.

[0078] Floaters (reconstruction artifact): Erroneous, detached, or semi-transparent geometric elements that appear to float in the reconstructed 3D scene. These artifacts can arise in 3D reconstruction methods like 3DGS when input images have significant photometric inconsistencies or other issues.

[0079] Systems and methods are described herein for harmonizing or canonicalizing the appearance of multi- view images, particularly for use in 3D scene reconstruction processes. The described approaches aim to address photometric inconsistencies that commonly occur in casually captured image sets due to factors such as varying camera settings (e.g., exposure, white balance) and changing scene illumination. Unlike scene- specific methods that optimize appearance embedding jointly with 3D reconstruction, the techniques described herein provide a generalizable, feed-forward model for image harmonization. This model is learned from a diverse dataset of scenes exhibiting various photometric conditions, including both synthetic and real-world data.

[0080] Figure 1 is a schematic representation of a generalizable bilateral grid approach for appearance embedding, according to an example. A multi- view consistent generalizable transformer model 101 is configured to receive input comprising a sequence of multi- view images of a scene and a reference frame, 103. The output of this transformer model 101 comprises a bilateral grid used to perform appearance embedding on the input images that can be sued in the generation of primitives for 3DGS. The approach is trained on a diverse dataset and can apply to new scenes, and is multi-view consistent, such that it can produce harmonized images that are consistent across different viewpoints.

[0081] Accordingly, a multi-view consistent generalizable transformer model predicts at least one bilateral grid, which in turn simulates appearance embedding(s). The model can transform a sequence of images, potentially captured under varying photometric conditions, to match the appearance characteristics (e.g., style, colour, exposure) of a given reference image or a set of target photometric parameters. This harmonization is performed as a feed-forward step, effectively fixing the image appearance before or during the 3D reconstruction (e.g., 3D Gaussian Splatting). This decoupling avoids the computational overhead and optimization challenges, such as increased risk of local minima, associated with joint scene-specific appearance learning, and allows for rapid prediction of appearance embeddings without introducing significant latency into the 3DGS process.

[0082] F igure 2 is a schematic representation of method, according to an example. In the example of figure 2 a plurality of images 201 of an object or scene are provided. At least one of the images may exhibit photometric inconsistencies (e.g., variations in exposure, white balance, illumination) relative to other images, therefore requiring appearance harmonization.

[0083] A reference frame 203 comprises an image that serves as the exemplar for the desired visual appearance. Its function is to define a set of target photometric parameters (e.g., specific white balance, exposure level, overall colour tone, and style) to which the input images 201 will be conformed. The selection of this frame dictates the final look of the harmonized sequence. In an example, the reference frame 203 comprises one of the input images 201 that has been selected by a user as representing the exemplar for the desired visual appearance.

[0084] The input frames are initially processed by a transformer encoder 205. This encoder utilizes an approach similar to VGGT (Visual Geometry Grounded Transformer), which is adept at handling alternative frames (different views of the scene) and incorporates global attention mechanisms. Global attention allows the encoder to consider information from all input viewssimultaneously, capturing a comprehensive understanding of the scene context and inter-frame relationships, which is crucial for multi- view consistency.

[0085] The transformer also includes a decoder block 207 where cross-attention is applied. This provides a mechanism for the style transfer aspect. Specifically, in the cross-attention mechanism:

[0086] Key and value components are derived from the reference frame 203 (which embodies the target style).

[0087] The query component is derived from a source image 201 (the input image currently being processed and needing style harmonization). This configuration allows the decoder to effectively query the reference style and apply it to the source frame features.

[0088] The transformer model, through its encoder-decoder architecture with global and cross-attention, analyses the input images 201 and the reference frame 203. For each input image (source frame), it predicts the necessary transformations to make its appearance consistent with the reference frame 203.

[0089] For each processed source frame, the transformer decoder 209 outputs a set of patch-specific bilateral grids (one grid for each image patch of the target frame) 211, and a set of patch-specific uncertainty grids (one grid for each image patch of the target frame) 211. Post-decoder processing takes these outputs from the transformer decoder 207 and the original source frame to generate the final inputs for 3D reconstruction.

[0090] For example, the set of patch-specific bilateral grids 209, along with the original source frame (specifically, its pixel values for sampling / guidance), are processed. This operation involves using each patch- specific bilateral grid to transform the corresponding patch of the source frame. This results in the corrected frames 215, which comprise the input images after their appearance has been transformed to match the target photometric parameters and style of the reference frame 203.

[0091] During the training of the transformer model, corrected frames can be evaluated against ground truth images using an uncertainty-based RGB loss 219. This loss function typically compares the RGB values of the corrected frame to a ground truth image 221, with the contribution of each pixel or region weighted by the corresponding uncertainty (derived from the uncertainty maps), giving less weight to highly uncertain corrections.

[0092] The set of patch-specific uncertainty grids 211, along with the original source frame (potentially for spatial alignment), are processed. This operation involves assembling or interpolating the patch- specific uncertainty grids into a complete map for the frame. This results in the uncertainty maps 217.

[0093] These maps accompany each corrected frame, indicating the model’s assessed uncertainty or reliability of the appearance correction for different regions.

[0094] Both the corrected frames 215 and the uncertainty maps 217 are input to module 223 for 3D scene reconstruction. The uncertainty map helps in robustly optimizing the 3D Gaussian primitives by down- weighting contributions from image regions with high uncertainty (low reliability) in the appearance correction.

[0095] The generation of corrected frames and uncertainty maps is directly dependent on the patch- specific bilateral grids and uncertainty grids output by the transformer decoder, as well as the slicing and slicing and applying operations that utilize the original target frame."Slicing" / "Slicing and Applying" comprise operations that use the predicted grids and the target frame to generate full corrected frames and uncertainty maps. "Slicing" refers to how the grid values are sampled or interpolated based on pixel coordinates and / or intensities from the target frame patch. "Applying" refers to using the sampled transformation from the bilateral grid to modify the pixel colours of the target frame patch.

[0096] More specifically, the transformer decoder 207 predicts a patch-specific bilateral grid 209 for each small patch of the target input image 201. This grid is a data structure (often 3D or higher-dimensional) that holds learned parameters. These parameters define how to transform the colour / appearance of pixels. The "bilateral" aspect means the transformation can depend on both the pixel's spatial location (x, y coordinates within the patch) and its original intensity or colour values.

[0097] For each pixel within an image patch of the original target frame, its corresponding transformation parameters are determined from the patch-specific bilateral grid. "Slicing" refers to the process of looking up or sampling these parameters from the grid. This lookup is typically guided by the pixel's original properties: its (x, y) coordinates within the patch and its original colour / intensity values. Because the grid is discrete, this can involve interpolation between the grid's cells to get a smooth transformation for each specific input pixel.

[0098] "Applying": Once the specific transformation parameters (e.g., an affine transformation matrix, or scale and bias values for each colour channel) are retrieved from the bilateral grid for a given pixel (via "slicing"), these parameters are then mathematically applied to the original colour values of that pixel. This application changes the pixel's colour to match the target appearance. This is done for all pixels in the patch, using its dedicated bilateral grid.

[0099] After "slicing and applying" the respective bilateral grids to all patches of a target frame, the "Corrected Frame" results.

[0100] Similar to the bilateral grid, the transformer decoder 207 also predicts a patch-specific uncertainty grid 211 for each image patch. This grid stores values that represent the model's confidence (or lack thereof, i.e., uncertainty) about the appearance correction it has predicted for that patch.

[0101] When used for the uncertainty grids, slicing refers to the process of sampling or interpolating values from the patch-specific uncertainty grids to create a continuous, full-frame "Uncertainty Map." For each pixel in the corrected frame, its corresponding uncertainty value is determined by looking up (and possibly interpolating) from the relevant patch- specific uncertainty grid based on the pixel's (x,y) coordinates.

[0102] After "slicing" all the patch-specific uncertainty grids and assembling them (or their sampled values) spatially, the full "Uncertainty Map" for the entire corrected frame is obtained. This map gives a pixel- wise or region- wise indication of how reliable the appearance correction is deemed to be.

[0103] Accordingly, slicing enables intelligently query of the predicted grids (bilateral or uncertainty) using information from the original image pixels (like their position and colour) to get specific values or parameters out of those grids.

[0104] Applying (in the context of bilateral grids) enables use of those retrieved parameters to perform the actual mathematical colour transformation on the original pixel values.

[0105] Figure 3 is a schematic representation of method, according to an example. In the example of figure 3 a pretrained transformer model is provided, as described above with reference to figure 2 for example.Multiple images are fed to the transformer encoder 205. As before, the reference frame 203 provides the target style; its features are used for cross-attention in the decoder.

[0106] Encoder processing employs a VGGT-like approach with global attention for multi-view context from sequential frames as described above, and decoder processing (style transfer via cross-attention) uses cross-attention with keys / values from the reference frame 203 and queries from a source frame 201 features to infuse the target style.

[0107] As with figure 2, on a patch- wise basis, a set of patch-specific bilateral grids and a set of patch-specific uncertainty grids are output. The set of patch- specific bilateral grids, the set of patch- specific uncertainty grids (both from the decoder), and the original source frame (from the input stage).

[0108] Each patch-specific bilateral grid is used in conjunction with the corresponding image patch from the original source frame. As above, slicing refers to sampling or looking up transformation parameters from the bilateral grid based on the characteristics (e.g., pixel coordinates, intensity values) of the source frame patch. Applying refers to using these transformation parameters to modify the colours / appearance of the source frame patch.

[0109] Accordingly, for each source (or target) frame (an input image to be corrected), the system processes it on a patch- wise basis. This involves dividing the target frame into a plurality of image patches. For each image patch, a neural network model, such as a multi-view consistent transformer architecture, predicts two outputs: a patch- specific bilateral grid and a patch- specific confidence score. The patch- specific bilateral grid encodes an appearance transformation function designed to align the appearance of that specific image patch with the target photometric parameters defined by the reference frame. These bilateral grids can then be sampled to produce per-pixel affine transforms which are applied to the original image colours to match the reference frame. The patch-specific confidence score indicates the model's reliability or certainty in the predicted appearance transformation for that patch. This is particularly useful for handling regions that are difficult to recover, such as severely underexposed or overexposed areas (uncertain regions).

[0110] After predicting these for all patches of a target frame, the plurality of predicted patch- specific bilateral grids are applied to their corresponding image patches within the target frame. This application transforms the target frame into a corrected target frame, where its appearance is harmonized according to the desired target photometric parameters. The plurality of predicted patch-specific confidence scores are used to derive a confidence map for the corrected target frame.

[0111] Finally, the corrected target frame and its associated confidence map are provided for use in a 3D scene reconstruction process, such as 3D Gaussian Splatting. The confidence map can be used, for example, to weight a photometric loss function during the 3D reconstruction, giving less importance to regions where the appearance correction was less certain, thereby improving the stability and accuracy of the reconstruction, especially for uncertain regions like over-exposed areas.

[0112] The neural network model, particularly when implemented as a transformer model, can be designed with specific architectural features to enhance its performance. For example, the transformer model may comprise a certain number of encoder layers and decoder layers (e.g., 3 encoder layers and 3 decoder layers). Multi-head attention mechanisms (e.g., using 8 attention heads) can be employed, and token embeddings may have a specific dimensionality (e.g., 384). The prediction of the patch-specific bilateral grid can involve a multilayer perceptron (MLP) that utilizes a Tanh activation function in its terminal layers to appropriately bound the output values of the grid. Layer Normalization can be incorporated within the attention blocks of the transformer for stable training.To achieve multi- view consistency, the neural network model can apply global attention mechanisms that look across frames, allowing the model to consider information from all views when making predictions for a specific frame or patch. Furthermore, an encoder of the transformer model may utilize an alternating self-attention and global attention framework. The model might comprise a single output head configured to predict both the patch- specific bilateral grid and the patch- specific confidence score from the same processed features, promoting efficiency.

[0113] Cross-attention mechanisms can be implemented within a decoder of the transformer model. This cross-attention can occur between features derived from the reference frame and features derived from the target frame, effectively replacing global attention within the decoder for the task of style transfer or appearance harmonization. In such a setup, key and value inputs for the cross-attention mechanism may be derived from the reference frame, while the query input is derived from the target frame. This allows the model to learn how to transform the target frame to match the style or photometric characteristics of the reference frame.

[0114] The confidence map, derived from the plurality of predicted patch-specific confidence scores, can be generated using various techniques, such as a slicing or sampling approach from the patch-specific scores.

[0115] As mentioned, the confidence map can play a role in the subsequent 3D scene reconstruction process. For example, it can be used to weight a photometric loss function. This loss function might be calculated as a product of the confidence map and a photometric difference between a rendered image (from the 3D model) and a ground truth image (one of the corrected input images).

[0116] The neural network model is typically pretrained on a large and diverse dataset comprising scenes with varying exposure, white balance, and illumination, using both synthetic and real- world data. This pretraining can use a loss function that includes, for example, an LI loss between a corrected image produced by the model and a ground truth consistent image, where this LI loss is weighted by a confidence map. Additionally, a total variation loss can be applied to the predicted bilateral grids to encourage spatial smoothness in the transformations.

[0117] The process of dividing the target frame into a plurality of image patches can involve creating non-overlapping image patches. The size of these patches can vary; for example, each image patch might have dimensions of 8x8 pixels, potentially extracted from a larger input image region (e.g., 224x224 pixels) that is processed by the model.

[0118] Image processing throughout this pipeline is typically performed in a standard colour space, such as the sRGB colour space, to ensure consistency.

[0119] The reference frame itself can be selected in various ways. It could be, for example, the first frame of a sequence of input images, or it could be an image selected by a user, preferably one that is well-exposed and represents the desired overall appearance for the scene.

[0120] As noted with reference to figures 2 and 3 for example, the pretrained transformer model receives input frames (target frames) and a reference frame. The model outputs corrected frames and confidence maps. T hese outputs are then fed into a 3D Gaussian Splatting optimization process. The pipeline may also include a further appearance tuning step, where, after the initial 3D reconstruction, the transformer model can be used again to apply arbitrary reference styles to novel views rendered from the reconstructed 3D scene, allowing for quick stylistic adjustments. The efficiency of the feed-forward harmonization (e.g., processing hundreds of frames in under a minute, or tens of frames in seconds) contributes to a significantly faster overall workflow compared to methods requiring per-scene optimization of appearance.Examples in the present disclosure can be provided as methods, systems or machine-readable instructions, such as any combination of software, hardware, firmware or the like. Such machine-readable instructions may be included on a computer readable storage medium (including but not limited to disc storage, CD-ROM, optical storage, etc.) having computer readable program codes therein or thereon.

[0121] The present disclosure is described with reference to flow charts and / or block diagrams of the method, devices and systems according to examples of the present disclosure. Although the flow diagrams described above show a specific order of execution, the order of execution may differ from that which is depicted. Blocks described in relation to one flow chart may be combined with those of another flow chart. In some examples, some blocks of the flow diagrams may not be necessary and / or additional blocks may be added. It shall be understood that each flow and / or block in the flow charts and / or block diagrams, as well as combinations of the flows and / or diagrams in the flow charts and / or block diagrams can be realized by machine readable instructions.

[0122] The machine-readable instructions may, for example, be executed by a machine such as a general-purpose computer, a platform comprising user equipment such as a smart device, e.g., a smart phone, a special purpose computer, an embedded processor or processors of other programmable data processing devices to realize the functions described in the description and diagrams. In particular, a processor or processing apparatus may execute the machine-readable instructions. Thus, modules of apparatus (for example, a module implementing a converter for converting one precision format to another, such as float32 to Hoatl6 and so on, and / or a module for performing a matrix multiplication, and / or a module for applying a scaling factor to a matrix) may be implemented by a processor executing machine readable instructions stored in a memory, or a processor operating in accordance with instructions embedded in logic circuitry. The term 'processor' is to be interpreted broadly to include a CPU, processing unit, ASIC, logic unit, or programmable gate set etc. The methods and modules may all be performed by a single processor or divided amongst several processors.

[0123] Such machine-readable instructions may also be stored in a computer readable storage that can guide the computer or other programmable data processing devices to operate in a specific mode. For example, the instructions may be provided on a non-transitory computer readable storage medium encoded with instructions, executable by a processor.

[0124] Figure 4 is a schematic representation of a machine according to an example. The machine 400 can be, e.g., a system or apparatus, user equipment, or part thereof. The machine 400 comprises a processor 403, and a memory 405 to store instructions 402, executable by the processor 403. The machine comprises a storage 409 that can be used to store data 401 as described above with reference to figures 1 to 3 for example.

[0125] The instructions 407, executable by the processor 403, can cause the machine 400 to receive a plurality of input images of a scene, wherein at least one input image exhibits a photometric inconsistency relative to other input images in the plurality, and to receive a reference frame, the reference frame defining a set of target photometric parameters for harmonizing the plurality of input images, for each of at least a subset of the input images, referred to as source frames, divide the source frame into a plurality of image patches, for each image patch of the plurality of image patches, predict, using a neural network model a respective patch-specific bilateral grid, the respective patch-specific bilateral grid encoding an appearance transformation function to align an appearance of the image patch with the set of target photometric parameters defined by the reference frame, and a respective patch-specific confidence score, the respective patch-specific confidence score indicating a reliability of the predicted appearance transformation for the image patch, apply the plurality of predicted patch- specific bilateral grids to the source frame, wherein each predicted patch-specific bilateral grid transforms its corresponding image patch within the source frame, to generate a corrected source frame having an appearance harmonized according to the set of target photometricparameters, and provide the corrected source frame and a confidence map derived from the plurality of predicted patch-specific confidence scores for use in a 3D scene reconstruction process.

[0126] Accordingly, the machine 400 can implement a method for harmonizing appearance of multi-view images for 3D scene reconstruction.

[0127] The machine 400 can comprise at least one image sensor 410 configured to capture a sequence of two-dimensional input views of an object or scene. The at least one memory can store instructions that, when executed by the at least one processor, cause the machine to perform operations comprising receiving a current two-dimensional input view from the sequence of two-dimensional input views of the object or scene captured by the at least one image sensor. The machine can comprise a display 420 configured to display a rendered image to a user.

[0128] Such machine-readable instructions may also be loaded onto a computer or other programmable data processing devices, so that the computer or other programmable data processing devices perform a series of operations to produce computer-implemented processing, thus the instructions executed on the computer or other programmable devices provide an operation for realizing functions specified by flow(s) in the flow charts and / or block(s) in the block diagrams.

[0129] Further, the teachings herein may be implemented in the form of a computer or software product, such as a non-transitory machine-readable storage medium, the computer software or product being stored in a storage medium and comprising a plurality of instructions, e.g., machine readable instructions, for making a computer device implement the methods recited in the examples of the present disclosure.

[0130] In some examples, some methods can be performed in a cloud-computing or network-based environment. Cloud-computing environments may provide various services and applications via the Internet. These cloud-based services (e.g., software as a service, platform as a service, infrastructure as a service, etc.) may be accessible through a web browser or other remote interface of the user equipment for example. Various functions described herein may be provided through a remote desktop environment or any other cloud-based computing environment.

[0131] While various embodiments have been described and / or illustrated herein in the context of fully functional computing systems, one or more of these exemplary embodiments may be distributed as a program product in a variety of forms, regardless of the particular type of computer-readable-storage media used to actually carry out the distribution. The embodiments disclosed herein may also be implemented using software modules that perform certain tasks. These software modules may include script, batch, or other executable files that may be stored on a computer-readable storage medium or in a computing system. In some embodiments, these software modules may configure a computing system to perform one or more of the exemplary embodiments disclosed herein. In addition, one or more of the modules described herein may transform data, physical devices, and / or representations of physical devices from one form to another.

[0132] The preceding description has been provided to enable others skilled in the art to best utilize various aspects of the exemplary embodiments disclosed herein. This exemplary description is not intended to be exhaustive or to be limited to any precise form disclosed. Many modifications and variations are possible without departing from the spirit and scope of the instant disclosure. The embodiments disclosed herein should be considered in all respects illustrative and not restrictive. Reference should be made to the appended claims and their equivalents in determining the scope of the instant disclosure.

Claims

CLAIMS1. A method for harmonizing appearance of multi- view images for 3D scene reconstruction, the method comprising:receiving a plurality of input images of a scene, wherein at least one input image exhibits a photometric inconsistency relative to other input images in the plurality;receiving a reference frame, the reference frame defining a set of target photometric parameters for harmonizing the plurality of input images;for each of at least a subset of the input images, referred to as source frames:dividing the source frame into a plurality of image patches;for each image patch of the plurality of image patches, predicting, using a neural network model:a respective patch-specific bilateral grid, the respective patch-specific bilateral grid encoding an appearance transformation function to align an appearance of the image patch with the set of target photometric parameters defined by the reference frame, anda respective patch-specific confidence score, the respective patch-specific confidence score indicating a reliability of the predicted appearance transformation for the image patch;applying the plurality of predicted patch-specific bilateral grids to the source frame to generate a corrected source frame having an appearance harmonized according to the set of target photometric parameters, wherein each predicted patchspecific bilateral grid transforms its corresponding image patch within the source frame; andproviding the corrected source frame and a confidence map derived from the plurality of predicted patch-specific confidence scores for use in a 3D scene reconstruction process.

2. The method of claim 1, wherein the set of target photometric parameters comprises at least one of white balance, exposure level, or colour grading.

3. The method of claim 1 or 2, wherein the neural network model is a transformer model.

4. The method of claim 3, wherein predicting the respective patch-specific bilateral grid comprises using a multilayer perceptron, MLP, utilizing a Tanh activation function in at least one of its terminal layers.

5. The method of claim 3 or 4, wherein the transformer model incorporates Layer Normalization within its attention blocks.

6. The method of any one of claims 3 to 5 , wherein an encoder of the transformer model utilizes an alternating self-attention and global attention framework, and wherein the transformer model comprises a single output head configured to predict both the respective patch-specific bilateral grid and the respective patch-specific confidence score.

7. The method of any preceding claim, further comprising:using the neural network model, applying global attention across a plurality of the input images simultaneously during processing.

8. The method of any one of claims 3 to 7, further comprising:using a decoder of the transformer model, implementing cross-attention between features derived from the reference frame and features derived from the source frame, the cross-attention replacing global attention within the decoder.

9. The method of claim 8, further comprising:deriving key and value inputs for the cross-attention from the reference frame, and a query input for the crossattention from the source frame.

10. The method of any preceding claim, further comprising;deriving the confidence map from the plurality of predicted patch- specific confidence scores using a slicing or sampling approach.

11. The method of any preceding claim, further comprising:utilizing the confidence map to weight a photometric loss function during the 3D scene reconstruction process, wherein the loss function is calculated as a product of the confidence map and a photometric difference between a rendered image and a ground truth image.

12. The method of any preceding claim, further comprising:pretraining the neural network model using a loss function comprising an LI loss between a corrected image and a ground truth consistent image, wherein the LI loss is weighted by a confidence map; andapplying a total variation loss to the plurality of predicted patch-specific bilateral grids.

13. The method of any preceding claim, wherein dividing the source frame into the plurality of image patches comprises dividing the source frame into a plurality of non-overlapping image patches.

14. The method of any preceding claim, wherein image processing is performed in the sRGB colour space.

15. The method of any preceding claim, wherein the reference frame is a first frame of a sequence of the input images or a user-selected image.

16. An apparatus for harmonizing appearance of multi- view images for 3D scene reconstruction, the apparatus configured to:receive a plurality of input images of a scene, wherein at least one input image exhibits a photometric inconsistency relative to other input images in the plurality, and to receive a reference frame, the reference frame defining a set of target photometric parameters for harmonizing the plurality of input images;the apparatus comprising at least one processor; andat least one memory storing instructions that, when executed by the at least one processor, cause the apparatus to perform operations comprising:for each of at least a subset of the input images, referred to as source frames:dividing the source frame into a plurality of image patches;for each image patch of the plurality of image patches, predicting, using a neural network model:a respective patch-specific bilateral grid, the respective patch-specific bilateral grid encoding an appearance transformation function to align an appearance of the image patch with the set of target photometric parameters defined by the reference frame, anda respective patch-specific confidence score, the respective patch-specific confidence score indicating a reliability of the predicted appearance transformation for the image patch;applying the plurality of predicted patch- specific bilateral grids to the source frame, wherein each predicted patch-specific bilateral grid transforms its corresponding image patch within the source frame, to generate a corrected source frame having an appearance harmonized according to the set of target photometric parameters; andproviding the corrected source frame and a confidence map derived from the plurality of predicted patchspecific confidence scores for use in a 3D scene reconstruction process.

17. The apparatus of claim 16, wherein the set of target photometric parameters comprises at least one of white balance, exposure level, or colour grading.

18. The apparatus of claim 16 or 17, wherein the neural network model is a transformer model.

19. The apparatus of claim 18, wherein the transformer model comprises 3 encoder layers and 3 decoder layers, and utilizes a multi-head attention mechanism comprising 8 attention heads.

20. The apparatus of claim 18 or 19, wherein an encoder of the transformer model utilizes an alternating selfattention and global attention framework, and wherein the transformer model comprises a single output head configured to predict both the respective patch-specific bilateral grid and the respective patch-specific confidence score.

21. The apparatus of any one of claims 18 to 20, wherein a decoder of the transformer model implements crossattention between features derived from the reference frame and features derived from the source frame, wherein key and value inputs for the cross-attention are derived from the reference frame, and a query input for the cross-attention is derived from the source frame.

22. The apparatus of any one of claims 16 to 21, wherein the instructions further cause the apparatus to: utilize the confidence map to weight a photometric loss function during a 3D scene reconstruction process performed by a 3D scene reconstruction module, the loss function calculated as a product of the confidence map and a photometric difference between a rendered image and a ground truth image.

23. The apparatus of any one of claims 16 to 22, wherein dividing the source frame into the plurality of image patches comprises dividing the source frame into a plurality of non-overlapping image patches.

24. A non- transitory computer-readable medium storing instructions that, when executed by at least one processor, cause an apparatus to perform a method for harmonizing appearance of multi-view images for 3D scene reconstruction as claimed in any one of claims 1 to 15.18