Iterative determination of NERF models

Abstract

There is provided techniques for iteratively determining a NeRF model of a scene. A method is performed by an image processing device. The method comprises obtaining an initial NeRF model of the scene. The initial NeRF model is generated from a set of images depicting the scene, and a set of masks, one for each of the images. The masks mark edges of objects depicted in the scene. The method comprises iteratively updating the NeRF model of the scene by repeatedly improving a target function that minimizes a difference between the set of images and a set of images rendered based on the NeRF model. The target function at least pertains to a depth-based loss function that uses collinearity loss as a regularization term. The collinearity loss is computed for groups of pixels constrained by the masks.

Description

[0001] ITERATIVE DETERMINATION OF NERF MODELS

[0002] TECHNICAL FIELD

[0003] Embodiments presented herein relate to a method, an image processing device, a computer program, and a computer program product for iteratively determining a Neural Radiance Field model of a scene.

[0004] BACKGROUND

[0005] Neural volumetric representations, such as Neural Radiance Fields (NeRFs), can be used to learn a three-dimensional (3D) scene representation £1 from sparse input views (as defined by a set of input images 1 and the corresponding camera poses C). By means of NeRF, the scene is modeled as a continuous five-dimensional (5D) function that outputs radiance emitted in each direction (9, <p), at each point in 3D space (x, y, z). In other words, NeRF learns a 5D vector-valued function whose input is a 3D location (x, y, z) and two-dimensional (2D) viewing direction (9, cp ) , and outputs an emitted color (or radiance) vector r and volume density (structure) o.

[0006] The resulting 3D model can be used for variety of industrial and consumer applications. Here, the 3D model should be constructed to allow measurement to be made in the digitized scene or and such that a 3D point cloud capturing the scene geometry can be extracted. These operations require precise modelling of depth values at every rendered image to avoid waviness of the modeled surfaces. One objective of NeRF learning is to produce realistic single-view image renders that correspond well to ground-truth images. Often, it is possible to find ways to produce such renders without the correct geometry, especially when some parts of the scene are captured from few view angles and / or have a monochromatic texture. Reference is here made to Fig. 1 in which is provided a schematic representation of two possible NeRF reconstructions 100a, 100b of a scene depicting a cube. If the cube sides are monochromatic, both reconstructions 100a, 100b are indistinguishable from a rendering perspective; geometric dents (as being examples of waviness), as present in the reconstruction 100b will not be visible. These dents, however, will be apparent in an extracted point cloud.

[0007] In this respect, the Multi-View Stereo (MVS) planarity assumption simplifies 3D reconstruction by assuming that small surface patches in a scene can be locallyapproximated as planar. This helps improve depth estimation by fitting local planes to depth data, reducing noise, and making it easier to establish correspondences across multiple camera views. MVS relies on finding pixel correspondences between images and assumes that these points lie on the same local plane, enforcing geometric consistency across viewpoints. This assumption is particularly effective for smooth, man-made surfaces but struggles with highly curved or intricate geometries where surface normals change rapidly. However, this assumption does not translate well to the context of NeRFs, where the scene is represented as a continuous volumetric density field rather than explicit surfaces. Additionally, since NeRF lacks explicit depth constraints, errors in density estimation can accumulate, leading to wavy or floating surfaces instead of well-defined geometry. Moreover, NeRF does not rely on explicit pixel correspondences, making MVS-style depth constraints difficult to integrate efficiently. This incompatibility also affects surface quality. Without explicit geometric priors, NeRF tends to overfit photometric loss, leading to depth estimation errors that propagate across views. This results in wavy surfaces in physical scenes modeled by NeRF. The resulting interpolations in the density field introduce high-frequency artifacts, producing wavy surfaces instead of sharp, well-defined geometry. This difference in representation makes traditional MVS-based assumptions ineffective for NeRF-based reconstruction.

[0008] One way to address this issue is to sample rays in large patches (such as 8-by-8 pixels) to regularize the geometry, see the paper titled “RegNeRF: Regularizing Neural Radiance Fields for View Synthesis from Sparse Inputs”, IEEE Conference on Computer Vision and Pattern Recognition (CVPR) 2022, authored by Michael Niemeyer, et al. In more detail, this paper addresses challenges NeRF faces when reconstructing scenes from a limited number of input images. Methods are proposed to regularize both the geometry and appearance of patches rendered from unobserved viewpoints, in an effort to enhance the quality of novel view synthesis in scenarios with sparse data. The approach includes annealing the ray sampling space during training and employing a normalizing flow model to regularize colors from unseen perspectives. However, introducing many co-dependent ray samples drastically increase computational complexity and negatively affects the overall quality of the reconstruction.Another way to address this issue is to enhance NeRF with per-pixel unconditional regularizers, putting a global smoothness constraint on the whole scene, see the paper titled “Stable Surface Regularization for Fast Few-Shot NeRF”, International Conference on 3D Vision (3DV) 2024, authored by Byeongin Joung et al. An annealing Signed Distance Function (ASDF) loss is introduced to regularize surfaces in a coarse-to-fine manner to accelerate convergence and improve stability during training. This method is aimed at addressing limitations of traditional geometric regularizations, such as the Eikonal loss, which often require dense training data and may underperform in few-shot scenarios. However, this may result in loosing resolution in the modeled scene.

[0009] Hence, there is still a need for improved determination of a NeRF model of a scene.

[0010] SUMMARY

[0011] An object of embodiments herein is to address the above issues and enable accurate determination of a NeRF model of a scene.

[0012] A particular object is to enable accurate determination of a NeRF model without relying on the MVS planarity assumption, without introducing many co-dependent ray samples, and without putting a global smoothness constraint on the whole scene.

[0013] According to a first aspect there is presented a method for iteratively determining a NeRF model of a scene. The method is performed by an image processing device. The method comprises obtaining an initial NeRF model of the scene, and a set of masks, one for each of the images. The initial NeRF model is generated from a set of images depicting the scene. The masks mark edges of objects depicted in the scene. The method comprises iteratively updating the NeRF model of the scene by repeatedly improving a target function that minimizes a difference between the set of images and a set of images rendered based on the NeRF model. The target function at least pertains to a depth-based loss function that uses collinearity loss as a regularization term. The collinearity loss is computed for groups of pixels constrained by the masks.

[0014] According to a second aspect there is presented an image processing device for iteratively determining a NeRF model of a scene. The image processing device comprises processing circuitry. The processing circuitry is configured to cause the image processing device to obtain an initial NeRF model of the scene, and a set ofmasks, one for each of the images. The initial NeRF model is generated from a set of images depicting the scene. The masks mark edges of objects depicted in the scene. The processing circuitry is configured to cause the image processing device to iteratively update the NeRF model of the scene by repeatedly improving a target function that minimizes a difference between the set of images and a set of images rendered based on the NeRF model. The target function at least pertains to a depthbased loss function that uses collinearity loss as a regularization term. The collinearity loss is computed for groups of pixels constrained by the masks.

[0015] According to a third aspect there is presented a computer program for iteratively determining a NeRF model of a scene. The computer program comprises computer code which, when run on processing circuitry of an image processing device, causes the image processing device to perform actions. One action comprises the image processing device to obtain an initial NeRF model of the scene, and a set of masks, one for each of the images. The initial NeRF model is generated from a set of images depicting the scene. The masks mark edges of objects depicted in the scene. One action comprises the image processing device to iteratively update the NeRF model of the scene by repeatedly improving a target function that minimizes a difference between the set of images and a set of images rendered based on the NeRF model. The target function at least pertains to a depth-based loss function that uses collinearity loss as a regularization term. The collinearity loss is computed for groups of pixels constrained by the masks.

[0016] According to a fourth aspect there is presented a computer program product comprising a computer program according to the third aspect and a computer readable storage medium on which the computer program is stored. The computer readable storage medium could be a non-transitory computer readable storage medium.

[0017] Advantageously, these aspects enable the generation of an accurate NeRF model of the scene.

[0018] Advantageously, these aspects in turn enable extraction of less noisy (in terms of scene geometry) 3D point clouds.Advantageously, these aspects in turn enable more accurate depth measurements to be made in the resulting rendered 3D scene.

[0019] Other objectives, features and advantages of the enclosed embodiments will be apparent from the following detailed disclosure, from the attached dependent claims as well as from the drawings.

[0020] Generally, all terms used in the claims are to be interpreted according to their ordinary meaning in the technical field, unless explicitly defined otherwise herein. All references to "a / an / the element, apparatus, component, means, module, step, etc." are to be interpreted openly as referring to at least one instance of the element, apparatus, component, means, module, step, etc., unless explicitly stated otherwise. The steps of any method disclosed herein do not have to be performed in the exact order disclosed, unless explicitly stated.

[0021] BRIEF DESCRIPTION OF THE DRAWINGS

[0022] The inventive concept is now described, by way of example, with reference to the accompanying drawings, in which:

[0023] Fig. 1 shows a schematic representation of two possible NeRF reconstructions of a scene according to examples;

[0024] Fig. 2 shows a comparison of two point clouds according to an embodiment;

[0025] Fig. 3 is a block diagram of an image processing device according to an embodiment;

[0026] Fig. 4 is a flowchart of methods according to embodiments;

[0027] Fig. 5 schematically illustrates a physical-world scene and a segment defined by three pixels sampled on an image depicting the scene according to an embodiment;

[0028] Fig. 6 schematically illustrates ray tracing according to an embodiment;

[0029] Fig. 7 is a schematic diagram showing structural units of an image processing device according to an embodiment; and

[0030] Fig. 8 shows one example of a computer program product comprising computer readable storage medium according to an embodiment.DETAILED DESCRIPTION

[0031] The inventive concept will now be described more fully hereinafter with reference to the accompanying drawings, in which certain embodiments of the inventive concept are shown. This inventive concept may, however, be embodied in many different forms and should not be construed as limited to the embodiments set forth herein; rather, these embodiments are provided by way of example so that this disclosure will be thorough and complete, and will fully convey the scope of the inventive concept to those skilled in the art. Like numbers refer to like elements throughout the description. Any step or feature illustrated by dashed lines should be regarded as optional.

[0032] As noted above, there is still a need for improved determination of a NeRF model of a scene. In particular, inaccurate NeRF models, in terms of incorrect scene geometry, may be rendered when some parts of the scene are captured from few view angles and / or have a monochromatic texture. To illustrate this, reference is here made to Fig. 2. In Fig. 2 is provided a comparison of a point cloud 200a extracted from a NeRF model having been generated, or learned, according to state of the art, and a point cloud 200b having been generated, or learned, according to the herein disclosed embodiments. In can be seen that the point cloud 200b exhibits less waviness than the point cloud 200a, and thus that the point cloud 200b is a more accurate representation of the physical-world scene.

[0033] Fig. 3 is a block diagram of an image processing device 300 configured for iteratively determining a NeRF model of a scene of interest according to an embodiment. The image processing device 300 comprises a number of blocks 310:328, which could be implemented and executed in hardware, software, or a combination thereof. The image processing device 300 takes as input a set of images 1 and an initial NeRF model fl, and produces as output an updated NeRF model fl'. The initial NeRF model fl may be either a pre-trained or an un-trained NeRF representation of the scene of interest. In the latter, the below described refinement stage coincides with (or replaces) the training stage. The updated NeRF model fl' can then be used to visualize the scene, extract the point cloud or render the depth images at predefined or novel views. The updated NeRF model fl' is obtained by first performing image analysis (block 310) on the set of images and then executing an iterative procedure in anoptimization loop (block 320). In this respect, the optimization loop block 320 comprises a number of blocks 322:328 that are iteratively executed. The optimization loop is defined by iteratively performing pixel sampling (block 322), ray sampling and rendering (block 324), target function computation (326) and backpropagation (block 328). The operations implemented and executed in each of the blocks 310:328 will be disclosed in detail below.

[0034] In general terms, NeRF is based on training, where a target function that minimizes the difference between the rendered image and the ground truth image is optimized. During NeRF training, the target function often includes several loss functions, such as rendering loss that measures how far the rendered pixels diverge from ground truth colours, or various regularizations. A commonly used loss function is a photometric reconstruction loss function (or colour reconstruction loss function), but additional regularization terms can be included to improve geometry and prevent artifacts. If no explicit depth supervision is available, the NeRF relies only on the photometric loss, which can lead to wavy or floating surfaces due to ambiguity in scene reconstruction. As disclosed above, the herein disclosed embodiments are therefore based on using a depth-based loss function that uses collinearity loss as a regularization term. The target function can then be regarded as a weighted sum of the different components. Denote by L* the loss function with respect to some given regularization term *. Then the target function, denoted L, can be expressed as:

[0035]

[0036] Here, Lcand Ldare the photometric reconstruction loss function and the depth-based loss function, respectively, and Acand Adare hyperparameters (or weight values) controlling the relative importance of each loss function. The values of the hyperparameters may change from one iteration to the next. For example, the value of Admay increase with increased number of iterations being performed, thus letting the depth-based loss function yield more impact on the target function as the number of iterations increases. The depth-based loss function can be used in conjunction with other existing targets of optimization, and hence the target function may have more terms. Further, the depth-based loss function may even be used as the only loss function and hence, L = Ad■ Ld.Fig. 4 is a flowchart illustrating embodiments of methods for iteratively determining a NeRF model of a scene. The methods are performed by the image processing device 300. The methods are advantageously provided as computer programs.

[0037] The solution proposes a NeRF optimization technique by means of a loss factor that induces a geometric prior onto scene surfaces. By means of the geometric prior, additional constraints or assumptions imposed on the scene geometry to guide the learning process towards a more accurate and stable surface representation, and thus a more accurate NeRF mode. In the context of NeRF, a surface prior is typically used to regularize the optimization and encourage the formation of well-defined surfaces, as opposed to a fully unconstrained volumetric density field. Here, the constraints are based on image data (as obtained from each single image) and are affecting the scene geometry. That is, geometric priors obtained from analysis of the images are applied to the NeRF model. In general terms, the loss factor is using a collinearity constraint and penalizes large variations in depth values along the same surface. The surface flatness is modeled as the zero curvature of all surface’s section curves. By means of the loss factor, surface waviness can be mitigated, and the measurement accuracy in the modeled scene can therefore be improved.

[0038] The process involves taking an initial NeRF representation of the scene as input (step S102) and improving it during a refinement stage (step S104).

[0039] S102: The image processing device 300 obtains an initial NeRF model of the scene and a set of masks, one for each of the images. The initial NeRF model is generated from a set of images depicting the scene. The initial NeRF model is either a pretrained NeRF model of the scene or a randomly initialized untrained NeRF model of the scene. The masks mark edges of objects depicted in the scene.

[0040] S104: The image processing device 300 iteratively updates the NeRF model of the scene by repeatedly improving a target function that minimizes a difference between the set of images and a set of images rendered based on the NeRF model. The target function at least pertains to a depth-based loss function that uses collinearity loss as a regularization term. The collinearity loss is computed for groups of pixels constrained by the masks. The groups of pixels define pixels for which the collinearity loss is computed.Step S104 thus defines a refinement procedure, or optimization loop. Step S104 is implemented and executed by block 320.

[0041] Embodiments relating to further details of iteratively determining a NeRF model of a scene as performed by the image processing device 300 will now be disclosed with continued reference to the block diagram in Fig. 3, as well as to the method disclosed with reference to Fig. 2.

[0042] Further details of the image analysis block 310 will be disclosed next.

[0043] The image analysis block 310 is configured to apply edge mask to each of the images. The edge mask may, for example, be represented by a Canny edge detector or similar edge detector. Each such edge detector may be associated with threshold parameters. As a non-limiting and illustrative example, for a Canny edge detector, thresholdi could be set to 100 and threshold2 could be set to 300, depending on the dimensions of the images, etc. The produced edge mask can also be widened, for example by applying dilation with a structuring element of a square.

[0044] This results in a set of masks

[0045]

[0046] with one mask for each image i e I. These masks mark the edges of the objects based on colour information, or at least luminance information, present in the images. Hence, in some embodiments, the set of masks are determined using edge detection based on colour or luminance information in the set of images. These masks can be used to reject applying refining operation over parts of the scene that cross the borders of the objects. Particular, when iteratively updating the NeRF model of the scene, the collinearity loss may be computed for different groups of pixels, and, according to the masks, each group of pixels only includes pixels which are not separated by any edge.

[0047] Further details of the optimization loop block 320 will be disclosed next.

[0048] One purpose of the optimization loop block 320 is to update the current NeRF model £1 (from a few to several thousand iterations, based on the application). After halting, the resulting updated NeRF model is returned as £1'. As disclosed above, the optimization loop block 320 comprises a number of blocks 322:328 that are iteratively executed.

[0049] Further details of the pixel sampling block 322 will be disclosed next.Generally, the pixel sampling block 322 is configured to perform random selection of pixels in randomly selected images in the set of images. Hence, in some embodiments, the pixels, for each iteration when iteratively updating the NeRF model of the scene, are sampled in randomly selected images in the set of images. In general terms, the random sampling is performed of batches of pixels in groups of three (i. e. , in triplets). Each such triplet (p0, p , p2) of pixels denotes a single segment on one image i E 1, with p being located between p0and p2. In particular, in some embodiments, using the collinearity loss as regularization term, each group of pixels is constrained to consist of a triplet of pixels p0, p , p2placed along a straight line 550 in one of the images 530. Hence, with respect to the masks

[0050]

[0051] the triplets are selected in such way that the corresponding segments do not intersect any edges masked by

[0052]

[0053] One reason for randomly selecting the pixels in the images is because the ray sampling and rendering NeRF procedure itself is based on random sampling (see, below). Another reason for randomly selecting the pixels in the images is because the content of the images is unknown, and therefore there is no assumption with regards to orientation, placement, size, etc. of objects depicted in the images.

[0054] There could be different ways to define each triplet of pixels. Two non-limiting examples will be disclosed next. Both examples are based on random selection of one or more pixels among all pixels of a given image i that are not masked as edges in

[0055]

[0056] Reference is here made to Fig. 5 A physical-world scene 500 comprises a cube 510. An image 530 has been captured of the cube from a camera 520 at some fixed location. Edge detection has been performed to detect the edges 540 of the cube (assuming the entire cube has the same colour). A segment defined by three pixels p0, Pi, P2 is sampled on the image 530 along a line 550 that does not cross the edges 540.

[0057] According to a first non-limiting example, two pixels are initially randomly selected in a given image among all pixels of this given image i that are not masked as edges in the mask

[0058]

[0059] for this given image i. As per above, by means of the mask for this given image, the pixels are not separated by any edge. These pixels are defined as end-point pixels pQ, p2. A third pixels, denoted mid-point pixel p is then defined by the midpoint pixel between the two end-point pixels (rounded to integer pixel coordinates).According to a second non-limiting examples, a first pixel p is initially randomly selected in a given image i among all pixels of this given image i that are not masked as edges in the mask

[0060]

[0061] for this given image i. Then, a 2D direction (orientation) u is randomly selected. Set

[0062]

[0063] where D is a desired maximum length of a segment. The largest radius

[0064] <

[0065]

[0066] Is then found such that all pixels of the segment within distance r from p do not cross any edges as defined by

[0067]

[0068] Then, p0, p2can be redefined as follows:

[0069] Po = Pi ~r■u> p2= P + r • u

[0070] Further details of the ray sampling and rendering block 324 will be disclosed next.

[0071] The term ray sampling in the area of Neural Radiance Fields corresponds to sampling of 3D points along a ray. Common techniques for this involve uniform sampling of N points until certain depth, or using a smaller proposal networks that produce point samples at high-density areas along the ray. Then, the NeRF model fl is queried at those points for the values of color, density, and potentially other learnable quantities such as normals. These values are then accumulated across each ray, producing single differentiable renders of those quantities for the sampled pixels. Density values are used to aggregate the distances to the sampled points into single depth values d*. During training, the depth values can be computed as expected depth values via a volume rendering equation, which assigns transmittance weights to all sampled points and accumulates the distances to each point of the ray with those weights.

[0072] A visualization of the ray sampling together with cast rays with depicted depth values is provided in Fig. 6. An image 630 has been captured of the cube from a camera 620 at some fixed location. Edge detection has been performed to detect the edges 640 of the cube (assuming the entire cube has the same colour). A segment defined by three pixels p0, pltp2is sampled on the image 630 along a line 650 that does not cross theedges 640. In Fig. 6 is further illustrated ray casting through the three pixels pQ, pltp2from the optical center 660 of the camera 620. In Fig. 6, three rays 670, as sampled through the three selected points on the image plane, are thus cast. In contrast to Fig.

[0073] 5, in Fig. 6 is illustrated the current reconstructed scene 600 as learned by the current NeRF model fl. The reconstructed scene 600 thus comprises a learned (or reconstructed) cube 610. It can be seen that, in comparison to the cube 510, the learned cube 610 comprises dents. This is in Fig. 6 illustrated by that the line 680 along which the depth values d0, d , d2are placed on is not straight.

[0074] Further details of the target function computation block 326 will be disclosed next.

[0075] For each iteration when iteratively updating the NeRF model of the scene, one value of the depth-based loss function is computed for each group (i.e., triplet) of pixels. With respect to the above-defined triplet of pixels p0, p , p2that includes two endpoint pixels p0, p2and one mid-point pixel p±with respect to a straight line 550, the depth-based loss function is computed for the mid-point pixel p±.

[0076] In some non-limiting examples, the depth-based loss function (as above denoted Ld) comprises three multiplicative parts:

[0077] Ld= 1 [within_thres] • color_diff(c0, c1(c2) • depth_diff(Ad),

[0078] Hence, if the difference between expected depth value of the mid-point pixel and the rendered depth value is below some threshold {l[within_thres]}, and if the pixel colors are not too far apart {color_diff(c0, cnc2)}, then a low loss value implies a low difference between the expected and the actually rendered depth values at the midpoint {depth_diff(Ad)}. That is, in some embodiments, the alignment between depth values of pixels in the set of images and corresponding pixels in the set of images rendered based on the NeRF model is defined by a depth difference Ad between the depth value d of the mid-point pixel p in one of the images 530 rendered based on the NeRF model and the depth value d expected for the midpoint pixel p, as computed based on the depth values d0, d2of the end-point pixels p0, p2in this one of the images 530 rendered based on the NeRF model.

[0079] In further detail, assume that each pixel pk, k = 0 ...2, is associated with the following attributes: ckis the ground truth color vector (in red-green-blue) of the pixel in theinput image, dkis the depth value at the pixel rendered from the NeRF model, and ukis the directional unit-vector that goes from the camera’s optic center through the center of pixel pk.

[0080] Then, in depth_diff(Ad), the parameter Ad can be defined as the difference between the rendered depth at mid-point pixel p and the depth d^ expected from the rendered depth values at end-point pixels p0and p2. In other words:

[0081] Ad — d- d-

[0082] The expectation is based on the assumption that all three points at their respective depth values along the rays lay on the same line, i.e. are collinear. Such an expected depth value may be computed using the following expression (that comes from planar geometry):

[0083]

[0084] Here, d thus represents what the depth value would be if p0, p , p2were all on a straight line (i.e., in the same plane).

[0085] The depth difference Ad may then be scaled via one of the arbitrary standard loss functions, such as a hyperbolic tangent depth_diff(Ad) = tanh T| Ad | for a parameter T (e.g. T = 4 in a normalized scene box), or simply the absolute value function depth_diff(Ad) = |Ad|. Hence, in some embodiments, the depth-based loss function pertains to a scale version of the depth difference Ad.

[0086] The remaining components of the depth-based loss function (i.e., {l[within_thres]}, and {color_diff(c0, q, c2)}) may be considered optional but may to improve the overall quality of the optimization.

[0087] In this respect, l[within_thres] is flag (with either value o which “deactivates” the inclusion of the loss function, or with value 1 which “activates” the inclusion of the loss function). This function can be used to verify that Ad is not too large relative to the rendered depth values, using a hard threshold. Therefore, in some examples,

[0088] within_thres[i] = 1[| Ad| < £ ■ min{d0, d1(d2}]for some threshold parameter e. In some non-limiting examples, 0.0025 < s < 0.05.

[0089] If the difference between depths is too large, the loss is not applied to such triplet.

[0090] Further, color_diff(c0, q, c2) maybe defined as:

[0091] IICI-COII2+IIC2~CIII2

[0092] color_diff(c0, q, c2) = e 2<r2

[0093] where <J G 6 is some parameter, e.g. <J = 0.1 for color values normalized to values from o to 1. The expression for color_diff(c0, q, c2) is similar to a gaussian distribution, but is not required to be so; it may be any function that decreases as the difference between the color vectors increases. The function color_diff(c0, q, c2) is used to reduce the influence of the loss on less monochromatic regions that were not captured by the edge detector.

[0094] Further details of the backpropagation block 328 will be disclosed next.

[0095] In some embodiments, gradients of the target function with respect to the NeRF model are computed as part of iteratively updating the NeRF model of the scene. Therefore, in some aspects, the backpropagation block 328 is configured to compute gradients of the target function and apply the gradients over the parameters of the NeRF model fl by performing a step of gradient descent. Hence, in some embodiments, the NeRF model is updated by performing a gradient descent step based on the gradients of the target function. In each iterations the parameters are shifted in a way to reduce the loss functions. The NeRF model fl with the updated parameters for the current iteration then replaces the NeRF model fl obtained during the previous iteration. If some stopping criterion is reached, the process halts. The current NeRF model fl then represents the reoptimized NeRF model fl' that is provided as output. Else, a new iteration is performed. That is, step S104 is entered again. Here, there can be different types of stopping criteria. One stopping criterion is that a desired number of iterations is reached. One stopping criterion is that the target function has reached a desired value.

[0096] Fig. 7 schematically illustrates, in terms of a number of structural units, the components of an image processing device 700 according to an embodiment.

[0097] Processing circuitry 710 is provided using any combination of one or more of asuitable central processing unit (CPU), multiprocessor, microcontroller, digital signal processor (DSP), etc., capable of executing software instructions stored in a computer program product 810 (as in Fig. 8), e.g. in the form of a storage medium 730. The processing circuitry 710 may further be provided as at least one application specific integrated circuit (ASIC), or field programmable gate array (FPGA).

[0098] Particularly, the processing circuitry 710 is configured to cause the image processing device 700 to perform a set of operations, or steps, as disclosed above. For example, the storage medium 730 may store the set of operations, and the processing circuitry 710 may be configured to retrieve the set of operations from the storage medium 730 to cause the image processing device 700 to perform the set of operations. The set of operations may be provided as a set of executable instructions.

[0099] Thus, the processing circuitry 710 is thereby arranged to execute methods as herein disclosed. The storage medium 730 may also comprise persistent storage, which, for example, can be any single one or combination of magnetic memory, optical memory, solid state memory or even remotely mounted memory. The image processing device 700 may further comprise a communications (comm.) interface 720 at least configured for communications with other entities, functions, nodes, and devices. For example, the communications interface 720 may be configured for communications with at least one camera, or storage unit, for obtaining a set of input images. For example, the communications interface 720 may be configured for communications with a computational unit for obtaining initial NeRF models of scenes of interest. For example, the communications interface 720 may be configured for communications with at least one application device, or storage unit, for providing determined NeRF models.

[0100] As such the communications interface 720 may comprise one or more transmitters and receivers, comprising analogue and digital components. The processing circuitry 710 controls the general operation of the image processing device 700 e.g. by sending data and control signals to the communications interface 720 and the storage medium 730, by receiving data and reports from the communications interface 720, and by retrieving data and instructions from the storage medium 730. Other components, as well as the related functionality, of the image processing device 700 are omitted in order not to obscure the concepts presented herein.A first portion of the instructions performed by the image processing device 700 may be executed in a first device, and a second portion of the of the instructions performed by the image processing device 700 may be executed in a second device; the herein disclosed embodiments are not limited to any particular number of devices on which the instructions performed by the image processing device 700 may be executed. Hence, the methods according to the herein disclosed embodiments are suitable to be performed by an image processing device 700 residing in a cloud computational environment. Therefore, although a single processing circuitry 710 is illustrated in Fig. 7 the processing circuitry 710 may be distributed among a plurality of devices, or nodes. The same applies to the computer program 820 of Fig. 8.

[0101] Fig. 8 shows one example of a computer program product 810 comprising computer readable storage medium 830. On this computer readable storage medium 830, a computer program 820 can be stored, which computer program 820 can cause the processing circuitry 710 and thereto operatively coupled entities and devices, such as the communications interface 720 and the storage medium 730, to execute methods according to embodiments described herein. The computer program 820 and / or computer program product 810 may thus provide means for performing any steps as herein disclosed.

[0102] In the example of Fig. 8, the computer program product 810 is illustrated as an optical disc, such as a CD (compact disc) or a DVD (digital versatile disc) or a Blu-Ray disc. The computer program product 810 could also be embodied as a memory, such as a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM), or an electrically erasable programmable read-only memory (EEPROM) and more particularly as a non-volatile storage medium of a device in an external memory such as a USB (Universal Serial Bus) memory or a Flash memory, such as a compact Flash memory. Thus, while the computer program 820 is here schematically shown as a track on the depicted optical disk, the computer program 820 can be stored in any way which is suitable for the computer program product 810.

[0103] The inventive concept has mainly been described above with reference to a few embodiments. However, as is readily appreciated by a person skilled in the art, otherembodiments than the ones disclosed above are equally possible within the scope of the inventive concept, as defined by the appended patent claims.

Claims

CLAIMS1. A method for iteratively determining a Neural Radiance Field, NeRF, model of a scene (500), the method being performed by an image processing device (300, 700), the method comprising:obtaining (S102) an initial NeRF model of the scene (500), wherein the initial NeRF model is generated from a set of images depicting the scene (500), and a set of masks, one for each of the images, wherein the masks mark edges of objects depicted in the scene (500); anditeratively updating (S104) the NeRF model of the scene (500) by repeatedly improving a target function that minimizes a difference between the set of images and a set of images rendered based on the NeRF model, wherein the target function at least pertains to a depth-based loss function that uses collinearity loss as a regularization term, and wherein the collinearity loss is computed for groups of pixels constrained by the masks.

2. The method according to claim 1, wherein the initial NeRF model is either a pretrained NeRF model of the scene (500) or a randomly initialized untrained NeRF model of the scene (500).

3. The method according to claim 1 or 2, wherein the set of masks are determined using edge detection based on colour or luminance information in the set of images.

4. The method according to any preceding claim, wherein, when iteratively updating the NeRF model of the scene (500), the collinearity loss is computed for different groups of pixels, and wherein, according to the masks, each group of pixels only includes pixels not separated by any edge.

5. The method according to claim 4, wherein the groups of pixels define pixels for which the collinearity loss is computed.

6. The method according to claim 4 or 5, wherein the pixels, for each iteration when iteratively updating the NeRF model of the scene (500), are sampled in randomly selected images in the set of images.

7. The method according to any of claims 4 to 6, wherein using the collinearity loss as regularization term constrains each group of pixels to consist of a triplet of pixels (p0, Pi, P2) placed along a straight line (550, 650) in one of the images (530, 630).

8. The method according to any of claims 4 to 7, wherein, for each iteration when iteratively updating the NeRF model of the scene (500), one value of the depth-based loss function is computed for each group of pixels.

9. The method according to claim 7, wherein each of the triplet of pixels (p0, p1(p2) includes two end-point pixels (p0, p2) and one mid-point pixel (p ) with respect to the straight line (550, 650), and wherein the depth-based loss function is computed for the mid-point pixel (p .

10. The method according to claim 9, wherein the alignment between depth values of pixels in the set of images and corresponding pixels in the set of images rendered based on the NeRF model is defined by a depth difference (Ad) between the depth value (di) of the mid-point pixel (pi) in one of the images (530, 630) rendered based on the NeRF model and the depth value (d expected for the mid-point pixel (p- as computed based on the depth values (d0, d2) of the end-point pixels (p0, p2) in said one of the images (530, 630) rendered based on the NeRF model.

11. The method according to claim 10, wherein the depth-based loss function pertains to a scale version of the depth difference (Ad).

12. The method according to any preceding claim, wherein gradients of the target function with respect to the NeRF model are computed as part of iteratively updating the NeRF model of the scene (500).

13. The method according to claim 12, wherein the NeRF model is updated by performing a gradient descent step based on the gradients of the target function.

14. An image processing device (300, 700) for iteratively determining a Neural Radiance Field, NeRF, model of a scene (500), the image processing device (300, 700) comprising processing circuitry (710), the processing circuitry being configured to cause the image processing device (300, 700) to:obtain an initial NeRF model of the scene (500), wherein the initial NeRF model is generated from a set of images depicting the scene (500), and a set of masks, one for each of the images, wherein the masks mark edges of objects depicted in the scene (500); anditeratively update the NeRF model of the scene (500) by repeatedly improving a target function that minimizes a difference between the set of images and a set of images rendered based on the NeRF model, wherein the target function at least pertains to a depth-based loss function that uses collinearity loss as a regularization term, and wherein the collinearity loss is computed for groups of pixels constrained by the masks.

15. The image processing device (300, 700) according to claim 14, further being configured to perform the method according to any of claims 2 to 13.

16. A computer program (820) for iteratively determining a Neural Radiance Field, NeRF, model of a scene (500), the computer program comprising computer code which, when run on processing circuitry (710) of an image processing device (300, 700), causes the image processing device (300, 700) to:obtain (S102) an initial NeRF model of the scene (500), wherein the initial NeRF model is generated from a set of images depicting the scene (500), and a set of masks, one for each of the images, wherein the masks mark edges of objects depicted in the scene (500); anditeratively update (S104) the NeRF model of the scene (500) by repeatedly improving a target function that minimizes a difference between the set of images and a set of images rendered based on the NeRF model, wherein the target function at least pertains to a depth-based loss function that uses collinearity loss as a regularization term, and wherein the collinearity loss is computed for groups of pixels constrained by the masks.

17. A computer program product (810) comprising a computer program (820) according to claim 16, and a computer readable storage medium (830) on which the computer program is stored.

Images