Apparatus and method for generating novel views with human-object interaction

Abstract

A method (700) for reconstructing a human-object interaction model, comprising: receiving images depicting a human-object interaction; using a candidate model, modelling (701) the human using a first set of three-dimensional Gaussians and modelling (702) the object using a second set of three-dimensional Gaussians; separately rendering (703, 704) the sets to form first and second rendered outputs; jointly rendering (705) the sets to form a third rendered output; combining (706) the first, second and third rendered outputs to determine region(s) where the human is occluded by the object; in dependence on the region(s), forming (707) a rendered image; comparing (708) the rendered image with a ground truth; updating (709) model parameters based on the comparison. This method models the human and object separately using Gaussian splats, providing the flexibility of controlling human and object while being able to model complex 3D shapes and achieve photorealistic renderings.

Description

[0001] APPARATUS AND METHOD FOR GENERATING NOVEL VIEWS WITH HUMAN-OBJECT INTERACTION

[0002] FIELD OF THE INVENTION

[0003] This invention relates to novel view synthesis for human-object interactions, which may be used in computer vision applications.

[0004] BACKGROUND

[0005] A classical way to control and render humans, objects and scenes in three dimensions (3D) is to use meshes. Mesh-based templates have been used for 3D human shape and pose recovery from monocular and multi- view RGB images. Once estimated, these templates can be rendered in novel poses. However, the mesh-based representation does not generally allow for photorealistic renderings.

[0006] The photorealism of neural rendering techniques has recently progressed significantly thanks to radiance fields models, introduced by NeRF, as described in Ben Mildenhall, Pratul P. Srinivasan, Matthew Tancik, Jonathan T. Barron, Ravi Ramamoorthi, and Ren Ng, “Nerf: Representing scenes as neural radiance fields for view synthesis”, European Conference on Computer Vision, 2020. This approach combines a neural scene representation with volumetric rendering to address novel view synthesis (NVS) of static scenes from posed images. Despite appealing image rendering quality, NeRF is limited by its rendering speed because many 3D points need to be processed to compute the color of a single pixel. The high computational cost of NeRF often limits its deployment across various applications.

[0007] Another important milestone has been the release of 3D Gaussian Splatting (3DGS), as described in Bernhard Kerbl, Georgios Kopanas, Thomas Leimkuhler, and George Drettakis, “3d gaussian splatting for real-time radiance field rendering”, ACM Transactions on Graphics, 42 (4), 2023. In this approach, the scene is not represented by a neural network, but by an explicit set of primitives shaped as 3D Gaussians. The main strength of 3DGS is its efficiency to render gaussians primitives into RGB images by using a variant of the splatting algorithm, as described in Lee Alan Westover, “Splatting: a parallel, feed- forward volume rendering algorithm”, PhD thesis, USA, 1992. Radiance Fields are originally designed for static scenes, but can be extended to replay dynamic content by incorporating time in the representation.

[0008] The above approaches do not generally model human-object interactions. Human-object interaction is another recurrent topic of study in computer vision and graphics. Early works modelled affordances and human-object interactions using monocular RGB. The collection of several recent human-object interaction datasets, such as that described in Bharat Lal Bhatnagar, Xianghui Xie, Ilya A Petrov, Cristian Sminchisescu, Christian Theobalt, and Gerard Pons-Moll, “Behave: Dataset and method for tracking human object interactions”, Proceedings of the IEEE / CVF Conference on Computer Vision and Pattern Recognition, pages 15935-15946, 2022, has allowed the computer vision community to make significant progress in joint 3D reconstruction of human-object interactions. These datasets have also led to the development of methods that synthesize object-conditioned controllable human motion using 3D mesh approaches. As such, these approaches inherit all the limitations of mesh-based representations, including their inability to generate photorealistic images.

[0009] There is currently a lack of suitable solutions that address a problem of controllable human-object novel view synthesis that enables full control of body and object, together with free camera navigation.

[0010] Most current work focuses on reconstruction of static 3D scenes, where there are no temporal changes. This can result in a lack of control, because when dynamics are considered, the temporal change cannot be controlled. Furthermore, most existing methods for controllable rendering focus only on the human and ignore the environment, such as objects in the background and the 3D scene, and may not allow reconstruction when there is heavy occlusion in the ground truth images.

[0011] In existing methods, if an object is considered, it is generally assumed to be an extension of the human hand, as in the approach described in Liu et al., “Hosnerf: Dynamic human-object-scene neural radiance fields from a single video”, Conference on Computer Vision and Pattern Recognition 2024, and remains attached to the hand throughout novel view rendering. This limits existing work to interactions such as waving an object, and most human-object interactions such as sitting or grabbing cannot be modelled.

[0012] Additionally, many prior approaches ignore photo-realism. As they model the body using skeletons or meshes, they cannot be rendered in a photo-realistic manner.

[0013] It is desirable to develop an approach that may overcome at least some of the above issues.

[0014] SUMMARY OF THE INVENTION

[0015] According to a first aspect, there is provided a method for reconstructing a model for human-object interaction, the model defining three-dimensional Gaussian primitives, the method comprising: receiving multiple input images, each input image depicting a human interacting with an object; based on the input images, using a candidate model, modelling the human using a first set of multiple three-dimensional Gaussians; based on the input images, using the candidate model, modelling the object using a second set of multiple three-dimensional Gaussians; separately rendering the first set of three-dimensional Gaussians and the second set of three-dimensional Gaussians to form a first rendered output and a second rendered output respectively; jointly rendering the first set of three-dimensional Gaussians and the second set of three-dimensional Gaussians to form a third rendered output; combining the first, second and third rendered outputs to determine one or more regions where the human is occluded by the object; in dependence on the one or more regions where the human is occluded by the object, forming a rendered image of an interaction between the human and the object; comparing the rendered image with a ground truth image; and updating parameters of the candidate model based on the comparison between the rendered image and the ground truth image.

[0016] This method models the human and object separately using Gaussian splats, which provide the flexibility of controlling both the human and the object while being able to model complex 3D shapes along with renderings having a photorealistic appearance.

[0017] Combining the first, second and third rendered outputs may be used to form an occlusion map which indicates one or more regions where the human is occluded by the object. The occlusion map may be used to determine loss functions during training.

[0018] The rendered image may be formed by combining and jointly rendering the first set of three-dimensional Gaussians and the second set of three-dimensional Gaussians. The rendering may be performed based on a loss function determined in dependence on the one or more regions where the human is occluded by the object.

[0019] The method may comprise: selecting an input image of the multiple input images, wherein the selected input image has the least occlusion between the human and the object of the multiple input images; for the selected input image, determine a coarse location for the object; using the coarse location for the object, initialize a random set of Gaussian parameters; optimize the random set of Gaussian parameters to determine a static reconstruction of the object; and in dependence on the static reconstruction of the object, determine a rigid transformation for the object from the selected input image to one or more subsequent image frames. This may allow a 3D object to be reconstructed and tracked.

[0020] The method may comprise determining the coarse object location by triangulating a three-dimensional point from a plurality of input images, each of the plurality of input images being captured by a different imaging device (for example, camera). This may allow the coarse object location to be conveniently determined.

[0021] The determined rigid transformation of the object from the input image to a subsequent image frame may be such that rendered images formed by rendering the second set of three-dimensional Gaussians deformed according to the rigid object transformation for multiple camera poses match respective original images available at the subsequent image frame for the multiple camera poses. The rigid transformation of the object may be determined by: forming one or more preliminary outputs by rendering the second set of three-dimensional Gaussians deformed according to a preliminary rigid object transformation; comparing the one or more preliminary rendered outputs with a corresponding ground-truth image to determine a respective image similarity measurement for the preliminary output; and updating the preliminary rigid object transformation in dependence on the image similarity measurement(s). The comparison of the rendered outputs with ground truth images may allow the model to be updated to improve the accuracy of the model.

[0022] The method may further comprise: initialising a first set of Gaussian parameters for the human and a second set of Gaussian parameters for the object in a canonical space; extracting a first set of features for the object in the canonical space; extracting a second set of features for the object in the canonical space; mapping the first set of features and the second set of features onto the first and second sets of Gaussian parameters respectively in the canonical space; and modifying the first and second sets of Gaussian parameters in the canonical space to a posed space. By performing these steps in a canonical space, high-quality 3D Gaussian splats may be reconstructed even when the number of cameras is sparse.

[0023] The method may comprise modifying the first set of Gaussian parameters to the posed space using linear blend skinning and / or modifying the second set of Gaussian parameters to the posed space using the determined rigid transformation for the object. This may allow the Gaussian for the human to be deformed to represent the pose of the human.

[0024] The Gaussian parameters may comprise a colour, scale, opacity and rotation for each Gaussian. The use of Gaussian primitives having such parameters may allow for more photorealistic rendering.

[0025] The first and second sets of features may be extracted in the canonical space using learnt triplane features. Modifying the Gaussian parameters in the canonical space to a posed space may comprise deforming the Gaussians having the Gaussian parameters using a determined pose of the human from a ground truth image. By performing these steps in a canonical space, high-quality 3D Gaussian splats may be reconstructed even when the number of cameras is sparse.

[0026] The method may comprise optimizing the first and second sets of three-dimensional Gaussians such that one or more parts of the first and second sets of three-dimensional Gaussians remain in contact with each other. This may allow for more realistic rendering at regions where the human and object are in contact.

[0027] The one or more parts of the first and second sets of three-dimensional Gaussians may comprise one or more pre-defined regions of three-dimensional space around known positions of the hands of the human. This may allow the parts of the human and object that are in contact to be determined. According to another aspect, there is provided a device for reconstructing a model for human-object interaction, the model defining three-dimensional Gaussian primitives, the device comprising one or more processors configured to: receive multiple input images, each input image depicting a human interacting with an object; based on the input images, using a candidate model, model the human using a first set of multiple three-dimensional Gaussians; based on the input images, using the candidate model, model the object using a second set of multiple three-dimensional Gaussians; separately render the first set of three-dimensional Gaussians and the second set of three-dimensional Gaussians to form a first rendered output and a second rendered output respectively; jointly render the first set of three-dimensional Gaussians and the second set of three-dimensional Gaussians to form a third rendered output; combine the first, second and third rendered outputs to determine one or more regions where the human is occluded by the object; in dependence on the one or more regions where the human is occluded by the object, form a rendered image of an interaction between the human and the object; compare the rendered image with a ground truth image; and update parameters of the candidate model based on the comparison between the rendered image and the ground truth image.

[0028] Such a device can model the human and object separately using Gaussian splats, which provide the flexibility of controlling both the human and the object while being able to model complex 3D shapes along with renderings having a photorealistic appearance.

[0029] According to a further aspect, there is provided one or more computer programs for instructing a computer comprising one or more processors to implement the method above.

[0030] According to a further aspect there is provided a data carrier storing in non-transitory form the one or more computer programs above.

[0031] BRIEF DESCRIPTION OF THE FIGURES

[0032] Figure 1 schematically illustrates using multi- view camera images to create a human-object interaction setup which can be controlled by changing the object and human pose and rendered from new camera viewpoints.

[0033] Figure 2 schematically 3D object tracking.

[0034] Figure 3 schematically illustrates mapping of features extracted from a canonical space onto Gaussian parameters in canonical space. These canonical parameters are transformed in a posed space and rendered to match provided ground-truth images. Figure 4 schematically illustrates the model training process.

[0035] Figure 5 schematically illustrates the process at test time, where are test human post and test object pose are input and a rasterized image is output by the trained model.

[0036] Figure 6 schematically illustrates test-time finetuning of the model.

[0037] Figure 7 shows the steps of an exemplary method for reconstructing a model for human-object interaction.

[0038] Figure 8 shows an example of a device for implementing the methods described herein and some of its associated components.

[0039] DETAILED DESCRIPTION

[0040] Embodiments of the present invention concern a controllable human-object Gaussian splatting method for generating controllable renderings of human-object interactions. Given a set of multi- view RGB images of human-object interactions, the method can reconstruct deformable templates of the human and the object, represented as separate sets of Gaussian splats, and allows for joint renderings of these templates from novel camera viewpoints.

[0041] Unlike most existing work, which solely focuses on human reconstruction and ignores object interactions as background, the method described herein explicitly reconstructs both human and object, thereby allowing for controllable renderings of humanobject interactions in different poses from novel camera viewpoints. An occlusion guided loss may be used that allows for reconstructions under significant occlusions. Furthermore, unlike existing work which either predicts the parameters of the individual Gaussian splats directly or uses static features, pose-guided features are used which allow for reconstruction of fine-grained pose dependent deformations.

[0042] In general, there are two separate Gaussians splats learnt: one static that does not change, apart from a rigid transformation between image frames (for the object), and another dynamic which changes with pose (for the human). During reconstruction of the model, the Gaussian splats used to generate rendered images can be constrained to be displaced by a learnable function that operates on the set of canonical Gaussians. This ensures temporal coherency during reconstructions. Furthermore, by simply changing the parameters of the deformation function after training, the method can be used to generate renderings of human-object interaction in novel poses from novel camera viewpoints. Since the method models humans and objects separately, this allows to retarget object motion from one sequence to another and generate animations of humans interacting with different objects.

[0043] Figure 1 shows the use of multi- view camera images to create a human-object interaction setup which can be controlled by changing the object and human pose and rendered from new camera viewpoints. The method can use multi- view human-object interaction RGB data to recreate controllable human-object interaction which can be rendered from novel camera views.

[0044] During training, the multi- view camera images 101 are used to train a model. At test time, poses for the human 102 and the object 103 are input (which can be input to the model as a driving human pose signal and a driving object pose signal respectively) and a rendered image 104 of the human interacting with the object according to the input human and object poses, 102 and 103 respectively, is produced. The images can be rendered from novel viewpoints, resulting in controllable humanobject interaction image synthesis for novel view synthesis. The system allows for full control of body and object while allowing for free-camera viewpoint navigation, giving an automatic, control-driven system for rendering human-object interaction from novel views in multiple poses and interaction setups.

[0045] As will be described in more detail, the method learns parameters of two separate sets of Gaussian primitives which are then combined together to generate RGB images of human-object interaction. The rendered images are compared to ground truth images and the comparison is used to update the model parameters until the model converges.

[0046] The method uses multi-view data of humans interacting with objects (shown at 101 in Figure 1). These RGB images are used to train two separate Gaussian sets (each comprising multiple 3D Gaussian primitives) in canonical space, which can be reposed and re-rendered from novel views.

[0047] The method can use a template mesh model with linear skinning weights to represent the human subject.

[0048] In the particular implementations described herein, the Skinned Multi-Person Linear (SMPL) body model (as described in Matthew Loper, Naureen Mahmood, Javier Romero, Gerard Pons-Moll, and Michael J. Black, “SMPL: A skinned multi-person linear model”, ACM Trans. Graphics (Proc. SIGGRAPH Asia), 34(6):248: 1-248: 16, 2015) may be used to represent the human subject.

[0049] The method may use a differentiable function M(9, t,p): JR>72 X3 X1° J^689OX3mapSpOseg global translation t and shape pl parameters to the vertices of a human mesh. The final output is a watertight triangulated mesh M (Af (0, t, / J ), F) where

[0050]

[0051] F describes the 13776 faces of the mesh. The underlying skeleton (for example, the skeleton of SMPL) may have 23 joints. The pose parameters θ ∈ ℝ72correspond to the relative orientation of each joint in the skeleton, expressed in axis-angles. The shape parameters β ∈ ℝ10are the PCA coefficients of a shape space learnt from a corpus of registered scans. The notation Mn(θ, t, β) ∈ ℝ3is used to indicate the nthvertex.

[0052] 3D Gaussian splatting, 3DGS (for example as described in Bernhard Kerbl, Georgios Kopanas, Thomas Leimkuhler, and George Drettakis, “3D Gaussian splatting for real-time radiance field rendering”, ACM Transactions on Graphics, 42 (4), 2023) uses a set of 3D Gaussian primitives Q (referred to as a Gaussian ‘splat’ of multiple 3D Gaussians) to represent static 3D scenes which can be rendered in real-time using rasterization. Each 3D Gaussian in the set 풢 is parametrized by its mean x, covariance Σ, opacity α and color parametrized by spherical harmonics coefficients p. To ensure positive semi-definiteness, the covariance matrix is decomposed into this scaling S, transpose T and rotation components R.

[0053] Σ = RSSTRT

[0054] A quaternion is used, q ∈ ℝ4, to represent rotation, which can be trivially converted to a valid covariance matrix to ensure that the matrix represents a valid rotation. The scale matrix is parametrized by a scaling vector s.

[0055] Herein, the following notations are used to denote a set of standard 3D Gaussian splats:

[0056] {xk, qk, αk, sk, pk}Kk=1

[0057] where K is the total number of Gaussian splats and can vary from one scene to another.

[0058] The 3D Gaussians are projected to the 2D image plane via alpha blending. Given a viewing transformation W and the Jacobian of the affine approximation of the projective transformation J, the 2D covariance matrix in camera co-ordinates is given by: Σ' = JWΣ WTJT1:2,1:2. The pixel color C is computed by blending 3D Gaussian splats that overlap at the given pixel, sorted according to their depth:

[0059] C = Σi(α'iΠi-1j=1(1 − α'j))ci

[0060] where α'idenotes the learned opacity αiweighted by the probability density of i-th projected 2D Gaussian at the target pixel location, c denotes the view-dependent color obtained from spherical harmonics coefficients p. The parameters of the individual 3D Gaussians are optimized by comparing rendered pixels against provided ground truth images via a photometric loss. During optimization, 3D Gaussian splatting adaptively controls the number of 3D Gaussians via periodic densification and pruning.

[0061] Given a set of multi- view RGB images of humans inter-acting with objects, captured from N cameras at T temporal frames, {c}t=i c=ialong with estimates of camera parameters {y£c}£“^“™, human pose {0GT}£=1and the object 6D pose {£GT}£=1, the method described herein aims to learn a deformable function f(f), ip; y) that maps new human and object pose parameters to RGB images rendered from novel camera viewpoints.

[0062] Two sets of deformable Gaussian splat parameters 풢Oand 풢Hare estimated in canonical space for the object and human respectively. These are deformed using the provided human and object pose parameters to the rendering space and learnt using an occlusion guided photometric loss, as will be described in more detail below. The method performs feature-based reconstruction in canonical space. Instead of learning per-frame Gaussian splat parameters, as in traditional methods, all per-frame rendered images are constrained to be deformations of a set of Gaussian parameters 풢Oand 풢Hfor the object and human respectively. This allows for temporal coherency and for the generation of renderings of novel human object poses by replacing the deformation parameters.

[0063] Due to significant occlusion between humans and objects, learning the canonical Gaussian parameters, as in the original 3DGS formulation, directly can lead to blurry results. Instead, canonical Gaussian parameters are learned in feature space which allows to incorporate spectral priors in the learning process and hence to deal with occlusion. Specifically, H x W I -dimensional planes FOJFH6 R3x / / xl’ / x / are learned.

[0064] Next, each splat location for the human or object xkis projected onto each of the three feature planes to obtain feature vectors via bilinear interpolation. These features are then concatenated along the feature dimension. This operation may be denoted as

[0065]

[0066] fk= π(xk; F) ∈ ℝ3l.

[0067] Using multiple shallow multi-layer perceptrons (MLPs), these feature vectors are mapped onto Gaussian parameters. Instead of learning the full position of the Gaussian ykdirectly, only a displacement on top of the original position xkis learned. For the human Gaussians, in addition to the original Gaussian splat parameters, linear blend skinning weights wkare also learned which are then used for pose-guided deformations.

[0068] Hence the two sets of canonical Gaussian splats are defined as:

[0069] c — iv" „H „ll. JI

[0070] t)H — IVk - Rk ^k >sk ■Pk 'Mk Sk=l

[0071] — (yronor,o -O ^.0

[0072]

[0073] JO — lyk -c:lk -ak -sk -Pk -Mk Sk=l

[0074] where all the defined human and object parameters are learnt by mapping the queried features fkand f° at positions xkand xkonto the requisite human and object Gaussian parameters.

[0075] Although the canonical Gaussians are represented using standard 3DGS parameters, these parameters are not learned. Instead, the network weights and initial Gaussian splat locations xkare learned that are mapped onto the 3DGS parameters.

[0076] The approach allows for the learning of dynamic Gaussians. Learning static canonical Gaussians works well for rigid objects which do not change temporally. However, humans change dynamically with pose and there are pose-dependent deformations that linear-blend skinning alone cannot model. As a consequence, the above formulation may in some implementations lead to blurry and oversmooth results for humans. To further address this, the canonical human Gaussians may be decomposed into static and dynamic components.

[0077] Specifically, the ground truth human pose represented as axis angles θGTis first transformed to its rotation matrix representation and this rotation matrix is normalized by subtracting the rotation matrices corresponding to the rest pose from each frame Normalized pose as input Rt- pose corresponding to the Ground truth frame, r = Rt− R*.

[0078] Augmented feature vectors for all canonical Gaussians are defined as f'k= [fk;r], Note that though the pose part is same for all Gaussians, it varies from one temporal frame to another. The augmented feature is mapped using an-other set of small MLPs to dynamic appearance independent Gaussian parameters which change the geometry of the original canonical Gaussian y'k=

[0079]

[0080] 9x(f fcX Q k ~ 9x(f fcX Sk— Rx(f k)x The final canonical human Gaussian is defined as:

[0081]

[0082] GH = {y" +y'k>q'k° • s’k,pk,^

[0083] where the operator ° is used to denote a quaternion obtained by converting the matrix product of two quaternions represented as matrices.

[0084] This design choice allows to split the underlying geometry of a human into a shape dependent and a pose-dependent component. One can think of this decomposition as learning a mixture-of-experts of canonical Gaussians splats where each expert learns to respond to one particular human pose.

[0085] The two canonical Gaussian splats for the human and the object are then deformed to represent human and object splats in posed space. In this example, for deforming the human splat, the standard Linear Blend Skinning algorithm with the learnt skinning weights is used. The blend skinning weights are used to obtain a rotation matrix for every individual Gaussian splat from the rotation matrices of every individual joint of the SMPL skeleton. This rotation matrix is used to rotate the original Gaussian splat locations, as well their orientations.

[0086] For each individual Gaussian splat k, the learned skinning weights are used to produce a convex combination of the J rotation matrices associated with the J skeleton joints.

[0087] J

[0088]

[0089] J=i

[0090] where the function RHj(θ) maps a SMPL pose vector expressed in axis-angles to the corresponding rotation matrix representation of its jthjoint. Using this rotation matrix, the geometric components of the canonical human gaussian are transformed as

[0091] 풯(풢H; θ) = {RHk(θ)yHk+ t(θ), RHk(θ)qHk}Kk=1

[0092] Similarly, every object Gaussian splat is rigidly transformed using the object pose parameters (see object tracking below) to posed space.

[0093]

[0094] 풯(풢O; φ) = {ROk(φ)yOk+ t(φ), ROk(φ)qOk}Kk=1

[0095] Here, the notation Rqis used to denote the quaternion resulting from multiplying a rotation matrix with a quaternion.

[0096] The deformation transformations only affect the position y and orientations q of the two canonical Gaussians 풢Hand 풢O. The other Gaussian parameters (scale, spherical harmonics and opacities) remain unchanged.

[0097] To deform the Gaussians in the posed space for the object, 3D object reconstruction and tracking can be performed as follows to obtain a rigid transformation of the object from a first frame to subsequent frames such that the rendered image matches the multi-view ground truth images available at each frame.

[0098] Figure 2 schematically illustrates steps of the 3D object reconstruction and tracking. As illustrated at 201, a keyframe with least occlusion between the human and the object is selected. This may be performed using any suitable method. For example, the frame may be selected manually, or the total pixels in all camera frames with object masks may be computed. In the latter case, the frame with the most pixels has least occlusion and is selected.

[0099] Further, an 2D object segmentation method is applied to the camera views at 202 for the keyframe selected, obtaining thus a coarse object mask for all the camera views. Based on this object segmentation in 2D, a point cloud initialization can be performed in 3D.

[0100] At 203, the point cloud initialization is performed. The point cloud comprises multiple 3D points that define the positions of the 3D Gaussians. For this selected frame with least occlusion and the object segmentation masks or bounding boxes, using multiple images from multiple cameras, a triangulation procedure is performed for a 3D point to find a coarse object location. Two frames diametrically opposite to each other may be selected and one 3D point may be selected in both these frames. The input image that has least occlusion and any other image that depicts the object at the same position can be used for triangulation. The 3D point can be determined automatically, for example by selecting the centre of a bounding box containing a mask for the object.

[0101] At 204, using the coarse 3D object location, a random set of Gaussian parameters is initialised using Gaussians randomly centred at the 3D point corresponding to the coarse object location which are then optimized to find a static reconstruction of the object. Using only static cameras for the initial keyframe, the Gaussian parameters are optimized to reconstruct the object using the original 3DGS rasterization pipeline.

[0102] At 205, the 3D object tracking may use iterative reconstruction. The object parameters for Gaussians at timestep t are initialised with parameters from timestep t-1. This reconstruction is then used to find a rigid transform from the first image frame to subsequent image frames.

[0103] At 207, the rigid transformation of the object is determined. Here, T and R are transpose and rotation components respectively. At 207, the rigid object transformation is output.

[0104] Mathematically, using one temporal frame t* with minimal occlusion, the static Gaussian splat parameters that best explain the segmented object pixels in the multi- view RGB setup can be found using the following:

[0105] . N

[0106] ΣNc=1‖ℛ(풢t*O; γc) − Oct*· Ict*‖2

[0107]

[0108] arg min풢

[0109] Once the Gaussian splat parameters describing one frame have been obtained, the rigid transformation that maps the set of Gaussian splats at timestep t* to the renderings at subsequent timesteps at t.

[0110] N

[0111] yiK^’; 0£); / c-o^i£c||2

[0112] arg minφ

[0113]

[0114]

[0115] The pose parameters at timestep t may be initialised with the pose parameters from timestep t — 1. Another simultaneous problem addressed by the present approach is the 3D reconstruction of humans and objects under significant occlusion. As mentioned previously, the method may use occlusion-aware Gaussian fitting, via occlusion guided feature-based reconstruction in canonical space.

[0116] In addition to the ground truth images captured by camera c at timestep t, it is assumed that a human mask ff£, and an object mask O£care available. Once the two deformed Gaussian splats (J), = T; yt), Qg = T (Qo; <f>t) are obtained for the human and the object respectively, two individually rendered images are obtained for the human and the object, yc)-Iot=R(db’' yc), and a jointly rendered image for the human and the object (J yc). Here, 7?(.; y) is u

[0117]

[0118] sed to denote a Gaussian rasterizer with fixed camera parameters y.

[0119] The two separate images are compared against ground truth human and object images, with the pixels for the other entity (i.e. object for human and vice-versa) ignored.

[0120] For timestep t and camera c, the losses ℒ for the human ℒctH, the object ℒctOand the jointly rendered image ℒctAare given as:

[0121] ℒctH= ‖Ict· Hct· (1 − Oct) − IcH· (1 − Oct)‖2 ℒctO= ‖Ict· Oct· (1 − Hct) − IcO· (1 − Hct)‖2

[0122]

[0123] ℒctA= ‖Ict· (Oct+ Hct) − IcA‖2The loss is formulated such that there is no weight on the pixels for the regions of known-occlusion (i.e. for the pixels known to correspond to the object). The regions of known occlusion of the human by the object may be determined by combining the separately and jointed rendered Gaussians for the human and the object. This may be expressed in the form of an occlusion map. The occlusion map, indicating one or more regions where the human is occluded by the object, may be used to formulate the loss functions by indicating pixels where there is occlusion of the human by the object. When the human is rendered, the Gaussian parameters are free to be whatever they want and vice-versa when the object is rendered. As a feature-based formulation is used and multi- view consistency is enforced, this method can generate coherent solutions.

[0124] Figure 3 schematically illustrates the mapping of features extracted from a canonical space onto Gaussian parameters in the canonical space. These canonical Gaussian parameters are transformed in a posed space and rendered to match provided ground-truth images. The object-occlusion-aware human reconstruction recovers the appearance of the human.

[0125] Since the camera setup may be sparse (for example, four views), instead of directly mapping points onto Gaussian splat parameters, a learnable feature grid (shown as triplane features F in Figure 3) is first queried and features extracted from the grid are used to map points onto Gaussian splat parameters. This allows the network to learn a smoothness prior in feature space which ensures acceptable results even for sparse camera setups. These triplane features can be seen as a large grid in the 3D space. For each Gaussian, given its position, features can be extracted by interpolating features from the grid. This is a way to ensure that Gaussians close in space have similar parameters.

[0126] It may be assumed that the SMPL parameters are available from an existing off-the-shelf method and the object rigid transformation is available.

[0127] Two sets of Gaussian parameters are initialized in a canonical frame: one for the human and one for the object. Using the x, y, z position points for the object and human Gaussians, shown at 301 and 302 respectively in Figure 3, sets of features 305 and 306 are extracted in canonical space for the object and the human using learnt triplane features 303 and 304. The point-wise features 305, 306 are extracted for both object and human in the canonical space. The features 305, 306 are then mapped to Gaussian parameters 307, 308 used for RGB image rendering using an occlusion aware loss. By performing all computational steps in a canonicalized feature space, the method allows for the reconstruction of high-quality 3D Gaussians Splats even when the number of cameras are sparse.

[0128] Each Gaussian position point in the canonical space has three dimensions x, y, z. The three planes of the triplane features are defined as the xy, xz, yz planes of a feature grid. In one example, at each 2D point of these feature planes, a 256-dimensional feature vector is stored. This vector can be optimized. The 3D coordinate frame is centred at the origin. Each Gaussian position point in the 3D canonical space is projected onto these three feature planes. Once the point is projected, the three 256-dimensional feature vectors in these feature planes are lifted. The extracted features are then mapped onto the Gaussian parameters in the canonical space.

[0129] These Gaussian parameters 307 and 308 for the object and the human respectively can then be modified using the object rigid transform 309 (for the object) and linear blend skinning or SMPL body pose 311 (for the human) into world coordinate systems (posed space), shown at 310 and 312 for the object Gaussians and the human Gaussians respectively.

[0130] The two sets of Gaussian splats 310 and 312 are rendered separately and jointly, i.e. the Gaussian splats for the human 312 and the object 310 are rendered separately and together, by human Gaussian rasterizer 313, object Gaussian rasterizer 314 and joint Gaussian rasterizer 315. The resulting renderings are combined to determine one or more regions where the human is occluded by the object, which can be used to formulate the training loss functions. The two sets of Gaussians are rendered jointly to form a rendered image of the human interacting with the object, which is then compared with a ground truth image.

[0131] This allows to impose occlusion aware losses on the human and the object rendering for occlusion guided reconstruction, as shown at 316. The resulting trained model is indicated at 317. The trained model 317 can then be used for inference.

[0132] Since the parameters are learned in feature space and multi-view consistency is enforced, this allows the method to generate coherent results. This allows for the reconstruction of 3D humans and objects in canonical space that can be easily re-posed to novel poses and re-rendered from novel camera views

[0133] Figure 4 shows a further overview of the training process.

[0134] Using the x, y, z position points for the human and object Gaussians, shown at 401 and 402 respectively, sets of features 403 and 404 are extracted in canonical space for the human and the object using learnt triplane features 405 and 406.

[0135] The point-wise features 403, 404 are extracted for both human and object in the canonical space. The features 403, 404 are then mapped to Gaussian parameters 407, 408.

[0136] The Gaussian parameters 407, 408 in the canonical space are deformed using the human and object poses. The human pose 412 and the object pose 414 are estimated from the ground truth image 411. The Gaussian parameters 407 and 408 are deformed to the posed space in dependence on the human pose 412 using linear blend skinning, as shown at 413, and the rigid transformation for the object 415 respectively.

[0137] The two sets of Gaussian splats, shown at 409 and 410, for the human and the object respectively, are deformed to the posed space and then rendered separately and jointly, i.e. the Gaussian splats for the human 409 and the object 410 are rendered separately and together, by human Gaussian rasterizer 416, object Gaussian rasterizer 417 and joint Gaussian rasterizer 418. The two sets of Gaussians for the human and the object combined to form the rendered image 419 of the human interacting with the object, which is then compared with a ground truth image.

[0138] Figure 5 summarises the inference process, at test time. The test human pose 501 and the test object pose 502 are input. These are used to deform the learned Gaussian splats for the human and the object, illustrated at 503. A Gaussian rasterizer 504 then outputs a rendered image of the human-object interaction 505.

[0139] The reconstructed human and object Gaussian splats in 3D space may not fit perfectly with each other. For example, there may be intersections and regions where the Gaussian splats overlap.

[0140] A test-time contact guided fine-tunning for Gaussian splatting may be performed to guarantee the human is in contact with the object.

[0141] To ensure that the two separately reconstructed Gaussian splats for the human and the object do not overlap, it may be enforced that Gaussians close to SMPL contact regions (which may be close to the SMPL hands of the human) remain in contact with the object at frames where the human is known to be in contact with the object. For example, the splats may be optimized such that when there is contact-annotation provided in the dataset, the two splats remain in touch with each other.

[0142] At each temporal frame of the test-time video, it may be assumed that either 1 ) the user provides whether the human and object are in contact (for example, the user may indicate when the human-object contact starts), or 2) this is provided in the data.

[0143] The contact between the human and the object is detected. The user may also provide the rough region of the object which remains in contact with the human: for example, the handle of a brush or the surface of a couch. At each temporal frame, the closest Gaussian location in the human splat corresponding to the Gaussian splats located in this user specified region is found. The location of the object splat can then be optimized such that the distance between these points is minimized.

[0144] This test-time finetuning of the trained model 600 to produce the final rendered image is schematically illustrated in Figure 6.

[0145] At 601, contact detection is performed. In this example, it is assumed that either the user provides the frames in which contact happens, or a simple check can be performed to see if the velocity and acceleration of the feet or hip remain unchanged to detect contacts.

[0146] At 602, a human contact region search is performed. In this example, it is assumed that it is known which region(s) the contacts happen in - i.e the feet, hips, hands etc. The SMPL per vertex segmentations may be known. Using this information, segmentation information is mapped onto human Gaussians.

[0147] At 603, the nearest neighbour in the object is determined. For each splat in the object, the nearest human splat in the segmented subset of the human splat is looked up (e.g. only find the nearest neighbour in the hand or feet or hip).

[0148] At 604, contact guided optimization is performed. The object Gaussian splat is optimized to minimize distance between the object splat and nearest splat in the segmented part of the human.

[0149] At 605 rasterization is performed by a Gaussian rasterizer and the final rendered image is output at 606. Practically, the method may be implemented in the open-source PyTorch framework. For all mappings from canonical features to Gaussian parameters, multi-layer perceptrons (MLPs) with three hidden layers may be used. The camera setup may have, for example, 4 to 60 cameras. The Adam optimizer may be used for all aforementioned optimization problems with decaying learning rates.

[0150] Figure 7 shows an example of a method for reconstructing a model of a 3D conversational head of a subject. At step 701, the method comprises receiving multiple input images, each input image depicting a human interacting with an object. At step 702, based on the input images, using a candidate model, the method comprises modelling the human using a first set of multiple three-dimensional Gaussians. At step 703, the method comprises, based on the input images, using the candidate model, modelling the object using a second set of multiple three-dimensional Gaussians. At step 704, the method comprises separately rendering the first set of three-dimensional Gaussians and the second set of three-dimensional Gaussians to form a first rendered output and a second rendered output respectively. At step 705, the method comprises jointly rendering the first set of three-dimensional Gaussians and the second set of three-dimensional Gaussians to form a third rendered output. At step 706, the method comprises combining the first, second and third rendered outputs to determine one or more regions where the human is occluded by the object. At step 707, the method comprises, in dependence on the one or more regions where the human is occluded by the object, forming a rendered image of an interaction between the human and the object. At step 708, the method comprises comparing the rendered image with a ground truth image. At step 709, the method comprises updating parameters of the candidate model based on the comparison between the rendered image and the ground truth image.

[0151] Figure 8 shows an example of a device 800 configured to output and / or implement a model reconstructed using the methods described herein. The device 800 comprises a processor 801 and a memory 802. The memory 802 stores in a non-transient way code that is executable by the processor 801 to implement the respective entity in the manner described herein. The device 800 may be implemented by hardware or may be service-based computing device, for example it may be implemented as a cloudbased computing device. Therefore, the method may be deployed in multiple ways, for example in the cloud, on the device, or in dedicated hardware.

[0152] The device 800 may in some implementations also comprise a transceiver that is capable of communicating over a network with other entities. For example, the device may receive videos and / or images from other entities. Those entities may be physically remote from the device 800. The network may be a publicly accessible network such as the internet. The entities may in some cases be based in the cloud. These entities may be logical entities. In practice they may each be provided by one or more physical devices such as servers and data stores, and the functions of two or more of the entities may be provided by a single physical device. Each physical device implementing an entity comprises a processor and a memory.

[0153] The method allows for the synthesis of RGB images of human-object interaction. By conditioning on the SMPL pose, and the object pose the method allows for the synthesis of images with a variety of human and object poses. This method models the body and object using Gaussian splats which provide the flexibility of modelling complex 3D shapes along with photorealistic renderings.

[0154] Several experiments on human-object datasets (including the Behave dataset, referred to previously) have demonstrated that the method allows for high-quality reconstruction of human and object templates under significant occlusion and for the synthesis controllable renderings of human-object interactions from novel camera viewpoints.

[0155] The applicant hereby discloses in isolation each individual feature described herein and any combination of two or more such features, to the extent that such features or combinations are capable of being carried out based on the present specification as a whole in the light of the common general knowledge of a person skilled in the art, irrespective of whether such features or combinations of features solve any problems disclosed herein, and without limitation to the scope of the claims. The applicant indicates that aspects of the present invention may consist of any such individual feature or combination of features. In view of the foregoing description, it will be evident to a person skilled in the art that various modifications may be made within the scope of the invention.

Claims

CLAIMS1. A method (700) for reconstructing a model for human-object interaction, the model defining three-dimensional Gaussian primitives, the method comprising:receiving (701) multiple input images, each input image depicting a human interacting with an object;based on the input images, using a candidate model, modelling (702) the human using a first set of multiple three-dimensional Gaussians;based on the input images, using the candidate model, modelling (703) the object using a second set of multiple three-dimensional Gaussians;separately rendering (704) the first set of three-dimensional Gaussians and the second set of three-dimensional Gaussians to form a first rendered output and a second rendered output respectively;jointly rendering (705) the first set of three-dimensional Gaussians and the second set of three-dimensional Gaussians to form a third rendered output;combining (706) the first, second and third rendered outputs to determine one or more regions where the human is occluded by the object;in dependence on the one or more regions where the human is occluded by the object, forming (707) a rendered image of an interaction between the human and the object;comparing (708) the rendered image with a ground truth image; andupdating (709) parameters of the candidate model based on the comparison between the rendered image and the ground truth image.

2. The method as claimed in claim 1, wherein the method comprises:selecting an input image of the multiple input images, wherein the selected input image has the least occlusion between the human and the object of the multiple input images;for the selected input image, determining a coarse location for the object;using the coarse location for the object, initialize a random set of Gaussian parameters;optimizing the random set of Gaussian parameters to determine a static reconstruction of the object; and in dependence on the static reconstruction of the object, determining a rigid transformation for the object from the selected input image to one or more subsequent image frames.

3. The method as claimed in claim 2, wherein the method comprises determining the coarse object location by triangulating a three-dimensional point from a plurality of input images, each of the plurality of input images being captured by a different imaging device.

4. The method as claimed in claim 2 or claim 3, wherein the determined rigid transformation of the object from the input image to a subsequent image frame issuch that rendered images formed by rendering the second set of three-dimensional Gaussians deformed according to the rigid object transformation for multiple camera poses match respective original images available at the subsequent image frame for the multiple camera poses.

5. The method as claimed in any of claims 2 to 4, wherein the rigid transformation of the object is determined by:forming one or more preliminary outputs by rendering the second set of three-dimensional Gaussians deformed according to a preliminary rigid object transformation;comparing the one or more preliminary rendered outputs with a corresponding ground-truth image to determine a respective image similarity measurement for the preliminary output; andupdating the preliminary rigid object transformation in dependence on the image similarity measurement(s).

6. The method as claimed in any preceding claim, wherein the method further comprises:initialising a first set of Gaussian parameters for the human and a second set of Gaussian parameters for the object in a canonical space;extracting a first set of features for the object in the canonical space;extracting a second set of features for the object in the canonical space;mapping the first set of features and the second set of features onto the first and second sets of Gaussian parameters respectively in the canonical space; andmodifying the first and second sets of Gaussian parameters in the canonical space to a posed space.

7. The method as claimed in claim 6 as dependent on any of claims 2 to 5, wherein the method comprises modifying the first set of Gaussian parameters to the posed space using linear blend skinning and / or modifying the second set of Gaussian parameters to the posed space using the determined rigid transformation for the object.

8. The method as claimed in claim 6 or claim 7, wherein the Gaussian parameters comprise a colour, scale, opacity and rotation for each Gaussian.

9. The method as claimed in any of claims 6 to 8, wherein the first and second sets of features are extracted in the canonical space using learnt triplane features.

10. The method as claimed in any of claims 6 to 9, wherein modifying the Gaussian parameters in the canonical space to a posed space comprises deforming the Gaussians having the Gaussian parameters using a determined pose of the human from a ground truth image.

11. The method as claimed in any preceding claim, wherein the method comprises optimizing the first and second sets of three-dimensional Gaussians such that one or more parts of the first and second sets of three-dimensional Gaussians remain in contact with each other.

12. The method as claimed in claim 11, wherein the one or more parts of the first and second sets of three-dimensional Gaussians comprise one or more pre-defined regions of three-dimensional space around known positions of the hands of the human.

13. A device (800) for reconstructing a model for human-object interaction, the model defining three-dimensional Gaussian primitives, the device comprising one or more processors (801) configured to:receive (701) multiple input images, each input image depicting a human interacting with an object;based on the input images, using a candidate model, model (702) the human using a first set of multiple three-dimensional Gaussians;based on the input images, using the candidate model, model (703) the object using a second set of multiple three-dimensional Gaussians;separately render (704) the first set of three-dimensional Gaussians and the second set of three-dimensional Gaussians to form a first rendered output and a second rendered output respectively;jointly render (705) the first set of three-dimensional Gaussians and the second set of three-dimensional Gaussians to form a third rendered output;combine (706) the first, second and third rendered outputs to determine one more regions where the human is occluded by the object;in dependence on the one or more regions where the human is occluded by the object, form (707) a rendered image of an interaction between the human and the object;compare (708) the rendered image with a ground truth image; andupdate (709) parameters of the candidate model based on the comparison between the rendered image and the ground truth image.

14. One or more computer programs for instructing a computer comprising one or more processors (801) to implement the method as claimed in any of claims 1 to 12.

15. A data carrier (802) storing in non-transitory form the one or more computer programs as claimed in claim 14.

Images