Volumetric parametric attribute coding for radiance rendering primitives

Abstract

An apparatus includes at least one processor; and at least one memory storing instructions that, when executed by the at least one processor, cause the apparatus at least to: receive as input appearance attribute data of a rendering primitive and geometry attribute data of the rendering primitive; estimate at least one function that predicts an appearance attribute in a volume of a cell of a representation associated with the rendering primitive, using the appearance attribute data of the rendering primitive and the geometry attribute data of the rendering primitive; and code, into an output bitstream, the at least one function that predicts the appearance attribute in the volume of the cell of the representation associated with the rendering primitive.

Description

VOLUMETRIC PARAMETRIC ATTRIBUTE CODING FOR RADIANCE RENDERING PRIMITIVESTECHNICAL FIELD

[0001] The examples and non-limiting embodiments relate generally to volumetric parametric attribute coding for radiance rendering primitives.BACKGROUND

[0002] It is known to process and render video in a multimedia system.BRIEF DESCRIPTION OF THE DRAWINGS

[0003] The foregoing embodiments and other features are explained in the following description, taken in connection with the accompanying drawings, wherein:

[0004] FIG. 1 is an illustration of a Gaussian splat.

[0005] FIG. 2 is an illustration of spherical harmonics on the left, and a visual comparison with DCT basis functions on the right.

[0006] FIG. 3 shows examples of appearance attribute prediction with a volumetric function.

[0007] FIG 4 is an illustration of appearance attributes functions for different LoDs.

[0008] FIG. 5 shows a coding scheme for a selected LoD.

[0009] FIG. 6 shows an inter-LOD coding scheme.

[0010] FIG. 7 shows a decoding scheme for a specific LOD.

[0011] FIG. 8 shows an inter-LOD decoding scheme.

[0012] FIG. 9 shows an encoder according to an embodiment.

[0013] FIG. 10 shows a decoder according to an embodiment.

[0014] FIG. 11 is a block diagram illustrating a system in accordance with an example.

[0015] FIG. 12 is an example apparatus configured to implement the examples described herein.

[0016] FIG. 13 shows a representation of an example of non-volatile memory media used to store instructions that implement the examples described herein

[0017] FIG. 14 is an example method based on the examples described herein.

[0018] FIG. 15 is an example method based on the examples described herein.DETAILED DESCRIPTION OF EXAMPLE EMBODIMENTS

[0019] Volumetric Content

[0020] There are many ways to capture and represent a volumetric scene. The format used to capture and represent it depends on the processing to be performed on it, and the target application using it. Some exemplary representations are listed below.

[0021] A volumetric frame can be represented as a point cloud. A point cloud is a set of unstructured points in 3D space, where each point is characterized by its position in a 3D coordinate system (e.g. Euclidean), and some corresponding attributes (e.g. color information provided as RGBA value, or normal vectors).

[0022] A volumetric frame can be represented as images, with or without depth, captured from multiple view points in 3D space. In other words, it can be represented by one or more view frames where a view is a projection of a volumetric scene on to a plane (the camera plane) using a real or virtual camera with known / computed extrinsics and intrinsics. Each view may be represented by a number of components (e.g. geometry, color, transparency, and occupancy picture), which may be part of the geometry picture or represented separately.

[0023] A volumetric frame can be represented as a mesh. Mesh is a collection of points, called vertices, and connectivity information between vertices, called edges. Vertices along with edges form faces. The combination of vertices, edges and faces can uniquelyapproximate shapes of objects.

[0024] A volumetric frame can be represented as an implicit neural radiance field such as the Neural Radiance Field (NeRF) representation for example. A NeRF is a representation of a 3D scene that is parameterized by a deep neural network (DNN). The network predicts a volume density and view-dependent emitted radiance given the spatial location (x, y, z) and viewing direction, e.g., in Euler angles (0, <P) of the camera. Volume rendering techniques can produce an image by sampling points along rays intercepted by the camera projection model.

[0025] A volumetric frame can be represented as a radiance field such as the Gaussian Splatting (a.k.a., 3DGS, 4DGS, Gaussian Splat) representation. This representation can be seen as a point cloud representation with additional attributes that model the geometry and appearance of the scene in the close neighborhood of a position (x, y, z). Rendering of radiance fields represented as Gaussian splats and other variants using modified versions of the attributes may be performed by simple and fast rasterization or more accurate volume rendering using ray tracing.

[0026] Depending on the capture, a volumetric frame can provide viewers the ability to navigate a scene with six degrees of freedom, i.e., both translational and rotational movement of their viewing pose (which includes yaw, pitch and roll). The data to be coded for a volumetric frame can also be significant, as a volumetric frame can contain large numbers of objects, and the positioning and movement of these objects in the scene can result in many dis-occluded regions. Furthermore, the interaction of light and materials in objects and surfaces in a volumetric frame can generate complex light fields that can produce texture variations for even a slight change of pose.

[0027] A sequence of volumetric frames is a volumetric video. Due to large amount of information, storage and transmission of a volumetric video requires compression.

[0028] Point Cloud

[0029] A point cloud is a data structure that can be used to represent a three-dimensional scene. Point clouds consist of a set of points in a coordinate system. Each point is defined by its three coordinates and may be associated with zero or more attributes. Attributes canhave different types and are used to describe the properties of a point, such as its color or reflectance. Point clouds are created by sensors such as Lidar (“light detection and ranging”) or Radar (“radio detection and ranging”) sensors or by depth cameras or by multi-camera setups or by ultrasound devices or by other distance sensing devices or combinations one or more of the above and, optionally, other devices. There are many use-case scenarios for point clouds. An example use case is a vehicle that uses one or multiple sensors to scan its environment to create a point cloud describing its surroundings. The point clouds can then be used for further processing, e.g., for creating a virtual mapping of the environment, for autonomous driving use cases, or for teleoperation. In some cases, further processing has to be done on a different device than the device that captured the point cloud. In those cases, point clouds or sequences of point clouds have to be transmitted over a network. Due to the large amount of data that point cloud sequences produce, transmitting the data in real-time is challenging. In addition to the use of efficient codecs to reduce the data rate, it may be beneficial to lower the bandwidth requirements by prioritizing certain parts of a point cloud and allowing other parts to be removed from the point cloud before transmission. Moreover, transmission channels (such as wireless networks, packet networks, or other potentially unreliable channels) may not deliver all point cloud data intact, in which case, possibly selective, repair mechanisms may be beneficial.

[0030] Geometry-based point cloud compression (G-PCC)

[0031] G-PCC is an encoding mechanism defined by the Moving Picture Experts Group (MPEG) to reduce the size of a point cloud. G-PCC encodes point clouds with a number of tools that are especially efficient for sparse point cloud data. The geometry of the point cloud is first encoded as an octree and attributes are coded depending on the geometry, in other words, the attribute coding relies on reconstructed geometry data that can be reproduced exactly at encoder and decoder so as to avoid coding drifts and errors for attributes. Different tools are available after octree decomposition within G-PCC specifications, such as “trisoup” that consists in a surface approximation based on a set of triangles defined on octree leaves, and such as predictive occupancy octree coding. Attributes are typically transferred from the original point cloud to the reconstructed geometry. Attributes may be coded with a LOD-based approach combined with a predictive transform and optionally a Lifting Transform. As an alternative approach to LoDs, a Region- Adaptive HierarchicalTransform (RAHT) introduced interesting inter prediction compression performances. In the emerging G-PCC edition 2, these tools have been extended and complemented, offering more access granularity to LoDs, adaptive quantization per LoD, cross-attribute prediction within a LOD, LoD attribute slices, enhanced coding of DC and AC components of RAHT, and overall, a better point cloud inter prediction performance.

[0032] Visual Volumetric Video-base Coding (V3C) - ISO / IEC 23090-5

[0033] ISO / IEC 23090-5 specifies the syntax, semantics, and process for coding volumetric video. The specified syntax is designed to be generic so that it can be reused for a variety of applications. Point clouds, immersive video with depth, and mesh representations can all use ISO / IEC 23090-5 standard with extensions that deal with the specific nature of the final representation. The purpose of the specification is to define how to decode and interpret the associated data (for example atlas data in ISO / IEC 23090-5) which tells a Tenderer how to interpret 2D frames to reconstruct a volumetric frame.

[0034] Two applications of V3C (ISO / IEC 23090-5) have been defined, V-PCC (ISO / IEC 23090-5) and MIV (ISO / IEC 23090-12). MIV and V-PCC use number of V3C syntax elements with a slightly modified semantics.

[0035] MPEG 3DG (ISO SC29 WG7) group has started work on a third application of V3C - the mesh compression (ISO / IEC 23090-29). It is also envisaged that mesh coding reuses V3C syntax as much as possible and can also slightly modify the semantics.

[0036] Gaussian splats

[0037] Gaussian splats can be seen as point clouds with additional attributes as illustrated on Figure 1. These attributes of the Gaussian Splats are: Position (x,y,z); Scale (sx,sy,sz), i.e., non-uniform scale factor of the Gaussian splat; Rotation (rx,ry,rz) or (qx,qy,qz,qw), i.e., the rotation of the Gaussian splat with respect to world coordinates, the rotation is expressed by Euler angles or quaternions; optionally, Normal (nx, ny, nz), i.e., the normal vector of the Gaussian splat; Opacity (alpha), i.e., a factor that indicates the transparency of the Gaussian splat; Color (r,g,b), i.e., the primary color of the Gaussian splat; Spherical harmonics (e.g. SH[0:48] or SH[0: 3*(degree+l)A2]), i.e., the reflectance of the Gaussian splat expressed in spherical harmonics; Sometimes color is represented by the DCcomponent of spherical harmonics, and spherical harmonics contain 45 non-DC components. Spherical harmonics can be seen as DCT basis functions defined on the sphere (see Figure 2).

[0038] The rendering function for a Gaussian splat is as follows:

[0039] In the above equation, S is the covariance matrix and is equal to:2 = RSS7’ / ?7’

[0040] In the above equation, R is a 3x3 rotation matrix expressed with 4 quaternions and S is a diagonal scaling matrix with 3 parameters for scale.

[0041] FIG. 1 is an illustration of a Gaussian splat 100. Position information includes the center 102 of the Gaussian splat 100, where the center 102 is highlighted by a circle. The size and orientation of the shape of the splat 100 is determined by the scale and rotation. The color, transparency and reflectivity are defined by the spherical harmonics and opacity.

[0042] FIG. 2 is an illustration of Spherical Harmonics on the left (202) and a visual comparison with DCT basis functions on the right (204). Similarly to DCT that enables low to high frequency variations on the horizontal and vertical axis of an image, increasing order of spherical harmonics shows increasing frequency variations in vertical and horizontal directions on the sphere. The first component (m=0, 1=0), referred to as DC represents a constant value over the sphere, the second component (m=0, 1=1) enables a variation from south pole to north pole following cos(theta), the third component (m=±l, 1=1) enables a variation from one side of the equator to the other side, etc. Multiplying the three color components RGB to these 15 basis functions on the sphere leads to 45 spherical harmonics components as illustrated on FIG. 2. Some approaches achieve a fast estimation of the Spherical harmonics transform and inverse transform by relying on the FFT.

[0043] Other variants of radiance fields representations include:

[0044] Spherical point clouds, which are similar to Gaussian splats except that the shapeis spherical, i.e., there is only one scale factor and there is no rotation attribute data and no falloff due to the Gaussian probability distribution.

[0045] Ellipsoids, while sharing similar attributes with Gaussian splats, the rendering includes a cut-off at the ellipsoid scale-driven boundary and a constant opacity in the ellipsoid.

[0046] Billboards, where the attributes are common with Gaussian splats but the rendering equation is defined on a rectangle.

[0047] 2DGS or surfels, similar to Gaussian splats but where the smallest scale is set to zero and a normal is defined. These are typically defined on surfaces in the tangent space. The rendering equation is the same as for Gaussian splats.

[0048] Gaussian splats with neural appearance attributes, where view dependent appearance is represented by latent variables. The rendering equation requires a MLP to decode the latent appearance.

[0049] Coding of Gaussian splats

[0050] A few examples of coding formats that are available for Gaussian splat coding are as follows.

[0051] .spz file format for static Gaussian splats. The .spz format is a gzipped stream of data consisting of a 16-byte header followed by the gaussian data. This data is organized by attribute in the following order: positions, scales, rotations, alphas, colors, spherical harmonics.

[0052] The header is as follows in little-endian.

[0053] Values are specified as follows:

[0054] magic: 0x5053474e

[0055] version: 2

[0056] numPoints: number of encoded Gaussians

[0057] shDegree: The degree of spherical harmonics. This must be between 0 and 3 (inclusive).

[0058] fractionalBits: number of bits used to store the fractional part of coordinates in the fixed-point encoding.

[0059] flags: bit field containing flags. 0x1: whether the splat was trained with antialiasing.

[0060] reserved: Reserved for future use. Must be 0.

[0061] Positions are represented as (x, y, z) coordinates, each as a 24-bit fixed point signed integer. The number of fractional bits is determined by the fractionalBits field in the header.

[0062] Scales are represented as (x, y, z) components, each represented as an 8-bit log-encoded integer.

[0063] Rotations are represented as the (x, y, z) components of the normalized rotation quaternion. The w component can be derived from the others and is not stored. Each component is encoded as an 8-bit signed integer.

[0064] Alphas are represented as 8-bit unsigned integers.

[0065] Colors are stored as (r, g, b) values, where each color component is represented as an unsigned 8-bit integer.

[0066] Spherical Harmonics. Depending on the degree of spherical harmonics for the splat, this can contain 0 (for degree 0), 9 (for degree 1), 24 (for degree 2), or 45 (for degree3) coefficients per gaussian.

[0067] The coefficients for a gaussian are organized such that the color channel is the inner (faster varying) axis, and the coefficient is the outer (slower varying) axis, i.e. for degree 1, the order of the 9 values is: shlnl_r, shlnl_g, shlnl_b, shlO_r, shlO_g, shlO_b, shlpl r, shlpl g, shlpl b.

[0068] Each coefficient is represented as an 8-bit signed integer. Additional quantization can be performed to attain a higher compression ratio. This library currently uses 5 bits of precision for degree 0 and 4 bits of precision for degrees 1 and 2, but this may be changed in the future without breaking backwards compatibility.

[0069] Extension of G-PCC. G-PCC can be used to encode arbitrary attributes. An extension of G-PCC requires a simple modification to code quaternions and add the required metadata for pre- and post-processing attributes.

[0070] The current extension of G-PCC for Gaussian Splats is using following notations:• geometry: xyz positions of points• shO: DC values of the spherical harmonics• shl: first 3 AC values of the spherical harmonics• shl: next 5 AC values of the spherical harmonics• sh3: last 7 AC values of the spherical harmonics• opacity: opacity• scale: scale• rotation: rotation

[0071] Table 1 summarizes the number of bits allocated per component (or in other words, attribute).Table 1: G-PCC extension bits allocation per attribute

[0072] G-PCC is used to encode positions as point clouds and the additional Gaussian splat attributes with (Region- Adaptive Hierarchical Transform) RAHT. The G-PCC extension also uses a preprocessing step that voxelizes attributes and components with specific bitdepths that are illustrated on Table 1.

[0073] Bitstream signaling. In this disclosure, the similar terminology is borrowed for encoding syntax as ISO / IEC 23090-5 with for example:

[0074] u(n): unsigned integer using n bits. When n is "v" in the syntax table, the number of bits varies in a manner dependent on the value of other syntax elements. The parsing process for this descriptor is specified by the return value of the function read_bits( n ) interpreted as a binary representation of an unsigned integer with the most significant bit written first. When n is equal to 0, the associated syntax element is not present in the bitstream and its corresponding value shall be inferred to be equal to 0 unless specified otherwise in the semantics.

[0075] ue(v): unsigned integer O-th order Exp-Golomb-coded syntax element with the left bit first.

[0076] The problem consists in efficiently encoding appearance attributes of rendering primitives in a hierarchical way that exploits the redundancy of attributes between primitives. While most methods focus on points with attributes, these methods do not take into account the fact that the primitives represent a volume in space and the primitives’ attributes impact their full volume, not just their central point. This prevents from efficiently capturing the redundancy between primitives based on their volume and orientation rather than only their central point proximity.

[0077] Appearance attributes in a local volume are typically correlated due to the fact the primitives belong to the same object or because or lighting and / or inter-object reflections. Considering rendering primitives as points does not allow to capture these aspects as such primitives are sparser than point clouds for instance. The volume and orientation of the primitive captures the signal that would be represented by a cloud of points and not only a single central point.

[0078] Described herein are methods to code rendering primitive appearance attributes by defining and using prediction mechanisms that capture their volumetric nature. For example, parametric functions may be used to predict attributes in a volume of the scene space given that the corresponding primitives positions and geometry are already coded (or decoded) for a given Level-of-Detail.

[0079] FIG. 3 shows examples of appearance attribute prediction with a volumetric function; (a) shows a voxelized Gaussian splat model consisting of three Gaussian splats (including Gaussian splat 302, Gaussian splat 304, Gaussian splat 306) located at different positions, with different sizes, orientation and appearance inside a grid 308 that can be part of a hierarchical volume discretization such as an octree for example, (b) shows the same grid and the three positions corresponding to the Gaussian splats in (a) where a volumetric gradient function 316 (linear regression 316) is depicted to approximate the appearance attribute in the grid cell, such as appearance attribute 318. Position 310 corresponds to Gaussian splat 302, position 312 corresponds to Gaussian splat 304, and position 314 corresponds to Gaussian splat 306. Positions (e.g. position 310, position 312, position 314) of the Gaussian splats (e.g. Gaussian splat 302, Gaussian splat 304, Gaussian splat 306) can be used to sample the function (e.g. function 316) to predict the corresponding Gaussiansplats appearance attribute (e.g. color appearance attribute 318 of Gaussian splat 302). (c) shows, similarly to (b) the appearance attribute (e.g. color appearance attribute 320 of Gaussian splat 302) being modeled by a function 322, here a second-degree polynomial.

[0080] In FIG. 3, the main concept is illustrated in 2D for the ease of drawing. A set of primitives (the primitives being Gaussian splat 302, Gaussian splat 304, and Gaussian splat 306) are depicted in a voxelized grid 308 and two example functions (316, 322) are also illustrated to approximate the appearance (e.g. color attribute 318 of Gaussian splat 302 and color attribute 320 of Gaussian splat 302) of the set of primitives inside the grid cell, such as grid cell 324. The appearance attributes (e.g. attribute 318) of the primitives (e.g. primitive 302, primitive 304, and primitive 306) can then be predicted by sampling the function (e.g. function 316 or function 322) at the corresponding position (e.g. position 310, position 312, or position 314) of the primitives, or at multiple positions inside the geometric extent of the primitive in the cell (volume of the primitive in space), such as cell 324, or grid 308. This function (e.g. function 316 or function 322) can be estimated on the grid 308 at various levels, if the grid 308 is represented by a hierarchical tree structure such as an octree. It follows that the function can be defined at various levels of detail. In some embodiments, the functions may be coded using a dictionary to achieve different tradeoffs between the quality of the prediction offered by the function and the required bitrate to potentially encode residuals and to signal the function.

[0081] If the grid 308 is represented by a hierarchical tree structure as an octree, instead of having independent cells in the grid (like pixels in an image), an octree gives you a parent / child / brother relationship between cells. These relationships may give more importance to some cells than others, or help finding cells that are close to each other in the grid.

[0082] FIG. 4 is an illustration of appearance attributes functions for different LoDs; (a) shows on the left two different LoDs for the same Gaussian Splat model, LOD i 402 on the right shows one way to aggregate primitives from the LOD i+1 404 on the left, (b) shows functions (namely function 406 and function 408) to approximate the appearance attributes for LOD i and LOD i+1 respectively (e.g. appearance attribute 410 of LOD i 402 and appearance attribute 412 of LOD i+1 404), as well as a potential prediction mechanism 414between the functions of consecutive LODs.

[0083] In FIG. 4, the LOD aspect of the function-based approximation of appearance attributes is illustrated. It is also possible to define such functions for each LOD and / or to define a function at a given LOD to predict the function parameters at another LOD.

[0084] This description provides embodiments for the encoding, decoding and signaling of appearance attributes within a volume at specified Level-of-Details.

[0085] The following terms are used to describe the herein described example embodiments:

[0086] - Rendering primitive: it is a geometric object that can be used for rendering and it is characterized by a position in a N-dimensional space, possibly some additional geometric attributes (e.g. the scales and the rotations for an ellipsoid or a gaussian, or the dimensions and rotation of a rectangle, cube, plane, hypercube, etc.) and possibly some additional attributes (e.g. color, representation learned through gradient descent or a neural network, semantic information, material information, etc.) that is referred to as nongeometric or appearance attributes. One example of rendering primitive is a 3D Gaussian splat.

[0087] - Level-of-Detail (LOD) describes a layer in a scalable and hierarchical representation. A LOD is usually linked to resolution scalability, and for the case of point clouds or rendering primitives to different number of primitives, or density of primitives. Accordingly, LOD 0 represents the coarser and sparser representation of an object or scene (useful when viewed from a large distance to optimize primitive to screen pixel usage). LOD 1 is then a refined version of LOD 0 that contains more primitives, and forms a denser representation. Typically LOD i contains N times more primitives than LOD i+1, with for example N being a power of two, but can also be a smaller or larger amount that may not be uniform in all the scene (some parts of the scene may be denser than others due to the content itself).

[0088] - Children and Parents primitives: these two define a hierarchical relationship between primitives where the Parent is obtained by aggregating the information of the children from LOD i (parent) to LOD i+1 (children). The parent could be the same or adifferent type of primitive w.r.t. the children and it might have different attributes.

[0089] 1 Coding embodiments

[0090] In one embodiment, a pre-processing algorithm takes as input a set of input views for example obtained by multiple cameras, and learns a static or dynamic model of the captured scene as a rendering primitive model, such as for example, 2D, 3D or 4D Gaussian splats, Billboards, Gaussian Textures, etc. This learning mechanism can be for example based on a gradient descent and a differentiable rendering approach.

[0091] In another embodiment, the pre-processing algorithm directly ingests a rendering primitives static or dynamic model, without necessarily having access to input, capture views. This for example can be the case of a model being obtained without access to the capture data, or because the model was designed directly using a synthetic content production toolset for example or because the model was obtained by converting another representation of the scene such as one or multiple static or dynamic mesh(es), point clouds, signed distance functions or an implicit neural representation.

[0092] In one embodiment, a hierarchical tree representation of the model is obtained by for example first voxelizing the model positions and / or geometry attributes according to a selected or set of selected bit depth and accuracy for each of the attributes. The hierarchical representation is built on the voxelized content by applying a hierarchical structure such as for example an octree or any other type of tree structure composed of parent and child relationships. The hierarchical tree does not necessarily partition space uniformly, nor partitions the voxelization grid along world coordinate axes in regular patterns. It is assumed within this description that the hierarchical representation is described fully by a set of parameters and metadata that then encoder signals in or along the bitstream it produces as an output.

[0093] In another embodiment, the scene is represented by a set of separate or overlapping hierarchical tree representations that provide spatial random access capabilities for a rendering application.

[0094] In one embodiment, the hierarchical representation (or each hierachical representation of the scene) is decomposed into Levels of Detail (LoD or LOD) which allowto reconstruct the whole scene or parts of the scene at different resolutions. For example, a LoD 0 represents the coarsest representation of the scene and contains the root(s) of the hierachical representation. The next LoDs contain children of the previous LoD and represent additional data that allows to render the scene with higher spatial resolution, accuracy or for example temporal frames in case of a dynamic representation.

[0095] 1.1 Per LoD encoder

[0096] FIG. 5 shows a coding scheme for a selected LoD. More particularly, an encoder 500 that implements an encoding scheme for a specific LoD of a rendering primitive hierarchical representation is depicted in FIG. 5. The inputs of the encoder comprise the rendering primitives of the selected LoD, together with their geometry attributes 502 and their appearance attributes 504.

[0097] Geometry Coding

[0098] In one embodiment the geometry attributes 502 such as the position, and for example rotation and scale for the case of Gaussian Splats, are encoded by a geometry coding module 506. The geometry coding module 506 could for example use an octree decomposition module that is governed by a Controller module 508. Other alternatives can be thought of as coding geometry attributes with a triangle soup approach, or signed distance coding algorithm etc.

[0099] Function Estimation

[0100] The encoder also comprises a function estimation module 510 that takes as input the rendering primitives appearance attribute data 504, the geometry attribute data 502 and metadata describing the hierarchical representation. For each attribute component or a set of these components, the function estimation module 510 learns or estimates a model that approximates the appearance attribute in the volume of a given cell of the hierarchical representation. A cell is for example a region in the space of the scene that is described by one node or a clique of nodes of the hierarchical representation. This means that the cell can be any set of grid elements, i.e. one or multiple ones. One criterion to select one or several elements is to check if the rendering primitive volume (the ellipsoid) intersects with one or more of this grid elements. The set of instersected grid elements is the “cell”. Forexample, in one embodiment the cell may be all grid elements covered by the volume of one or multiple primitives whose position(s) is / are inside the grid.

[0101] In one 2D example provided in FIG. 3 at (a), the cell consists in a set of four by four voxelized position or grid elements. In another example, the cell may consist in voxels containing 4x4x4 voxelized positions in the grid. In another example, the cell is any volume preferably connected to the volume region of the grid.

[0102] The function estimation 510 may run a set of learning algorithms that are specific to each appearance attribute component or set of components due to an encoder configuration that is used by the Controller module 508. In one embodiment, one of these function estimation algorithms is a linear or polynomial regression module that estimates the attributes in the cell volume by sampling the rendering primitives volume as described by their geometry attributes. The degree and coefficients of the polynomial obtained represent the function parameters for the specified cell and specified attribute.

[0103] In one embodiment, the sampling of the primitives in the volume of the cell is performed by sampling the attribute at the primitive central point for estimating the function that predicts these attributes. In another embodiment, the sampling of the primitive is performed by sampling each primitive at several points that are contained in the volume represented by the primitive. In another embodiment, the sampling may be irregular in the grid and follow for example a Monte Carlo sampling method, for which the pseudo-random generator seed and the number of samples need to be signaled to the decoder. The contribution of the samples of a given primitive to the function estimation may be weighted equally or normalized for a given primitive such that the weight of a sample further away from the center receives a smaller weight than those closer to the center. In another embodiment, the number of samples used for one primitive is proportional to the volume of the primitive and may be different for several primitives in a cell. Another embodiment may sample the cell volume in finer grid subsamples to evaluate the attributes in the cell volume in a regular manner, verifying if a primitive volume is occupying the sample and what attribute contribution it provides. An alternative is to use a non linear regression module and use iterative approaches such as the Levenberg-Marquardt or Gauss-Newton algorithms for example on the least squares difference between the function and the data, by selectingamong a number of function models such as for example the exponential, logarithmic, power-law, trigonometric, hyperbolic or logistic models. In one embodiment, the encoder estimates the models for each cell in brute force to obtain the most compact function parameters and smallest residual signal. Alternatively to using least squares as a loss for minimizing the function error model, the encoder may select a maximum likelihood criterion or Bayesian inference when functions have already been learned at another LoD for example.

[0104] Another alternative consists in using a neural network such as a MLP to regress the appearance attribute component or set of appearance attribute components in the given cell. In particular this would be done with an implicit representation network that takes as input all or part of the geometric attributes of the primitives and outputs the appearance attributes. This network would thus act as a function that fits well the attributes of the primitives and that can properly exploit the correlation between primitives with similar geometry when this is present.

[0105] Function Coding

[0106] Once the function estimated, the function type should be signaled as well as its parameters or coefficients. In case of polynomial function, the degree and coefficients need to be coded for the cell and appearance attribute components. Additionally the degree of the polynomial can be encoded (e.g. with function coding 512) with higher efficiency by considering the number of points in the cell (e.g. if there are only 3 points then the function can be a hyperplane thus bounding the maximum degree of the function). In case of a non linear model, the type of model (exponential, power-law, etc) needs to be signalled and coded as well as the parameters of the model. The same applies to MLPs and other neural network or probabilistic models.

[0107] In case a MLP or other neural network is used to model the set of appearnace attribute components, the network connectivity and weights can for example be coded using ISO / IEC 15938-17 NNC.

[0108] In one embodiment, instead of directly coding the parameters of a given function, a dictionary or codebook can be encoded by means of vector quantization on the parameters.While this can reduce the accuracy of the prediction and increase residual signal energy, this could also reduce the bitrate required to code the functions themselves.

[0109] In another embodiment a Look Up Table (LUT) can be used to code model coefficients or parameters in case the same model is used in several cells.

[0110] Since different attributes might be correlated also the functions approximating them might be. To properly exploit this correlation in one embodiment the function parameters are coded autoregressively.

[0111] Appearance Attribute Prediction

[0112] In the appearance attribute prediction module 514, both appearance and geometry attribute types are inputs of the process. Moreover, the estimated model or function is also provided as an input from the Function Estimation module 510. The estimated model or function is sampled to predict the corresponding appearance attribute components of the rendering primitives within the hierarchical tree representation cell. This can be performed by evaluating the model or function at positions that correspond to the geometry positions of the rendering primitives. In another embodiment, the sample positions for a given rendering primitive are taken at all cell grid centers that are covered by this rendering primitive (therefore taking into account the volume of the rendering primitive). The resulting appearance attribute for the rendering primitive is obtained by a weighted average of the samples that depends on the rendering primitive type. For example, in cases of a Gaussian Splat, weights can be made proportional to the corresponding GS Gaussian function. The encoder needs to signal if multiple samples are used and which method is used to obtain the prediction from the samples.

[0113] In another embodiment, instead of the input geometry attributes, the appearance attributes prediction module 514 takes the decoded and reconstructed geometry attributes by decoding the output of the Geometry Coding module 506. This enables ensuring that encoder and decoder build the same predictions.

[0114] Residual Coding

[0115] The Residual Coding module 516 takes the predictions from the AppearanceAttribute Prediction module 514 and subtracts them from the input appearance attributes 504 to generate a residual signal per cell and per set of appearance attribute components. Once the residual signal is obtained, it can potentially be transformed using for example a Discrete Cosine Transform within the cell.

[0116] Transformed or not, the residuals can be entropy encoded using CAB AC or other alternatives.

[0117] Metadata Coding

[0118] The metadata obtained during the encoding process is signalled and encoded by metadata coding 518 into a specific syntax that is further detailed with respect to the signaling embodiments described herein.

[0119] Output substreams

[0120] The Geometry Coding module 506 produces a geometry substream 520. The output of the Function Coding module 512 and the output of the Residual Coding module are coded in an appearance substream 522. The Metadata Coding module 518 generates a metadata substream 524. The three substreams (namely the geometry substream 520, the appearance substream 522, and the metadata substream 524) may be multiplexed in a LoD i bitstream 526 (LoD i bitstream may also be referred to as an LoD i substream, e.g. in an inter-LoD coding scenario) that is on its turn multiplexed with the other LoD substreams for each LoD of the coded representation.

[0121] In an alternative embodiment, as shown in FIG. 5, the coding into geometry substream 520, appearance substream 522, and metadata substream 524 is not implemented, where the output of geometry coding 506 is coded directly into LoD i bitstream 526 as item 532, the output of function coding 512 is coded directly into LoD i bitstream 526 as item 534, the output of residual coding 516 is coded directly into LoD i bitstream 526 as item 536, and the output of metadata coding 518 is coded directly into LoD i bitstream 526 as item 538.

[0122] 1.2 Inter LoD coding

[0123] FIG. 6 shows an encoder 600 that implements an inter-LOD coding scheme. FIG.6 illustrates one example of an Inter LoD encoder 600. The encoder 600 is similar to the Per LoD encoder 500 and the differences between the per LoD encoder 500 and the inter-LOD encoder 600 are described.

[0124] The encoder 600 also ingests another LoD previously encoded, therefore, typically a coarser LOD level. This could be the previous LoD; for example, LoD i-1 if the current LoD index is equal to i. In another embodiment, it could be LoD i-2 or any other LoD that would be found suitable. One LoD could be used as additional input by several Inter LoD encoders. In FIG. 6, the another LoD previously encoded that is input to the inter-LOD encoder 600 is encoded within LOD j bitstream 601. The LOD j bistream 601 may be an LOD j appearance bitstream, and LOD j appearance substream, or an LOD j substream.

[0125] Inter-LOD coding by inter-LOD encoder 600 comprises several embodiments such as providing a prediction or initialization for the appearance attribute function estimation module 610 thus requiring to code only the residuals of the parameters of the function instead of all the coefficients. In another embodiment, the reference LoD information can be used to better code the function in the function coding module, by providing contexts, dictionary entries, etc.

[0126] Decoder 603 decodes the previously encoded LoD j that is input to the inter-LOD encoder 600 from LOD j bitstream 601, and provides the previously encoded LoD j to function estimation module 610 and function coding 612.

[0127] In FIG. 6, LOD i primitives geometry attributes 602 is similar to LOD i primitive geometry attributes 502, LOD i primitives appearance attributes 604 is similar to LOD primitives appearance attributes 504, function estimation 610 is similar to function estimation 510, appearance attributes prediction 614 is similar to appearance attributes prediction 514, controller 608 is similar to controller 508, geometry coding 606 is similar to geometry coding 506, function coding 612 is similar to function coding 512, residual coding 616 is similar to residual coding 516, metadata coding 618 is similar to metadata coding 518, geometry substream 620 is similar to geometry substream 520, appearance substream 622 is similar to appearance substream 522, metadata substream 624 is similar to metadata substream 524, and LOD i bitstream 626 is similar to LoD i bitstream 526. LoD i bistream 626 may also be referred to as LoD i substream 626 e.g. in an inter-LoD coding scenario.

[0128] In an alternative embodiment, as shown in FIG. 6, the coding into geometry substream 620, appearance substream 622, and metadata substream 624 is not implemented, where the output of geometry coding 606 is coded directly into LoD i bitstream 626 as item 632, the output of function coding 612 is coded directly into LoD i bitstream 626 as item 634, the output of residual coding 616 is coded directly into LoD i bitstream 626 as item 636, and the output of metadata coding 618 is coded directly into LoD i sub stream 626 as item 638.

[0129] 2 Decoder embodiments

[0130] 2.1 Per LoD decoder

[0131] An illustration of the decoding embodiments is provided in FIG. 7. FIG. 7 shows a decoder 700 that implements a decoding scheme for a specific LOD.

[0132] The decoder 700 receives a bitstream containing multiple LoDs of a volumetric or radiance field scene. This bitstream may contain information such as headers and specific NAL units that describe the scene hierarchical representation, its number of LoDs and their possible inter-dependencies.

[0133] The bitstream containing the LoD i related information and data (such as LoD i bitstream 526, LoD i bitstream 626, or LoD i bitstream 726) is extracted and demultiplexed into a number of substreams for example a geometry substream 720, an appearance substream 722 and a metadata substream 724. It is understood by a person skilled in the art that other partitions into substreams can be used, mutas mutandis, in the disclosed decoding scheme.

[0134] For example, a controller module 708, which can be distributed in all other modules or be centralized uses the metadata decoded by the metadata decoding module 718. The geometry decoding module 706 extracts and decodes the geometry substream 720.

[0135] Using the decoded metadata and part of the geometry attributes, the Function decoding module 712 and the residual decoding module 716 demultiplex, extract and decode the data contained in the appearance-related substream 722.

[0136] In an alternative embodiment, as shown in FIG. 7, the separation into substreamsis not implemented, where the geometry substream 720, the appearance substream 722, and the metadata substream 724 could be in the same bitstream without clear substream separation. In this alternative embodiment, as shown in FIG. 7, LoD i bitstream (526, 626, 726) is directly connected as item 732 to geometry decoding 706 which extracts geometry information directly from LoD i bitstream (526, 626, 726), as item 734 to function decoding 712 which extracts appearance information directly from LoD i bitstream (526, 626, 726), as item 736 to residual decoding 716 which extracts appearance information directly from LoD i bitstream (526, 626, 726), and as item 738 to metadata decoding 718 which extracts metadata information directly from LoD i bitstream (526, 626, 726).

[0137] The appearance attribute prediction module 714 uses the decoded function per appearance attribute component or set of components, to produce an appearance attribute component (or set of components) prediction.

[0138] The appearance attributes reconstruction module 715 uses the predictions and residuals that correspond to the same attribute component(s) to reconstruct them.

[0139] The output of the LOD decoder comprise the rendering primitives belonging to this LoD including their geometry and appearance attributes, specifically the output of the LOD decoder includes LOD i primitives geometry attributes 702 and LOD i primitives appearance attributes 704.

[0140] 2.2 Inter-LoD Decoder

[0141] FIG. 8 shows a decoder 800 that implements an inter-LOD decoding scheme. FIG.8 illustrates one possible scheme for providing inter LoD decoding functionality to the present disclosure.

[0142] The difference between this decoder 800 and the decoder 700 depicted in FIG. 7 relates to the fact the LoD j bitstream 801 is also ingested and decoded first. The LoD j bitstream 801 may be an LoD j appearance bitstream, an LoD j appearance substream, or an LoD substream. The output of the LoD j decoder 803 is used by the Function Decoding module 812 to predict its function model or weights.

[0143] In FIG. 8, LoD i bitstream 826 is similar to LoD i bitstream 726, geometrysubstream 820 is similar to geometry substream 720, appearance substream 822 is similar to appearance substream 722, metadata substream 824 is similar to metadata substream 724, controller 808 is similar to controller 708, geometry decoding 806 is similar to geometry decoding 706, function decoding 812 is similar to function decoding 712, residual decoding 816 is similar to residual decoding 716, metadata decoding 818 is similar to metadata decoding 718, appearance attributes prediction 814 is similar to appearance attributes prediction 714, appearance attributes reconstruction 815 is similar to appearance attributes reconstruction 715, LOD i primitives geometry attributes 802 is similar to LOD i primitives geometry attributes 702, and LOD i primitives appearance attributes 804 is similar to LOD i primitives appearance attributes 704.

[0144] In an alternative embodiment, as shown in FIG. 8, the separation into sub streams is not implemented, meaning the geometry substream 820, the appearance substream 822, and the metadata substream 824 could be in the same bitstream without clear substream separation. In this alternative embodiment, as shown in FIG. 8, LoD i bitstream (526, 626, 826) is directly connected as item 832 to geometry decoding 806 which extracts geometry information directly from LoD i bitstream (526, 626, 826), as item 834 to function decoding 812 which extracts appearance information directly from LoD i bitstream (526, 626, 826), as item 836 to residual decoding 816 which extracts appearance information directly from LoD i bitstream (526, 626, 826), and as item 838 to metadata decoding 818 which extracts metadata information directly from LoD i bitstream (526, 626, 826).

[0145] In another embodiment, as shown in FIG. 8, LoD i substream is directly connected to decoder 803 as item 840. In this another embodiment, decoder 803 contains geometry decoding 806, function decoding 812, residual decoding 816, and metadata decoding 818. In this case, items 832, 834, 836, and 838 are coupled to decoder 803, as shown in FIG. 8.

[0146] LoD i bistream 726 may be referred to as LoD i substream 726 (e.g. in an inter-LoD decoding scenario), and LoD i bitstream 826 may be referred to as LoD i substream 826 (e.g. in an inter-LoD decoding scenario).

[0147] 3 Signaling Embodiments

[0148] The type of primitive must be signaled in the metadata to be encoded into orextracted from bitstream. Table A details the primitive types.Table A Primitive Type

[0149] The type of attributes must be signaled for the primitive as detailed in Table BTable B Attribute types

[0150] The hierarchical representation LoD structure may be signaled as follows.<<<<><<<<

[0151] Table C tree types

[0152] An appearance function information syntax element may be fined as follows.<<

[0153] Table D Function type

[0154] Table E function sampling method

[0155] tree type defines the type of tree and takes values as the following Table C. It impacts the variable hsi_node_children_count_log2 that defines the number of nodes that are children of a parent node in a tree. For tree type ADAPTIVE, this number of nodes needs to be signalled by node_children_count_log2.

[0156] node_children_count_log2 specifies the number of children for a node in the treewith ADAPTIVE type.

[0157] lod count minusl + 1 specifies the number of level of details of the representation.

[0158] inter layer _flag[i] set to 1 specifies if the lod i is inter coded and when set to zero if it is intra coded.

[0159] reference_layer[i] specifies the lod that is used as reference for the inter coding of lod i.

[0160] primitive_type[i] specifies the primitive type at lod i. primitive types are specified in Table A.

[0161] geom etry attrib ute count minusl plus 1 specifies the number of attributes with a geometry nature (as described in Table B)

[0162] appearance attribute count minusl plus 1 specifies the number of attributes with an appearance nature (as described in Table B)

[0163] primitive_count_minusl[i][j] plus 1 specifies the number of primitives of lod i within its cell j .

[0164] parent_position_bit_depth[i] specifies the bit depth accuracy of the position geometry information of parents in LOD i.

[0165] The appearance function information may be signaled as follows.

[0166] function type takes values in Table D

[0167] function sampling method takes values in Table E

[0168] sampling_grid_rate_log2 indicates the sampling rate in case the function sampling method is equal to REGULAR SAMPLING. Rate is equal to sampling grid rate = 1 « sampling_grid_rate_log2 and indicates that the cell is subdivided along each axis in sampling grid rate number of samples.

[0169] coefficients count minusl plus 1 specifies the number of coefficients for afunction of type POLYNOMIAL

[0170] coefficient_qp_minus4 plus 4 specifies the QP at which the coefficients are quantized

[0171] LUT enabled flag set to 1 specifies that a Look-up-Table entry is available for the function

[0172] coefficients LUT index specifies the index of the coefficients in the Look Up Table

[0173] coefficient_numerator_minusl[i] specifies the numerator of the ith coefficient of the function

[0174] coefficient_denominator_minusl[i] specifies the denominator of the ith coefficient of the function

[0175] param eters count minusl plus 1 specifies the number of parameters for a function of type EXPONENTIAL, LOGARITHMIC, POWER-LAW, TRIGONOMETRIC, HYPERBOLIC or LOGISTIC.

[0176] parameters_qp_minus4 plus 4 specifies the QP at which the function parameters are quantized.

[0177] param eter LUT index specifies the index of the parameters in the Look Up Table.

[0178] neural network LUT index specifies the index of the neural network function representation in the neural network Look Up Table.

[0179] The examples described herein may be relevant to ISO / IEC 23090.

[0180] FIG. 9 shows an encoder 900 according to an embodiment. FIG. 9 illustrates an image to be encoded (In), a predicted representation of an image block (P'n), a prediction error signal (Dn), a reconstructed prediction error signal (D'n), a preliminary reconstructed image (I'n), a final reconstructed image (R'n), a transform (T) and inverse transform fU1), a quantization (Q) and inverse quantization (Q-1), entropy encoding (E), a reference frame memory (RFM), inter prediction (Pinter), intra prediction (Pintra), mode selection (MS) andfiltering (F).

[0181] FIG. 10 shows a decoder 1000 according to an embodiment. FIG. 10 illustrates a predicted representation of an image block (P'n), a reconstructed prediction error signal (D'n), a preliminary reconstructed image (I'n), a final reconstructed image (R'n), an inverse transform (T-1), an inverse quantization (fT1), an entropy decoding (E"1), a reference frame memory (RFM), a prediction (either inter or intra) (P), and filtering (F).

[0182] Referring to FIG. 9, the output of E comprises coded video data and / or in some embodiments, level of detail geometry attributes data and / or appearance attributes data. Function estimation 902 uses the output of E to estimate a function, which estimated function is encoded into bitstream 920 by function encoding 918.

[0183] Referring to FIG. 10, function decoding 1002 may decode a function from at least the bitstream 920, which function is used by appearance attributes prediction and reconstruction 1010 to predict appearance attributes and to reconstruct appearance attributes into the one or more reconstructed frame(s) R'n. The one or more reconstructed frame(s) R'n may be input to further video processing 1014.

[0184] A video encoder transforms the input video into a compressed representation suited for storage / transmission and a video decoder decompresses the compressed video representation back into a viewable form. Typically, an encoder discards some information in the original video sequence in order to represent the video in a more compact form (that is, at lower bitrate).

[0185] A video encoder may encode the video information in two phases. Firstly, pixel values in a certain picture area (or “block”) are predicted for example by motion compensation means (finding and indicating an area in one of the previously coded video frames that corresponds closely to the block being coded) or by spatial means (using the pixel values around the block to be coded in a specified manner). Secondly the prediction error, e.g., the difference between the predicted block of pixels and the original block of pixels, is coded. This is typically done by transforming the difference in pixel values using a specified transform (e.g., Discrete Cosine Transform (DCT) or a variant of it), quantizing the coefficients and entropy coding the quantized coefficients. By varying the fidelity of thequantization process, the encoder can control the balance between the accuracy of the pixel representation (picture quality) and size of the resulting coded video representation (file size or transmission bitrate).

[0186] Inter prediction, which may also be referred to as temporal prediction, motion compensation, or motion-compensated prediction, exploits temporal redundancy. In inter prediction the sources of prediction are previously decoded pictures (a.k.a. reference pictures).

[0187] Intra prediction utilizes the fact that adjacent pixels within the same picture are likely to be correlated. Intra prediction can be performed in spatial or transform domain, e.g., either sample values or transform coefficients can be predicted. Intra prediction is typically exploited in intra coding, where no inter prediction is applied.

[0188] An intra picture may be defined as a coded picture that is decoded using intra prediction only, or in other words, does not make use of inter prediction in decoding. An intra picture may be interchangeably called an intra frame.

[0189] An inter picture may be defined as a coded picture whose decoding may include intra prediction and inter prediction. An inter picture may be interchangeably called an inter frame.

[0190] FIG. 11 is a block diagram illustrating a system 1100 in accordance with several examples. In an example, the encoder 1130 is used to encode an image or video from the scene 1115, which scene 1115 may be captured with one or more cameras, and the encoder 1130 is implemented in a transmitting apparatus 1180. The encoder 1130 produces a bitstream 1110 comprising signaling that is received by the receiving apparatus 1182, which implements a decoder 1140. The encoder 1130 sends the bitstream 1110 that comprises the herein described signaling. The decoder 1140 forms the image or video for the scene 1115-1, and the receiving apparatus 1182 would present this to the user, e.g., via a smartphone, television, or projector among many other options.

[0191] In some examples, the transmitting apparatus 1180 and the receiving apparatus 1182 are at least partially within a common apparatus, and for example are located within a common housing 1150. In other examples the transmitting apparatus 1180 and the receivingapparatus 1182 are at least partially not within a common apparatus and have at least partially different housings. Therefore in some examples, the encoder 1130 and the decoder 1140 are at least partially within a common apparatus, and for example are located within a common housing 1150. For example the common apparatus comprising the encoder 1130 and decoder 1140 implements a codec. In other examples the encoder 1130 and the decoder 1140 are at least partially not within a common apparatus and have at least partially different housings, but when together still implement a codec.

[0192] In some examples, 3D media from the capture (e.g., volumetric capture) at a viewpoint 1112 of the scene 1115, which includes a person 1113) is converted via projection to a series of 2D representations with occupancy, geometry, attributes and / or displacements. Additional atlas information is also included in the bitstream to enable inverse reconstruction. For decoding, the received bitstream 1110 is separated into its components with atlas information; occupancy, geometry, displacement, and attribute 2D representations. A 3D reconstruction is performed to reconstruct the scene 1115-1 created looking at the viewpoint 1112-1 with a “reconstructed” person 1113-1. The “-1” are used to indicate that these are reconstructions of the original.

[0193] In some examples, the scene 1115 and reconstructed scene 1115-1 are not limited to volumetric outside-in video as indicated by the eye representing viewpoint 1112 circling the person 1113 and the eye representing viewpoint 1112-1 circling the reconstructed person 1113-1. For example, the scene 1115 may be a 2D frame or an image, and the reconstructed scene 1115-1 may be a 2D frame or an image.

[0194] As indicated at 1120, the decoder 1140 performs an action or actions based on the received signaling.

[0195] Encoding 1190 performs the examples described herein related to volumetric parametric attribute coding for radiance rendering primitives. Decoding 1192 performs the examples described herein related to volumetric parametric attribute decoding for radiance rendering primitives.

[0196] FIG. 12 is an example apparatus 1200, which may be implemented in hardware, configured to implement the examples described herein. The apparatus 1200 comprises atleast one processor 1202 (e.g., an FPGA and / or CPU and / or GPU), one or more memories 1204 including computer program code 1205, the computer program code 1205 having instructions to carry out the methods described herein, wherein the at least one memory 1204 and the computer program code 1205 are configured to, with the at least one processor 1202, cause the apparatus 1200 to implement circuitry, a process, component, module, or function (implemented with control module 1206) to implement the examples described herein.

[0197] Apparatus 1200 may be a smartphone, personal digital device or assistant, smart television, laptop, pad, tablet, head-mounted display (HMD), or other user device or terminal device. The memory 1204 may be a non-transitory memory, a transitory memory, a volatile memory (e.g. RAM), or a non-volatile memory (e.g., ROM).

[0198] Optionally included attribute coding for radiance rendering primitives 1230 implements the examples described herein related to volumetric parametric attribute coding for radiance rendering primitives. Optionally included attribute decoding for radiance rendering primitives 1240 implements the examples described herein related to volumetric parametric attribute decoding for radiance rendering primitives.

[0199] The apparatus 1200 includes a display and / or I / O interface 1208, which includes user interface (UI) circuitry and elements, that may be used to display features or a status of the methods described herein (e.g., as one of the methods is being performed or at a subsequent time), or to receive input from a user such as with using a keypad, camera, touchscreen, touch area, microphone, biometric recognition, one or more sensors, etc. The apparatus 1200 includes one or more communication e.g. network (N / W) interfaces (I / F(s)) 1210. The communication I / F(s) 1210 may be wired and / or wireless and communicate over the Internet / other network(s) via any communication technique including via one or more links 1224. The communication I / F(s) 1210 may comprise one or more transmitters or one or more receivers.

[0200] The transceiver 1216 comprises one or more transmitters 1218 and one or more receivers 1220. The transceiver 1216 and / or communication I / F(s) 1210 may comprise standard well-known components such as an amplifier, filter, frequency-converter, (de)modulator, and encoder / decoder circuitries and one or more antennas, such as antennas1214 used for communication over wireless link 1226.

[0201] The control module 1206 of the apparatus 1200 comprises one of or both parts 1206-1 and / or 1206-2, which may be implemented in a number of ways. The control module 1206 may be implemented in hardware as control module 1206-1, such as being implemented as part of the one or more processors 1202. The control module 1206-1 may be implemented also as an integrated circuit or through other hardware such as a programmable gate array. In another example, the control module 1206 may be implemented as control module 1206-2, which is implemented as computer program code (having corresponding instructions) 1205 and is executed by the one or more processors 1202. For instance, the one or more memories 1204 store instructions that, when executed by the one or more processors 1202, cause the apparatus 1200 to perform one or more of the operations as described herein. Furthermore, the one or more processors 1202, one or more memories 1204, and example algorithms (e.g., as flowcharts and / or signaling diagrams), encoded as instructions, programs, or code, are means for causing performance of the operations described herein.

[0202] The apparatus 1200 to implement the functionality of control 1206 may correspond to any of the apparatuses depicted herein. Alternatively, apparatus 1200 and its elements may not correspond to any of the other apparatuses depicted herein, as apparatus 1200 may be part of a self-organizing / optimizing network (SON) node or other node, such as a node in a cloud.

[0203] The apparatus 1200 may also be distributed throughout the network including within and between apparatus 1200 and any network element (such as a base station and / or terminal device and / or user equipment).

[0204] Interface 1212 enables data communication and signaling between the various items of apparatus 1200, as shown in FIG. 12. For example, the interface 1212 may be one or more buses such as address, data, or control buses, and may include any interconnection mechanism, such as a series of lines on a motherboard or integrated circuit, fiber optics or other optical communication equipment, and the like. Computer program code (e.g. instructions) 1205, including control 1206 may comprise object-oriented software configured to pass data or messages between objects within computer program code 1205.Computer program code (e.g. instructions) 1205, including control 1206 may comprise procedural, functional, or scripting code. The apparatus 1200 need not comprise each of the features mentioned, or may comprise other features as well. The various components of apparatus 1200 may at least partially reside in a common housing 1228, or a subset of the various components of apparatus 1200 may at least partially be located in different housings, which different housings may include housing 1228.

[0205] FIG. 13 shows a schematic representation of non-volatile memory media 1300a (e.g. computer / compact disc (CD) or digital versatile disc (DVD)) and 1300b (e.g. universal serial bus (USB) memory stick) and 1300c (e.g. cloud storage for downloading instructions and / or parameters 1302 or receiving emailed instructions and / or parameters 1302) storing instructions and / or parameters 1302 which when executed by a processor allows the processor to perform one or more of the operations of the methods described herein. Instructions and / or parameters 1302 may represent or correspond to a non-transitory computer readable medium.

[0206] FIG. 14 is an example method 1400 based on the examples described herein. At 1410, the method includes receiving as input appearance attribute data of a rendering primitive and geometry attribute data of the rendering primitive. At 1420, the method includes estimating at least one function that predicts an appearance attribute in a volume of a cell of a representation associated with the rendering primitive, using the appearance attribute data of the rendering primitive and the geometry attribute data of the rendering primitive. At 1430, the method includes coding, into an output bitstream, the at least one function that predicts the appearance attribute in the volume of the cell of the representation associated with the rendering primitive. Method 1400 may be implemented with encoder 500, encoder 600, encoder 900, transmitting apparatus 1180 with encoder 1130, or apparatus 1200.

[0207] FIG. 15 is an example method 1500 based on the examples described herein. At 1510, the method includes decoding, from at least one bitstream, at least one function that predicts an appearance attribute in a volume of a cell of a representation associated with a rendering primitive, and geometry attribute data of the rendering primitive. At 1520, the method includes using the at least one function and the geometry attribute data of therendering primitive to predict the appearance attribute in the volume of the cell of the representation associated with the rendering primitive. At 1530, the method includes reconstructing the appearance attribute in the volume of the cell of the representation associated with the rendering primitive, based on the prediction of the appearance attribute in the volume of the cell of the representation associated with the rendering primitive using the at least one function and the geometry attribute data of the rendering primitive. Method 1500 may be implemented with decoder 700, decoder 800, decoder 1000, receiving apparatus 1182 with decoder 1140, or apparatus 1200.

[0208] The following examples are provided and described herein.

[0209] Example 1. An apparatus comprising: at least one processor; and at least one memory storing instructions that, when executed by the at least one processor, cause the apparatus at least to: receive as input appearance attribute data of a rendering primitive and geometry attribute data of the rendering primitive; estimate at least one function that predicts an appearance attribute in a volume of a cell of a representation associated with the rendering primitive, using the appearance attribute data of the rendering primitive and the geometry attribute data of the rendering primitive; and code, into an output bitstream, the at least one function that predicts the appearance attribute in the volume of the cell of the representation associated with the rendering primitive.

[0210] Example 2. The apparatus of claim 1, wherein the representation associated with the rendering primitive comprises a hierarchical representation.

[0211] Example 3. The apparatus of any of claims 1 to 2, wherein the at least one memory stores instructions that, when executed by the at least one processor, cause the apparatus at least to: receive as input metadata describing the representation associated with the rendering primitive; wherein the at least one function that predicts the appearance attribute in the volume of the cell of the representation associated with the rendering primitive is estimated using the metadata describing the representation associated with the rendering primitive.

[0212] Example 4. The apparatus of any of claims 1 to 3, wherein the at least one memory stores instructions that, when executed by the at least one processor, cause the apparatus atleast to: code the output bitstream into a level of detail bitstream associated with a level of detail that is associated with the appearance attribute.

[0213] Example 5. The apparatus of any of claims 1 to 4, wherein the at least one memory stores instructions that, when executed by the at least one processor, cause the apparatus at least to: use the at least one function to predict the appearance attribute in the volume of the cell of the representation associated with the rendering primitive; subtract the appearance attribute that was predicted in the volume of the cell of the representation associated with the rendering primitive from the input appearance attribute data of the rendering primitive to determine a residual signal; and code the residual signal into the output bitstream.

[0214] Example 6. The apparatus of any of claims 1 to 5, wherein the at least one memory stores instructions that, when executed by the at least one processor, cause the apparatus at least to: decode appearance attribute data from an input bitstream associated with a first level of detail; wherein the at least one function that predicts the appearance attribute in the volume of the cell of the representation associated with the rendering primitive is estimated using the appearance attribute data decoded from the input bitstream associated with the first level of detail; wherein the at least one function that predicts the appearance attribute in the volume of the cell of the representation associated with the rendering primitive that is estimated is associated with a second level of detail that is coarser or finer than the first level of detail.

[0215] Example 7. The apparatus of any of claims 1 to 6, wherein the at least one memory stores instructions that, when executed by the at least one processor, cause the apparatus at least to: decode appearance attribute data from an input bitstream associated with a first level of detail; wherein the at least one function that predicts the appearance attribute in the volume of the cell of the representation associated with the rendering primitive is coded into the output bitstream based on the appearance attribute data decoded from the input bitstream associated with the first level of detail; wherein the at least one function that predicts the appearance attribute in the volume of the cell of the representation associated with the rendering primitive that is estimated is associated with a second level of detail that is coarser or finer than the first level of detail.

[0216] Example 8. The apparatus of any of claims 1 to 7, wherein the at least one functionthat predicts the appearance attribute in the volume of the cell of the representation associated with the rendering primitive is coded into the output bitstream as a look up table, when the same model for the at least one function is used in two or more cells of the representation associated with the rendering primitive.

[0217] Example 9. The apparatus of any of claims 1 to 8, wherein the at least one memory stores instructions that, when executed by the at least one processor, cause the apparatus at least to signal, into a metadata bitstream using at least one syntax element, a type of the rendering primitive, wherein the type of the rendering primitive comprises one of: a multidimensional point with additional attributes, or a multidimensional shape with additional attributes, or a set of multidimensional points with common additional attributes, or a multidimensional point with a specified probability distribution, or a multidimensional Gaussian splat, or an ellipsoid, or a sphere, or a billboard, or a plane, or a convex shape, or a Gaussian, or a generalized Gaussian.

[0218] Example 10. The apparatus of any of claims 1 to 9, wherein the at least one memory stores instructions that, when executed by the at least one processor, cause the apparatus at least to signal, using at least one syntax element, a type of the appearance attribute, wherein the type of the appearance attribute comprises one of: a color, or an opacity, or a spherical harmonics band.

[0219] Example 11. The apparatus of any of claims 1 to 10, wherein the at least one memory stores instructions that, when executed by the at least one processor, cause the apparatus at least to: signal information related to the representation associated with the rendering primitive using at least one syntax element.

[0220] Example 12. The apparatus of any of claims 1 to 11, wherein the at least one memory stores instructions that, when executed by the at least one processor, cause the apparatus at least to signal, using at least one syntax element, a type of the at least one function that predicts the appearance attribute in the volume of the cell of the representation associated with the rendering primitive, wherein the type of the at least function that predicts the appearance attribute in the volume of the cell of the representation associated with the rendering primitive comprises one of: polynomial, or exponential, or logarithmic, or powerlaw, or trigonometric, or hyperbolic, or logistic, or neural.

[0221] Example 13. The apparatus of any of claims 1 to 12, wherein the appearance attribute in the volume of the cell of the representation associated with the rendering primitive is predicted using the at least one function by sampling the at least one function at a location corresponding to at least one position of the of the rendering primitive within the volume of the cell of the representation associated with the rendering primitive.

[0222] Example 14. The apparatus of any of claims 1 to 13, wherein the appearance attribute in the volume of the cell of the representation associated with the rendering primitive is predicted using the at least one function by sampling the at least one function at multiple positions inside a geometric extent of the rendering primitive in the cell corresponding to a volume of the primitive in space.

[0223] Example 15. An apparatus comprising: at least one processor; and at least one memory storing instructions that, when executed by the at least one processor, cause the apparatus at least to: decode, from at least one bitstream, at least one function that predicts an appearance attribute in a volume of a cell of a representation associated with a rendering primitive, and geometry attribute data of the rendering primitive; use the at least one function and the geometry attribute data of the rendering primitive to predict the appearance attribute in the volume of the cell of the representation associated with the rendering primitive; and reconstruct the appearance attribute in the volume of the cell of the representation associated with the rendering primitive, based on the prediction of the appearance attribute in the volume of the cell of the representation associated with the rendering primitive using the at least one function and the geometry attribute data of the rendering primitive.

[0224] Example 16. The apparatus of claim 15, wherein the representation associated with the rendering primitive comprises a hierarchical representation.

[0225] Example 17. The apparatus of any of claims 15 to 16, wherein the at least one memory stores instructions that, when executed by the at least one processor, cause the apparatus at least to: decode metadata describing the representation associated with the rendering primitive; wherein the metadata describing the representation associated with the rendering primitive is used to predict the appearance attribute in the volume of the cell of the representation associated with the rendering primitive.

[0226] Example 18. The apparatus of any of claims 15 to 17, wherein the at least one memory stores instructions that, when executed by the at least one processor, cause the apparatus at least to: decode appearance information from a level of detail bitstream associated with a level of detail that is associated with the appearance attribute; wherein the reconstructed appearance attribute in the volume of the cell of the representation associated with the rendering primitive is associated with the level of detail and is reconstructed based on the appearance information decoded from the level of detail bitstream.

[0227] Example 19. The apparatus of any of claims 15 to 18, wherein the at least one memory stores instructions that, when executed by the at least one processor, cause the apparatus at least to: decode, from the at least one bitstream, a residual signal that represents a difference between an appearance attribute that was predicted in the volume of the cell of the representation associated with the rendering primitive and appearance attribute data of the rendering primitive; and use the residual signal to reconstruct the appearance attribute in the volume of the cell of the representation associated with the rendering primitive.

[0228] Example 20. The apparatus of any of claims 15 to 19, wherein the at least one memory stores instructions that, when executed by the at least one processor, cause the apparatus at least to: decode appearance attribute data from a bitstream associated with a first level of detail; wherein the appearance attribute data decoded from the bitstream associated with the first level of detail is used to predict the appearance attribute in the volume of the cell of the representation associated with the rendering primitive; wherein the appearance attribute in the volume of the cell of the representation associated with the rendering primitive that is predicted using the appearance attribute data associated with the first level of detail is associated with a second level of detail that is coarser or finer than the first level of detail.

[0229] Example 21. The apparatus of any of claims 15 to 20, wherein the at least one memory stores instructions that, when executed by the at least one processor, cause the apparatus at least to: decode appearance attribute data from a bitstream associated with a first level of detail; wherein the at least one function used to predict the appearance attribute in the volume of the cell of the representation associated with the rendering primitive is based on the appearance attribute data decoded from the bitstream associated with the firstlevel of detail; wherein the appearance attribute in the volume of the cell of the representation associated with the rendering primitive that is predicted using the at least one function is associated with a second level of detail that is coarser or finer than the first level of detail.

[0230] Example 22. The apparatus of any of claims 15 to 21, wherein the at least one function that predicts the appearance attribute in the volume of the cell of the representation associated with the rendering primitive is decoded from the at least one bitstream as a look up table, when the same model for the at least one function is used in two or more cells of the representation associated with the rendering primitive.

[0231] Example 23. The apparatus of any of claims 15 to 22, wherein the at least one memory stores instructions that, when executed by the at least one processor, cause the apparatus at least to decode, from a metadata bitstream and from at least one syntax element, a type of the rendering primitive, wherein the type of the rendering primitive comprises one of: a multidimensional point with additional attributes, or a multidimensional shape with additional attributes, or a set of multidimensional points with common additional attributes, or a multidimensional point with a specified probability distribution, or a multidimensional Gaussian splat, or an ellipsoid, or a sphere, or a billboard, or a plane, or a convex shape, or a Gaussian, or a generalized Gaussian.

[0232] Example 24. The apparatus of any of claims 15 to 23, wherein the at least one memory stores instructions that, when executed by the at least one processor, cause the apparatus at least to decode, from at least one syntax element, signaling of a type of the appearance attribute, wherein the type of the appearance attribute comprises one of: a color, or an opacity, or a spherical harmonics band.

[0233] Example 25. The apparatus of any of claims 15 to 24, wherein the at least one memory stores instructions that, when executed by the at least one processor, cause the apparatus at least to: decode, from at least one syntax element, signaling of information related to the representation associated with the rendering primitive.

[0234] Example 26. The apparatus of any of claims 15 to 25, wherein the at least one memory stores instructions that, when executed by the at least one processor, cause theapparatus at least to decode, from at least one syntax element, signaling of a type of the at least one function that predicts the appearance attribute in the volume of the cell of the representation associated with the rendering primitive, wherein the type of the at least one function that predicts the appearance attribute in the volume of the cell of the representation associated with the rendering primitive comprises one of: polynomial, or exponential, or logarithmic, or power-law, or trigonometric, or hyperbolic, or logistic, or neural.

[0235] Example 27. The apparatus of any of claims 15 to 26, wherein the appearance attribute in the volume of the cell of the representation associated with the rendering primitive is predicted using the at least one function by sampling the at least one function at a location corresponding to at least one position of the of the rendering primitive within the volume of the cell of the representation associated with the rendering primitive.

[0236] Example 28. The apparatus of any of claims 15 to 27, wherein the appearance attribute in the volume of the cell of the representation associated with the rendering primitive is predicted using the at least one function by sampling the at least one function at multiple positions inside a geometric extent of the rendering primitive in the cell corresponding to a volume of the primitive in space.

[0237] Example 29. A method comprising: receiving as input appearance attribute data of a rendering primitive and geometry attribute data of the rendering primitive; estimating at least one function that predicts an appearance attribute in a volume of a cell of a representation associated with the rendering primitive, using the appearance attribute data of the rendering primitive and the geometry attribute data of the rendering primitive; and coding, into an output bitstream, the at least one function that predicts the appearance attribute in the volume of the cell of the representation associated with the rendering primitive.

[0238] Example 30. A method comprising: decoding, from at least one bitstream, at least one function that predicts an appearance attribute in a volume of a cell of a representation associated with a rendering primitive, and geometry attribute data of the rendering primitive; using the at least one function and the geometry attribute data of the rendering primitive to predict the appearance attribute in the volume of the cell of the representation associated with the rendering primitive; and reconstructing the appearance attribute in the volume ofthe cell of the representation associated with the rendering primitive, based on the prediction of the appearance attribute in the volume of the cell of the representation associated with the rendering primitive using the at least one function and the geometry attribute data of the rendering primitive.

[0239] Example 31. An apparatus comprising: means for receiving as input appearance attribute data of a rendering primitive and geometry attribute data of the rendering primitive; means for estimating at least one function that predicts an appearance attribute in a volume of a cell of a representation associated with the rendering primitive, using the appearance attribute data of the rendering primitive and the geometry attribute data of the rendering primitive; and means for coding, into an output bitstream, the at least one function that predicts the appearance attribute in the volume of the cell of the representation associated with the rendering primitive.

[0240] Example 32. An apparatus comprising: means for decoding, from at least one bitstream, at least one function that predicts an appearance attribute in a volume of a cell of a representation associated with a rendering primitive, and geometry attribute data of the rendering primitive; means for using the at least one function and the geometry attribute data of the rendering primitive to predict the appearance attribute in the volume of the cell of the representation associated with the rendering primitive; and means for reconstructing the appearance attribute in the volume of the cell of the representation associated with the rendering primitive, based on the prediction of the appearance attribute in the volume of the cell of the representation associated with the rendering primitive using the at least one function and the geometry attribute data of the rendering primitive.

[0241] Example 33. A computer readable medium comprising instructions stored thereon for performing at least the following: receiving as input appearance attribute data of a rendering primitive and geometry attribute data of the rendering primitive; estimating at least one function that predicts an appearance attribute in a volume of a cell of a representation associated with the rendering primitive, using the appearance attribute data of the rendering primitive and the geometry attribute data of the rendering primitive; and coding, into an output bitstream, the at least one function that predicts the appearance attribute in the volume of the cell of the representation associated with the renderingprimitive.

[0242] Example 34. A computer readable medium comprising instructions stored thereon for performing at least the following: decoding, from at least one bitstream, at least one function that predicts an appearance attribute in a volume of a cell of a representation associated with a rendering primitive, and geometry attribute data of the rendering primitive; using the at least one function and the geometry attribute data of the rendering primitive to predict the appearance attribute in the volume of the cell of the representation associated with the rendering primitive; and reconstructing the appearance attribute in the volume of the cell of the representation associated with the rendering primitive, based on the prediction of the appearance attribute in the volume of the cell of the representation associated with the rendering primitive using the at least one function and the geometry attribute data of the rendering primitive.

[0243] In the above, some embodiments have been described in relation to the term quantization parameter. It is to be understood that embodiments may be similarly realized with reference to any similar term, such as quantizer, or control variable that defines or determines fidelity of the (de)coded video or image signal. In the above, some embodiments assume that the lower the quantization parameter value is, the better fidelity is achieved. It is to be understood that embodiments may be similarly realized with a control variable that has an opposite way of mapping values, i.e., the higher the control variable value is, the better fidelity is achieved. For example, embodiments may be realized with JPEG compression where the higher the quality factor is, the better fidelity is achieved.

[0244] References to a ‘computer’, ‘processor’, etc. should be understood to encompass not only computers having different architectures such as single / multi-processor architectures and sequential / parallel architectures but also specialized circuits such as field-programmable gate arrays (FPGAs), application specific circuits (ASICs), signal processing devices and other processing circuitry. References to computer program, instructions, code etc. should be understood to encompass software for a programmable processor or firmware such as, for example, the programmable content of a hardware device such as instructions for a processor, or configuration settings for a fixed-function device, gate array or programmable logic device, etc.

[0245] The term “non-transitory,” as used herein, is a limitation of the medium itself (i.e., tangible, not a signal) as opposed to a limitation on data storage persistency (e.g., RAM vs. ROM).

[0246] As used herein, the term ‘circuitry’, ‘circuit’ and variants may refer to any of the following: (a) hardware circuit implementations, such as implementations in analog and / or digital circuitry, and (b) combinations of circuits and software (and / or firmware), such as (as applicable): (i) a combination of processor(s) or (ii) portions of processor(s) / software including digital signal processor(s), software, and one or more memories that work together to cause an apparatus to perform various functions, and (c) circuits, such as a microprocessor s) or a portion of a microprocessor s), that require software or firmware for operation, even when the software or firmware is not physically present. As a further example, as used herein, the term ‘circuitry’ would also cover an implementation of merely a processor (or multiple processors) or a portion of a processor and its (or their) accompanying software and / or firmware. The term ‘circuitry’ would also cover, for example and when applicable to the particular element, a baseband integrated circuit or applications processor integrated circuit for a mobile phone or a similar integrated circuit in a server, a cellular network device, or another network device. Circuitry or circuit may also be used to mean a function or a process used to execute a method.

[0247] It should be understood that the foregoing description is only illustrative. Various alternatives and modifications may be devised by those skilled in the art. For example, features recited in the various dependent claims could be combined with each other in any suitable combination(s). In addition, features from different embodiments described above could be selectively combined into a new embodiment. Accordingly, the description is intended to embrace all such alternatives, modifications and variances which fall within the scope of the appended claims.

[0248] The following acronyms and abbreviations that may be found in the specification and / or the drawing figures are defined as follows (the abbreviations may be appended with each other or with other characters using e.g. a hyphen, dash (-), or number (or abbreviations having a character may be the same with a character removed), and may be case insensitive):2D two-dimensional2DGS 2D Gaussian splat3D three-dimensional3DG 3D graphics3DGS three-dimensional Gaussian splat4D four-dimensional4DGS four-dimensional Gaussian splatAC alternating currentASIC application specific integrated circuitCAB AC context-adaptive binary arithmetic coding CPU central processing unitDC direct currentDCT Discrete Cosine TransformDNN deep neural networkFFT fast Fourier transformFPGA field programmable gate arrayG-PCC geometry based point cloud compression GPU graphics processing unitGS Gaussian splatHMD head-mounted displayhsi hierarchical structure information (e.g.hsi_node_children_count_log2)IEC International Electrotechnical Commission I / F interfaceI / O input / outputISO International Organization for Standardization JPEG Joint Photographic Experts GroupLIDAR light detection and rangingLOD level of detailLUT look up tableMIV MPEG immersive videoMLP multilayer perceptronMPEG moving picture experts groupNAL network abstraction layerNeRF neural radiance fieldNNC neural network codingN / W networkQP quantization parameterRAHT region adaptive hierarchical transformRAM random access memoryRFM reference frame memoryRGB red green blueRGBA red green blue alphaROM read only memorySC subcommitteeSH spherical harmonicsSON self-organizing / optimizing networku(n) unsigned integer using n bits (e.g. u(2))ue(v) unsigned integer O-th order Exp-Golomb-coded syntax element with the left bit first.UI user interfaceUSB universal serial busV3C visual volumetric video based codingV-PCC video based point cloud compressionWG working group

Claims

CLAIMSWhat is claimed is:

1. An apparatus comprising:at least one processor; andat least one memory storing instructions that, when executed by the at least one processor, cause the apparatus at least to:receive as input appearance attribute data of a rendering primitive and geometry attribute data of the rendering primitive;estimate at least one function that predicts an appearance attribute in a volume of a cell of a representation associated with the rendering primitive, using the appearance attribute data of the rendering primitive and the geometry attribute data of the rendering primitive;code, into an output bitstream, the at least one function that predicts the appearance attribute in the volume of the cell of the representation associated with the rendering primitive.

2. The apparatus of claim 1, wherein the representation associated with the rendering primitive comprises a hierarchical representation.

3. The apparatus of any of claims 1 to 2, wherein the at least one memory stores instructions that, when executed by the at least one processor, cause the apparatus at least to:receive as input metadata describing the representation associated with the rendering primitive;wherein the at least one function that predicts the appearance attribute in thevolume of the cell of the representation associated with the rendering primitive is estimated using the metadata describing the representation associated with the rendering primitive.

4. The apparatus of any of claims 1 to 3, wherein the at least one memory stores instructions that, when executed by the at least one processor, cause the apparatus at least to:code the output bitstream into a level of detail bitstream associated with a level of detail that is associated with the appearance attribute.

5. The apparatus of any of claims 1 to 4, wherein the at least one memory stores instructions that, when executed by the at least one processor, cause the apparatus at least to:use the at least one function to predict the appearance attribute in the volume of the cell of the representation associated with the rendering primitive;subtract the appearance attribute that was predicted in the volume of the cell of the representation associated with the rendering primitive from the input appearance attribute data of the rendering primitive to determine a residual signal; andcode the residual signal into the output bitstream.

6. The apparatus of any of claims 1 to 5, wherein the at least one memory stores instructions that, when executed by the at least one processor, cause the apparatus at least to:decode appearance attribute data from an input bitstream associated with a first level of detail;wherein the at least one function that predicts the appearance attribute in the volume of the cell of the representation associated with the rendering primitive is estimated using the appearance attribute data decoded from the input bitstream associated with the first level of detail;49wherein the at least one function that predicts the appearance attribute in the volume of the cell of the representation associated with the rendering primitive that is estimated is associated with a second level of detail that is coarser or finer than the first level of detail.

7. The apparatus of any of claims 1 to 6, wherein the at least one memory stores instructions that, when executed by the at least one processor, cause the apparatus at least to:decode appearance attribute data from an input bitstream associated with a first level of detail;wherein the at least one function that predicts the appearance attribute in the volume of the cell of the representation associated with the rendering primitive is coded into the output bitstream based on the appearance attribute data decoded from the input bitstream associated with the first level of detail;wherein the at least one function that predicts the appearance attribute in the volume of the cell of the representation associated with the rendering primitive that is estimated is associated with a second level of detail that is coarser or finer than the first level of detail.

8. The apparatus of any of claims 1 to 7, wherein the at least one function that predicts the appearance attribute in the volume of the cell of the representation associated with the rendering primitive is coded into the output bitstream as a look up table, when the same model for the at least one function is used in two or more cells of the representation associated with the rendering primitive.

9. The apparatus of any of claims 1 to 8, wherein the at least one memory stores instructions that, when executed by the at least one processor, cause the apparatus at least to signal, into a metadata bitstream using at least one syntax element, a type of the rendering primitive, wherein the type of the rendering primitive comprises one of:a multidimensional point with additional attributes, or50a multidimensional shape with additional attributes, ora set of multidimensional points with common additional attributes, ora multidimensional point with a specified probability distribution, ora multidimensional Gaussian splat, oran ellipsoid, ora sphere, ora billboard, ora plane, ora convex shape, ora Gaussian, ora generalized Gaussian.

10. The apparatus of any of claims 1 to 9, wherein the at least one memory stores instructions that, when executed by the at least one processor, cause the apparatus at least to signal, using at least one syntax element, a type of the appearance attribute, wherein the type of the appearance attribute comprises one of:a color, oran opacity, ora spherical harmonics band.

11. The apparatus of any of claims 1 to 10, wherein the at least one memory stores instructions that, when executed by the at least one processor, cause the apparatus at least to:signal information related to the representation associated with the rendering51primitive using at least one syntax element.

12. The apparatus of any of claims 1 to 11, wherein the at least one memory stores instructions that, when executed by the at least one processor, cause the apparatus at least to signal, using at least one syntax element, a type of the at least one function that predicts the appearance attribute in the volume of the cell of the representation associated with the rendering primitive, wherein the type of the at least function that predicts the appearance attribute in the volume of the cell of the representation associated with the rendering primitive comprises one of:polynomial, orexponential, orlogarithmic, orpower-law, ortrigonometric, orhyperbolic, orlogistic, orneural.

13. The apparatus of any of claims 1 to 12, wherein the appearance attribute in the volume of the cell of the representation associated with the rendering primitive is predicted using the at least one function by sampling the at least one function at a location corresponding to at least one position of the of the rendering primitive within the volume of the cell of the representation associated with the rendering primitive.

14. The apparatus of any of claims 1 to 13, wherein the appearance attribute in the volume of the cell of the representation associated with the rendering primitive is predicted using the at least one function by sampling the at least one function at multiple positions inside a geometric extent of the rendering primitive in the cell correspondingto a volume of the primitive in space.

15. An apparatus comprising:at least one processor; andat least one memory storing instructions that, when executed by the at least one processor, cause the apparatus at least to:decode, from at least one bitstream, at least one function that predicts an appearance attribute in a volume of a cell of a representation associated with a rendering primitive, and geometry attribute data of the rendering primitive;use the at least one function and the geometry attribute data of the rendering primitive to predict the appearance attribute in the volume of the cell of the representation associated with the rendering primitive; andreconstruct the appearance attribute in the volume of the cell of the representation associated with the rendering primitive, based on the prediction of the appearance attribute in the volume of the cell of the representation associated with the rendering primitive using the at least one function and the geometry attribute data of the rendering primitive.

16. The apparatus of claim 15, wherein the representation associated with the rendering primitive comprises a hierarchical representation.

17. The apparatus of any of claims 15 to 16, wherein the at least one memory stores instructions that, when executed by the at least one processor, cause the apparatus at least to:decode metadata describing the representation associated with the rendering primitive;wherein the metadata describing the representation associated with the rendering primitive is used to predict the appearance attribute in the volume of the cellof the representation associated with the rendering primitive.

18. The apparatus of any of claims 15 to 17, wherein the at least one memory stores instructions that, when executed by the at least one processor, cause the apparatus at least to:decode appearance information from a level of detail bitstream associated with a level of detail that is associated with the appearance attribute;wherein the reconstructed appearance attribute in the volume of the cell of the representation associated with the rendering primitive is associated with the level of detail and is reconstructed based on the appearance information decoded from the level of detail bitstream.

19. The apparatus of any of claims 15 to 18, wherein the at least one memory stores instructions that, when executed by the at least one processor, cause the apparatus at least to:decode, from the at least one bitstream, a residual signal that represents a difference between an appearance attribute that was predicted in the volume of the cell of the representation associated with the rendering primitive and appearance attribute data of the rendering primitive; anduse the residual signal to reconstruct the appearance attribute in the volume of the cell of the representation associated with the rendering primitive.

20. The apparatus of any of claims 15 to 19, wherein the at least one memory stores instructions that, when executed by the at least one processor, cause the apparatus at least to:decode appearance attribute data from a bitstream associated with a first level of detail;wherein the appearance attribute data decoded from the bitstream associated with the first level of detail is used to predict the appearance attribute in the volume ofthe cell of the representation associated with the rendering primitive;wherein the appearance attribute in the volume of the cell of the representation associated with the rendering primitive that is predicted using the appearance attribute data associated with the first level of detail is associated with a second level of detail that is coarser or finer than the first level of detail.

21. The apparatus of any of claims 15 to 20, wherein the at least one memory stores instructions that, when executed by the at least one processor, cause the apparatus at least to:decode appearance attribute data from a bitstream associated with a first level of detail;wherein the at least one function used to predict the appearance attribute in the volume of the cell of the representation associated with the rendering primitive is based on the appearance attribute data decoded from the bitstream associated with the first level of detail;wherein the appearance attribute in the volume of the cell of the representation associated with the rendering primitive that is predicted using the at least one function is associated with a second level of detail that is coarser or finer than the first level of detail.

22. The apparatus of any of claims 15 to 21, wherein the at least one function that predicts the appearance attribute in the volume of the cell of the representation associated with the rendering primitive is decoded from the at least one bitstream as a look up table, when the same model for the at least one function is used in two or more cells of the representation associated with the rendering primitive.

23. The apparatus of any of claims 15 to 22, wherein the at least one memory stores instructions that, when executed by the at least one processor, cause the apparatus at least to decode, from a metadata bitstream and from at least one syntax element, a type of the rendering primitive, wherein the type of the rendering primitive comprises one55of:a multidimensional point with additional attributes, ora multidimensional shape with additional attributes, ora set of multidimensional points with common additional attributes, ora multidimensional point with a specified probability distribution, ora multidimensional Gaussian splat, oran ellipsoid, ora sphere, ora billboard, ora plane, ora convex shape, ora Gaussian, ora generalized Gaussian.

24. The apparatus of any of claims 15 to 23, wherein the at least one memory stores instructions that, when executed by the at least one processor, cause the apparatus at least to decode, from at least one syntax element, signaling of a type of the appearance attribute, wherein the type of the appearance attribute comprises one of:a color, oran opacity, ora spherical harmonics band.

25. The apparatus of any of claims 15 to 24, wherein the at least one memory stores56instructions that, when executed by the at least one processor, cause the apparatus at least to:decode, from at least one syntax element, signaling of information related to the representation associated with the rendering primitive.

26. The apparatus of any of claims 15 to 25, wherein the at least one memory stores instructions that, when executed by the at least one processor, cause the apparatus at least to decode, from at least one syntax element, signaling of a type of the at least one function that predicts the appearance attribute in the volume of the cell of the representation associated with the rendering primitive, wherein the type of the at least one function that predicts the appearance attribute in the volume of the cell of the representation associated with the rendering primitive comprises one of:polynomial, orexponential, orlogarithmic, orpower-law, ortrigonometric, orhyperbolic, orlogistic, orneural.

27. The apparatus of any of claims 15 to 26, wherein the appearance attribute in the volume of the cell of the representation associated with the rendering primitive is predicted using the at least one function by sampling the at least one function at a location corresponding to at least one position of the of the rendering primitive within the volume of the cell of the representation associated with the rendering primitive.

28. The apparatus of any of claims 15 to 27, wherein the appearance attribute in the volume of the cell of the representation associated with the rendering primitive is predicted using the at least one function by sampling the at least one function at multiple positions inside a geometric extent of the rendering primitive in the cell corresponding to a volume of the primitive in space.

29. A method comprising:receiving as input appearance attribute data of a rendering primitive and geometry attribute data of the rendering primitive;estimating at least one function that predicts an appearance attribute in a volume of a cell of a representation associated with the rendering primitive, using the appearance attribute data of the rendering primitive and the geometry attribute data of the rendering primitive; andcoding, into an output bitstream, the at least one function that predicts the appearance attribute in the volume of the cell of the representation associated with the rendering primitive.

30. A method comprising:decoding, from at least one bitstream, at least one function that predicts an appearance attribute in a volume of a cell of a representation associated with a rendering primitive, and geometry attribute data of the rendering primitive;using the at least one function and the geometry attribute data of the rendering primitive to predict the appearance attribute in the volume of the cell of the representation associated with the rendering primitive; andreconstructing the appearance attribute in the volume of the cell of the representation associated with the rendering primitive, based on the prediction of the appearance attribute in the volume of the cell of the representation associated with the rendering primitive using the at least one function and the geometry attribute data of the rendering primitive.5831. An apparatus comprising:means for receiving as input appearance attribute data of a rendering primitive and geometry attribute data of the rendering primitive;means for estimating at least one function that predicts an appearance attribute in a volume of a cell of a representation associated with the rendering primitive, using the appearance attribute data of the rendering primitive and the geometry attribute data of the rendering primitive; andmeans for coding, into an output bitstream, the at least one function that predicts the appearance attribute in the volume of the cell of the representation associated with the rendering primitive.

32. An apparatus comprising:means for decoding, from at least one bitstream, at least one function that predicts an appearance attribute in a volume of a cell of a representation associated with a rendering primitive, and geometry attribute data of the rendering primitive;means for using the at least one function and the geometry attribute data of the rendering primitive to predict the appearance attribute in the volume of the cell of the representation associated with the rendering primitive; andmeans for reconstructing the appearance attribute in the volume of the cell of the representation associated with the rendering primitive, based on the prediction of the appearance attribute in the volume of the cell of the representation associated with the rendering primitive using the at least one function and the geometry attribute data of the rendering primitive.

33. A computer readable medium comprising instructions stored thereon for performing at least the following:receiving as input appearance attribute data of a rendering primitive and geometry attribute data of the rendering primitive;estimating at least one function that predicts an appearance attribute in a volume of a cell of a representation associated with the rendering primitive, using the appearance attribute data of the rendering primitive and the geometry attribute data of the rendering primitive; andcoding, into an output bitstream, the at least one function that predicts the appearance attribute in the volume of the cell of the representation associated with the rendering primitive.

34. A computer readable medium comprising instructions stored thereon for performing at least the following:decoding, from at least one bitstream, at least one function that predicts an appearance attribute in a volume of a cell of a representation associated with a rendering primitive, and geometry attribute data of the rendering primitive;using the at least one function and the geometry attribute data of the rendering primitive to predict the appearance attribute in the volume of the cell of the representation associated with the rendering primitive; andreconstructing the appearance attribute in the volume of the cell of the representation associated with the rendering primitive, based on the prediction of the appearance attribute in the volume of the cell of the representation associated with the rendering primitive using the at least one function and the geometry attribute data of the rendering primitive.60

Images