V-DMC based coding of gaussian splats

Abstract

The embodiments concern a method comprising receiving a bitstream comprising a mesh sequence, a Gaussian splat representation sequence, and metadata that associates the Gaussian splat representation sequence with the mesh sequence; extracting the metadata that associates the Gaussian splat representation sequence with the mesh sequence; decoding the mesh sequence; decoding the Gaussian splat representation sequence; and reconstructing one or more of the mesh sequence or the Gaussian splat representation sequence The embodiments also concern technical equipment for implementing the method.

Description

V-DMC BASED CODING OF GAUSSIAN SPLATSTECHNICAL FIELD

[0001] The examples and non-limiting embodiments relate generally to V-DMC based coding of Gaussian splats.BACKGROUND

[0002] It is known to process and render video in a multimedia system.BRIEF DESCRIPTION OF THE DRAWINGS

[0003] The foregoing embodiments and other features are explained in the following description, taken in connection with the accompanying drawings, wherein:

[0004] FIG. 1 shows an overview of a V-DMC encoder.

[0005] FIG. 2 shows a structure of a compressed Draco bitstream.

[0006] FIG. 3 shows displacement packing to a patch.

[0007] FIG. 4 shows an example of an attribute map generated by V-DMC.

[0008] FIG. 5 shows octahedral mapping.

[0009] FIG. 6 illustrates spherical harmonics.

[0010] FIG. 7 is an illustration of a Gaussian splat.

[0011] FIG. 8 shows rendering of a scene composed of 3D or 4D Gaussian splats through rasterization or ray casting.

[0012] FIG. 9 shows an example of Gaussian splats associated to a surface.

[0013] FIG. 10 shows an encoder scheme for coding dynamic meshes and 4DGS.

[0014] FIG. 11 shows a decoder scheme to extract dynamic mesh sequences and dynamicGaussian splats from a V3C bitstream.

[0015] FIG. 12 shows Gaussian splats (GSO, GS1, and GS2), base mesh triangle with vertices a, b, and c and normal n.

[0016] FIG. 13 shows mapping Gaussian splat elements from a base mesh primitive to subdivided mesh vertex primitives.

[0017] FIG. 14 shows an example syntax table used to signal 4DGS attributes.

[0018] FIG. 15A shows a first portion of an example syntax table used to signal 4DGS information.

[0019] FIG. 15B shows a second portion of the example syntax table used to signal 4DGS information.

[0020] FIG. 15C shows a third portion of the example syntax table used to signal 4DGS information, where the example syntax table is shown by FIG. 15 A, FIG. 15B, and FIG. 15C collectively.

[0021] FIG. 16 shows an encoder according to an embodiment.

[0022] FIG. 17 shows a decoder according to an embodiment.

[0023] FIG. 18 is a block diagram illustrating a system in accordance with an example.

[0024] FIG. 19 is an example apparatus configured to implement the examples described herein.

[0025] FIG. 20 shows a representation of an example of non-volatile memory media used to store instructions that implement the examples described herein

[0026] FIG. 21 is an example method based on the examples described herein.

[0027] FIG. 22 is an example method based on the examples described herein.DETAILED DESCRIPTION OF EXAMPLE EMBODIMENTS

[0028] Volumetric video

[0029] There are many ways to capture and represent a Volumetric frame. The format used to capture and represent it depends on the processing to be performed on it, and the target application using it. Some exemplary representations are listed below (1-3):

[0030] 1. A volumetric frame can be represented as a point cloud. A point cloud is a set of unstructured points in 3D space, where each point is characterized by its position in a 3D coordinate system (e.g. Euclidean), and some corresponding attributes (e.g. color information provided as RGBA value, or normal vectors).

[0031] 2. A volumetric frame can be represented as images, with or without depth, captured from multiple viewpoints in 3D space. In other words, it can be represented by one or more view frames (where a view is a projection of a volumetric scene on to a plane (the camera plane) using a real or virtual camera with known / computed extrinsics and intrinsics). Each view may be represented by a number of components (e.g. geometry, color, transparency, and occupancy picture), which may be part of the geometry picture or represented separately.

[0032] 3. A volumetric frame can be represented as a mesh. Mesh is a collection of points, called vertices, and connectivity information between vertices, called edges. Vertices along with edges form faces. The combination of vertices, edges and faces can uniquely approximate shapes of objects.

[0033] Depending on the capture, a volumetric frame can provide viewers the ability to navigate a scene with six degrees of freedom, i.e. both translational and rotational movement of their viewing pose (which includes yaw, pitch, and role). The data to be coded for a volumetric frame can also be significant, as a volumetric frame can contain many objects, and the positioning and movement of these objects in the scene can result in many dis- occluded regions. Furthermore, the interaction of light and materials in objects and surfaces in a volumetric frame can generate complex light fields that can produce texture variations for even a slight change of pose.

[0034] A sequence of volumetric frames is a volumetric video. Due to large amount of information, storage and transmission of a volumetric video requires compression. A way to compress a volumetric frame can be to project the 3D geometry and related attributes into a collection of 2D images along with additional associated metadata. The projected 2D images can then be coded using 2D video and image coding technologies, for example ISO / IEC 14496-10 (H.264 / AVC) and ISO / IEC 23008-2 (H.265 / HEVC). The metadata can be coded with technologies specified in specification such as ISO / IEC 23090-5. The coded images and the associated metadata can be stored or transmitted to a client that can decode and render the 3D volumetric frame.

[0035] Visual Volumetric Video-base Coding (V3C) - ISO / IEC 23090-5

[0036] ISO / IEC 23090-5 specifies the syntax, semantics, and process for coding volumetric video. The specified syntax is designed to be generic so that it can be reused for a variety of applications. Point clouds, immersive video with depth, and mesh representations can all use ISO / IEC 23090-5 standard with extensions that deal with the specific nature of the final representation. The purpose of the specification is to define how to decode and interpret the associated data (for example atlas data in ISO / IEC 23090-5) which tells a Tenderer how to interpret 2D frames to reconstruct a volumetric frame.

[0037] Two applications of V3C (ISO / IEC 23090-5) have been defined, V-PCC (ISO / IEC 23090-5) and MIV (ISO / IEC 23090-12). MIV and V-PCC use number of V3C syntax elements with a slightly modified semantics. An example on how the generic syntax element can be differently interpreted by the application is pdu_projection_id. In case of V-PCC the syntax element, pdu_projection_id specifies the index of the projection plane for the patch. There can be 6 or 18 projection planes in V-PCC, and they are implicit, i.e. pre-determined. In case of MIV pdu_projection_id corresponds to a view ID, i.e. identifies which view the patch originated from. View IDs and their related information is explicitly provided in MIV view parameters list and may be tailored for each content.

[0038] MPEG 3DG (ISO SC29 WG7) group has started work on a third application of V3C - the mesh compression. It is also envisaged that mesh coding re-uses V3C syntax as much as possible and can also slightly modify the semantics.

[0039] To differentiate between applications of V3C bitstream, that allow a client to properly interpret the decoded data, V3C uses the ptl_profile_toolset_idc parameter.

[0040] V3C introduces a concept of a map, i.e., an attribute map or a geometry map. Attribute map is an attribute frame containing attribute patch information projected at a particular depth indicated by the corresponding geometry map. Where geometry frame containing geometry patch information projected at a particular depth. Maps can be used to store multiple layers of surface data, resulting in denser point clouds in case of V-PCC.

[0041] V3C - V3C bitstream

[0042] V3C bitstream is a sequence of bits that forms the representation of coded volumetric frames and the associated data making one or more coded V3C sequences (CVS). Where CVS is a sequence of bits identified and separated by appropriate delimiters, and is required to start with a VPS, includes a V3C unit, and contains one or more V3C units with atlas sub-bitstream or video sub-bitstream. Video sub-bitstreams and atlas subbitstreams can be referred to as V3C sub-bitstreams. Which V3C sub-bitstream a V3C unit contains and how to interpret it is identified by a V3C unit header in conjunction with VPS information.

[0043] V3C bitstream can be stored according to Annex C of ISO / IEC 23090-5 which specifies syntax and semantics of a sample stream format to be used by applications that deliver some or all of the V3C unit stream as an ordered stream of bytes or bits within which the locations of V3C unit boundaries need to be identifiable from patterns in the data.

[0044] In V3C bitstream attribute maps and corresponding geometry maps are identified by vuh map index syntax element in V3C unit header. This syntax element indicates the map index of the current geometry or attribute stream. The number of maps in V3C bitstream is signaled for each atlas by vps map count minusl syntax element in V3C parameter set. In the current version of the specification the number of maps indicated by vps map count minusl tells how many maps there are of geometry and attribute.

[0045] Video-based dynamic mesh coding (V-DMC) - ISO / IEC 23090-29

[0046] V-DMC (ISO / IEC 23090-29) is another application form of V3C that aims onintegration of mesh compression into the V3C family of standards. The standard is under development and at WD stage (MDS23617_WG07_N00822)

[0047] The technology is based on multiresolution mesh analysis and coding. This approach consists of (1-6):

[0048] 1. generating a base-mesh that is a simplified (low resolution) mesh approximation of the original mesh, called base-mesh (this is done for all frames of the dynamic mesh sequence)

[0049] 2. performing several mesh subdivision iterative steps (e.g., each triangle is converted into four triangles by connecting the triangle edge midpoints on the generated base mesh, generating other approximation meshes)

[0050] 3. defining displacement vectors, also named error vectors, for each vertex of each mesh approximation. Each approximation can be seen as level of details (LoD) of the original mesh.

[0051] 4. For each subdivision level by adding the displacement vectors to the subdivided mesh vertices generates the best approximation of the original mesh at that resolution, given the base-mesh and prior subdivision levels.

[0052] 5. The displacement vectors may undergo a lazy wavelet transform prior to compression.

[0053] 6. The attribute map of the original mesh is transferred to the deformed mesh at the highest resolution (i.e., subdivision level) such that texture coordinates are obtained for the deformed mesh and a new attribute map is generated.

[0054] FIG. 1 shows an overview of a V-DMC encoder 100. The V-DMC encoder 100 generates compressed bitstreams, which later on are packed in V3C units and create V3C bitstream by concatenating V3C units (i-iv):

[0055] i. A sub-bitstream with the encoded base-mesh (124) using a mesh codec 116.

[0056] ii. A sub-bitstream with the displacement vectors (126): packed in an 2D frameand encoded using a video codec or image codec (118), or arithmetic encoded as defined in Annex J of WD ISO / IEC 23090-29.

[0057] iii. A sub-bitstream with the attribute map (128) encoded using a video codec (120).

[0058] iv. An atlas sub-bitstream (122) that contains all metadata required to decode and reconstruct the mesh sequence 102 based on the aforementioned sub-bitstreams. The signaling of the metadata is based on the V3C syntax and includes necessary extensions that are specific to meshes.

[0059] As shown in FIG. 1, the dynamic mesh sequence 102 is pre-processed with preprocessing 104 to produce atlas data 106, base mesh 108, displacements 110, and attributes 112. Altas encoder 114 encodes atlas data 106 into atlas sub-bitstream 122, base mesh encoder 116 encodes base mesh 108 into base mesh sub-bitstream 124, displacement encoder 118 encodes displacement data 110 into displacement sub -bitstream 126, and video encoder 120 encodes attribute data 112 into attribute sub-bitstream 128. Multiplexer 130 encodes and multiplexes atlas sub-bitstream 122, base mesh sub-bitstream 124, displacement sub-bitstream 126, and attribute sub-bitstream 128 into VC3 bitstream 132 with Gaussian splat signaling 134.

[0060] Base-mesh sub-bitstream (ISO / IEC 23090-29)

[0061] An elementary unit for the output of a base-mesh encoder (Annex H of ISO / IEC 23090-29) is a NAL unit.

[0062] A NAL unit may be defined as a syntax structure containing an indication of the type of data to follow and bytes containing that data in the form of an RBSP interspersed as necessary with emulation prevention bytes. A raw byte sequence payload (RBSP) may be defined as a syntax structure containing an integer number of bytes that is encapsulated in a NAL unit. An RBSP is either empty or has the form of a string of data bits containing syntax elements followed by an RBSP stop bit and followed by zero or more subsequent bits equal to 0.

[0063] NAL units can be categorized into Base-mesh Coding Layer (BMCL) NAL unitsand non-BMCL NAL units. BMCL NAL units can be coded sub-mesh NAL units. A non- BMCL NAL unit may be for example one of the following types: a base-mesh sequence parameter set, a base-mesh frame parameter set, a supplemental enhancement information (SEI) NAL unit, an access unit delimiter, an end of sequence NAL unit, an end of bitstream NAL unit, or a filler data NAL unit. Parameter sets may be needed for the reconstruction of decoded bas-mesh, whereas many of the other non-BMCL NAL units are not necessary for the reconstruction of decoded sample values.

[0064] V-DMC specifications may contain a set of constraints for associating data units (e.g. NAL units) into coded base-mesh access units.

[0065] NOTE: at the time of writing of the document there was no definition of coded base-mesh access unit in WD of V-DMC specification.

[0066] MPEG EdgeBreaker Static Mesh (ISO / IEC 23090-29 Annex I)

[0067] The MPEG EdgeBreaker Static Mesh codec specifies signaling and decoding processes for intra mesh coding based on the well-known EdgeBreaker algorithm, which was extended with several algorithmic enhancements allowing to handle a wider variety of mesh connectivity types and better geometry and UV texture coordinate predictions. It achieves comparable compression performances as the Draco compression that is introduced later in this document.

[0068] ISO / IEC 23090-29 Annex I specifies a mesh coding header, a mesh position coding payload and mesh attributes coding payload.

[0069] I.8.3.1 General Mesh coding syntax

[0070] The mesh coding header syntax is detailed in the following table and includes themesh codec type (i.e. the mesh connectivity traversal chosen, for the moment only the forward traversal is implemented), position encoding parameters (that are detailed later in this document), position dequantization information, the number of mesh attributes and for each of them its number of components (2 for texture coordinates, 3 for normal and colors, 1 for material type and a generic type with n components), and finally attribute coding and dequantization parameters that are detailed later in this document.

[0071] I.8.3.2 Mesh coding header syntax

[0072] The position coding parameters are specified as follows, and include information related to the position bitdepth, the EdgeBreaker CLERS symbols coding method chosen, the position prediction methods used, the position residuals encoding method chosen and finally the position duplicate removal method.

[0073] I.8.3.3 Mesh position encoding parameters syntax

[0074] The mesh attributes encoding parameters are provided per attribute (signaled by its index) in the following table and signal the attribute bitdepth, whether it is a per face attribute and if so if a separate index table is provided for these attributes (several attributes may be mapped to the same face and in that case it is suitable to provide a separate index table with a mapping between attribute value index and face index), the mesh attribute prediction method used and the mesh attribute residuals encoding method chosen.

[0075] I.8.3.5 Mesh attributes encoding parameters syntax

[0076] Normal attribute coding in MPEG Edgebreaker

[0077] The base mesh attributes such as the normal data are encoded but the MPEG Edgebreaker uses the following syntax and using octahedral mapping parameters. Octahedral mapping of normal attribute data is used instead of spherical coordinates, octahedral coordinates are illustrated on FIG. 5.

[0078] 1.8.3.11 Mesh extra attribute data syntax

[0079] 1.8.3.14 Mesh normal octahedral extra data syntax

[0080] Draco compression

[0081] Draco is a technology for compressing and decompressing 3D geometric meshes and point clouds. It is intended to improve the storage and transmission of 3D graphics. It supports compressing points, connectivity information, texture coordinates, color information, normals, and any other generic attributes associated with geometry. Draco may compress the 3D mesh either sequentially or using edgebreaker algorithm. The input to Draco encoder can be any 3D model and the encoder compresses it into a Draco bitstream. The compressed bitstream may be decoded back into the original 3D model, or if needed transcoded into a different format.

[0082] FIG. 2 shows a structure of compressed Draco bitstream 200. The compressedbitstream 200 is divided into four parts: Draco header 202, metadata 204, connectivity and attributes 208. The general structure of the compressed bitstream 200 is illustrated in FIG. 2, where three different alternative structures for the connectivity data have been separated, and are shown as connectivity data structure 206-1, connectivity data structure 206-2, and connectivity data structure 206-3.

[0083] The Draco header 202 contains high-level information about the bitstream and identifies the bitstream as Draco bitstream. The fields of the header are described below.ParseHeader() { draco string major version minor version encoder type encoder method flags}

[0084] The metadata part 204 allows associating per attribute metadata or file level metadata with the Draco bitstream 200. It could be for example used to describe attribute names. The metadata 204 consists of following information and enables recursively adding more levels of sub-metadata. The metadata 204 is represented by key-value pairs. void DecodeMetadata() {ParseMetadataCount(); for (i = 0; i < num_att_metadata; ++i) {ParseAttributeMetadatald(i);DecodeMetadataElement(att_metadata[i]);}DecodeMetadataElement(file metadata);} void ParseMetadataCount() {num att metadata} void ParseAttributeMetadatald(index) { att metadata idfindex]} void ParseMetadataElement(metadata) { metadata.num entries for (i = 0; i < metadata. num_entries; ++i) { sz = metadata.key_size[i] metadata.keyfi] sz = metadata.value sizefi] metadata.valuefi]} metadata, num sub metadata} void ParseSubMetadataKey(metadata, index) { sz = metadata. sub_metadata_key_size[index] metadata, sub metadata key [index]} void DecodeMetadataElement(metadata) { ParseMetadataElement(metadata); for (i = 0; i < metadata.num sub metadata; ++i) { ParseSubMetadataKey(metadata, i);DecodeMetadataElement(metadata.sub_metadata[i]);}}

[0085] The structure of the connectivity part 206 depends on the encoder method. Itconsists of either sequential or edgebreaker information. The type of connectivity information is defined in the Draco header 202 (encoder method). The sequential connectivity header (210-1, 210-2, 210-3) contains the following fields: void ParseSequentialConnectivityData() { num faces num_points connectivity method}

[0086] It contains information such as the number of faces and points in the connectivity bitstream as well as the connectivity method which identifies if the sequential connectivity bitstream consists of compressed indices (e.g. indices 211) or uncompressed indices (e.g. indices 211). The rest of the connectivity data contains the connectivity bitstream.

[0087] Alternatively, the compressed Draco bitstream may consist of edgebreaker encoded connectivity data instead of the sequential connectivity data. In this case the connectivity header contains information as defined by ParseEdgebreakerConnectivityData structure. void ParseEdgebreakerConnectivityData() { edgebreaker traversal type num encoded vertices num faces num attribute data num encoded sy mb ol s num encoded split sy mb ol s}

[0088] The header provides information such as the traversal type, which indicates the type of the edgebreaker connectivity bitstream. In current version this can be either standard edgebreaker (0) or valence edgebreaker (2). Additionally, it contains information such as the number of encoded vertices and attributes. It also provides information on the numberof encoder symbols and split symbols, which are required to decode different parts of the edgebreaker encoded connectivity bitstream. In addition to the connectivity header the connectivity data contains the connectivity bitstream.

[0089] The structure of the connectivity bitstream depends on the traversal type and can include encoded split data (212, 214), encoded edgebreaker symbol data (216, 218), encoded start face configuration data (220, 222) and the attribute connectivity data (224, 226). It could additionally include valence header 228 and context data 230 in case edgebreaker valence traversal type is used.

[0090] The attribute data 208 contains two sections. The first part is the attribute header 232, which indicates how many attributes need to be decoded as well as what components each attribute consists of. void ParseAttributeDecodersData() { num attributes decoders if (encoder method == MESH EDGEBREAKER ENCODING) { for (i = 0; i < num_attributes_decoders; ++i) { att dec data i d [i ] att_dec_decoder_type[i] att_dec_traversal_method[i]}} for (i = 0; i < num_attributes_decoders; ++i) { att dec num attributes [i ] for (j = 0; j < att_dec_num_attributes[i]; ++j) { att_dec_att_type[i] [j ] att_dec_data_type[i] [j ] att_dec_num_components[i] [j ] att_dec_normalized[i] [j ] att dec uni que i d [i ] [j ]} for (j = 0; j < att_dec_num_attributes[i]; ++j) {seq_att_dec_decoder_type[i] [j ]}}}

[0091] The second part of the attribute data is comprised of compressed attributes, such as positions, texture coordinates, normals, etc. Each attribute type section (including attribute type 234 and attribute type 236) is comprised of one or more unique components.

[0092] Displacement / geometry video sub-bitstream (ISO / IEC 23090-29)

[0093] During the encoding process the input mesh is simplified and a base-mesh is generated. This is done by performing several mesh subdivision iterative steps (e.g., each triangle is converted into four triangles by connecting the triangle edge midpoints on the generated base mesh, generating other approximation meshes). Next, displacement vectors, also named error vectors, for each vertex of each mesh approximation is computed. Each approximation can be seen as level of details (LoD) of the original mesh. By adding the displacement vectors to the subdivided mesh vertices generates the best approximation of the original mesh at that resolution, given the base-mesh and prior subdivision levels.

[0094] Displacement vectors are packed in a 2D frame and encoded using a video codec or image codec. How the displacements are packed and unpacked from video frame is defined in V-DMC (11.2.4 inverse image packing of transform coefficients).

[0095] Displacement for each vertex are separate to distinctive regions corresponding to the subdivision level as presented on FIG. 3. In particular, FIG. 3 shows displacement packing to a patch 300. Each block (such as block 302) corresponds to PatchPackingBlockSize* PatchPackingBlockSize pixels in video frame. A patch corresponds to a region in a video frame.

[0096] Attribute video sub-bitstream (ISO / IEC 23090-29)

[0097] In V-DMC, attribute maps are accessed by texture coordinates, also called UV coordinates, attached to vertices of the base mesh and interpolated for the subdivided mesh. Using texture coordinates, the color properties of a vertex v with texture coordinates (u,v)are defined by the pixel of the Texture Attribute Map A(u,v). As texture coordinates are normalized and quantized in the base mesh substream, the color information may relate to an exact pixel position in the texture video frame or a fractional pixel, i.e., a position in between two or four pixels, that is bilinearly interpolated as supported by GPUs.

[0098] FIG. 4 illustrates an attribute map in the case of a texture (V3C TEXTURE). Values of the attributes are accessed by texture coordinates, also called UV coordinates (V3C TEXTURE COORDINATES) that are conveyed by the base mesh sub-stream. FIG. 5 illustrates another example mapping, in this case of attribute normal vectors, using an octahedral mapping. Such octahedral mapping allows mapping of 3D normal coordinates values to 2D coordinates.

[0099] In particular, FIG. 4 shows an example of an attribute map generated by V-DMC for color attribute, where texture coordinates are mapped to vertices so as to identify to which pixel s / sub -pixels they correspond to in the attribute map.

[0100] In particular, FIG. 5 shows an octahedral mapping, despite not sampling all angles equally as the spherical coordinates. The octahedral mapping shown in FIG. 5 enables a simple packing to a 2D frame.

[0101] Dynamic Gaussian splats

[0102] Dynamic Gaussian Splatting, also known as 3DGS or 4DGS, is a volumetric representation that enables to render novel viewpoints from a 3D scene by rendering primitives that are called Gaussian Splats. Such Gaussian Splats are described by the following properties:

[0103] - Position (x,y,z); i.e., the center of the Gaussian splat

[0104] - Scale (sx,sy,sz); i.e., non-uniform scale factor of the Gaussian splat

[0105] - Rotation (rx,ry,rz) or (qx,qy,qz,qw); i.e., the rotation of the Gaussian splat with respect to world coordinates, the rotation is expressed by Euler angles or quaternions

[0106] - Normal (nx, ny, nz); i.e., the normal vector of the Gaussian splat

[0107] - Opacity (alpha), i.e., a factor that indicates the transparency of the Gaussian splat

[0108] - Color (r,g,b), i.e., the primary color of the Gaussian splat

[0109] - Spherical harmonics (SH[0:48]), i.e., the reflectance of the Gaussian splat expressed in spherical harmonics

[0110] Sometimes color is represented by the DC component of spherical harmonics, and spherical harmonics contain 45 non-DC components. Spherical harmonics can be seen as DCT basis functions defined on the sphere. The r, g and b color components[OHl] FIG. 6 is an illustration of spherical harmonics on the left (602) and visual comparison with DCT basis functions on the right (604). Similarly to DCT that enables low to high frequency variations on the horizontal and vertical axis of an image, increasing order of spherical harmonics shows increasing frequency variations in vertical and horizontal directions on the sphere. The first component (m=0, 1=0), referred to as DC represents a constant value over the sphere, the second component (m=0, 1=1) enables a variation from south pole to north pole following cos(theta), the third component enables a variation from one side of the equator to the other side, etc. Multiplying the three color components rgb to these 15 basis functions on the sphere leads to 45 spherical harmonics components. Some approaches achieve a fast estimation of the Spherical harmonics transform and inverse transform by relying on the FFT.

[0112] FIG. 7 is an illustration of a Gaussian splat 700. The position information is the one of its center 702 highlighted by a circle. The size and orientation of the shape of the splat 700 is determined by the scale and rotation. The color, transparency and reflectivity are defined by the spherical harmonics and opacity.

[0113] A collection of Gaussian splats can be seen as a point cloud with additional attributes that capture view-dependent effects such as opacity and spherical harmonics. The rendering of Gaussian splats borrows much from classical computer graphics rendering; in order to create a viewport, ray casting is used through the scene and intersections with Gaussian splats are detected. The opacity and color information is accumulated along the ray by taking into account the attributes of Gaussian splats it traverses such as opacity, color and spherical harmonics.

[0114] 4DGS is the name given to dynamic Gaussian splats, that can be seen as a sequence, where each frame consists in a set of Gaussian splats defined at the time corresponding to this frame. Efficient representations optimize the stability of the number of Gaussian Splats from frame to frame and enable their tracking. However, this is challenged by the fact that occlusions may hide Gaussian splats from previous frames or disocclusions may create new Gaussian splats in a new frame.

[0115] FIG. 8 shows rendering of a scene 802 composed of 3D or 4D Gaussian splats (including Gaussian splat 804, Gaussian splat 806, and Gaussian splat 808) through rasterization or ray casting. A ray 810 is cast from the viewport 812 to the 3D scene 802 and intersections with 4D Gaussian splats (including Gaussian splat 804, Gaussian splat 806, and Gaussian splat 808) are estimated. The color of the pixel is computed by accumulating the opactity, color and specularity -related attributes along the ray 810 from close to far depth. Centers of 3D Gaussians and / or 4D Gaussians (including center 814 of Gaussian splat 804, center 816 of Gaussian splat 806, and center 818 of Gaussian splat 808) are represented with dots, while the shape and gradient of the Gaussian splats (including Gaussian splat 804, Gaussian splat 806, and Gaussian splat 808) illustrate the other attributes such as scale, opacity, spherical harmonics and rotation.

[0116] Rendering 3D or 4D Gaussian Splats is performed by rasterization or ray casting and is similar to traditional Computer Graphics rendering of triangle meshes. It follows the 4DGS rendering can be done in real-time depending on the number of primitives.

[0117] There is currently neither a standard file format nor a coding standard for such 4DGS representations.

[0118] Visual compression of a 3D Gaussian splat may include embodiments to encode 3DGS or 4DGS using V3C projections using the spirit of V-PCC and MIV. However, this does not cover the fact that 4DGS may be represented on a dynamic surface mesh. Furthermore, it does not provide LoDs or a multiresolution representation of 4DGS.

[0119] There is currently no standard to compress Gaussian Splats (4DGS). The existing approaches in the state of the art represent 4DGS as dynamic point clouds with a point cloud or multiview representation. These approaches do not capture well the fact that Gaussiansplats associated to an object or person is on or close to the object or person outer surface, and that this surface may be represented with multiple levels of detail, i.e., multiple, adaptive resolutions. While it is possible to jointly learn and optimize Gaussian Splats and the underlying geometry or surface, there is no compression format that can efficiently code a hybrid representation consisting of the 4DGS and the underlying mesh(es) they can be associated with, neither in the static case nor in the dynamic case. FIG. 9 illustrates a set of Gaussian splats associated with a mesh surface 902.

[0120] In particular, FIG. 9 shows an example of Gaussian splats associated to a surface 902, where each Gaussian splat (such as Gaussian splat 904) is represented in FIG. 9 as an elliptical shape. 3D Gaussian splats contributing to the appearance of the surface 902 can lie on, below or above this surface. In FIG. 9, Gaussian splat 904 contributes to the appearance of surface 902 and lies on the surface 902, Gaussian splat 906 contributes to the appearance of surface 902 and lies above surface 902, and Gaussian splat 908 contributes to the appearance of surface 902 and lies below surface 902.

[0121] Described herein is the coding and decoding of 4DGS jointly with a dynamic mesh sequence. In a particular embodiment we show how this can be performed with a tailored extension of the V-DMC standard described herein. Other embodiments cover a frame-by- frame coding and decoding of the mesh and associated Gaussian Splats with static mesh codecs. The examples described herein include signaling, decoding, reconstruction and encoding embodiments for V-DMC but are not limited to this standard.

[0122] The advantages and technical effects of this approach are as follows: the example embodiments described herein relate to efficiently encoding 4DGS for captured people or objects represented as dynamic meshes, the examples described herein allow for efficiently encoding the geometry of the 4DGS scene by attaching 4DGS splats to the mesh representation, the examples described herein allow surface-based multi-resolution using adaptive LoDs that provides a LoD hierarchy for Gaussian splats, the examples described herein provide a natural way of encoding temporal motion of Gaussian splats and their attributes, and the examples described herein use hardware support for video-based coded GS and mesh attributes

[0123] Several embodiments are provided for: i) associating the 4DGS data and relatedparameters (or attributes) with static or dynamic mesh primitives, ii) providing means to access, decode and encode the 4DGS data associated to the dynamic mesh sequence for example based on a Material ID attribute that provides the mapping of GS parameters to their respective attributes, iii) providing means to decode and encode 4DGS related parameters into the base mesh sub-bitstream that is attached per primitive, i.e., base mesh face, edge or vertex, iv) providing means to signal the 4DGS and dynamic mesh sequence information in or along the V-DMC atlas sub-bitstream, v) providing pre-processing means to convert 4DGS attribute data to a format that is compatible with V-DMC (non-negative integers, clustering, projection to base mesh, etc.).

[0124] Each type of 4DGS parameters (position, scale, rotation, opacity, color, spherical harmonics and possibly normal) are encoded in either a parameterized approach (for example using texture coordinates or barycentric coordinates) inside the attribute subbitstream or in an indexed approach inside the displacement sub-bitstream. The herein described approach provides the flexibility to map part or all of the 4DGS attributes to attribute data as the optimal choice in terms of compression may be data-dependent. The advantage of displacement data is the possibility to use the lifting transform, while the advantage of using V3C attributes is that some 4DGS parameters such as color for example may be better encoded by a video codec without such lifting transform.

[0125] FIG. 10 illustrates the herein described encoder 1000 that jointly codes dynamic meshes and associated 4DGS data. Thus FIG. 10 shows an encoder scheme for coding dynamic meshes and 4DGS.

[0126] Referring to FIG. 10, pre-processing 1004 takes as input dynamic mesh sequence 1002 and dynamic Gaussian splats 1003, and generates atlas data 1006, 4DGS metadata 1007, base mesh 1008, 4DGS base mesh parameters 1009, displacements 1010, attributes 1012, and 4DGS UV-mapped parameters 1013. Atlas encoder 1014 codes input atlas data 1006 and 4DGS metadata 1007 into atlas sub-bitstream 1022. Base mesh encoder 1016 codes input base mesh 1008 and 4DGS base mesh parameters 1009 into base mesh subbitstream 1024. Displacement ender 1018 codes input displacements 1010 into displacement sub-bitstream 1026. Video encoder 1020 codes attributes 1012 and 4DGS UV-mapped parameters 1013 into attribute sub-bitstream 1028. Multiplexer 1030 codes theatlas sub-bitstream 1022, base mesh sub-bitstream 1024, displacement sub-bitstream 1026, and attribute sub-bitstream 1028 into the VC3 bitstream 1032 containing the Gaussian splat signaling 1034.

[0127] FIG. 11 illustrates the herein described decoder 1100 that enables reconstruction of dynamic meshes and associated dynamic Gaussian splats. Metadata (1116, 1007) and signaling information (1114, 1118) are extracted from the V3C parameter set 1104 and decoded atlas parameter sets (1006) to decode and extract the 4DGS related information. Thus FIG. 11 shows a decoder scheme to extract dynamic mesh sequences and dynamic Gaussian splats from a V3C bitstream.

[0128] Referring to FIG. 11, demultiplexer 1102 takes as input V3C bitstream 1032 containing Gaussian splat signaling 1034 and produces V3C parameter set 1104, atlas sub- bistream 1022, base mesh sub-bitstream 1024, displacement sub-bitstream 1026 and attribute sub-bitstream 1028. V3C parameter set 1104 includes signaling information 1114 and metadata 1116. Atlas decoder 1106 takes as input atlas sub-bitstream 1022 and produces signaling information 1118, atlas data 1006 and 4DGS metadata 1007. Base mesh decoder 1108 takes as input base mesh sub-bitstream 1024 to produce base mesh data 1008 and 4DGS base mesh parameters 1009. Displacement decoder takes as input displacement sub-bitstream 1026 to produce displacement data 1010. Video decoder 1112 takes as input attribute sub-bitstream 1028 to produce attribte parameters 1122, 4DGS UV-mapped parameters 1013, and attribute data 1012.

[0129] Reconstruction process 1124 takes as input signaling information 1118, atlas data 1006, 4DGS metadata 1007, base mesh data 1008, 4DGS base mesh parameters 1009, metadata 1116, displacement data 1010, signaling information 1114, attributre parametes 1122, 4DGS UV-mapped parameters 1013, and attribute data 1012 to produce reconstructed dynamic mesh sequence 1126 and dynamic Gussian splats 1128.

[0130] 1.1 Embodiment - mapping Gaussian Splats to a coded mesh bitstream

[0131] This embodiment describes how to map Gaussian Splat primitives to a mesh primitive; vertex, edge or face. Projecting Gaussian Splats to faces provides an intuitive means to perform this mapping. However, each type of primitives comes with its ownattributes and its own way of encoding / decoding them for static mesh codecs and V-DMC. The following embodiments describe each of the three primitive cases. This mapping may be performed by the Gaussian Splat index, or when such index is implicit (non-indexed representation) then the central position of the Gaussian Splat is mapped instead. These are equivalent provided that no more than one Gaussian Splat share the same central position.

[0132] 1.1.1 Using triangle primitives as a reference

[0133] Associating a Gaussian Splat center position or index with a base mesh triangle abc is described hereafter.

[0134] FIG. 12 describes how Gaussian Splats denoted as GSi (for i = 0, 1, 2), or GSo, GSi, and GS2, can be associated with a given base mesh frame triangle 1202 with vertices a, b and c. The central position (1204, 1206, 1208) of the Gaussian Splat is projected on the triangle along the triangle normal n and this projection leads to a point pi that lies on the triangle surface. Such point that can be represented in barycentric coordinates of the triangle abc or if present, texture coordinates interpolated at this position. The distance along the projection axis for each Gaussian Splat GSi is denoted zi and can be positive, negative or equal to zero.

[0135] One embodiment encodes the central position of a Gaussian splat GSO as a displacement from pO to GSO along the normal n: GS0(x,y,z) = p0(x,y,z) + n(x,y,z) * zO

[0136] The projected point pO can be represented by barycentric coordinates alpha and beta such that: p0(x,y,z)= alpha * a(x,y,z) + beta * b(x,y,z) + (1 -alpha-beta) * c(x,y,z)

[0137] An alternative consists in describing pO with a 3D displacement d(x,y,z) in local coordinates such that: p0(x,y,z) = d(x,y,z) * n(x,y,z) + a(x,y,z) where n is the normal of face abc and for example, a is by convention the vertex with smallest index in abc. The 3D displacement d(x,y,z) can be provided in canonical coordinates (i.e. World Coordinates) using p0(x,y,z) = d(x,y,z) + a(x,y,z). An alternative embodiment instead of the vertex with smallest index is to select the center of gravity of the triangle abc noted g : g(x,y,z) = (a(x,y ,z)+b(x,y ,z)+c(x,y ,z)) / 3.0.

[0138] Then pO = d(x,y,z) *n(x,y,z)+g(x,y,z) in local coordinates or p0(x,y,z)= d(x,y,z)+g(x,y,z) in canonical coordinates.

[0139] In an encoder embodiment, the encoder may compute all base mesh face barycenters and perform a kd-tree search to identify a set of K closest faces for a given Gaussian Splat GSO. The encoder then iteratively tests whether the projection of the GSO center position falls inside one of the K faces. In case the projection does not fall inside any of these faces, then the encoder tests iteratively faces that are connected to the K nearest faces, and so on until a projection is found.

[0140] Thus FIG. 12 shows Gaussian Splats (GSO, GS1 and GS2) and base mesh triangle 1202 with vertices a, b and c and normal n 1210. Points pO, pl and p2 represent the projection of the Gaussian Splats respectively GSO, GS1 and GS2 on the base mesh triangle abc. The projection is performed along the normal vector n (1210) of triangle abc. Variables zO, zl, z2 represent the distance between the Gaussian Splat central point position of GSO, GS1, and GS2 respectively, along their projection axis. Variables zi can be positive (GS is above the triangle surface), negative (GS position is under the triangle surface, or zero (GS center position is on the triangle surface). Projected points pi can be represented with barycentric coordinates of vertices a, b and c in the triangle abc. Projected points can also be represented with a displacement vector based on one of the face vertices or based on the face center of gravity.

[0141] The 4dgs elements are listed in the coded mesh as follows: for (i = 0; i < 4dgs_count_minusl; i++) {4dgs_element[i] = {4dgs_index[i], basemesh face index, alpha, beta, z}}

[0142] Alternatively, UV coordinates are computed based on the barycentric coordinates alpha and beta and the list becomes: for (i = 0; i < 4dgs_count_minusl; i++) {4dgs_element[i] = {4dgs_index[i], basemesh face index, texture_coordinate[0], texture_coordinate[l], z} }

[0143] In the above examples the 4dgs_index[i] may be kept implicit and removed from the 4dgs_element.

[0144] 1.1.2 Using a vertex primitive as a reference

[0145] In order to associate a 4DGS GSO with a base mesh vertex, an encoder may for example select the vertex vO that is the closest to the GS through a kd-tree search for example using a shortest distance criterion. When two or more vertices are at the same distance, the one where the normal is aligned towards GSO is preferred and if this criterion is not sufficient, then the vertex with the lowest index is selected. It is not necessary that all vertices of the base mesh are mapped to a Gaussian Splat and it is not an issue that several Gaussian Splats are mapped to the same vertex.

[0146] The 3D displacement vector between a Gaussian Splat GSO and the vertex vO it is mapped to is provided by GS0_position (x,y,z) - v0(x,y,z).

[0147] The coded mesh then contains the following: for (i = 0; i < 4dgs_count_minusl; i++) {4dgs_element[i] = {4dgs_index[i], basemesh_vertex_index, displacement_vector[0]; displacement_vector

[0001] , displacement_vector[2] } }

[0148] In this case, the UV coordinates of the 4dgs_element are set to the ones of the base mesh vertex they are mapped to.

[0149] In the above examples the 4dgs_index[i] may be kept implicit and removed from the 4dgs_element.

[0150] 1.1.3 Using an edge primitive as a reference

[0151] A base mesh edge e (vO-vl) connecting base mesh vertices vO and vl can be mapped to a 4DGS element as follows.

[0152] The midpoint of edge e, given by (v0+vl) / 2 is used to compute a displacementvector d towards the GS central position. The 3D displacement may be represented in canonical coordinate system or in a local coordinate system using the edge midpoint vertex normal.

[0153] The coded mesh then contains the following: for (i = 0; i < 4dgs_count_minusl; i++) {4dgs_element[i] = {4dgs_index[i], basemesh edge index, displacement_vector[0]; displacement vectorf 1 ], displacement_vector[2] }}

[0154] In this case, the UV coordinates of the 4dgs_element are set to the ones of the base mesh edge midpoint they are mapped to.

[0155] In the above examples the 4dgs_index[i] may be kept implicit and removed from the 4dgs_element.

[0156] 1.1.4 Using a mixture of primitives as a reference

[0157] In another embodiment, the base mesh may link 4dgs elements to any of the primitive types in order to improve compression performance, i.e., some elements are mapped to a vertex, others to an edge and the remaining ones to faces. for (i = 0; i < 4dgs_count_minusl; i++) {4dgs_element[i] = {4dgs_index[i], primitive_type, primitive_index, displacement_vector[0]; displacement vectorf 1 ], displacement_vector[2] } }

[0158] In the above syntax, primitive type equal to 0 indicates a vertex, equal to 1 indicates an edge and equal to 2 indicates a face.

[0159] 2. Mapping other GS attributes with a material ID attribute

[0160] In one embodiment, the different parameters of the Gaussian Splats (rotation, scale, opacity, color, spherical harmonics) may be associated with a dedicated attribute map. This creates a list of attributes that needs to be defined in the mesh coded representation, for example in the V-DMC framework or in a static mesh codec such as Draco or the MPEGEdge Breaker (V-DMC Annex I).

[0161] Mesh attributes may be associated with faces, vertices and sometimes with edges thanks to UV coordinates. Different UV coordinate tables can be defined for meshes and attached to vertices, faces (e.g., the vertices of the face) and edges (e.g., the edge vertices or the midpoint virtual vertex of the edge).

[0162] In one embodiment, a material ID attribute lists a mapping between the Gaussian Splat parameters, the texture coordinate map (face, vertex, edge, multiple maps per vertex etc.) and the index of the mesh attribute map that can be coded as an image.

[0163] In another embodiment, the attribute is not coded as an image but rather encoded or decoded similarly to mesh geometry, i.e., using similar predictors as the geometry coding of vertex positions. In one embodiment the face attribute can be coded as an image or as Edge-breaker comer attribute coding together with the geometry bitstream.

[0164] For Gaussian Splat parameters containing a number of components between 1 and 3, a custom V3C attribute or mesh vertex / face / edge attribute may be used to code the information as an image.

[0165] The generation of the corresponding attribute map at the encoder is similar to the one for the texture map, i.e., rasterizing the image based on the UV-map and the mesh connectivity and filling the non-occupied background with approaches such as push pull or diffusion algorithms or simply set to a constant value. The Gaussian parameter data must be scaled and quantized to positive integers in the most suitable chroma sampling and bitdepth.

[0166] A material ID attribute may be chosen to list the mapping between Gaussian Parameters and coded mesh attributes as in the following example.

[0167] GS Attributes may map to more than one attribute map as they may contain a large number of components. In that case an index list is coded.

[0168] The index does not necessarily link to an attribute coded image but can link to a geometry-embedded coded attribute for codecs using approaches such as Edge Breaker (Draco, V-DMC Annex I).

[0169] In another embodiment, a material ID is associated with vertices, faces and edges of a mesh to allow different texture coordinates per portion of the mesh or primitive. In another embodiment, the material ID may be used to flag the absence of some attribute such as the spherical harmonics to portions of the mesh where they do not bring higher quality. The material ID is then coded as an image for example for each frame of the sequence.

[0170] 2.1 V-DMC case

[0171] The above embodiments described the mapping of Gaussian Splats to mesh primitives and can be performed on the V-DMC base mesh sub-bitstream, including the coding of a material ID.

[0172] Given the coarse nature of the base mesh, in one embodiment, several Gaussian splats may be mapped to a single base mesh primitive. The following describes how to further link the Gaussian Splats to the V-DMC subdivided mesh primitives as illustrated on FIG. 13.

[0173] In particular, FIG. 13 shows a mapping gaussian splat elements from a base mesh primitive on the left (here a face 1300) to the subdivided mesh vertex primitives on the right (closest vertex within the face the Gaussian splat center is projected to). FIG. 13 shows center 1310 of Gaussian splat 1301a and center 1312 of Gaussian splat 1301b.

[0174] FIG. 13 shows that face 1300 is divided into face 1301 and face 1302. FIG. 13 further shows Gaussian splat 1301a mapped to face 1300 and Gaussian splat 1301b (that corresponds to Gaussian splat 1301a) mapped to a subdivision 1303 and subdivision 1304 of face 1300.

[0175] 2.2.1 Subdivided Mesh

[0176] The asve subdivision iteration count and asve subdivision method list metadata are used to perform the subdivision of the reconstructed base mesh. For each 4dgs_element, the closest subdivided vertex is estimated based on the linked base mesh primitive. In case of a face primitive, the barycentric coordinates enable computation of the closest subdivided vertex. In case of a vertex or an edge, the respectively corresponding UV coordinates of the vertex, of the edge midpoint are selected as the UV coordinates for the 4DGS element. FIG. 13 illustrates these aspects.

[0177] 2.1.2 V3C Attributes and Material ID

[0178] For some types of Gaussian Splat parameters, with N components and N larger than 3, the attribute may be muxed for example to ceil(N / 3) attribute maps. Alternatively, as not all Gaussian splats may contain Spherical Harmonics for example, material IDs can be used to generate a smaller attribute map resolution for the Spherical Harmonics.

[0179] In one embodiment, N components of a given parameter may be encoded as a smaller number of components, for example using dimensionality reduction (such as Principal Components Analysis) or learned distillation of higher frequency spherical harmonics coefficients to lower frequency spherical harmonics coefficients, or local linear embeddings (LLE) etc. The method used to reduce the number of encoded components and their transformation may be signaled by an SEI message for example.

[0180] In all these example dimensionality reduction methods, additional information needs to be coded such as principal axes for PCA, or for example neural networks encoded for example with the NNC standard.

[0181] 2.1.3 Lifting Transform

[0182] In an extension of V-DMC the lifting scheme can be used to transform the Gaussian Splat mapped Attribute data. I.e. the parameter from a Gaussian Splat mapped to vertices from LOD 0 can be used to predict the parameters of Gaussian Splats mapped to vertices at LODI and so on prior to generate the attribute map.

[0183] 3. General signaling information embodiments for V-DMC

[0184] The presence of 4DGS data in the V3C bitstream must be signaled, for example with an SEI message or by extending the atlas sequence parameter set. The presence of 4DGS data in the Base Mesh sub-bitstream must be signaled and the base mesh primitive type (vertex, face or edge) to which the 4DGS data is associated to must be signaled. The presence of a 4DGS parameters must be signaled (e.g., not all 4DGS contain spherical harmonics).

[0185] Signaling the presence of 4DGS data in the V-DMC bitstream can be for example be done along the atlas sub-bitstream atlas parameter sets and / or with a dedicated SEI message.

[0186] The number of 4D Gaussian Splats per sequence and / or per frame can be signaled at atlas sequence parameter set asps or atlas frame parameter set afps:

[0187] vps ext bmesh data present flag indicates the presence of 4DGS primitives encoded per frame in the V-DMC sequence.

[0188] asve_4dgs_data_present_flag indicates the presence of 4DGS primitives encoded per frame in the V-DMC sequence.

[0189] asve_4dgs_count_minusl indicates the number of 4DGS primitives encoded per frame in the V-DMC sequence.

[0190] afve_4dgs_count_minusl indicates the number of 4DGS primitives encoded in the current V-DMC frame.

[0191] The list of 4D Gaussian Splats attribute / parameter types that are encoded in the V- DMC sequence, where the attribute types are listed in the Table hereafter

[0192] asve_4dgs_parameter_type_count_minusl indicates the number of 4DGS attributes / parameters encoded in the V-DMC sequence

[0193] asve_4dgs_parameter_type_list indicates the list of 4DGS attributes / parameters encoded in the V-DMC sequence. The length of the sequence is equal to asve_4dgs_parameter_type_count_minusl

[0194] For each 4DGS parameter type, the number of components must be provided. In one embodiment, this number of components is provided by the following Table as well.

[0195] For each 4DGS attribute / parameter type, how it is encoded in the V-DMC subbitstreams (indexed or parameterized).

[0196] asve_4dgs_parameter_type_mapping_list indicates the mapping of 4DGS attributes / parameters encoded in the V-DMC sequence to their corresponding subbitstream. The length of the sequence is equal to asve_4dgs_parameter_type_count_minusl . The following table provides examples of mappings between 4DGS parameters and V3C attribute types.

[0197] In one embodiment, a Material ID attribute is used to indicate the UV coordinates index table and corresponding attribute map. In another embodiment, such a table is provided by an SEI message.

[0198] The V3C parameter set V-DMC extension syntax 8.3.4.11 can be used and extended to separate V-DMC attributes and 4DGS attributes to enable to partially decode and reconstruct them (the example syntax table shown below is also shown in FIG. 14 including items 1402, 1404, and 1406 that show signaling of 4DGS information):

[0199] vps_ext_bmesh_4dgs_attribute_index[ j ][ i ] indicates the attribute index for a 4DGS parameter

[0200] vps_ext_bmesh_4dgs_attribute_bit_depth_minusl[ j ][ i ] indicates the bitdepth for a 4DGS parameter

[0201] vps_ext_bmesh_4dgs_attribute_msb_align _flag[ j ][ i ] if equal to 1 indicates the alignment of the MSB is performed for a 4DGS parameter

[0202] vps_ext_bmesh_4dgs_attribute_type[ j ][ i ] indicates the type of 4DGS attribute.

[0203] vps_ext_4dgs_attribute_frame_width[ j ][ i ] indicates the frame width of a 4DGS attribute map

[0204] vps_ext_4dgs_attribute_frame_height[ j ] [ i ] indicates the frame height of a 4DGS attribute map

[0205] Extensions to the asps 8.3.6.1.3 (the example syntax table is also shown in FIG. 15 A, FIG. 15B, and FIG. 15C including item 1502 that shows signaling of 4DGS information):

[0206] The mapping of a 4DGS parameter to displacement or attribute sub-bitstreams need to be signaled for example with a material ID and / or a SEI message.

[0207] The use of texture coordinates or barycentric coordinates per attribute is signaled in the base mesh sub-bitstream and or in an SEI message.

[0208] Metadata for the reconstruction of the 4DGS parameters and attributes needs to be signaled in atlas metadata

[0209] 4. Embodiment - Packed video coding of V-DMC attributes and displacements and 4DGS parameters

[0210] Frame packing is used on all V-DMC attributes and displacements related to the mesh and on the 4DGS parameters as well. Mesh patch data unit signal the association of a parameter or attribute to a region of the packed video.

[0211] The examples described herein may be relevant to standardization activities for the 4DGS representation, extensions of current standards such as G-PCC, and V-DMC extensions.

[0212] Throughout this description, ISO / IEC 23090-29 means the output document MDS24196_WG07_N00960 of MPEG145.

[0213] FIG. 16 shows an encoder 1600 according to an embodiment. FIG. 16 illustrates an image to be encoded (In), a predicted representation of an image block (P'n), a prediction error signal (Dn), a reconstructed prediction error signal (D'n), a preliminary reconstructed image (I'n), a final reconstructed image (R'n), a transform (T) and inverse transform (T-1), a quantization (Q) and inverse quantizationentropy encoding (E), a reference frame memory (RFM), inter prediction (Pinter), intra prediction (Pintra), mode selection (MS) and filtering (F).

[0214] Gaussian splat signaling 1610 implements the examples described herein related to signaling of Gaussian splats. Gaussian splat processing 1612 implements the examples described herein related to encoder-side Gaussian splat processing.

[0215] FIG. 17 shows a decoder 1700 according to an embodiment. FIG. 17 illustrates a predicted representation of an image block (P'n), a reconstructed prediction error signal (D'n), a preliminary reconstructed image (I'n), a final reconstructed image (R'n), an inverse transform (T-1), an inverse quantizationan entropy decoding (E1), a reference frame memory (RFM), a prediction (either inter or intra) (P), and filtering (F).

[0216] Gaussian splat signaling 1710 implements the examples described herein related to receiving Gaussian splat signaling. Gaussian splat processing 1712 implements the examples described herein related to decoder-side Gaussian splat processing.

[0217] A video encoder transforms the input video into a compressed representation suited for storage / transmission and a video decoder decompresses the compressed videorepresentation back into a viewable form. Typically, an encoder discards some information in the original video sequence in order to represent the video in a more compact form (that is, at lower bitrate).

[0218] A video encoder may encode the video information in two phases. Firstly, pixel values in a certain picture area (or “block”) are predicted for example by motion compensation means (finding and indicating an area in one of the previously coded video frames that corresponds closely to the block being coded) or by spatial means (using the pixel values around the block to be coded in a specified manner). Secondly the prediction error, e.g., the difference between the predicted block of pixels and the original block of pixels, is coded. This is typically done by transforming the difference in pixel values using a specified transform (e.g., Discrete Cosine Transform (DCT) or a variant of it), quantizing the coefficients and entropy coding the quantized coefficients. By varying the fidelity of the quantization process, the encoder can control the balance between the accuracy of the pixel representation (picture quality) and size of the resulting coded video representation (file size or transmission bitrate).

[0219] Inter prediction, which may also be referred to as temporal prediction, motion compensation, or motion-compensated prediction, exploits temporal redundancy. In inter prediction the sources of prediction are previously decoded pictures (a.k.a. reference pictures).

[0220] Intra prediction utilizes the fact that adjacent pixels within the same picture are likely to be correlated. Intra prediction can be performed in spatial or transform domain, e.g., either sample values or transform coefficients can be predicted. Intra prediction is typically exploited in intra coding, where no inter prediction is applied.

[0221] An intra picture may be defined as a coded picture that is decoded using intra prediction only, or in other words, does not make use of inter prediction in decoding. An intra picture may be interchangeably called an intra frame.

[0222] An inter picture may be defined as a coded picture whose decoding may include intra prediction and inter prediction. An inter picture may be interchangeably called an inter frame.

[0223] FIG. 18 is a block diagram illustrating a system 1800 in accordance with several examples. In an example, the encoder 1830 is used to encode an image or video from the scene 1815, and the encoder 1830 is implemented in a transmitting apparatus 1880. The encoder 1830 produces a bitstream 1810 comprising signaling that is received by the receiving apparatus 1882, which implements a decoder 1840. The encoder 1830 sends the bitstream 1810 that comprises the herein described signaling. The decoder 1840 forms the image or video for the scene 1815-1, and the receiving apparatus 1882 would present this to the user, e.g., via a smartphone, television, or projector among many other options.

[0224] In some examples, the transmitting apparatus 1880 and the receiving apparatus 1882 are at least partially within a common apparatus, and for example are located within a common housing 1850. In other examples the transmitting apparatus 1880 and the receiving apparatus 1882 are at least partially not within a common apparatus and have at least partially different housings. Therefore in some examples, the encoder 1830 and the decoder 1840 are at least partially within a common apparatus, and for example are located within a common housing 1850. For example the common apparatus comprising the encoder 1830 and decoder 1840 implements a codec. In other examples the encoder 1830 and the decoder 1840 are at least partially not within a common apparatus and have at least partially different housings, but when together still implement a codec.

[0225] In some examples, 3D media from the capture (e.g., volumetric capture) at a viewpoint 1812 of the scene 1815, which includes a person 1813) is converted via projection to a series of 2D representations with occupancy, geometry, attributes and / or displacements. Additional atlas information is also included in the bitstream to enable inverse reconstruction. For decoding, the received bitstream 1810 is separated into its components with atlas information; occupancy, geometry, displacement, and attribute 2D representations. A 3D reconstruction is performed to reconstruct the scene 1815-1 created looking at the viewpoint 1812-1 with a “reconstructed” person 1813-1. The “-1” are used to indicate that these are reconstructions of the original.

[0226] As indicated at 1820, the decoder 1840 performs an action or actions based on the received Gaussian splat signaling.

[0227] Encoding 1890 performs the examples described herein related to encoding ofGaussian splat information to be used during the reconstruction process. Decoding 1892 performs the examples described herein related to decoding Gaussian splat information and usage of the Gaussian splat information to generate reconstructed scene 1815-1.

[0228] FIG. 19 is an example apparatus 1900, which may be implemented in hardware, configured to implement the examples described herein. The apparatus 1900 comprises at least one processor 1902 (e.g., an FPGA and / or CPU and / or GPU), one or more memories 1904 including computer program code 1905, the computer program code 1905 having instructions to carry out the methods described herein, wherein the at least one memory 1904 and the computer program code 1905 are configured to, with the at least one processor 1902, cause the apparatus 1900 to implement circuitry, a process, component, module, or function (implemented with control module 1906) to implement the examples described herein.

[0229] Apparatus 1900 may be a smartphone, personal digital device or assistant, smart television, laptop, pad, tablet, head-mounted display (HMD), or other user device or terminal device. The memory 1904 may be a non-transitory memory, a transitory memory, a volatile memory (e.g. RAM), or a non-volatile memory (e.g., ROM).

[0230] Gaussian splat signaling 1930 implements the examples described herein related to signaling Gaussian splat information. Gaussian splat processing 1940 implements the examples described herein related to processing of Gaussian splat information.

[0231] The apparatus 1900 includes a display and / or I / O interface 1908, which includes user interface (UI) circuitry and elements, that may be used to display features or a status of the methods described herein (e.g., as one of the methods is being performed or at a subsequent time), or to receive input from a user such as with using a keypad, camera, touchscreen, touch area, microphone, biometric recognition, one or more sensors, etc. The apparatus 1900 includes one or more communication e.g. network (N / W) interfaces (I / F(s)) 1910. The communication I / F(s) 1910 may be wired and / or wireless and communicate over the Internet / other network(s) via any communication technique including via one or more links 1924. The communication I / F(s) 1910 may comprise one or more transmitters or one or more receivers.

[0232] The transceiver 1916 comprises one or more transmitters 1918 and one or more receivers 1920. The transceiver 1916 and / or communication I / F(s) 1910 may comprise standard well-known components such as an amplifier, filter, frequency-converter, (de)modulator, and encoder / decoder circuitries and one or more antennas, such as antennas 1914 used for communication over wireless link 1926.

[0233] The control module 1906 of the apparatus 1900 comprises one of or both parts 1906-1 and / or 1906-2, which may be implemented in a number of ways. The control module 1906 may be implemented in hardware as control module 1906-1, such as being implemented as part of the one or more processors 1902. The control module 1906-1 may be implemented also as an integrated circuit or through other hardware such as a programmable gate array. In another example, the control module 1906 may be implemented as control module 1906-2, which is implemented as computer program code (having corresponding instructions) 1905 and is executed by the one or more processors 1902. For instance, the one or more memories 1904 store instructions that, when executed by the one or more processors 1902, cause the apparatus 1900 to perform one or more of the operations as described herein. Furthermore, the one or more processors 1902, one or more memories 1904, and example algorithms (e.g., as flowcharts and / or signaling diagrams), encoded as instructions, programs, or code, are means for causing performance of the operations described herein.

[0234] The apparatus 1900 to implement the functionality of control 1906 may correspond to any of the apparatuses depicted herein. Alternatively, apparatus 1900 and its elements may not correspond to any of the other apparatuses depicted herein, as apparatus 1900 may be part of a self-organizing / optimizing network (SON) node or other node, such as a node in a cloud.

[0235] The apparatus 1900 may also be distributed throughout the network including within and between apparatus 1900 and any network element (such as a base station and / or terminal device and / or user equipment).

[0236] Interface 1912 enables data communication and signaling between the various items of apparatus 1900, as shown in FIG. 19. For example, the interface 1912 may be one or more buses such as address, data, or control buses, and may include any interconnectionmechanism, such as a series of lines on a motherboard or integrated circuit, fiber optics or other optical communication equipment, and the like. Computer program code (e.g. instructions) 1905, including control 1906 may comprise object-oriented software configured to pass data or messages between objects within computer program code 1905. Computer program code (e.g. instructions) 1905, including control 1906 may comprise procedural, functional, or scripting code. The apparatus 1900 need not comprise each of the features mentioned, or may comprise other features as well. The various components of apparatus 1900 may at least partially reside in a common housing 1928, or a subset of the various components of apparatus 1900 may at least partially be located in different housings, which different housings may include housing 1928.

[0237] FIG. 20 shows a schematic representation of non-volatile memory media 2000a (e.g. computer / compact disc (CD) or digital versatile disc (DVD)) and 2000b (e.g. universal serial bus (USB) memory stick) and 2000c (e.g. cloud storage for downloading instructions and / or parameters 2002 or receiving emailed instructions and / or parameters 2002) storing instructions and / or parameters 2002 which when executed by a processor allows the processor to perform one or more of the operations of the methods described herein. Instructions and / or parameters 2002 may represent or correspond to a non-transitory computer readable medium.

[0238] FIG. 21 is an example method 2100 based on the examples described herein. At 2110, the method includes receiving an input mesh sequence. At 2120, the method includes receiving a Gaussian splat representation sequence. At 2130, the method includes producing metadata that associates the Gaussian splat representation sequence with the input mesh sequence. At 2140, the method includes jointly coding the input mesh sequence, the Gaussian splat representation sequence, and the metadata that associates the Gaussian splat representation sequence with the input mesh sequence into an output bitstream. Method 2100 may be performed with encoder 100, encoder 1000, encoder 1600, transmitting apparatus 1880 with encoder 1830, or apparatus 1900.

[0239] FIG. 22 is an example method 2200 based on the examples described herein. At 2210, the method includes receiving a bitstream comprising a mesh sequence, a Gaussian splat representation sequence, and metadata that associates the Gaussian splat representationsequence with the mesh sequence. At 2220, the method includes extracting the metadata that associates the Gaussian splat representation sequence with the mesh sequence. At 2230, the method includes decoding the mesh sequence. At 2240, the method includes decoding the Gaussian splat representation sequence. At 2250, the method includes reconstructing one or more of the mesh sequence or the Gaussian splat representation sequence. Method 2200 may be performed with decoder 1100, decoder 1700, receiving apparatus 1882 with decoder 1840, or apparatus 1900.

[0240] The following examples are provided and described herein.

[0241] Example 1. An apparatus including: at least one processor; and at least one memory storing instructions that, when executed by the at least one processor, cause the apparatus at least to: receive an input mesh sequence; receive a Gaussian splat representation sequence; produce metadata that associates the Gaussian splat representation sequence with the input mesh sequence; and jointly code the input mesh sequence, the Gaussian splat representation sequence, and the metadata that associates the Gaussian splat representation sequence with the input mesh sequence into an output bitstream.

[0242] Example 2. The apparatus of example 1, wherein the apparatus is further caused to: encode Gaussian splat data associated to the input mesh sequence, wherein the Gaussian splat data is related to the Gaussian splat representation sequence.

[0243] Example 3. The apparatus of example 2, wherein the Gaussian splat data associated to the input mesh sequence comprises a material identifier that provides a mapping of Gaussian splat parameters to respective attributes.

[0244] Example 4. The apparatus of any of examples 1 to 3, wherein the apparatus is further caused to: encode Gaussian splat parameters related to the Gaussian splat representation sequence into a base mesh sub-bitstream; wherein a Gaussian splat parameter of the Gaussian splat parameters is mapped to a face, edge, or vertex of a base mesh.

[0245] Example 5. The apparatus of any of examples 1 to 4, wherein the apparatus is further caused to: encode Gaussian splat data associated with the Gaussian splat representation sequence and information associated with the input mesh sequence into or along an atlas sub-bitstream.

[0246] Example 6. The apparatus of example 5, wherein the atlas sub-bitstream is a videobased dynamic mesh coding atlas sub-bitstream.

[0247] Example 7. The apparatus of any of examples 1 to 6, wherein the apparatus is further caused to: convert Gaussian splat attribute data to a format that is compatible with the output bitstream, wherein the output bitstream is a video-based dynamic mesh coding bitstream.

[0248] Example 8. The apparatus of example 7, wherein the Gaussian splat attribute data that is converted to the format that is compatible with the video-based dynamic mesh coding bitstream comprises one or more of: at least one non-negative integer data, clustering data, or data related to projection to a base mesh.

[0249] Example 9. The apparatus of any of examples 1 to 8, wherein the apparatus is further caused to: signal information that indicates whether or not Gaussian splat data associated with the Gaussian splat representation sequence is present in the output bitstream.

[0250] Example 10. The apparatus of any of examples 1 to 9, wherein frame packing is used on video-based dynamic mesh coding attributes related to the input mesh sequence, displacements related to the input mesh sequence, and Gaussian splat parameters associated with the Gaussian splat representation sequence.

[0251] Example 11. The apparatus of any of examples 1 to 10, wherein the apparatus is further caused to: encode a mapping of Gaussian splat two-dimensional texture coordinate parameters to attributes into an attribute sub-bitstream; and encode the attribute subbitstream comprising the Gaussian splat two-dimensional texture coordinate parameters into the output bitstream.

[0252] Example 12. The apparatus of any of examples 1 to 11, wherein the apparatus is further configured to: encode, into a base mesh sub-bitstream, information that links a Gaussian splat position to a base mesh.

[0253] Example 13. The apparatus of example 12, wherein the information that links the Gaussian splat position to the base mesh comprises information that associates a Gaussian splat center position or index with a triangle of the base mesh.

[0254] Example 14. An apparatus including: at least one processor; and at least one memory storing instructions that, when executed by the at least one processor, cause the apparatus at least to: receive a bitstream comprising a mesh sequence, a Gaussian splat representation sequence, and metadata that associates the Gaussian splat representation sequence with the mesh sequence; extract the metadata that associates the Gaussian splat representation sequence with the mesh sequence; decode the mesh sequence; decode the Gaussian splat representation sequence; and reconstruct one or more of the mesh sequence or the Gaussian splat representation sequence.

[0255] Example 15. The apparatus of example 14, wherein the apparatus is further caused to: decode Gaussian splat data associated to the input mesh sequence, wherein the Gaussian splat data is related to the Gaussian splat representation sequence.

[0256] Example 16. The apparatus of example 15, wherein: the Gaussian splat data associated to the input mesh sequence comprises a material identifier that provides a mapping of Gaussian splat parameters to respective attributes, and the one or more of the mesh sequence or the Gaussian splat representation sequence is reconstructed based on the material identifier that provides a mapping of the Gaussian splat parameters to respective attributes.

[0257] Example 17. The apparatus of any of examples 14 to 16, wherein the apparatus is further caused to: decode a base mesh sub-bitstream from the bitstream; decode Gaussian splat parameters related to the Gaussian splat representation sequence from the base mesh sub-bitstream; wherein a Gaussian splat parameter of the Gaussian splat parameters is mapped to a face, edge, or vertex of a base mesh; wherein one or more of the mesh sequence or the Gaussian splat representation sequence is reconstructed from: the Gaussian splat parameters related to the Gaussian splat representation sequence decoded from the base mesh sub-bitstream, including the respective Gaussian splat parameter mapped to the face, edge, or vertex of the base mesh.

[0258] Example 18. The apparatus of any of examples 14 to 17, wherein the apparatus is further caused to: decode signaling of: Gaussian splat data associated with the Gaussian splat representation sequence and information associated with the input mesh sequence from an atlas sub-bitstream; wherein one or more of the mesh sequence or the Gaussian splatrepresentation sequence is reconstructed from the Gaussian splat data associated with the Gaussian splat representation sequence decoded from the atlas sub-bitstream, or the information associated with the input mesh sequence decoded from the atlas sub-bitstream.

[0259] Example 19. The apparatus of example 18, wherein the atlas sub-bitstream is a video-based dynamic mesh coding atlas sub-bitstream.

[0260] Example 20. The apparatus of any of examples 14 to 19, wherein the Gaussian splat attribute data is in a format that is compatible with the bitstream, wherein the bitstream is a video-based dynamic mesh coding bitstream.

[0261] Example 21. The apparatus of example 20, wherein the Gaussian splat attribute data that is in the format that is compatible with the video-based dynamic mesh coding bitstream comprises one or more of: at least one non-negative integer data, clustering data, or data related to projection to a base mesh.

[0262] Example 22. The apparatus of any of examples 14 to 21, wherein the apparatus is further caused to: decode signaling of information that indicates whether or not Gaussian splat data is present in the bitstream; wherein the one or more of the mesh sequence or the Gaussian splat representation sequence is reconstructed based on the information that indicates whether or not Gaussian splat data is present in the bitstream.

[0263] Example 23. The apparatus of any of examples 14 to 22, wherein the one or more of the mesh sequence or the Gaussian splat representation sequence is reconstructed based on the metadata that associates the Gaussian splat representation sequence with the mesh sequence.

[0264] Example 24. The apparatus of any of examples 14 to 23, wherein the apparatus is further caused to: decode an attribute sub-bitstream from the bitstream; decode a mapping of Gaussian splat two-dimensional texture coordinate parameters to attributes from the attribute sub-bitstream; and wherein the one or more of the mesh sequence or the Gaussian splat representation sequence is reconstructed from the mapping of the Gaussian splat two- dimensional texture coordinate parameters to attributes decoded from the attribute subbitstream.

[0265] Example 25. The apparatus of any of examples 14 to 24, wherein the apparatus is further caused to: decode a base mesh sub-bitstream from the bitstream; decode, from the base mesh sub-bitstream, information that links a Gaussian splat position to a base mesh.

[0266] Example 26. The apparatus of example 25, wherein the information that links the Gaussian splat position to the base mesh comprises information that associates a Gaussian splat center position or index with a triangle of the bash mesh.

[0267] Example 27. A method including: receiving an input mesh sequence; receiving a Gaussian splat representation sequence; producing metadata that associates the Gaussian splat representation sequence with the input mesh sequence; and jointly coding the input mesh sequence, the Gaussian splat representation sequence, and the metadata that associates the Gaussian splat representation sequence with the input mesh sequence into an output bitstream.

[0268] Example 28. A method including: receiving a bitstream comprising a mesh sequence, a Gaussian splat representation sequence, and metadata that associates the Gaussian splat representation sequence with the mesh sequence; extracting the metadata that associates the Gaussian splat representation sequence with the mesh sequence; decoding the mesh sequence; decoding the Gaussian splat representation sequence; and reconstructing one or more of the mesh sequence or the Gaussian splat representation sequence.

[0269] Example 29. An apparatus including: means for receiving an input mesh sequence; means for receiving a Gaussian splat representation sequence; means for producing metadata that associates the Gaussian splat representation sequence with the input mesh sequence; and means for jointly coding the input mesh sequence, the Gaussian splat representation sequence, and the metadata that associates the Gaussian splat representation sequence with the input mesh sequence into an output bitstream.

[0270] Example 30. An apparatus including: means for receiving a bitstream comprising a mesh sequence, a Gaussian splat representation sequence, and metadata that associates the Gaussian splat representation sequence with the mesh sequence; means for extracting the metadata that associates the Gaussian splat representation sequence with the mesh sequence; means for decoding the mesh sequence; means for decoding the Gaussian splatrepresentation sequence; and means for reconstructing one or more of the mesh sequence or the Gaussian splat representation sequence.

[0271] Example 31. A computer readable medium including instructions stored thereon for performing at least the following: receiving an input mesh sequence; receiving a Gaussian splat representation sequence; producing metadata that associates the Gaussian splat representation sequence with the input mesh sequence; and jointly coding the input mesh sequence, the Gaussian splat representation sequence, and the metadata that associates the Gaussian splat representation sequence with the input mesh sequence into an output bitstream.

[0272] Example 32. A computer readable medium including instructions stored thereon for performing at least the following: receiving a bitstream comprising a mesh sequence, a Gaussian splat representation sequence, and metadata that associates the Gaussian splat representation sequence with the mesh sequence; extracting the metadata that associates the Gaussian splat representation sequence with the mesh sequence; decoding the mesh sequence; decoding the Gaussian splat representation sequence; and reconstructing one or more of the mesh sequence or the Gaussian splat representation sequence.

[0273] References to a ‘computer’, ‘processor’, etc. should be understood to encompass not only computers having different architectures such as single / multi-processor architectures and sequential / parallel architectures but also specialized circuits such as field- programmable gate arrays (FPGAs), application specific circuits (ASICs), signal processing devices and other processing circuitry. References to computer program, instructions, code etc. should be understood to encompass software for a programmable processor or firmware such as, for example, the programmable content of a hardware device such as instructions for a processor, or configuration settings for a fixed-function device, gate array or programmable logic device, etc.

[0274] The term “non-transitory,” as used herein, is a limitation of the medium itself (i.e., tangible, not a signal) as opposed to a limitation on data storage persistency (e.g., RAM vs. ROM).

[0275] As used herein, the term ‘circuitry’, ‘circuit’ and variants may refer to any of thefollowing: (a) hardware circuit implementations, such as implementations in analog and / or digital circuitry, and (b) combinations of circuits and software (and / or firmware), such as (as applicable): (i) a combination of processor(s) or (ii) portions of processor(s) / software including digital signal processor(s), software, and one or more memories that work together to cause an apparatus to perform various functions, and (c) circuits, such as a microprocessor s) or a portion of a microprocessor s), that require software or firmware for operation, even when the software or firmware is not physically present. As a further example, as used herein, the term ‘circuitry’ would also cover an implementation of merely a processor (or multiple processors) or a portion of a processor and its (or their) accompanying software and / or firmware. The term ‘circuitry’ would also cover, for example and when applicable to the particular element, a baseband integrated circuit or applications processor integrated circuit for a mobile phone or a similar integrated circuit in a server, a cellular network device, or another network device. Circuitry or circuit may also be used to mean a function or a process used to execute a method.

[0276] It should be understood that the foregoing description is only illustrative. Various alternatives and modifications may be devised by those skilled in the art. For example, features recited in the various dependent claims could be combined with each other in any suitable combination(s). In addition, features from different embodiments described above could be selectively combined into a new embodiment. Accordingly, the description is intended to embrace all such alternatives, modifications and variances which fall within the scope of the appended claims.

[0277] The following acronyms and abbreviations that may be found in the specification and / or the drawing figures are defined as follows (the abbreviations may be appended with each other or with other characters using e.g. a hyphen, dash (-), or number (or abbreviations having a character may be the same with a character removed), and may be case insensitive):ID one-dimensional2D two-dimensional3D three-dimensional3DG 3D graphics3DGS three dimensional Gaussian splat4DGS dynamic Gaussian splat, or four dimensional Gaussian splat, or 3DGaussian splat in a moving sequence ae(v) context-adaptive arithmetic entropy-coded syntax element.AC alternating current afve atlas frame parameter set V-DMC extension afps atlas frame parameter setASIC application specific integrated circuit asps atlas sequence parameter set asve Atlas Sequence Parameter Set V-DMC extensionATTR attributeAVC advanced video coding b blueBMCL base mesh coding layer bmesh base meshCLERS string having symbols C, L, E, R, SCPU central processing unitCVS coded V3C sequencesDC direct currentDCT Discrete Cosine Transform ext extensionFFT fast Fourier transformFPGA field programmable gate array g greenG-PCC geometry-based point cloud compressionGPU graphics processing unitGS Gaussian splatH.2xx family of video coding standards (e.g. H.264, H.265)HEVC high efficiency video codingHMD head-mounted displayID identifierIEC International Electrotechnical CommissionI / F interfaceI / O input / outputISO International Organization for Standardization kd k-dimensional (e.g. kd-tree)LLE local linear embeddingLoD level of detailMDS multimedia description schemeMIV MPEG immersive videoMPEG moving picture experts groupNAL network abstraction layerNNC neural network codingN / W networkPCA principal components analysis r redRAM random access memoryRBSP raw byte sequence payloadRGBA red green blue alphaRFM reference frame memoryROM read only memorySC subcommitteeSEI supplemental enhancement information se(v) signed integer Exp-Golomb-coded syntax element with the left bit firstSH spherical harmonicsSON self-organizing / optimizing network ue(v) unsigned integer Exp-Golomb-coded syntax element with the left bit first u(n) unsigned integer using n bits (e.g. u(2))UI user interfaceUSB universal serial busUV "U" and "V" denote the axes of the 2D textureV3C visual volumetric video-based codingV-DMC, vdmc video-based dynamic mesh codingV-PCC video-based point cloud compression vui video usability information vi(v) signed integer using a variable number of bytes as groups of 8 bits. vu(v) unsigned integer using a variable number of bytes as groups of 8 bitsVPS V3C parameter setWD working draftWG working group

Claims

CLAIMSWhat is claimed is:

1. An apparatus comprising: at least one processor; and at least one memory storing instructions that, when executed by the at least one processor, cause the apparatus at least to: receive an input mesh sequence; receive a Gaussian splat representation sequence; produce metadata that associates the Gaussian splat representation sequence with the input mesh sequence; and jointly code the input mesh sequence, the Gaussian splat representation sequence, and the metadata that associates the Gaussian splat representation sequence with the input mesh sequence into an output bitstream.

2. The apparatus of claim 1, wherein the apparatus is further caused to: encode Gaussian splat data associated to the input mesh sequence, wherein the Gaussian splat data is related to the Gaussian splat representation sequence.

3. The apparatus of claim 2, wherein the Gaussian splat data associated to the input mesh sequence comprises a material identifier that provides a mapping of Gaussian splat parameters to respective attributes.

4. The apparatus of any of claims 1 to 3, wherein the apparatus is further caused to: encode Gaussian splat parameters related to the Gaussian splat representationsequence into a base mesh sub-bitstream; wherein a Gaussian splat parameter of the Gaussian splat parameters is mapped to a face, edge, or vertex of a base mesh.

5. The apparatus of any of claims 1 to 4, wherein the apparatus is further caused to: encode Gaussian splat data associated with the Gaussian splat representation sequence and information associated with the input mesh sequence into or along an atlas sub-bitstream.

6. The apparatus of claim 5, wherein the atlas sub-bitstream is a video-based dynamic mesh coding atlas sub-bitstream.

7. The apparatus of any of claims 1 to 6, wherein the apparatus is further caused to: convert Gaussian splat attribute data to a format that is compatible with the output bitstream, wherein the output bitstream is a video-based dynamic mesh coding bitstream.

8. The apparatus of claim 7, wherein the Gaussian splat attribute data that is converted to the format that is compatible with the video-based dynamic mesh coding bitstream comprises one or more of: at least one non-negative integer data, clustering data, or data related to projection to a base mesh.

9. The apparatus of any of claims 1 to 8, wherein the apparatus is further caused to: signal information that indicates whether or not Gaussian splat data associated with the Gaussian splat representation sequence is present in the output bitstream.

10. The apparatus of any of claims 1 to 9, wherein frame packing is used on videobased dynamic mesh coding attributes related to the input mesh sequence, displacements related to the input mesh sequence, and Gaussian splat parameters associated with the Gaussian splat representation sequence.

11. The apparatus of any of claims 1 to 10, wherein the apparatus is further caused to:55encode a mapping of Gaussian splat two-dimensional texture coordinate parameters to attributes into an attribute sub-bitstream; and encode the attribute sub-bitstream comprising the Gaussian splat two- dimensional texture coordinate parameters into the output bitstream.

12. The apparatus of any of claims 1 to 11, wherein the apparatus is further configured to: encode, into a base mesh sub-bitstream, information that links a Gaussian splat position to a base mesh.

13. The apparatus of claim 12, wherein the information that links the Gaussian splat position to the base mesh comprises information that associates a Gaussian splat center position or index with a triangle of the base mesh.

14. An apparatus comprising: at least one processor; and at least one memory storing instructions that, when executed by the at least one processor, cause the apparatus at least to: receive a bitstream comprising a mesh sequence, a Gaussian splat representation sequence, and metadata that associates the Gaussian splat representation sequence with the mesh sequence; extract the metadata that associates the Gaussian splat representation sequence with the mesh sequence; decode the mesh sequence; decode the Gaussian splat representation sequence; and reconstruct one or more of the mesh sequence or the Gaussian splat representation sequence.5615. The apparatus of claim 14, wherein the apparatus is further caused to: decode Gaussian splat data associated to the input mesh sequence, wherein the Gaussian splat data is related to the Gaussian splat representation sequence.

16. The apparatus of claim 15, wherein: the Gaussian splat data associated to the input mesh sequence comprises a material identifier that provides a mapping of Gaussian splat parameters to respective attributes, and the one or more of the mesh sequence or the Gaussian splat representation sequence is reconstructed based on the material identifier that provides a mapping of the Gaussian splat parameters to respective attributes.

17. The apparatus of any of claims 14 to 16, wherein the apparatus is further caused to: decode a base mesh sub-bitstream from the bitstream; decode Gaussian splat parameters related to the Gaussian splat representation sequence from the base mesh sub-bitstream; wherein a Gaussian splat parameter of the Gaussian splat parameters is mapped to a face, edge, or vertex of a base mesh; wherein one or more of the mesh sequence or the Gaussian splat representation sequence is reconstructed from: the Gaussian splat parameters related to the Gaussian splat representation sequence decoded from the base mesh sub-bitstream, including the respective Gaussian splat parameter mapped to the face, edge, or vertex of the base mesh.

18. The apparatus of any of claims 14 to 17, wherein the apparatus is further caused to: decode signaling of: Gaussian splat data associated with the Gaussian splat representation sequence and information associated with the input mesh sequence from an atlas sub-bitstream;57wherein one or more of the mesh sequence or the Gaussian splat representation sequence is reconstructed from the Gaussian splat data associated with the Gaussian splat representation sequence decoded from the atlas sub-bitstream, or the information associated with the input mesh sequence decoded from the atlas sub-bitstream.

19. The apparatus of claim 18, wherein the atlas sub-bitstream is a video-based dynamic mesh coding atlas sub-bitstream.

20. The apparatus of any of claims 14 to 19, wherein the Gaussian splat attribute data is in a format that is compatible with the bitstream, wherein the bitstream is a videobased dynamic mesh coding bitstream.

21. The apparatus of claim 20, wherein the Gaussian splat attribute data that is in the format that is compatible with the video-based dynamic mesh coding bitstream comprises one or more of: at least one non-negative integer data, clustering data, or data related to projection to a base mesh.

22. The apparatus of any of claims 14 to 21, wherein the apparatus is further caused to: decode signaling of information that indicates whether or not Gaussian splat data is present in the bitstream; wherein the one or more of the mesh sequence or the Gaussian splat representation sequence is reconstructed based on the information that indicates whether or not Gaussian splat data is present in the bitstream.

23. The apparatus of any of claims 14 to 22, wherein the one or more of the mesh sequence or the Gaussian splat representation sequence is reconstructed based on the metadata that associates the Gaussian splat representation sequence with the mesh sequence.

24. The apparatus of any of claims 14 to 23, wherein the apparatus is further caused to: decode an attribute sub-bitstream from the bitstream;decode a mapping of Gaussian splat two-dimensional texture coordinate parameters to attributes from the attribute sub-bitstream; and wherein the one or more of the mesh sequence or the Gaussian splat representation sequence is reconstructed from the mapping of the Gaussian splat two- dimensional texture coordinate parameters to attributes decoded from the attribute subbitstream.

25. The apparatus of any of claims 14 to 24, wherein the apparatus is further caused to: decode a base mesh sub-bitstream from the bitstream; decode, from the base mesh sub-bitstream, information that links a Gaussian splat position to a base mesh.

26. The apparatus of claim 25, wherein the information that links the Gaussian splat position to the base mesh comprises information that associates a Gaussian splat center position or index with a triangle of the bash mesh.

27. A method comprising: receiving an input mesh sequence; receiving a Gaussian splat representation sequence; producing metadata that associates the Gaussian splat representation sequence with the input mesh sequence; and jointly coding the input mesh sequence, the Gaussian splat representation sequence, and the metadata that associates the Gaussian splat representation sequence with the input mesh sequence into an output bitstream.

28. A method comprising: receiving a bitstream comprising a mesh sequence, a Gaussian splat representation sequence, and metadata that associates the Gaussian splat representationsequence with the mesh sequence; extracting the metadata that associates the Gaussian splat representation sequence with the mesh sequence; decoding the mesh sequence; decoding the Gaussian splat representation sequence; and reconstructing one or more of the mesh sequence or the Gaussian splat representation sequence.

29. An apparatus comprising: means for receiving an input mesh sequence; means for receiving a Gaussian splat representation sequence; means for producing metadata that associates the Gaussian splat representation sequence with the input mesh sequence; and means for jointly coding the input mesh sequence, the Gaussian splat representation sequence, and the metadata that associates the Gaussian splat representation sequence with the input mesh sequence into an output bitstream.

30. An apparatus comprising: means for receiving a bitstream comprising a mesh sequence, a Gaussian splat representation sequence, and metadata that associates the Gaussian splat representation sequence with the mesh sequence; means for extracting the metadata that associates the Gaussian splat representation sequence with the mesh sequence; means for decoding the mesh sequence; means for decoding the Gaussian splat representation sequence; andmeans for reconstructing one or more of the mesh sequence or the Gaussian splat representation sequence.

31. A computer readable medium comprising instructions stored thereon for performing at least the following: receiving an input mesh sequence; receiving a Gaussian splat representation sequence; producing metadata that associates the Gaussian splat representation sequence with the input mesh sequence; and jointly coding the input mesh sequence, the Gaussian splat representation sequence, and the metadata that associates the Gaussian splat representation sequence with the input mesh sequence into an output bitstream.

32. A computer readable medium comprising instructions stored thereon for performing at least the following: receiving a bitstream comprising a mesh sequence, a Gaussian splat representation sequence, and metadata that associates the Gaussian splat representation sequence with the mesh sequence; extracting the metadata that associates the Gaussian splat representation sequence with the mesh sequence; decoding the mesh sequence; decoding the Gaussian splat representation sequence; and reconstructing one or more of the mesh sequence or the Gaussian splat representation sequence.61

Images