Silent packages animation bitstream format and compression for avatar representation format

Abstract

A method, in a receiver, is disclosed. An avatar animation bitstream is received from a transmitter. The avatar animation bitstream comprises at least one avatar animation unit (AAU). A header of the at least one avatar animation unit (AAU) is decoded. It is then determined from the header that the AAU is a silent AAU, a silent AAU indicating to the receiver that the transmitter is still alive and not sending any animation data. Responsive to the determining, a first syntax element indicating a number of ticks per second and a second syntax element indicating a duration of the silent AAU are decoded. A target avatar is rendered taking into account the first and second syntax elements.

Description

[0001] Docket No. 2025P00183WO

[0002] SILENT PACKAGES ANIMATION BITSTREAM FORMAT AND COMPRESSION FOR AVATAR REPRESENTATION FORMAT

[0003] CROSS REFERENCE TO RELATED APPLICATIONS

[0004] This application claims the benefit of European Application No 25305422.5, filed on March 25, 2025 which is incorporated herein by reference in its entirety.

[0005] BACKGROUND

[0006] The present application is related to the encoding and decoding of avatar animation data.

[0007] BRIEF SUMMARY

[0008] Briefly stated, in one embodiment, a method, in a receiver, is disclosed. An avatar animation bitstream is received from a transmitter. The avatar animation bitstream comprises at least one avatar animation unit (AAU). A header of the at least one avatar animation unit (AAU) is decoded. It is then determined from the header that the AAU is a silent AAU, a silent AAU indicating to the receiver that the transmitter is not sending any animation data. Responsive to the determining, a first syntax element indicating a number of ticks per second and a second syntax element indicating a duration of the silent AAU are decoded. A target avatar is rendered taking into account the first and second syntax elements.

[0009] In another embodiment, a method, in a transmitter, is disclosed. It is determined that the transmitter is not sending any animation data to a receiver. In a header of an avatar animation unit (AAU) information is encoded that indicates that the AAU is a silent AAU.

[0010] A first syntax element indicating a number of ticks per second and a second syntax element indicating a duration of the silent AAU are then coded in a payload of the AAU. The avatar animation bitstream comprising the AAU is then transmitted to the receiver.

[0011] Corresponding apparatuses are also disclosed.

[0012] In one embodiment, a signal is disclosed that comprises a header of an avatar animation unit comprising information indicating that the AAU is a silent AAU and a payload comprising a first syntax element indicating a number of ticks per second and a second syntax element indicating a duration of the silent AAU, wherein the silent AAU indicates to a receiver of the signal that a transmitter of the signal is not sending any animation data.Docket No. 2025P00183WO BRIEF DESCRIPTION OF THE DRAWINGS

[0013] The following detailed description will be beter understood when read in conjunction with the appended drawings, in which there are shown examples of one or more of the multiple embodiments of the present disclosure. It should be understood, however, that the embodiments described herein are not limited to the precise arrangements and instrumentalities shown in the drawings. In the drawings:

[0014] FIG. 1 illustrates a binary structure using keyframe and non-keyframe AAUs according to one or more embodiments of the present disclosure;

[0015] FIG.2 is a block diagram illustrating an example of a method for decoding avatar parameters according to one or more embodiments of the present disclosure;

[0016] FIG. 3 is a block diagram illustrating an example of a method for encoding avatar parameters according to one or more embodiments of the present disclosure;

[0017] FIG. 4 illustrates a binary structure using keyframe and non-keyframe AAUs when using silent AAU ;

[0018] FIG. 5 is a block diagram illustrating an example of a method in a receiver according to one or more embodiments of the present disclosure;

[0019] FIG. 6 is a block diagram illustrating an example of a method in a transmiter of an avatar animation unit using quantization and compression according to one or more embodiments of the present disclosure;

[0020] FIG. 7 is a block diagram illustrating an example of an encoding method of an avatar animation unit using quantization and compression according to one or more embodiments of the present disclosure;

[0021] FIG. 8 is a block diagram illustrating an example of a decoding method of an avatar animation unit using quantization and compression according to one or more embodiments of the present disclosure;

[0022] FIG. 9 shows an example architecture of a device which may be configured to implement methods according to one or more embodiments of the present disclosure; and

[0023] FIG. 10 shows an example of an embodiment of the syntax of a stream when the data are transmited over a packet-based transmission protocol according to one or more embodiments of the present disclosure.

[0024] DETAILED DESCRIPTION

[0025] In describing the various embodiments of the present disclosure, certain terminology is used herein for convenience only and should not be considered as limiting such embodiments. InDocket No. 2025P00183WO

[0026] the drawings, the same reference numerals are employed for designating the same elements throughout the several figures and the present description.

[0027] The current effort in MPEG for the streaming of avatar animation is limited to animation of volumetric meshes. Volumetric meshes are animated by pre-computing correspondences to a template (shadow mesh) for every single frame of the volumetric sequence. A shadow mesh is a mesh that contains a lower fidelity than the volumetric mesh. The correspondences between the two are performed in such a way that each vertex of the volumetric mesh maps to a polygon of the shadow mesh. The objective of the shadow mesh is to provide additional semantical information to the volumetric mesh, e.g., skeletal structure and lower polygon mesh representation, permitting the transferring of animations of the shadow mesh onto the dependent volumetric mesh.

[0028] The correspondences between the volumetric mesh and the shadow mesh are performed in such a way that each vertex of the volumetric mesh maps to a polygon of the shadow mesh. At the application side, the volumetric mesh follows the motion of the shadow mesh as a result of the mapping between the two meshes. This behavior is achieved by defining a correspondence schema. For example, in a glTF file, the correspondence schema may be defined according to the syntax in Table 1.

[0029]

[0030] “MPEG mesh linking” extension is dependent on circular buffers (“MPEG buffer circular”) and timed accessors (“MPEG accessor timed”) to produce timed animations that are dependent on the shadow mesh. The parameters defined in Table 1 have to be transmitted to the decoder. The parameter “Correspondence” provides a reference to the accessor that describes the buffer where the correspondence between a dependent mesh and a shadow mesh is established. The parameter “Mesh” provides a reference to a mesh in the list of meshes that are referred to as “shadow mesh” and to which the correspondences are established.Docket No. 2025P00183WO

[0031] The parameter “Pose” provides a reference to the accessor that describes the buffer, that contains the transformation matrices of all the nodes of the dependent mesh to be re-animated. The parameter “Weights” provides a reference to the accessor that describes the buffer where the morph targets weights of the shadow mesh associated with the dependent mesh is available. Animating avatar representations and streaming the animation parameters over a network, either for re-enactment or for driving other avatar representations that are supported on other platforms is not straightforward. Current solutions, for example in MPEG, lack the flexibility for such modality and only focuses on a very niche use case of dynamic mesh sequences.

[0032] Avatar representation format (ARF) arrived with rich properties, able to provide realistic renderings, simplistic animations and easy parametrizations that allow users to personalize and animate their characters freely. Its format is created with the intention of providing first interoperable format capable of representing and efficiently transmitting avatar related properties. The properties may be animation parameters, e.g., UV-based animation parameters related to texture formats, skeletal joints animations, blendshapes, keypoints, and more sophisticated controller mechanisms. The increased demand for video teleconferencing solutions requires efficient solutions for streaming of a diverse range of technologies, and MPEG has yet to provide an efficient solution for the problem for the streaming of avatar animation parameters. Methods are disclosed hereinafter for generation (creation) and transmission of a bitstream for UV-based texture animations. UV mapping is the 3D modeling process of projecting a 3D model's surface to a 2D image for texture mapping. Said otherwise, UV mapping is the process of unwrapping a 3D model to create a 2D layout, allowing textures to be applied accurately. This layout, known as the UV map, is a 2D layout of a 3D model’s surface that helps apply textures to the model. The "U" and "V" represent the horizontal and vertical coordinates of the texture. It works like unfolding a 3D object (like a box) onto a flat surface so you can accurately place a 2D image onto it. The methods provide a mechanism for transmitting UV-based texture animation parameters, e.g., generated by a capturing device in a bitstream format that can be used to store, load, and stream this information to a receiver to animate a corresponding digital representation loaded by the receiver.

[0033] An avatar animation bitstream is composed of a sequence of avatar animation units (AAUs), which may also be referred to as NAL units. The general syntax structure for an AAU is shown in Table 1, where the data types used for the definition of different fields in the syntax structures are as follows: uimsbf: unsigned integer with most significant bit first, vlc8: variable length character string that contains string data stored as a character array encoded in UTF-8, boolean: aDocket No. 2025P00183WO

[0034] single bit that represents a Boolean value, Hoat32: a 32-bit floating point value represented according to the IEEE 754 specification.

[0035] Each avatar animation unit (AAU) contains a header and a payload. More precisely, the AAU comprises an AAU header containing at least a field that indicates the unit type, and a field that indicates the AAU pay load as in Table 2. The contents of the payload depend on the type of the AAU, where ByteAlignment is a padding with up to seven bits set to 0 for the AAU payload to be byte-aligned.

[0036] FIG. 1 illustrates a binary structure using keyframe and non-keyframe AAUs. A keyframe AAU represents the main structure of the bitstream format including detailed information about the frame being transmitted, e.g., quantization of the data of unit type different from configuration type (AAU CONFIG), and other properties of the payload defined in this document. A non-key frame contains the same properties of the payload as for the keyframe with the exception that the data being transmitted is coded (reconstructed respectively) relative to a previous keyframe AAU. To this end, the data of a non-keyframe is a difference between a current frame and a last keyframe, using the aau quantization as a precision parameter to encode (reconstruct respectively) the current frame.

[0037] FIG. 2 illustrates a method 20 for encoding avatar parameters according to one embodiment. At step 21, parameters of a source avatar parameters and parameters of a target avatar are obtained from at least a source. At step 22, the parameters of the source avatar are mapped to the parameters of the target avatar. At step 23, the mapped parameters and the source avatar parameters are encoded in a data stream.

[0038] FIG. 3 illustrates a method 30 for decoding avatar parameters according to one embodiment. At step 31, parameters of a source avatar and mapped parameters are decoded from a data stream, e.g., the one generated by the encoding method of FIG. 2. At step 32, the mapping (e.g., mapped parameters) obtained in a previous packet are used to compute the target avatar parameters (e.g., weights) given the source avatar parameters (e.g., weights). At step 33, a target avatar mesh is generated and may be rendered.

[0039] The general syntax structure for an AAU is shown the following tables. Each animation unit contains a header and a payload. An AAU header contains at least a field that indicates the unit type. The contents of the pay load depend on the type of the unit.Docket No. 2025P00183WO

[0040] In one embodiment, the framing of the units may be done by signaling the unit (or the unit’s pay load) length / size in bytes in the AAU header. In another embodiment, the framing of the AAU is done externally through a separate framing mechanism.

[0041]

[0042] Table 2 - Syntax of avatar_animation_unit()

[0043] ByteAlignment is a padding with up to seven bits set to 0 for the AAU payload to be byte-aligned.

[0044] The syntax structure of the AAU header is as detailed in the following table:

[0045]

[0046] Table 3 - Syntax of aau headerQ

[0047] The aau headerQ syntax construct contains the following syntax elements:

[0048] • aau_unit_type: indicates the type of the AAU. The possible values are described in the following table 4.

[0049] • aau_unit_length: indicates the size of the AAU payload in bytes.

[0050] • aau_keyframe: indicates whether the AAU payload is a keyframe.

[0051] • aau quantization: defines a quantization scale for the carried data. The quantization processing model (a.k.a. encoding method) is described with reference to FIG. 7.

[0052]

[0053] Docket No. 2025P00183WO

[0054]

[0055] Table 4 - Avatar Animation Unit type codes and corresponding payloads The aau_payload() is defined as shown in the following table 5. As can be seen from the definition, depending on the value of auh type in the aau_header() syntax construct, the payload content will be different.

[0056]

[0057] Table 5 - Syntax of aau_payload()

[0058] The aau_payload() syntax construct contains the following syntax element aau timestamp is the timestamp of the AAU in ticks. The timestamp in seconds can be calculated as timestamp / timescale, where timescale is signalled in the configuration AAU.

[0059] A target avatar configuration AAU is an AAU whose auh_type field is set to AAU_TARGET. The payload of such AAU is defined as shown in Table 6.

[0060]

[0061] Docket No. 2025P00183WO

[0062]

[0063] Table 6 - Syntax of aau target unitQ

[0064] The aau_config_unit() syntax construct contains the following syntax elements:

[0065] • acu avatar id: is an integer identifying the avatar to animate. In some embodiments, this field may not be present. In some embodiments, this field may be replaced by two fields encoding a string, the first one being the length of the string and the second one the characters of the string.

[0066] • acu lod id: is an integer identifying the Level Of Detail (LOD) of the avatar to animate.

[0067] In some embodiments, this field may not be present.

[0068] • acu_skeleton_id: is an integer identifying the skeleton of the avatar to animate. In some embodiments, this field may be named "acujointset index". In some embodiments, this field may not be present.

[0069] • acu_controller_set_id: is an integer identifying the controller set of the avatar to animate.

[0070] In some embodiments, this field may not be present.

[0071] • acu keypoint set id: is an integer identifying the keypoint set of the avatar to animate. In some embodiments, this field may not be present.

[0072] In another embodiment, the target avatar information may alternatively be signalled as part of the information in a configuration AAU similar to the one defined in the following subsection.

[0073] A configuration AAU is an AAU whose auh_type field is set to AAU_CONFIG. The payload of such AAU is defined as shown in Table 7.

[0074]

[0075] Docket No. 2025P00183WO

[0076]

[0077] Table 7 - Syntax of aau_config^unit()

[0078] The aau_config_unit() syntax construct contains the following syntax elements:Docket No. 2025P00183WO

[0079] • acu_profile_length: is the number of characters in the profile string signalled by acu_animation_profile.

[0080] • acu_animation_profile: is a character string with the name of the profile that generated stream conforms to.

[0081] • acu_timescale: is the number of ticks per second.

[0082] • acu_control_precision_minusl : plus 1 specifies the size in bytes of the target avatar index in control AAUs. The value of this field shall be greater than 0 and smaller than or equal to 3.

[0083] • acu_controller_correspondence_flag: is a flag indicating whether controller correspondence information is signalled in this configuration AAU. Value 0 indicates that no correspondence information is present. Value 1 indicates that controller correspondence information is present in the configuration AAU.

[0084] • acu_ controller Joint_index[i]: is the index of the target controller corresponding to the source avatar’s i-th controller.

[0085] • acu_ morph target _correspondence_flag: is a flag indicating whether morph target correspondence information is signalled in this configuration AAU. Value 0 indicates that no correspondence information is present. Value 1 indicates that morph target correspondence information is present in the configuration AAU.

[0086] • acu target morph target_index[i] : is the index of the target morph target corresponding to the source avatar’s i-th morph target.

[0087] • acujoint_correspondence_flag: is a flag indicating whether joint correspondence information is signalled in this configuration AAU. Value 0 indicates that no correspondence information is present. Value 1 indicates that joint correspondence information is present in the configuration AAU.

[0088] • acu_targetjoint_index[i]: is the index of the target avatar skeleton joint corresponding to the source avatar’s i-th joint.

[0089] • acu keypoint correspondence flag: is a flag indicating whether keypoint correspondence information is signalled in this configuration AAU. Value 0 indicates that no correspondence information is present. Value 1 indicates that keypoint correspondence information is present in the configuration AAU.

[0090] • acu_target_keypoint_index[i] : is the index of the target keypoint corresponding to the source avatar’s i-th keypoint.acu_keypoint_correspondence_flag: is a flag indicating whether keypoint correspondence information is signalled in this configuration AAU.Docket No. 2025P00183WO

[0091] Value 0 indicates that no correspondence information is present. Value 1 indicates that keypoint correspondence information is present in the configuration AAU.

[0092] • acu_target_keypoint_index[i] : is the index of the target keypoint corresponding to the source avatar’s i-th key point.

[0093] • acu_keypoint2d_correspondence_flag: is a flag indicating whether 2D keypoint correspondence information is signalled in this configuration AAU. Value 0 indicates that no correspondence information is present. Value 1 indicates that 2D keypoint correspondence information is present in the configuration AAU.

[0094] • acu_target_keypoint2d_index[i]: is the index of the target 2D keypoint corresponding to the source avatar’s i-th 2D keypoint.

[0095] • acu landmarks correspondence flag: is a flag indicating whether landmark correspondence information is signalled in this configuration AAU. Value 0 indicates that no correspondence information is present. Value 1 indicates that landmarks correspondence information is present in the configuration AAU.

[0096] • acu landmarkset id: is an integer identifying which collection of landmarks is used from the avatar.

[0097] • acu_target_landmark_index[i]: is the index of the target avatar skeleton joint corresponding to the source avatar’s i-th joint.

[0098] • acu_keypoint2dset _id: is an integer identifying which collection of 2D keypoints is used from the avatar.

[0099] • acu_keypointset_id: is an integer identifying which collection of keypoints is used from the avatar.

[0100] • acujointset_id: is an integer identifying which collection of joints is used from the avatar.

[0101] • acu_morphset _id: is an integer identifying which collection of morphtargets is used from the avatar.

[0102] • acu_controllerset_id: is an integer identifying which collection of animation controllers is used from the avatar.

[0103] • acu_reserved_correspondence_flags: are reserved flags for the correspondences of future avatar components.

[0104] • acu_reserved_unspecified_flags: are flags for unspecified features.

[0105] In another embodiment, the configuration AAU may explicitly signal the index of the source component (controller, morph target, joint, keypoint or keypoint 2D) in addition to the index of the target component (same). The corresponding syntax for this is shown inTable 8, where the syntax element acu_source_<component>_index[i] indicates the i-th index of a sourceDocket No. 2025P00183WO

[0106] component signalled in the control AAU and acu_target_<component>_index[i] indicates the corresponding remove avatar component index.

[0107]

[0108] Docket No. 2025P00183WO

[0109]

[0110] Table 8 - Syntax of aau_config_unit() with explicit signalling of source indices In another embodiment, in addition to the information signaled in the previous two embodiments, the configuration AAU may include information on the target avatar instead of having a separate AAU type for that information. An example configuration AAU based on this embodiment is as in Table 9.

[0111]

[0112] Docket No. 2025P00183WO

[0113]

[0114] A control AAU is an AAU whose auh_type field is set to AAU_CONTROL. The payload of such AAU is defined as shown in Table 10.

[0115]

[0116] Table 10 - Syntax of aau_control_unit()

[0117] The aau control unitQ syntax construct contains the following syntax elements:Docket No. 2025P00183WO

[0118] • atu controller count minusl: plus 1 indicates the number of controller weights present in the control AAU.

[0119] • atu_controller_index[i]: is the index of the i-th source controller whose weight is signalled in the control AAU.

[0120] • atu_control_weight[i] : is the weight to be applied for the target controller corresponding to the i-th source controller in the control AAU.

[0121] A morph target AAU is an AAU whose auh type field is set to AAU MORPH. The payload of such AAU as shown in Table 11.

[0122]

[0123] Table 11 - Syntax of aau_morph_unit()

[0124] The aau morph unitQ syntax construct contains the following syntax elements:

[0125] • amu morph target count minusl: plus 1 indicates the number of morph targets whose weights are signalled in this morph target AAU.

[0126] • amu_morph_target_index[i] is the index of the i-th morph target whose weight is signalled in the control AAU.

[0127] • amu_morph_target_weight[i] is the weight of the i-th morph target whose weight is signalled in the control AAU.

[0128] A joint transform AAU is an AAU whose auh_type field is set to AAU_JOINT. The payload of such AAU is defined as shown in Table 12.

[0129]

[0130] Docket No. 2025P00183WO

[0131] Table 12 - Syntax of aau_joint_unit()

[0132] The aaujoint_unit() syntax construct contains the following syntax elements:

[0133] • ajujoint count minusl: plus 1 indicate the number of joint transformations signalled in the joint transform AAU.

[0134] • aju_sourcejoint_index[i]: indicates the source joint index for the i-thjoint signalled in the joint transform AAU.

[0135] • aju_targetjoint_index[i]: indicates the target joint index for the i-thjoint signalled in the joint transform AAU.

[0136] • ajujoint_transform[i]: is the transformation matrix for the target joint corresponding to the source joint whose index is signalled by the field aju_sourcejoint_index[i].

[0137] In some embodiments, the ajujoint_transform[i] field may be replaced by an encoding of optional translation, rotation and scale, as presented in Table 13.

[0138]

[0139] Table 13 - Syntax of aau_joint_unit() with translation, rotation, and scale

[0140] • ajujoint_translation_flag[i]: is a flag indicating whether a translation is signalled for the i-thjoint.Docket No. 2025P00183WO

[0141] • ajujoint_translation[i]: is the translation for the target joint corresponding to the source joint whose index is signalled by the field aju_sourcejoint_index[i].

[0142] • ajujoint_rotation_flag[i]: is a flag indicating whether a rotation is signalled for the i-th joint.

[0143] • ajujoint_ rotation [i] : is the rotation for the target joint corresponding to the source joint whose index is signalled by the field aju_sourcejoint_index[i].

[0144] • ajujoint_quatemion_flag[i]: is a flag indicating whether a quaternion type of rotation is signalled for the i-th joint.

[0145] • aj u J oint quatemion [i] : is the quaternion type of rotation for the target j oint corresponding to the source joint whose index is signalled by the field aju_sourcejoint_index[i],

[0146] • ajujoint_scale_flag[i]: is a flag indicating whether a scale is signalled for the i-th joint.

[0147] • ajujoint_scale[i]: is the scale for the target joint corresponding to the source joint whose index is signalled by the field aju_sourcejoint_index[i].

[0148] A keypoint AAU is an AAU whose auh_type field is set to AAU_KEYPOINT. The payload of such AAU is defined as shown in Table 14.

[0149]

[0150] Table 14 - Syntax of aau keypoint unitQ

[0151] The aau keypoint unitQ syntax construct contains the following syntax elements:

[0152] • aku keypoint count minusl: plus 1 indicates the number of keypoint transformations signalled in the kypoint transform AAU.

[0153] • aku_source_keypoint_index[i]: indicates the source keypoint index for the i-th keypoint signalled in the keypoint transform AAU.

[0154] • aku_keypoint_transform[i] : is the transformation matrix for the target keypoint corresponding to the source keypoint whose index is signalled by the field aku_source_keypoint_index[i] .Docket No. 2025P00183WO

[0155] In some embodiments, the aku_keypoint_transform[i] field may be replaced by an encoding of optional translation, rotation, and scale, as presented in Table 15.

[0156]

[0157] Table 15 - Syntax of aau_keypoint_unit() with translation, rotation and scale

[0158] • aku_keypoint_translation_flag[i]: is a flag indicating whether a translation is signalled for the i-th keypoint.

[0159] • aku_keypoint_translation[i] : is the translation for the target keypoint corresponding to the source keypoint whose index is signalled by the field aku_source_keypoint_index[i], • aku_keypoint_rotation_flag[i] : is a flag indicating whether a rotation is signalled for the i- th keypoint.

[0160] • aku_keypoint_ rotation [i]: is the rotation for the target keypoint corresponding to the source keypoint whose index is signalled by the field aku_source_keypoint_index[i], • aku_keypoint_scale_flag[i] : is a flag indicating whether a scale is signalled for the i-th keypoint.

[0161] • aku_keypoint_scale[i]: is the scale for the target keypoint corresponding to the source keypoint whose index is signalled by the field aku_source_keypoint_index[i],

[0162] A 2D keypoint AAU is an AAU whose auh_type field is set to AAU_KEYPOINT2D. The payload of such AAU is defined as shown in Table 16.

[0163]

[0164] Docket No. 2025P00183WO

[0165]

[0166] Table 16 - Syntax of aau_keypoint2d_unit()

[0167] The aau_keypoint2d_unit() syntax construct contains the following syntax elements:

[0168] • ak2u_keypoint2d_count_minusl: plus 1 indicate the number of 2D key point transformations signalled in the joint transform AAU.

[0169] • ak2u_source_keypoint2d_index[i] : indicates the source 2D keypoint index for the i-th 2D keypoint signalled in the 2D keypoint transform AAU.

[0170] • ak2u_keypoint2d_transform[i]: is the transformation matrix for the target 2D keypoint corresponding to the source 2D keypoint whose index is signalled by the field ak2u_source_keypoint2d_index[i] .

[0171] In some embodiments, the ak2u_keypoint2d_transform[i] field may be replaced by an encoding of optional translation, rotation, and scale, as presented in Table 17.

[0172]

[0173] Table 17 - Syntax of aau_keypoint2d_unit() with translation, rotation and scale • ak2u_keypoint2d_translation_flag[i] : is a flag indicating whether a translation is signalled for the i-th 2D keypoint.Docket No. 2025P00183WO

[0174] • ak2u_keypoint2d_translation[i] : is the translation for the target 2D keypoint corresponding to the source 2D keypoint whose index is signalled by the field ak2u_source_keypoint2d_index[i] .

[0175] • ak2u_keypoint2d_rotation_flag[i]: is a flag indicating whether a 2D rotation is signalled for the i-th 2D keypoint.

[0176] • ak2u_keypoint2d_rotation [i]: is the rotation for the target 2D keypoint corresponding to the source 2D keypoint whose index is signalled by the field ak2u_source_keypoint2d_index[i] .

[0177] • ak2u_keypoint2d_scale_flag[i] : is a flag indicating whether a scale is signalled for the i-th 2D keypoint.

[0178] • ak2u_keypoint2d_scale[i]: is the scale for the target 2D keypoint corresponding to the source 2D keypoint whose index is signalled by the field ak2u_source_keypoint2d_index[i] .

[0179] A landmark AAU is an AAU whose aau_unit_type field is set to AAU_LANDMARK. The payload of such AAU is defined as shown in Table 18.

[0180]

[0181] Table 18 - Syntax of aau landmark unitQ with translation, rotation, and scale The aau landmark unitQ syntax construct contains the following syntax elements:

[0182] • alu landmark count minusl: plus 1 indicate the number of landmark transformations signalled in the landmark animation AAU.

[0183] • alu_source_landmark_index[i]: indicates the source landmark index for the i-th landmark signalled in the landmark animation AAU.

[0184] • alu_target_landmark_index[i]: indicates the target landmark index for the i-th landmark signalled in the landmark animation AAU.Docket No. 2025P00183WO

[0185] • alu_landmark_transform[i] : is the transformation matrix for the target landmark corresponding to the source landmark whose index is signalled by the field aju_source_ landmark_index[i] .

[0186] Alternatively, the alu_landmark_transform[i] field may be replaced by an encoding of optional translation, rotation, scale, velocity, inertia and uncertainty as presented in Table 19.

[0187]

[0188] Table 19 - Alternative syntax of aau_landmark_unit()

[0189] • alu_landmark_translation_flag[i] : is a flag indicating whether a translation is signalled for the i-th landmark.

[0190] • alu_landmark_translation[i] : is the translation for the target landmark corresponding to the source landmark whose index is signalled by the field alu_source_landmark_index[i]. • alu_landmark_position_flag[i]: is a flag indicating whether a position is signalled for the i-th landmark.

[0191] • alu_landmark_position [i]: is the absolute position for the target landmark corresponding to the absolute position of the source landmark whose index is signalled by the field alu_source_landmark_index[i].

[0192] • alu_uncertainty_flag[i]: is a flag indicating whether an uncertainty is signalled for the i-th landmark.Docket No. 2025P00183WO

[0193] • alu landmark uncertainty [i]: is the uncertainty of the prediction for the animation parameter for the target landmark corresponding to the source landmark whose index is signalled by the field alu_source_landmark_index[i].

[0194] A control AAU is an AAU whose aau_unit_type field is set to AAU_SILENT. The payload of such AAU is defined in Table 20.

[0195]

[0196] The asu_silent_unit() syntax construct contains the following syntax elements:

[0197] • asu timescale: number of ticks per second. The aau timestamp in seconds becomes aau_timestamp / asu_timescale.

[0198] • asu duration: The duration of the silent packet. The asu duration is in the asu timescale, i.e., asu_duration / asu_timescale indicates the duration in seconds.

[0199] FIG. 4 illustrates the composition of the bitstream for key frames and non-key frames streams when using silent unit types. The header provides the context of the payload and if the payload is for a “silent unit” then the payload only contains the “timestamp”, “timescale” and “duration”. These units in the payload tell the receiver that the sender (a.k.a. transmitter) is still alive but is sending no data related to the animation streams, therefore it tells the receiver to not close the connections. The behavior of the sender when sending silent unit packages is unknown and not relevant to the context of the present principles. More precisely, timescale (asu timescale) tells the rate that the message is being sent, and duration, i.e., how long it has passed since a first packet. The receiver may only acknowledge this information and understand that the sender is still alive. Consequently, the receiver will not close its connections.

[0200] FIG. 5 is a block diagram illustrating an example of a method 500, in a receiver, according to one or more embodiments of the present disclosure.

[0201] An avatar animation bitstream is received (502) from a transmitter. The bitstream comprises at least one avatar animation unit (AAU).

[0202] A header of the at least one avatar animation unit (AAU) is decoded (504).Docket No. 2025P00183WO

[0203] From the header, it is determined (506) that the AAU is a silent AAU, a silent AAU indicating to the receiver that the transmitter is not sending any animation data but is still alive.

[0204] Responsive to the determining, a first syntax element indicating a number of ticks per second and a second syntax element indicating a duration of the silent AAU are decoded (508).

[0205] A target avatar may be rendered taking into account the first and second syntax elements, i.e., taking into account the fact that the sender is alive but not sending animation data. In such a situation the transmitter is thus aware not to close its connections with the transmitter. The transmitter may continue rendering the target avatar, e.g., the state of the avatar may return gradually to a neutral / initial state, so it does not freeze and thus look wrong.

[0206] FIG. 6 is a block diagram illustrating an example of a method (600), in a transmitter, according to one or more embodiments of the present disclosure.

[0207] The transmitter determines (602) that it is not sending any animation data to a receiver.

[0208] In a header of an avatar animation unit (AAU) information indicating that the AAU is a silent AAU is encoded (604).

[0209] A first syntax element indicating a number of ticks per second and a second syntax element indicating a duration of the silent AAU are encoded (606) in a pay load of the AAU.

[0210] The transmitter is then transmitting (608) an avatar animation bitstream comprising the AAU to the receiver.

[0211] FIG. 7 is a block diagram illustrating an example of an encoding method (e.g., a processing model) for an avatar animation unit using quantization and compression. The method uses quantization to code and compress the bitstream format. The quantization flag in one example can be fixed and normative with a defined default value for all the data parameters transmitted in the payload.

[0212] In a different embodiment, the quantization flag is transmitted and unique to all data parameters in the payload, which means all data in the payload is quantified to an integer value to the same number of bits e.g., quantization of 10 bits.

[0213] In yet another embodiment, the quantization flag is transmitted per keyframe and different for every data parameter transmitted in the payload, making the bitstream size variable depending on the type of data being transmitted in the payload.Docket No. 2025P00183WO

[0214] FIG. 7 shows the encoding of an aau header and an aau_payload function (S40). At S42, the header (aau header) is checked to determine whether the aau is a keyframe aau or a non keyframe aau.

[0215] If the aau is not a keyframe, residual values in the aau_payload are computed (S44):

[0216]

[0217] Where “data” can be represented by any data floating value present in the “aau_payload” with a “aau type” equals to “AAU ANIMATION”, “AAU JOINT” or “AAU LANDMARK” e.g., “amu_animation_target_weights[i]” that represent 32 bits of type Hoat32, but not limited to. The data of the previous frame dataprevis subtracted from the current frame data, which results in applying quantization to the difference between two frames, wherein the previous frame is a keyframe.

[0218] If the aau is a keyframe, values are not modified:

[0219] data' = data

[0220] Then, in all cases, values (residual or absolute) are quantized (S46) using aau quantization defined in the aau header:

[0221]

[0222] The QS can be defined by any variable integer value e.g., QS = 5, or QS = 10 or any positive number.

[0223] The resulted dataqin one embodiment is used to define a number of bits for the data that needs to be compressed and transmitted.

[0224] Finally, a compression mechanism of integers (S48) is used, this can typically be entropy encoding, but not limited too.

[0225] FIG. 8 is a block diagram illustrating an example of a decoding method (e.g., a processing model) for an avatar animation unit reconstruction.

[0226] A decompression mechanism of integers is used (S50), consistent with the one used in the encoding stage (S48), e.g., using entropy decoding. The result is an aau_header and an aau_payload where floating values are quantized.

[0227] The floating values are reconstructed / dequantized (S52) using the aau_quantization defined in the aau header:Docket No. 2025P00183WO

[0228] data' = (1 « QS)dataq

[0229] If the aau is not a keyframe, absolute values in the aau_payload are computed (S56):

[0230]

[0231] If the aau is a keyframe, values are not modified:

[0232] data = data'

[0233] An avatar may then be rendered once AAUs are reconstructed.

[0234] FIG. 9 shows an example architecture of a device 60 which may be configured to implement methods according to embodiments of the present principles. The device is linked with other devices via their bus 61 and / or via I / O interface 66.

[0235] Device 60 comprises following elements that are linked together by a data and address bus 61 :

[0236] - a processor 62 (or CPU), which is, for example, a DSP (or Digital Signal Processor); - a ROM (or Read Only Memory) 63;

[0237] - a RAM (or Random Access Memory) 64;

[0238] - a storage interface 65;

[0239] - an I / O interface 66 for reception of data to transmit, from an application; and

[0240] - a power supply (not represented), e.g. a battery.

[0241] In accordance with an example, the power supply is external to the device. In each of mentioned memory, the word « register » used in the specification may correspond to area of small capacity (some bits) or to very large area (e.g. a whole program or large amount of received or decoded data). The ROM 63 comprises at least a program and parameters. The ROM 63 may store algorithms and instructions to perform techniques in accordance with present principles. When switched on, the CPU 62 uploads the program in the RAM and executes the corresponding instructions.

[0242] The RAM 64 comprises, in a register, the program executed by the CPU 62 and uploaded after switch-on of the device 60, input data in a register, intermediate data in different states of the method in a register, and other variables used for the execution of the method in a register. The implementations described herein may be implemented in, for example, a method or a process, an apparatus, a computer program product, a data stream, or a signal. Even if only discussed in the context of a single form of implementation (for example, discussed only as a method or a device), the implementation of features discussed may also be implemented in otherDocket No. 2025P00183WO

[0243] forms (for example a program). An apparatus may be implemented in, for example, appropriate hardware, software, and firmware. The methods may be implemented in, for example, an apparatus such as, for example, a processor, which refers to processing devices in general, including, for example, a computer, a microprocessor, an integrated circuit, or a programmable logic device. Processors also include communication devices, such as, for example, computers, cell phones, portable / personal digital assistants ("PDAs"), and other devices that facilitate communication of information between end-users.

[0244] Device 60 is linked, for example via bus 61 to a set of sensors 67 and to a set of rendering devices 68. Sensors 67 may be, for example, cameras, microphones, temperature sensors, Inertial Measurement Units, GPS, hygrometry sensors, IR or UV light sensors or wind sensors.

[0245] Rendering devices 68 may be, for example, displays, speakers, vibrators, heat, fan, etc.

[0246] In accordance with examples, the device 60 is configured to implement a method according to the present principles of managing a representation of the real environment of a user of an XR application, and belongs to a set comprising:

[0247] - a mobile device;

[0248] - a communication device;

[0249] - a game device;

[0250] - a tablet (or tablet computer);

[0251] - a laptop;

[0252] - a still picture camera;

[0253] - a video camera.

[0254] FIG. 10 shows an example of an embodiment of the syntax of a stream when the data are transmitted over a packet-based transmission protocol. FIG. 10 shows an example structure 7 of a stream encoding point clouds according to the present principle. The structure consists in a container which organizes the stream in independent elements of syntax. The structure may comprise a header part 71 which is a set of data common to every syntax element of the stream. For example, the header part comprises some of metadata about syntax elements, describing the nature and the role of each of them. The structure comprises a payload comprising an element of syntax 72 and at least one element of syntax 73 (there may be an element of syntax 7 for each type of attribute data, for instance one for the color, one for the reflectance, one for the normal vectors, etc.). Syntax element 72 comprises data representative of the geometry of the pointDocket No. 2025P00183WO

[0255] cloud, that is, for example, a series of bits representative of the 3D blocks, for example represented as a trisoup.

[0256] The implementations described herein may be implemented in, for example, a method or a process, an apparatus, a computer program product, a data stream, or a signal. Even if only discussed in the context of a single form of implementation (for example, discussed only as a method or a device), the implementation of features discussed may also be implemented in other forms (for example a program). An apparatus may be implemented in, for example, appropriate hardware, software, and firmware. The methods may be implemented in, for example, an apparatus such as, for example, a processor, which refers to processing devices in general, including, for example, a computer, a microprocessor, an integrated circuit, or a programmable logic device. Processors also include communication devices, such as, for example, Smartphones, tablets, computers, mobile phones, portable / personal digital assistants ("PDAs"), and other devices that facilitate communication of information between end-users.

[0257] The implementations described herein may be implemented in, for example, a method or a process, an apparatus, a computer program product, a data stream, or a signal. Even if only discussed in the context of a single form of implementation (for example, discussed only as a method or a device), the implementation of features discussed may also be implemented in other forms (for example a program). An apparatus may be implemented in, for example, appropriate hardware, software, and firmware. The methods may be implemented in, for example, an apparatus such as, for example, a processor, which refers to processing devices in general, including, for example, a computer, a microprocessor, an integrated circuit, or a programmable logic device. Processors also include communication devices, such as, for example, Smartphones, tablets, computers, mobile phones, portable / personal digital assistants ("PDAs"), and other devices that facilitate communication of information between end-users.

[0258] Implementations of the various processes and features described herein may be embodied in a variety of different equipment or applications, particularly, for example, equipment or applications associated with data encoding, data decoding, view generation, texture processing, and other processing of images and related texture information and / or depth information.

[0259] Examples of such equipment include an encoder, a decoder, a post-processor processing output from a decoder, a pre-processor providing input to an encoder, a video coder, a video decoder, a video codec, a web server, a set-top box, a laptop, a personal computer, a cell phone, a PDA, andDocket No. 2025P00183WO

[0260] other communication devices. As should be clear, the equipment may be mobile and even installed in a mobile vehicle.

[0261] Additionally, the methods may be implemented by instructions being performed by a processor, and such instructions (and / or data values produced by an implementation) may be stored on a processor-readable medium such as, for example, an integrated circuit, a software carrier or other storage device such as, for example, a hard disk, a compact diskette (“CD”), an optical disc (such as, for example, a DVD, often referred to as a digital versatile disc or a digital video disc), a random access memory (“RAM”), or a read-only memory (“ROM”). The instructions may form an application program tangibly embodied on a processor-readable medium. Instructions may be, for example, in hardware, firmware, software, or a combination. Instructions may be found in, for example, an operating system, a separate application, or a combination of the two. A processor may be characterized, therefore, as, for example, both a device configured to carry out a process and a device that includes a processor-readable medium (such as a storage device) having instructions for carrying out a process. Further, a processor-readable medium may store, in addition to or in lieu of instructions, data values produced by an implementation.

[0262] As will be evident to one of skill in the art, implementations may produce a variety of signals formatted to carry information that may be, for example, stored or transmitted. The information may include, for example, instructions for performing a method, or data produced by one of the described implementations. For example, a signal may be formatted to carry as data the rules for writing or reading the syntax of a described embodiment, or to carry as data the actual syntaxvalues written by a described embodiment. Such a signal may be formatted, for example, as an electromagnetic wave (for example, using a radio frequency portion of spectrum) or as a baseband signal. The formatting may include, for example, encoding a data stream and modulating a carrier with the encoded data stream. The information that the signal carries may be, for example, analog or digital information. The signal may be transmitted over a variety of different wired or wireless links, as is known. The signal may be stored on a processor-readable medium.

[0263] A number of implementations have been described. Nevertheless, it will be understood that various modifications may be made. For example, elements of different implementations may be combined, supplemented, modified, or removed to produce other implementations. Additionally, one of ordinary skill will understand that other structures and processes may be substituted for those disclosed and the resulting implementations will perform at least substantially the same function(s), in at least substantially the same way(s), to achieve at least substantially the sameDocket No. 2025P00183WO

[0264] result(s) as the implementations disclosed. Accordingly, these and other implementations are contemplated by this application.

Claims

Docket No. 2025P00183WOCLAIMSWhat is claimed is:

1. A method, in a receiver, comprising:receiving, from a transmitter, an avatar animation bitstream comprising at least one avatar animation unit (AAU) ;decoding a header of the at least one avatar animation unit (AAU);determining from the header that the AAU is a silent AAU, a silent AAU indicating to the receiver that the transmitter is still alive and not sending any animation data ;responsive to the determining, decoding a first syntax element indicating a number of ticks per second and a second syntax element indicating a duration of the silent AAU; andrendering a target avatar taking into account the first and second syntax elements.

2. The method of claim 1, wherein rendering the target avatar taking into account the first and second syntax elements comprises rendering the target avatar so that the target avatar returns gradually to a neutral state.

3. A method, in a transmitter, comprising:determining that the transmitter is still alive and not sending any animation data to a receiver ; encoding in a header of an avatar animation unit (AAU) information indicating that the AAU is a silent AAU;encoding a first syntax element indicating a number of ticks per second and a second syntax element indicating a duration of the silent AAU in a payload of the AAU; andtransmitting an avatar animation bitstream comprising the AAU to the receiver.

4. A receiver comprising one or more processors and at least one memory coupled to said one or more processors, wherein said one or more processors are configured to perform: receiving, from a transmitter, an avatar animation bitstream comprising at least one avatar animation unit (AAU) ;decoding a header of the at least one avatar animation unit (AAU);determining from the header that the AAU is a silent AAU, a silent AAU indicating to the receiver that the transmitter is still alive and not sending any animation data ;Docket No. 2025P00183WOresponsive to the determining, decoding a first syntax element indicating a number of ticks per second and a second syntax element indicating a duration of the silent AAU; andrendering a target avatar taking into account the first and second syntax elements.

5. The receiver of claim 4, wherein rendering the target avatar taking into account the first and second syntax elements comprises rendering the target avatar so that the target avatar returns gradually to a neutral state.

6. A transmitter comprising one or more processors and at least one memory coupled to said one or more processors, wherein said one or more processors are configured to perform: determining that the transmitter is still alive and not sending any animation data to a receiver ; encoding, in an avatar animation bitstream, in a header of an avatar animation unit (AAU) indicating that the AAU is a silent AAU;encoding a first syntax element indicating a number of ticks per second and a second syntax element indicating a duration of the silent AAU in a payload of the AAU; andtransmitting an avatar animation bitstream comprising the AAU to the receiver.

7. A computer program comprising program code instructions for implementing the method according to any one of claims 1-3 when executed by a processor.

8. A computer readable storage medium having stored thereon instructions for implementing the method of any one of claims 1-3.

9. A signal comprising a header of an avatar animation unit (AAU) comprising information indicating that the AAU is a silent AAU and a payload comprising a first syntax element indicating a number of ticks per second and a second syntax element indicating a duration of the silent AAU, wherein the silent AAU indicates to a receiver of the signal that a transmitter of the signal is still alive and not sending any animation data.

Images