Animation bitstream format and compression for avatar representation format

Abstract

An encoding method is disclosed. An avatar animation unit (AAU) is obtained that comprises a header and a payload. From the header, it is determined whether the AAU is a keyframe AAU or a non-keyframe AAU. The AAU is finally encoded responsive to the determining.

Description

[0001] Docket No. 2025P00015WQ

[0002] ANIMATION BITSTREAM FORMAT AND COMPRESSION FOR AVATAR REPRESENTATION FORMAT

[0003] CROSS REFERENCE TO RELATED APPLICATIONS

[0004] This application claims the benefit of European Application No 25305040.5, filed on January 14, 2025 which is incorporated herein by reference in its entirety.

[0005] BACKGROUND

[0006] The present application is related to the encoding and decoding of avatar animation data.

[0007] BRIEF SUMMARY

[0008] Briefly stated, in one embodiment, an encoding method is disclosed. An avatar animation unit (AAU) is obtained that comprises a header and a payload. From the header, it is determined whether the AAU is a keyframe AAU or a non-keyframe AAU. The AAU is finally encoded responsive to the determining. An encoding apparatus is also disclosed.

[0009] A decoding method is also disclosed. A header and a payload of an avatar animation unit (AAU) are decoded. It is determined from the header whether the AAU is a keyframe AAU or a non-keyframe AAU. The AAU is finally reconstructed responsive to the determining. A decoding apparatus is also disclosed.

[0010] BRIEF DESCRIPTION OF THE DRAWINGS

[0011] The following detailed description will be better understood when read in conjunction with the appended drawings, in which there are shown examples of one or more of the multiple embodiments of the present disclosure. It should be understood, however, that the embodiments described herein are not limited to the precise arrangements and instrumentalities shown in the drawings. In the drawings:

[0012] FIG. 1 illustrates a binary structure using keyframe and non-keyframe AAUs according to one or more embodiments of the present disclosure;

[0013] FIG.2 is a block diagram illustrating an example of a method for decoding avatar parameters according to one or more embodiments of the present disclosure;

[0014] FIG. 3 is a block diagram illustrating an example of a method for encoding avatar parameters according to one or more embodiments of the present disclosure;Docket No. 2025P00015WQ

[0015] FIG. 4 is a block diagram illustrating an example of an encoding method of an avatar animation unit using quantization and compression according to one or more embodiments of the present disclosure;

[0016] FIG. 5 is a block diagram illustrating an example of a decoding method of an avatar animation unit using quantization and compression according to one or more embodiments of the present disclosure;

[0017] FIG. 6 shows an example architecture of a device which may be configured to implement methods according to one or more embodiments of the present disclosure; and

[0018] FIG. 7 shows an example of an embodiment of the syntax of a stream when the data are transmitted over a packet-based transmission protocol according to one or more embodiments of the present disclosure.

[0019] DETAILED DESCRIPTION

[0020] In describing the various embodiments of the present disclosure, certain terminology is used herein for convenience only and should not be considered as limiting such embodiments. In the drawings, the same reference numerals are employed for designating the same elements throughout the several figures and the present description.

[0021] The current effort in MPEG for the streaming of avatar animation is limited to animation of volumetric meshes. Volumetric meshes are animated by pre-computing correspondences to a template (shadow mesh) for every single frame of the volumetric sequence. A shadow mesh is a mesh that contains a lower fidelity than the volumetric mesh. The correspondences between the two are performed in such a way that each vertex of the volumetric mesh maps to a polygon of the shadow mesh. The objective of the shadow mesh is to provide additional semantical information to the volumetric mesh, e.g., skeletal structure and lower polygon mesh representation, permitting the transferring of animations of the shadow mesh onto the dependent volumetric mesh.

[0022] The correspondences between the volumetric mesh and the shadow mesh are performed in such a way that each vertex of the volumetric mesh maps to a polygon of the shadow mesh. At the application side, the volumetric mesh follows the motion of the shadow mesh as a result of the mapping between the two meshes. This behavior is achieved by defining a correspondence schema. For example, in a glTF file, the correspondence schema may be defined according to the syntax in Table 1.Docket No. 2025P00015WQ

[0023]

[0024] “MPEG mesh linking” extension is dependent on circular buffers (“MPEG buffer circular”) and timed accessors (“MPEG accessor timed”) to produce timed animations that are dependent on the shadow mesh. The parameters defined in Table 1 have to be transmitted to the decoder. The parameter “Correspondence” provides a reference to the accessor that describes the buffer where the correspondence between a dependent mesh and a shadow mesh is established. The parameter “Mesh” provides a reference to a mesh in the list of meshes that are referred to as “shadow mesh” and to which the correspondences are established. The parameter “Pose” provides a reference to the accessor that describes the buffer, that contains the transformation matrices of all the nodes of the dependent mesh to be re-animated. The parameter “Weights” provides a reference to the accessor that describes the buffer where the morph targets weights of the shadow mesh associated with the dependent mesh is available. Animating avatar representations and streaming the animation parameters over a network, either for re-enactment or for driving other avatar representations that are supported on other platforms is not straightforward. Current solutions, for example in MPEG, lack the flexibility for such modality and only focuses on a very niche use case of dynamic mesh sequences.

[0025] Avatar representation arrived with rich properties, able to provide realistic renderings, simplistic animations and easy parametrizations that allow users to personalize and animate their characters freely. The properties considered hereinafter are animation parameters, such as skeletal joints animations, blendshapes, keypoints, and more sophisticated controller mechanisms. The increased demand for video teleconferencing solutions requires efficient solutions for streaming of a diverse range of technologies, and MPEG has yet to provide an efficient solution for the problem for the streaming of avatar animation parameters.

[0026] According to the present principles, a solution is provided to map two different avatar representations and to drive the animation of a target avatar over the network based on theDocket No. 2025P00015WQ

[0027] animation of a source avatar using a streaming protocol. More precisely, a mechanism for structuring avatar animation information generated by a capturing device in a bitstream format that can be used to store, load, and stream this information to a receiver to animate a corresponding digital representation loaded by the receiver. In another example, a solution is provided to stream correspondence between avatar properties, animation data related to an avatar representation format and quantize and compress the data, this solution allows mapping different avatars representations including their shape, semantics, articulation and animation parameters (e.g., morph targets, blendshapes, landmarks).

[0028] An avatar animation bitstream is composed of a sequence of avatar animation units (AAUs), which may also be referred to as NAL units. The general syntax structure for an AAU is shown in Table 1, where the data types used for the definition of different fields in the syntax structures are as follows: uimsbf: unsigned integer with most significant bit first, vlc8: variable length character string that contains string data stored as a character array encoded in UTF-8, boolean: a single bit that represents a Boolean value, Hoat32: a 32-bit floating point value represented according to the IEEE 754 specification.

[0029] Each avatar animation unit (AAU) contains a header and a payload. An AAU header contains at least a field that indicates the unit type and a field that indicates the AAU payload. The contents of the payload depend on the type of the AAU, where ByteAlignment is a padding with up to seven bits set to 0 for the AAU payload to be byte-aligned.

[0030] FIG. 1 illustrates a binary structure using keyframe and non-keyframe AAUs. A keyframe AAU represents the main structure of the bitstream format including detailed information about the frame being transmitted, e.g., quantization of the data of unit type different from configuration type (AAU CONFIG), and other properties of the payload defined in this document. A non-key frame contains the same properties of the payload as for the keyframe with the exception that the data being transmitted is coded (reconstructed respectively) relative to a previous keyframe AAU. To this end, the data of a non-keyframe is a difference between a current frame and a last keyframe, using the aau quantization as a precision parameter to encode (reconstruct respectively) the current frame.

[0031] FIG. 2 illustrates a method 20 for encoding avatar parameters according to an embodiment. At step 21, parameters of a source avatar parameters and parameters of a target avatar are obtained from at least a source. At step 22, the parameters of the source avatar are mapped to the parameters of the target avatar. At step 23, the mapped parameters and the source avatar parameters are encoded in a data stream.Docket No. 2025P00015WQ

[0032] FIG. 3 illustrates a method 30 for decoding avatar parameters according to an embodiment. At step 31, parameters of a source avatar and mapped parameters are decoded from a data stream, e.g., the one generated by the encoding method of FIG. 2. At step 32, the mapping (e.g., mapped parameters) obtained in a previous packet are used to compute the target avatar parameters (e.g., weights) given the source avatar parameters (e.g., weights). At step 33, a target avatar mesh is generated and may be rendered.

[0033] The general syntax structure for an AAU is shown the following tables. Each animation unit contains a header and a payload. An AAU header contains at least a field that indicates the unit type. The contents of the pay load depend on the type of the unit.

[0034] In one embodiment, the framing of the units may be done by signaling the unit (or the unit’s pay load) length / size in bytes in the AAU header. In another embodiment, the framing of the AAU is done externally through a separate framing mechanism.

[0035]

[0036] Table 2 - Syntax of avatar animation unitQ

[0037] The syntax structure of the AAU header is as detailed in the following table:

[0038]

[0039] Table 3 - Syntax of aau headerQ

[0040] The aau headerQ syntax construct contains the following syntax elements:Docket No. 2025P00015WQ

[0041] • aau unit type: indicates the type of the AAU. The possible values are described in the following table 4.

[0042] • aau_unit_length: indicates the size of the AAU payload in bytes.

[0043] • aau_keyframe: indicates whether the AAU payload is a keyframe.

[0044] • aau quantization: defines a quantization scale for the carried data. The quantization processing model (a.k.a. encoding method) is described with reference to FIG. 4.

[0045]

[0046] Table 4 - Avatar Animation Unit type codes and corresponding payloads The aau_payload() is defined as shown in the following table 5. As can be seen from the definition, depending on the value of auh type in the aau_header() syntex construct, the payload content will be different.

[0047]

[0048] Docket No. 2025P00015WQ

[0049] Table 5 - Syntax of aau_payload()

[0050] The aau_payload() syntax construct contains the following syntax elements:

[0051] • aau timestamp: is the timestamp of the AAU in ticks. The timestamp in seconds can be calculated as timestamp / timescale, where timescale is signalled in the configuration AAU.

[0052] • ByteAlignment: is a padding with up to seven bits set to 0 for the AAU payload to be byte- aligned.

[0053] A target avatar configuration AAU is an AAU whose auh_type field is set to AAU_TARGET. The pay load of such AAU is defined as shown in Table 6.

[0054]

[0055] Table 6 - Syntax of aau_target_unit()

[0056] The aau_config_unit() syntax construct contains the following syntax elements:

[0057] • acu avatar id: is an integer identifying the avatar to animate. In some embodiments, this field may not be present. In some embodiments, this field may be replaced by two fields encoding a string, the first one being the length of the string and the second one the characters of the string.

[0058] • acu lod id: is an integer identifying the Level Of Detail (LOD) of the avatar to animate.

[0059] In some embodiments, this field may not be present.

[0060] • acu_skeleton_id: is an integer identifying the skeleton of the avatar to animate. In some embodiments, this field may be named "acujointset index". In some embodiments, this field may not be present.

[0061] • acu_controller_set_id: is an integer identifying the controller set of the avatar to animate.

[0062] In some embodiments, this field may not be present.

[0063] • acu keypoint set id: is an integer identifying the keypoint set of the avatar to animate. In some embodiments, this field may not be present.

[0064] In another embodiment, the target avatar information may alternatively be signalled as part of the information in a configuration AAU similar to the one defined in the following subsection.Docket No. 2025P00015WQ

[0065] A configuration AAU is an AAU whose auh_type field is set to AAU_CONFIG. The payload of such AAU is defined as shown in Table 7.

[0066] <

[0067] <

[0068] <

[0069] <

[0070] <

[0071]

[0072] Docket No. 2025P00015WQ

[0073] <

[0074]

[0075] The aau_config_unit() syntax construct contains the following syntax elements:

[0076] • acu_profile_length: is the number of characters in the profile string signalled by acu_animation_profile.

[0077] • acu_animation_profile: is a character string with the name of the profile that generated stream conforms to.

[0078] • acu_timescale: is the number of ticks per second.

[0079] • acu_control_precision_minusl : plus 1 specifies the size in bytes of the target avatar index in control AAUs. The value of this field shall be greater than 0 and smaller than or equal to 3.

[0080] • acu_controller_correspondence_flag: is a flag indicating whether controller correspondence information is signalled in this configuration AAU. Value 0 indicates that no correspondence information is present. Value 1 indicates that controller correspondence information is present in the configuration AAU.

[0081] • acu_ controller Joint_index[i]: is the index of the target controller corresponding to the source avatar’s i-th controller.

[0082] • acu_ morph target _correspondence_flag: is a flag indicating whether morph target correspondence information is signalled in this configuration AAU. Value 0 indicates that no correspondence information is present. Value 1 indicates that morph target correspondence information is present in the configuration AAU.

[0083] • acu target morph target_index[i] : is the index of the target morph target corresponding to the source avatar’s i-th morph target.

[0084] • acujoint_correspondence_flag: is a flag indicating whether joint correspondence information is signalled in this configuration AAU. Value 0 indicates that noDocket No. 2025P00015WQ

[0085] correspondence information is present. Value 1 indicates that joint correspondence information is present in the configuration AAU.

[0086] • acu_targetjoint_index[i]: is the index of the target avatar skeleton joint corresponding to the source avatar’s i-th joint.

[0087] • acu keypoint correspondence flag: is a flag indicating whether keypoint correspondence information is signalled in this configuration AAU. Value 0 indicates that no correspondence information is present. Value 1 indicates that keypoint correspondence information is present in the configuration AAU.

[0088] • acu_target_keypoint_index[i] : is the index of the target keypoint corresponding to the source avatar’s i-th keypoint.acu_keypoint_correspondence_flag: is a flag indicating whether keypoint correspondence information is signalled in this configuration AAU. Value 0 indicates that no correspondence information is present. Value 1 indicates that keypoint correspondence information is present in the configuration AAU.

[0089] • acu_target_keypoint_index[i] : is the index of the target keypoint corresponding to the source avatar’s i-th key point.

[0090] • acu_keypoint2d_correspondence_flag: is a flag indicating whether 2D keypoint correspondence information is signalled in this configuration AAU. Value 0 indicates that no correspondence information is present. Value 1 indicates that 2D keypoint correspondence information is present in the configuration AAU.

[0091] • acu_target_keypoint2d_index[i]: is the index of the target 2D keypoint corresponding to the source avatar’s i-th 2D keypoint.

[0092] • acu landmarks correspondence flag: is a flag indicating whether landmark correspondence information is signalled in this configuration AAU. Value 0 indicates that no correspondence information is present. Value 1 indicates that landmarks correspondence information is present in the configuration AAU.

[0093] • acu landmarkset id: is an integer identifying which collection of landmarks is used from the avatar.

[0094] • acu_target_landmark_index[i]: is the index of the target avatar skeleton joint corresponding to the source avatar’s i-th joint.

[0095] • acu_keypoint2dset _id: is an integer identifying which collection of 2D keypoints is used from the avatar.

[0096] • acu_keypointset_id: is an integer identifying which collection of keypoints is used from the avatar.

[0097] • acujointset_id: is an integer identifying which collection of joints is used from the avatar.Docket No. 2025P00015WQ

[0098] • acu morphset _id: is an integer identifying which collection of morphtargets is used from the avatar.

[0099] • acu_controllerset_id: is an integer identifying which collection of animation controllers is used from the avatar.

[0100] • acu_reserved_correspondence_flags: are reserved flags for the correspondences of future avatar components.

[0101] • acu_reserved_unspecified_flags: are flags for unspecified features.

[0102] In another embodiment, the configuration AAU may explicitly signal the index of the source component (controller, morph target, joint, key point or key point 2D) in addition to the index of the target component (same). The corresponding syntax for this is shown inTable 8, where the syntax element acu_source_<component>_index[i] indicates the i-th index of a source component signalled in the control AAU and acu_target_<component>_index[i] indicates the corresponding remove avatar component index.

[0103] <

[0104] <

[0105] <

[0106]

[0107] Docket No. 2025P00015WQ

[0108] <

[0109] <

[0110] <

[0111]

[0112] Table 8 - Syntax of aau_config_unit() with explicit signalling of source indices In another embodiment, in addition to the information signaled in the previous two embodiments, the configuration AAU may include information on the target avatar instead of having a separate AAU type for that information. An example configuration AAU based on this embodiment is as in Table 9.

[0113]

[0114] Docket No. 2025P00015WQ

[0115] <

[0116] <

[0117] <

[0118] <

[0119] <

[0120] <

[0121]

[0122] Table 9Docket No. 2025P00015WQ

[0123] A control AAU is an AAU whose auh_type field is set to AAU_CONTROL. The payload of such AAU is defined as shown in Table 10.

[0124] <

[0125]

[0126] Table 10 - Syntax of aau_control_unit()

[0127] The aau_control_unit() syntax construct contains the following syntax elements:

[0128] • atu controller count minusl: plus 1 indicates the number of controller weights present in the control AAU.

[0129] • atu_controller_index[i]: is the index of the i-th source controller whose weight is signalled in the control AAU.

[0130] • atu_control_weight[i] : is the weight to be applied for the target controller corresponding to the i-th source controller in the control AAU.

[0131] A morph target AAU is an AAU whose auh type field is set to AAU MORPH. The payload of such AAU as shown in Table 11.

[0132] <

[0133]

[0134] Table 11 - Syntax of aau_morph_unit()

[0135] The aau_morph_unit() syntax construct contains the following syntax elements:

[0136] • amu morph target count minusl: plus 1 indicates the number of morph targets whose weights are signalled in this morph target AAU.

[0137] • amu_morph_target_index[i] is the index of the i-th morph target whose weight is signalled in the control AAU.Docket No. 2025P00015WQ

[0138] • amu_morph_target_weight[i] is the weight of the i-th morph target whose weight is signalled in the control AAU.

[0139] A joint transform AAU is an AAU whose auh_type field is set to AAU_JOINT. The payload of such AAU is defined as shown in Table 12.

[0140] <

[0141]

[0142] Table 12 - Syntax of aau_joint_unit()

[0143] The aaujoint_unit() syntax construct contains the following syntax elements:

[0144] • ajujoint count minus 1: plus 1 indicate the number of joint transformations signalled in the j oint transform AAU.

[0145] • aju_sourcejoint_index[i]: indicates the source joint index for the i-thjoint signalled in the joint transform AAU.

[0146] • aju_targetjoint_index[i]: indicates the target joint index for the i-thjoint signalled in the joint transform AAU.

[0147] • ajujoint_transform[i]: is the transformation matrix for the target joint corresponding to the source joint whose index is signalled by the field aju_sourcejoint_index[i].

[0148] In some embodiments, the ajujoint_transform[i] field may be replaced by an encoding of optional translation, rotation and scale, as presented in Table 13.

[0149] <

[0150]

[0151] Docket No. 2025P00015WQ

[0152]

[0153] Table 13 - Syntax of aau_joint_unit() with translation, rotation, and scale

[0154] • ajujoint_translation_flag[i]: is a flag indicating whether a translation is signalled for the i-th joint.

[0155] • ajujoint_translation[i]: is the translation for the target joint corresponding to the source joint whose index is signalled by the field aju_sourcejoint_index[i].

[0156] • ajujoint_rotation_flag[i]: is a flag indicating whether a rotation is signalled for the i-th joint.

[0157] • ajujoint_ rotation [i] : is the rotation for the target joint corresponding to the source joint whose index is signalled by the field aju_sourcejoint_index[i].

[0158] • ajujoint_quatemion_flag[i]: is a flag indicating whether a quaternion type of rotation is signalled for the i-th joint.

[0159] • aj u J oint quatemion [i] : is the quaternion type of rotation for the target j oint corresponding to the source joint whose index is signalled by the field aju_sourcejoint_index[i],

[0160] • ajujoint_scale_flag[i]: is a flag indicating whether a scale is signalled for the i-th joint.

[0161] • ajujoint_scale[i]: is the scale for the target joint corresponding to the source joint whose index is signalled by the field aju_sourcejoint_index[i].

[0162] A keypoint AAU is an AAU whose auh_type field is set to AAU_KEYPOINT. The payload of such AAU is defined as shown in Table 14.

[0163]

[0164] < Docket No. 2025P00015WQ

[0165]

[0166] Table 14 - Syntax of aau keypoint unitQ

[0167] The aau keypoint unitQ syntax construct contains the following syntax elements:

[0168] • aku keypoint count minusl: plus 1 indicates the number of keypoint transformations signalled in the kypoint transform AAU.

[0169] • aku_source_keypoint_index[i]: indicates the source keypoint index for the i-th keypoint signalled in the keypoint transform AAU.

[0170] • aku_keypoint_transform[i] : is the transformation matrix for the target keypoint corresponding to the source keypoint whose index is signalled by the field aku_source_keypoint_index[i] .

[0171] In some embodiments, the aku_keypoint_transform[i] field may be replaced by an encoding of optional translation, rotation, and scale, as presented in Table 15.

[0172] <

[0173]

[0174] Table 15 - Syntax of aau keypoint unitQ with translation, rotation and scale

[0175] • aku_keypoint_translation_flag[i]: is a flag indicating whether a translation is signalled for the i-th keypoint.

[0176] • aku_keypoint_translation[i] : is the translation for the target keypoint corresponding to the source keypoint whose index is signalled by the field aku_source_keypoint_index[i].Docket No. 2025P00015WQ

[0177] • aku_keypoint_rotation_flag[i] : is a flag indicating whether a rotation is signalled for the i- th keypoint.

[0178] • aku_keypoint_ rotation [i]: is the rotation for the target keypoint corresponding to the source keypoint whose index is signalled by the field aku_source_keypoint_index[i], • aku_keypoint_scale_flag[i] : is a flag indicating whether a scale is signalled for the i-th keypoint.

[0179] • aku_keypoint_scale[i]: is the scale for the target keypoint corresponding to the source keypoint whose index is signalled by the field aku_source_keypoint_index[i],

[0180] A 2D keypoint AAU is an AAU whose auh_type field is set to AAU_KEYPOINT2D. The payload of such AAU is defined as shown in Table 16.

[0181] <

[0182]

[0183] Table 16 - Syntax of aau_keypoint2d_unit()

[0184] The aau_keypoint2d_unit() syntax construct contains the following syntax elements:

[0185] • ak2u_keypoint2d_count_minusl: plus 1 indicate the number of 2D key point transformations signalled in the joint transform AAU.

[0186] • ak2u_source_keypoint2d_index[i] : indicates the source 2D keypoint index for the i-th 2D keypoint signalled in the 2D keypoint transform AAU.

[0187] • ak2u_keypoint2d_transform[i]: is the transformation matrix for the target 2D keypoint corresponding to the source 2D keypoint whose index is signalled by the field ak2u_source_keypoint2d_index[i] .

[0188] In some embodiments, the ak2u_keypoint2d_transform[i] field may be replaced by an encoding of optional translation, rotation, and scale, as presented in Table 17.

[0189] <

[0190]

[0191] Docket No. 2025P00015WQ

[0192]

[0193] Table 17 - Syntax of aau_keypoint2d_unit() with translation, rotation and scale • ak2u_keypoint2d_translation_flag[i] : is a flag indicating whether a translation is signalled for the i-th 2D keypoint.

[0194] • ak2u_keypoint2d_translation[i] : is the translation for the target 2D keypoint corresponding to the source 2D keypoint whose index is signalled by the field ak2u_source_keypoint2d_index[i] .

[0195] • ak2u_keypoint2d_rotation_flag[i] : is a flag indicating whether a 2D rotation is signalled for the i-th 2D keypoint.

[0196] • ak2u_keypoint2d_rotation [i]: is the rotation for the target 2D keypoint corresponding to the source 2D keypoint whose index is signalled by the field ak2u_source_keypoint2d_index[i] .

[0197] • ak2u_keypoint2d_scale_flag[i] : is a flag indicating whether a scale is signalled for the i-th 2D keypoint.

[0198] • ak2u_keypoint2d_scale[i]: is the scale for the target 2D keypoint corresponding to the source 2D keypoint whose index is signalled by the field ak2u_source_keypoint2d_index[i] .

[0199] A landmark AAU is an AAU whose aau_unit_type field is set to AAU_LANDMARK. The payload of such AAU is defined as shown in Table 18.

[0200]

[0201] < Docket No. 2025P00015WQ

[0202]

[0203] Table 18 - Syntax of aau landmark unitQ with translation, rotation, and scale The aau landmark unitQ syntax construct contains the following syntax elements:

[0204] • alu landmark count minusl: plus 1 indicate the number of landmark transformations signalled in the landmark animation AAU.

[0205] • alu_source_landmark_index[i]: indicates the source landmark index for the i-th landmark signalled in the landmark animation AAU.

[0206] • alu_target_landmark_index[i] : indicates the target landmark index for the i-th landmark signalled in the landmark animation AAU.

[0207] • alu_landmark_transform[i] : is the transformation matrix for the target landmark corresponding to the source landmark whose index is signalled by the field aju_source_ landmark_index[i] .

[0208] Alternatively, the alu_landmark_transform[i] field may be replaced by an encoding of optional translation, rotation, scale, velocity, inertia and uncertainty as presented in Table 19.

[0209] <

[0210]

[0211] Table 19Docket No. 2025P00015WQ

[0212] • alu_landmark_translation_flag[i] : is a flag indicating whether a translation is signalled for the i-th landmark.

[0213] • alu_landmark_translation[i] : is the translation for the target landmark corresponding to the source landmark whose index is signalled by the field alu_source_landmark_index[i]. • alu_landmark_position_flag[i]: is a flag indicating whether a position is signalled for the i-th landmark.

[0214] • alu_landmark_position [i]: is the absolute position for the target landmark corresponding to the absolute position of the source landmark whose index is signalled by the field alu_source_landmark_index[i] .

[0215] • alu_uncertainty_flag[i]: is a flag indicating whether an uncertainty is signalled for the i-th landmark.

[0216] • alu landmark uncertainty [i]: is the uncertainty of the prediction for the animation parameter for the target landmark corresponding to the source landmark whose index is signalled by the field alu_source_landmark_index[i].

[0217] FIG. 4 is a block diagram illustrating an example of an encoding method (e.g., a processing model) for the avatar animation unit using quantization and compression. The method uses quantization to code and compress the bitstream format. The quantization flag in one example can be fixed and normative with a defined default value for all the data parameters transmitted in the payload.

[0218] In a different embodiment, the quantization flag is transmitted and unique to all data parameters in the payload, which means all data in the payload is quantified to an integer value to the same number of bits e.g., quantization of 10 bits.

[0219] In yet another embodiment, the quantization flag is transmitted per keyframe and different for every data parameter transmitted in the payload, making the bitstream size variable depending on the type of data being transmitted in the payload.

[0220] FIG. 4 shows the encoding of an aau header and an aau_payload function (S40). At S42, the header (aau header) is checked to determine whether the aau is a keyframe aau or a non keyframe aau.

[0221] If the aau is not a keyframe, residual values in the aau_payload are computed (S44):

[0222]

[0223] Docket No. 2025P00015WQ

[0224] Where “data” can be represented by any data floating value present in the “aau_payload” with a “aau type” equals to “AAU ANIMATION”, “AAU JOINT” or “AAU LANDMARK” e.g., “amu_animation_target_weights[i]” that represent 32 bits of type float32, but not limited to. The data of the previous frame dataprevis subtracted from the current frame data, which results in applying quantization to the difference between two frames, wherein the previous frame is a keyframe.

[0225] If the aau is a keyframe, values are not modified:

[0226] data' = data

[0227] Then, in all cases, values (residual or absolute) are quantized (S46) using aau quantization defined in the aau header:

[0228]

[0229] The QS can be defined by any variable integer value e.g., QS = 5, or QS = 10 or any positive number.

[0230] The resulted dataqin one embodiment is used to define a number of bits for the data that needs to be compressed and transmitted.

[0231] Finally, a compression mechanism of integers (S48) is used, this can typically be entropy encoding, but not limited too.

[0232] FIG. 5 is a block diagram illustrating an example of a decoding method (e.g., a processing model) for the avatar animation unit reconstruction.

[0233] A decompression mechanism of integers is used (S50), consistent with the one used in the encoding stage (S48), e.g., using entropy decoding. The result is an aau_header and an aau_payload where floating values are quantized.

[0234] The floating values are reconstructed / dequantized (S52) using the aau_quantization defined in the aau header:

[0235] data' = (1 « QS)dataq

[0236] If the aau is not a keyframe, absolute values in the aau_payload are computed (S56):

[0237]

[0238] If the aau is a keyframe, values are not modified:Docket No. 2025P00015WQ

[0239] data = data'

[0240] An avatar may then be rendered once AAUs are reconstructed.

[0241] FIG. 6 shows an example architecture of a device 60 which may be configured to implement methods according to embodiments of the present principles. The device is linked with other devices via their bus 61 and / or via I / O interface 66.

[0242] Device 60 comprises following elements that are linked together by a data and address bus 61 :

[0243] - a processor 62 (or CPU), which is, for example, a DSP (or Digital Signal Processor); - a ROM (or Read Only Memory) 63;

[0244] - a RAM (or Random Access Memory) 64;

[0245] - a storage interface 65;

[0246] - an I / O interface 66 for reception of data to transmit, from an application; and

[0247] - a power supply (not represented), e.g. a battery.

[0248] In accordance with an example, the power supply is external to the device. In each of mentioned memory, the word « register » used in the specification may correspond to area of small capacity (some bits) or to very large area (e.g. a whole program or large amount of received or decoded data). The ROM 63 comprises at least a program and parameters. The ROM 63 may store algorithms and instructions to perform techniques in accordance with present principles. When switched on, the CPU 62 uploads the program in the RAM and executes the corresponding instructions.

[0249] The RAM 64 comprises, in a register, the program executed by the CPU 62 and uploaded after switch-on of the device 60, input data in a register, intermediate data in different states of the method in a register, and other variables used for the execution of the method in a register. The implementations described herein may be implemented in, for example, a method or a process, an apparatus, a computer program product, a data stream, or a signal. Even if only discussed in the context of a single form of implementation (for example, discussed only as a method or a device), the implementation of features discussed may also be implemented in other forms (for example a program). An apparatus may be implemented in, for example, appropriate hardware, software, and firmware. The methods may be implemented in, for example, an apparatus such as, for example, a processor, which refers to processing devices in general, including, for example, a computer, a microprocessor, an integrated circuit, or a programmable logic device. Processors also include communication devices, such as, for example, computers,Docket No. 2025P00015WQ

[0250] cell phones, portable / personal digital assistants ("PDAs"), and other devices that facilitate communication of information between end-users.

[0251] Device 60 is linked, for example via bus 61 to a set of sensors 67 and to a set of rendering devices 68. Sensors 67 may be, for example, cameras, microphones, temperature sensors, Inertial Measurement Units, GPS, hygrometry sensors, IR or UV light sensors or wind sensors.

[0252] Rendering devices 68 may be, for example, displays, speakers, vibrators, heat, fan, etc.

[0253] In accordance with examples, the device 60 is configured to implement a method according to the present principles of managing a representation of the real environment of a user of an XR application, and belongs to a set comprising:

[0254] - a mobile device;

[0255] - a communication device;

[0256] - a game device;

[0257] - a tablet (or tablet computer);

[0258] - a laptop;

[0259] - a still picture camera;

[0260] - a video camera.

[0261] FIG. 7 shows an example of an embodiment of the syntax of a stream when the data are transmitted over a packet-based transmission protocol. FIG. 7 shows an example structure 7 of a stream encoding point clouds according to the present principle. The structure consists in a container which organizes the stream in independent elements of syntax. The structure may comprise a header part 71 which is a set of data common to every syntax element of the stream. For example, the header part comprises some of metadata about syntax elements, describing the nature and the role of each of them. The structure comprises a payload comprising an element of syntax 72 and at least one element of syntax 73 (there may be an element of syntax 7 for each type of attribute data, for instance one for the color, one for the reflectance, one for the normal vectors, etc.). Syntax element 72 comprises data representative of the geometry of the point cloud, that is, for example, a series of bits representative of the 3D blocks, for example represented as a trisoup.

[0262] The implementations described herein may be implemented in, for example, a method or a process, an apparatus, a computer program product, a data stream, or a signal. Even if only discussed in the context of a single form of implementation (for example, discussed only as aDocket No. 2025P00015WQ

[0263] method or a device), the implementation of features discussed may also be implemented in other forms (for example a program). An apparatus may be implemented in, for example, appropriate hardware, software, and firmware. The methods may be implemented in, for example, an apparatus such as, for example, a processor, which refers to processing devices in general, including, for example, a computer, a microprocessor, an integrated circuit, or a programmable logic device. Processors also include communication devices, such as, for example, Smartphones, tablets, computers, mobile phones, portable / personal digital assistants ("PDAs"), and other devices that facilitate communication of information between end-users.

[0264] The implementations described herein may be implemented in, for example, a method or a process, an apparatus, a computer program product, a data stream, or a signal. Even if only discussed in the context of a single form of implementation (for example, discussed only as a method or a device), the implementation of features discussed may also be implemented in other forms (for example a program). An apparatus may be implemented in, for example, appropriate hardware, software, and firmware. The methods may be implemented in, for example, an apparatus such as, for example, a processor, which refers to processing devices in general, including, for example, a computer, a microprocessor, an integrated circuit, or a programmable logic device. Processors also include communication devices, such as, for example, Smartphones, tablets, computers, mobile phones, portable / personal digital assistants ("PDAs"), and other devices that facilitate communication of information between end-users.

[0265] Implementations of the various processes and features described herein may be embodied in a variety of different equipment or applications, particularly, for example, equipment or applications associated with data encoding, data decoding, view generation, texture processing, and other processing of images and related texture information and / or depth information.

[0266] Examples of such equipment include an encoder, a decoder, a post-processor processing output from a decoder, a pre-processor providing input to an encoder, a video coder, a video decoder, a video codec, a web server, a set-top box, a laptop, a personal computer, a cell phone, a PDA, and other communication devices. As should be clear, the equipment may be mobile and even installed in a mobile vehicle.

[0267] Additionally, the methods may be implemented by instructions being performed by a processor, and such instructions (and / or data values produced by an implementation) may be stored on a processor-readable medium such as, for example, an integrated circuit, a software carrier or other storage device such as, for example, a hard disk, a compact diskette (“CD”), an optical disc (suchDocket No. 2025P00015WQ

[0268] as, for example, a DVD, often referred to as a digital versatile disc or a digital video disc), a random access memory (“RAM”), or a read-only memory (“ROM”). The instructions may form an application program tangibly embodied on a processor-readable medium. Instructions may be, for example, in hardware, firmware, software, or a combination. Instructions may be found in, for example, an operating system, a separate application, or a combination of the two. A processor may be characterized, therefore, as, for example, both a device configured to carry out a process and a device that includes a processor-readable medium (such as a storage device) having instructions for carrying out a process. Further, a processor-readable medium may store, in addition to or in lieu of instructions, data values produced by an implementation.

[0269] As will be evident to one of skill in the art, implementations may produce a variety of signals formatted to carry information that may be, for example, stored or transmitted. The information may include, for example, instructions for performing a method, or data produced by one of the described implementations. For example, a signal may be formatted to carry as data the rules for writing or reading the syntax of a described embodiment, or to carry as data the actual syntaxvalues written by a described embodiment. Such a signal may be formatted, for example, as an electromagnetic wave (for example, using a radio frequency portion of spectrum) or as a baseband signal. The formatting may include, for example, encoding a data stream and modulating a carrier with the encoded data stream. The information that the signal carries may be, for example, analog or digital information. The signal may be transmitted over a variety of different wired or wireless links, as is known. The signal may be stored on a processor-readable medium.

[0270] A number of implementations have been described. Nevertheless, it will be understood that various modifications may be made. For example, elements of different implementations may be combined, supplemented, modified, or removed to produce other implementations. Additionally, one of ordinary skill will understand that other structures and processes may be substituted for those disclosed and the resulting implementations will perform at least substantially the same function(s), in at least substantially the same way(s), to achieve at least substantially the same result(s) as the implementations disclosed. Accordingly, these and other implementations are contemplated by this application.

Claims

Docket No. 2025P00015WQCLAIMSWhat is claimed is:

1. A method comprising:obtaining an avatar animation unit (AAU) comprising a header and an AAU payload, wherein the header of the AAU comprises information indicating a type of AAU, information indicating a size of the AAU payload, information indicating whether the AAU payload is a keyframe and, in a case where the AAU is a keyframe, a quantization scale;responsive to determining from the header that the AAU is a keyframe AAU:quantizing data of payload of the keyframe AAU using the quantization scale and encoding the quantized data in a bitstream ; andresponsive to determining from the header that the AAU is a non-keyframe AAU encoded based on a previous keyframe AAU:computing residual data by subtracting data of payload of the previous keyframe AAU from data of pay load of the non-key frame AAU, quantizing the residual data using the quantization scale of the previous keyframe AAU and encoding the quantized data in the bitstream.

2. The method of claim 1, wherein the quantization scale is different for every data in the AAU payload.

3. The method of claim 1, wherein the quantization scale is unique to all data in the AAU pay load.

4. The method of any one of claims 1 to 3, wherein the information indicating a type of AAU indicates the AAU is a landmark AAU.

5. The method of claim 4, wherein data of the payload of the AAU comprises data representative of a number of landmark transformations signaled in the AAU, and, for each landmark signalled in the AAU, an index of a source landmark, an index of a target landmark and a transformation for the target landmark corresponding to the source landmark.27Docket No. 2025P00015WQ6. The method of claim 5 wherein the transformation is any one of a transformation matrix, a translation, a rotation, a scale, a velocity, an inertia and / or an uncertainty.

7. The method of any one of claims 1 to 6, wherein determining from the header whether AAU is a keyframe AAU or a non-keyframe AAU comprises determining that the AAU is a keyframe AAU in a case where a binary syntax element of the header is equal to one and a non-keyframe otherwise.

8. A method comprising :decoding an avatar animation unit (AAU) comprising a header and an AAU payload, wherein the header of the AAU comprises information indicating a type of AAU, information indicating a size of the AAU payload, information indicating whether the AAU payload is a keyframe and, in a case where the AAU is a keyframe, a quantization scale;responsive to determining from the header that the AAU is a keyframe AAU:dequantizing data of payload of the keyframe AAU using the quantization scale to obtain a reconstructed payload ; andresponsive to determining from the header that the AAU is a non-keyframe AAU encoded based on a previous keyframe AAU:dequantizing residual data using the quantization scale of the previous keyframe AAU and reconstructing payload of the non-keyframe AAU based on data of payload of the previous keyframe AAU and dequantized residual data.

9. The method of claim 8, wherein the quantization scale is different for every data in the AAU payload.

10. The method of claim 8, wherein the quantization scale is unique to all data in the AAU pay load.

11. The method of any one of claims 8 to 10, wherein the information indicating a type of AAU indicates the AAU is a landmark AAU.

12. The method of claim 11 , wherein data of the pay load of the AAU comprises data representative of a number of landmark transformations signaled in the AAU, and, for each landmark signalledDocket No. 2025P00015WQin the AAU, an index of a source landmark, an index of a target landmark and a transformation for the target landmark corresponding to the source landmark.

13. The method of claim 12, wherein the transformation is any one of a transformation matrix, a translation, a rotation, a scale, a velocity, an inertia and / or an uncertainty.

14. The method of any one of claims 8 to 13, wherein determining from the header whether AAU is a keyframe AAU or a non-keyframe AAU comprises determining that the AAU is a keyframe AAU in a case where a binary syntax element of the header is equal to one and a non-keyframe otherwise.

15. An encoding apparatus comprising one or more processors and at least one memory coupled to said one or more processors, wherein said one or more processors are configured to perform: obtaining an avatar animation unit (AAU) comprising a header and an AAU payload, wherein the header of the AAU comprises information indicating a type of AAU, information indicating a size of the AAU payload, information indicating whether the AAU payload is a keyframe and, in a case where the AAU is a keyframe, a quantization scale;responsive to determining from the header that the AAU is a keyframe AAU:quantizing data of payload of the keyframe AAU using the quantization scale and encoding the quantized data in a bitstream ; andresponsive to determining from the header that the AAU is a non-keyframe AAU encoded based on a previous keyframe AAU:computing residual data by subtracting data of payload of the previous keyframe AAU from data of pay load of the non-key frame AAU, quantizing the residual data using the quantization scale of the previous keyframe AAU and encoding the quantized data in the bitstream.

16. A decoding apparatus comprising one or more processors and at least one memory coupled to said one or more processors, wherein said one or more processors are configured to perform: an avatar animation unit (AAU) comprising a header and an AAU payload, wherein the header of the AAU comprises information indicating a type of AAU, information indicating a size of the AAU payload, information indicating whether the AAU payload is a keyframe and, in a case where the AAU is a keyframe, a quantization scale;responsive to determining from the header that the AAU is a keyframe AAU:Docket No. 2025P00015WQdequantizing data of payload of the keyframe AAU using the quantization scale to obtain a reconstructed payload ; andresponsive to determining from the header that the AAU is a non-keyframe AAU encoded based on a previous keyframe AAU:dequantizing residual data using the quantization scale of the previous keyframe AAU and reconstructing payload of the non-keyframe AAU based on data of payload of the previous keyframe AAU and dequantized residual data.

17. A computer program comprising program code instructions for implementing the method according to any one of claims 1-14 when executed by a processor.

18. A computer readable storage medium having stored thereon instructions for implementing the method of any one of claims 1-14.

19. A signal comprising a bitstream, formed by performing the method of any one of claims 1 to 7.

Images