Mesh data encoding device, mesh data encoding method, mesh data decoding device, and mesh data decoding method

Abstract

A decoding method according to embodiments may comprise the steps of: decoding a base mesh within a bitstream; decoding displacements within the bitstream; and decoding attributes within the bitstream. An encoding method according to embodiments may comprise the steps of: encoding a base mesh of mesh data; encoding displacements of the mesh data; and encoding attributes of the mesh data.

Description

Mesh data encoding device, mesh data encoding method, mesh data decoding device and mesh data decoding method

[0001] The embodiments provide a method for providing Point Cloud content to provide various services to users, such as VR (Virtual Reality), AR (Augmented Reality), MR (Mixed Reality), and autonomous driving services.

[0002] A point cloud is a collection of points in 3D space. There is a problem in that it is difficult to generate point cloud data because there are many points in 3D space.

[0003] Mesh data refers to a form of data in which connectivity information between the vertices of a mesh is added to point cloud data.

[0004] There is a problem in that a large amount of throughput is required to transmit and receive dynamic mesh data.

[0005] The technical problem according to the embodiments is to provide a point mesh data transmission device, a transmission method, a mesh data reception device, and a reception method for efficiently transmitting and receiving mesh data in order to solve the aforementioned problems, etc.

[0006] The technical problem according to the embodiments is to provide a point mesh transmission device, a transmission method, a mesh data receiving device, and a receiving method for solving latency and encoding / decoding complexity.

[0007] However, the scope of rights of the embodiments is not limited to the technical problems described above, and may be extended to other technical problems that can be inferred by a person skilled in the art based on the entire content of this document.

[0008] To achieve the above-described purpose and other advantages, the decoding method according to the embodiments may include the step of decoding a base mesh in a bitstream; the step of decoding a displacement in a bitstream; and the step of decoding an attribute in a bitstream. The encoding method according to the embodiments may include the step of encoding a base mesh of mesh data; the step of encoding a displacement of mesh data; and the step of encoding an attribute of mesh data.

[0009] A mesh data transmission method, a transmission device, a mesh data reception method, and a reception device according to the embodiments can provide a high-quality mesh service.

[0010] A mesh data transmission method, a transmission device, a mesh data reception method, and a reception device according to the embodiments can achieve various video codec methods.

[0011] The mesh transmission method, transmission device, mesh data reception method, and reception device according to the embodiments can provide general-purpose mesh content such as autonomous driving services.

[0012] Drawings are included to further understand the embodiments, and the drawings illustrate the embodiments along with descriptions related to the embodiments. For a better understanding of the various embodiments described below, one must refer to the description of the embodiments below in relation to the following drawings, which include parts corresponding to similar reference numerals throughout the drawings.

[0013] FIG. 1 shows a V-DMC-based encoder and decoder according to embodiments.

[0014] FIG. 2 shows a system for providing dynamic mesh content according to embodiments.

[0015] FIG. 3 illustrates a V-MESH compression method according to embodiments.

[0016] FIG. 4 shows the pre-processing of V-MESH compression according to the embodiments.

[0017] FIG. 5 illustrates a mid-edge subdivision method according to embodiments.

[0018] FIG. 6 illustrates a displacement generation process according to embodiments.

[0019] FIG. 7 illustrates the V-DMC encoding process according to the embodiments.

[0020] FIG. 8 illustrates a lifting conversion process for displacement according to embodiments.

[0021] FIG. 9 illustrates the process of packing conversion coefficients according to embodiments into a 2D image.

[0022] FIG. 10 illustrates the attribute transfer process of the V-MESH compression method according to the embodiments.

[0023] FIG. 11 illustrates a V-DMC decoding process according to embodiments.

[0024] FIG. 12 illustrates a V-DMC encoding process according to embodiments.

[0025] FIG. 13 illustrates a V-DMC decoding process according to embodiments.

[0026] FIG. 14 illustrates an example of a normal vector-based subdivision method according to embodiments.

[0027] FIG. 15 shows the mesh patch data unit syntax according to the embodiments.

[0028] FIG. 16 shows the configuration and operation of an encoder according to embodiments.

[0029] FIG. 17 shows the configuration and operation of a decoder according to embodiments.

[0030] FIG. 18 illustrates a encoding method according to embodiments.

[0031] FIG. 19 illustrates a decoding method according to embodiments.

[0032] Preferred embodiments of the embodiments are described in detail, and examples thereof are shown in the accompanying drawings. The following detailed description, with reference to the accompanying drawings, is intended to describe preferred embodiments of the embodiments rather than merely embodiments that may be implemented according to the embodiments. The following detailed description includes details to provide a thorough understanding of the embodiments. However, it is obvious to those skilled in the art that the embodiments may be practiced without these details.

[0033] Most terms used in the embodiments are selected from those commonly used in the field, but some terms are chosen at the applicant's discretion, and their meanings are described in detail in the following description as necessary. Accordingly, the embodiments should be understood based on the intended meaning of the terms, rather than their mere names or meanings.

[0034] FIG. 1 shows a V-DMC-based encoder and decoder according to embodiments.

[0035] The basic structure of the currently ongoing V-DMC (v-mesh) is shown in Fig. 1. The encoder and decoder according to Fig. 1 perform the encoding and decoding processes of media representing a dynamic mesh using V3C technology. The preprocessor converts the input dynamic mesh representation into several V3C components (base mesh, displacement set, 2D representation of attributes, and atlas). The original mesh is simplified into a base mesh. The base mesh can be encoded using any mesh codec. Displacement vectors can be represented by a profile or encoded into V3C geometric video components using any video codec via SEI messages. For example, depending on the profile, displacement vectors (displacement data) can be encoded using arithmetic coding. Attribute data may include additional attributes. For example, texture or material information may be included as additional attributes and can be encoded using any video codec. Atlas data contains information on how to perform inverse reconstruction and is provided to the V3C decoding and / or rendering system. For example, atlas data may include methods for performing subdivision of the base mesh, methods for applying displacement vectors to the vertices of the subdivided mesh, and methods for applying attributes to the reconstructed mesh.

[0036] The encoder may be composed of a memory and at least one processor connected to the memory. The at least one processor may be configured to perform operations such as a preprocessor, an atlas encoder, a basemesh encoder, a displacement vector encoder, a video encoder, and a multiplexer.

[0037] The atlas encoding unit encodes the atlas of the mesh data to generate an atlas bitstream. The basemesh encoding unit encodes the basemesh of the mesh data to generate a basemesh bitstream. The displacement vector encoding unit encodes the displacement vector of the mesh data to generate a displacement vector bitstream. The video encoding unit encodes the attributes of the mesh data to generate an attribute bitstream. The encoder generates parameter information (which may be referred to as signaling information, metadata, etc.) related to each encoding. The encoder can generate a bitstream containing parameter information, the atlas, the basemesh, the displacement vector, and / or attributes.

[0038] The decoder may be composed of a memory and at least one processor connected to the memory. The at least one processor may be configured to perform operations such as a demultiplexer, an atlas decoder, a basemesh decoder, a displacement vector decoder, and a video decoder.

[0039] The atlas decoder decodes the atlas within the bitstream. The basemesh decoder decodes the basemesh within the bitstream. The displacement vector decoder decodes the displacement vector within the bitstream. The video decoder decodes the attributes within the bitstream. The decoder can perform each decoding operation based on parameter information within the bitstream. The decoder can reconstruct dynamic mesh data based on the atlas, displacement vector, attributes, and basemesh.

[0040] Below, the operation of the V-DMC encoder and decoder of FIG. 1 is explained in more detail.

[0041] FIG. 2 shows a system for providing dynamic mesh content according to embodiments.

[0042] The system of FIG. 2 includes a point cloud data transmission device (100) and a point cloud data reception device (110) according to embodiments. The point cloud data transmission device may include a dynamic mesh video acquisition unit (101), a dynamic mesh video encoder (102), a file / segment encapsulator (103), and a transmitter (104). The point cloud data reception device (110) may include a receiver (111), a file / segment decapsulator (112), a dynamic mesh video decoder (113), and a renderer (114). Each component of FIG. 1 may correspond to hardware, software, a processor, and / or a combination thereof. Hereinafter, the point cloud data transmission device according to embodiments may be interpreted as a term referring to the transmission device (100) or the dynamic mesh video encoder (hereinafter, encoder) (102). The point cloud data receiving device according to the embodiments may be interpreted as a term referring to the receiving device (110) or the dynamic mesh video decoder (hereinafter, decoder) (113).

[0043] The system of Fig. 2 can perform video-based dynamic mesh compression and decompression.

[0044] With advancements in 3D capture, modeling, and rendering, users can access various forms of 3D content, such as AR, XR, the metaverse, and holograms, across multiple platforms and devices. 3D content represents objects more sophisticatedly and realistically to enable users to enjoy immersive experiences, and for this purpose, the creation and use of 3D models require a large amount of data. Among the various types of 3D content, 3D meshes are widely used for efficient data utilization and realistic object representation. The embodiments include a series of processing steps in a system that uses such mesh content.

[0045] First, the method for compressing dynamic mesh data originates from the V-PCC (Video-based point cloud compression) standard technology. Point cloud data consists of data containing color information along with vertex coordinates (X, Y, Z). Mesh data refers to data where connectivity information between vertices is added to this vertex data. Content can be created in a mesh data format from the outset when generating content. Point cloud data can be converted into mesh data and used by adding connectivity information.

[0046] Currently, the MPEG standards organization defines the data types for dynamic mesh data as the following two types: Category 1: Mesh data containing texture maps as color information. Category 2: Mesh data containing vertex colors as color information.

[0047] Mesh coding standards for Category 1 data are currently being developed, and standardization work for Category 2 data is also scheduled to proceed in the future. As shown in Fig. 1, the entire process for providing mesh content services may include an acquisition process, an encoding process, a transmission process, a decoding process, a rendering process, and / or a feedback process.

[0048] To provide mesh content services, 3D data acquired through multiple cameras or special cameras can be processed into a mesh data type through a series of processes and then generated as a video. The generated mesh video is transmitted after undergoing a series of processes, and at the receiving end, the received data can be processed back into a mesh video and rendered. Through this, the mesh video is provided to the user, and the user can use the mesh content according to their intention through interaction.

[0049] A mesh compression system may include a transmission device and a reception device. The transmission device can encode mesh video to output a bitstream and transmit it to the reception device via a digital storage medium or network in the form of a file or streaming (streaming segment). The digital storage medium may include various storage media such as USB, SD, CD, DVD, Blu-ray, HDD, and SSD.

[0050] The transmission device may schematically include a mesh video acquisition unit, a mesh video encoder, and a transmission unit. The receiving device may schematically include a receiver, a mesh video decoder, and a renderer. The encoder may be referred to as a mesh video / image / picture / frame encoding device, and the decoder may be referred to as a mesh video / image / picture / frame decoding device. The transmitter may be included in the mesh video encoder. The receiver may be included in the mesh video decoder. The renderer may include a display unit, and the renderer and / or the display unit may be composed of separate devices or external components. The transmission device and the receiving device may further include separate internal or external modules / units / components for a feedback process.

[0051] Mesh data represents the surface of an object using multiple polygons. Each polygon is defined by a vertex in 3D space and connectivity information indicating how those vertices are connected. It may also include vertex attributes such as color and normals. Mapping information, which enables the surface of the mesh to be mapped onto a 2D planar area, can also be included as an attribute of the mesh. Mapping can generally be described by a set of parametric coordinates, referred to as UV coordinates or texture coordinates, associated with the mesh vertices. The mesh contains a 2D attribute map, which can be used to store high-resolution attribute information such as textures, normals, and displacements.

[0052] The mesh video acquisition unit may include processing 3D object data acquired through a camera, etc., into a mesh data type having the attributes described above through a series of processes, and generating a video composed of such mesh data. In a mesh video, the attributes of the mesh, namely vertices, polygons, connectivity information between vertices, color, normals, etc., may change over time. A mesh video having attributes and connectivity information that change over time in this way can be described as a dynamic mesh video.

[0053] A mesh video encoder can encode input mesh video into one or more video streams. A single video may contain multiple frames, and a single frame may correspond to a still image or picture. In this document, the term "mesh video" may include mesh images, frames, or pictures, and the terms mesh video and mesh images, frames, or pictures may be used interchangeably. A mesh video encoder can perform a Video-based Dynamic Mesh (V-Mesh) Compression procedure. To improve compression and coding efficiency, a mesh video encoder can perform a series of procedures such as prediction, transformation, quantization, and entropy coding. The encoded data (encoded video / image information) can be output in the form of a bitstream.

[0054] The file / segment encapsulation module can encapsulate encoded mesh video data and / or mesh video-related metadata into a file or the like. Here, the mesh video-related metadata may be received from a metadata processing module or the like. The metadata processing module may be included in the mesh video encoder or may be configured as a separate component / module. The encapsulation module can encapsulate the data into a file format such as ISOBMFF or process it into other forms such as DASH segments. Depending on the embodiment, the encapsulation module may include mesh video-related metadata in the file format. Mesh video metadata may be included, for example, in various levels of boxes in the ISOBMFF file format or as data within a separate track in the file. Depending on the embodiment, the encapsulation module may encapsulate the mesh video-related metadata itself into a file.

[0055] The transmission processing unit can apply transmission processing to mesh video data encapsulated according to the file format. The transmission processing unit may be included in the transmission unit or may be configured as a separate component / module. The transmission processing unit can process mesh video data according to any transmission protocol. Processing for transmission may include processing for delivery via a broadcast network and processing for delivery via broadband. According to an embodiment, the transmission processing unit may receive mesh video-related metadata from the metadata processing unit in addition to mesh video data, and apply transmission processing to it.

[0056] The transmission unit can transmit encoded video / image information or data output in the form of a bitstream to the receiving unit of a receiving device via a digital storage medium or a network in the form of a file or streaming. The digital storage medium may include various storage media such as USB, SD, CD, DVD, Blu-ray, HDD, and SSD. The transmission unit may include elements for creating a media file through a predetermined file format and elements for transmission via a broadcasting / communication network. The receiving unit can extract the bitstream and transmit it to a decoding device.

[0057] The receiver can receive mesh video data transmitted by a mesh video transmission device. Depending on the transmission channel, the receiver may receive mesh video data via a broadcast network or via broadband. Alternatively, it may receive mesh video data via a digital storage medium.

[0058] The receiving processing unit can perform processing on the received mesh video data according to the transmission protocol. The receiving processing unit may be included in the receiving unit or may be configured as a separate component or module. Corresponding to the processing for transmission performed on the transmitting side, the receiving processing unit may perform the reverse process of the aforementioned transmission processing unit. The receiving processing unit may transmit the acquired mesh video data to the decapsulation processing unit and the acquired mesh video-related metadata to the metadata parser. The mesh video-related metadata acquired by the receiving processing unit may be in the form of a signaling table.

[0059] The file / segment decapsulation module can decapsulate mesh video data in file format received from the receiving module. The decapsulation module can decapsulate files based on ISOBMFF, etc., to obtain a mesh video bitstream or mesh video-related metadata (metadata bitstream). The obtained mesh video bitstream can be transmitted to a mesh video decoder, and the obtained mesh video-related metadata (metadata bitstream) can be transmitted to a metadata processing module. The mesh video bitstream may also contain metadata (metadata bitstream). The metadata processing module may be included in the mesh video decoder or configured as a separate component / module. The mesh video-related metadata obtained by the decapsulation module may be in the form of boxes or tracks within the file format. If necessary, the decapsulation module may receive metadata required for decapsulation from the metadata processing module. Mesh video-related metadata may be passed to a Mesh video decoder and used in the Mesh video decoding process, or passed to a renderer and used in the Mesh video rendering process.

[0060] The Mesh Video Decoder can decode video by receiving a bitstream as input and performing operations corresponding to those of the Mesh Video Encoder. The decoded Mesh Video can be displayed through a display unit. The user can view all or part of the rendered result through a VR / AR display or a standard display.

[0061] The feedback process may include the process of transmitting various feedback information obtainable during the rendering / display process to the transmitting side or to the decoder of the receiving side. Interactivity in mesh video consumption may be provided through the feedback process. According to an embodiment, head orientation information, viewport information indicating the area the user is currently viewing, etc., may be transmitted during the feedback process. According to an embodiment, the user may interact with elements implemented in a VR / AR / MR / autonomous driving environment, and in this case, information related to such interaction may be transmitted to the transmitting side or the service provider side during the feedback process. According to an embodiment, the feedback process may not be performed.

[0062] Head orientation information can refer to information regarding the user's head position, angle, movement, etc. Based on this information, viewport information—that is, information about the area the user is currently viewing within the mesh video—can be calculated.

[0063] Viewport information may be information about the area currently being viewed by the user within a mesh video. Through this, gaze analysis can be performed to determine how the user consumes the mesh video and to what extent they gaze at specific areas of the mesh video. Gaze analysis may be performed at the receiving end and transmitted to the transmitting end via a feedback channel. Devices such as VR / AR / MR displays can extract the viewport area based on the user's head position / orientation, the vertical or horizontal FOV supported by the device, etc.

[0064] According to an embodiment, the aforementioned feedback information may not only be transmitted to the transmitting side but may also be consumed at the receiving side. That is, decoding and rendering processes at the receiving side may be performed using the aforementioned feedback information. For example, using head orientation information and / or viewport information, only the mesh video for the area currently viewed by the user may be preferentially decoded and rendered.

[0065] This document relates to dynamic mesh video compression as described above. The methods / embodiments disclosed in this document may be applied to the MPEG (Moving Picture Experts Group) Video-based Dynamic Mesh Compression Method (V-Mesh) standard or next-generation video / image coding standards. Dynamic mesh video compression is a method for processing mesh connection information and attributes that change over time, and it can perform lossy and lossless compression for various applications such as real-time communication, storage, free-viewpoint video, and AR / VR.

[0066] The dynamic mesh video compression method described below is based on MPEG's V-Mesh method.

[0067] In this document, "picture" or "frame" generally refers to a unit representing a single image of a specific time period.

[0068] A pixel or pel may refer to the smallest unit that constitutes a picture (or image). Additionally, the term 'sample' may be used as a counterpart to pixel. A sample can generally represent a pixel or a pixel value, and may represent only the pixel / pixel value of the lumina component, only the pixel / pixel value of the chroma component, or only the pixel / pixel value of the depth component.

[0069] A unit may represent a basic unit of image processing. A unit may include at least one of a specific area of ​​a picture and information related to that area. Depending on the case, the term unit may be used interchangeably with terms such as block or area. In general, an MxN block may include samples (or sample arrays) or a set (or array) of transform coefficients consisting of M columns and N rows.

[0070] The encoding process of Fig. 2 is as follows.

[0071] Video-based dynamic mesh compression (V-Mesh) compression methods can provide a method for compressing dynamic mesh video data based on 2D video codecs such as HEVC and VVC. In the V-Mesh compression process, compression is performed by receiving the following data as input.

[0072] Input mesh: It includes the 3D coordinates (geometry) of the vertices constituting the mesh, normal information for each vertex, mapping information for mapping the mesh surface to a 2D plane, and connection information between the vertices constituting the surface. The surface of the mesh can be represented by triangles or polygons of greater size, and connection information between the vertices constituting each surface is stored according to a defined shape. The input mesh can be saved in the OBJ file format.

[0073] Attribute map: (Hereafter, Texture map is used with the same meaning): It contains information on the attributes (color, normals, displacement, etc.) of a mesh and stores data in the form of mapping the mesh surface onto a 2D image. Mapping which part of the mesh (surface or vertex) corresponds to each data point in this attribute map is based on the mapping information contained in the input mesh. Since the attribute map holds data for each frame of the mesh video, it can also be referred to as an attribute map video (or simply "attribute"). In the V-Mesh compression method, the attribute map primarily contains the mesh's color information and is stored in image file formats (PNG, BMP, etc.).

[0074] Material Library File: Contains material attribute information used in the mesh, specifically information that links the input mesh with its corresponding attribute map. This is saved in the Wavefront Material Template Library (MTL) file format.

[0075] In the V-Mesh compression method, the following data and information can be generated through the compression process.

[0076] Base Mesh: By simplifying (decimating) the input mesh through a preprocessing stage, objects from the input mesh are represented using the minimum number of vertices determined according to user criteria.

[0077] Displacement: Displacement information used to represent the input mesh as similarly as possible using the base mesh, expressed in the form of 3D coordinates.

[0078] Atlas information: This is metadata required to reconstruct a mesh using base mesh, displacement, and attribute map information. It can be generated and utilized as sub-units (sub-mesh, patch, etc.) that constitute the mesh.

[0079] Referring to FIGS. 3 to 7, a method for encoding mesh position information (vertices) is described, and referring to FIGS. 7-10 and others, a method for encoding attribute information (attribute map) by restoring mesh position information is described.

[0080] FIG. 3 illustrates a V-MESH compression method according to embodiments.

[0081] FIG. 3 illustrates the encoding process of FIG. 2, and the encoding process may include pre-processing and encoding processes. The encoder of FIG. 2 may include a pre-processor (200) and an encoder (201) as in FIG. 3. The transmitting device of FIG. 2 may be broadly referred to as an encoder, and the dynamic mesh video encoder of FIG. 2 may be referred to as an encoder. The V-Mesh compression method may include pre-processing (200) and encoding (201) processes as in FIG. 3. The pre-processor of FIG. 3 may be located in front of the encoder of FIG. 3. The pre-processor and encoder of FIG. 3 may be referred to as a single encoder.

[0082] The pre-processor can receive a static dynamic mesh and / or attribute map. The pre-processor can generate a base mesh and / or displacement through preprocessing. The pre-processor can receive feedback information from an encoder and generate a base mesh and / or displacement based on the feedback information.

[0083] The encoder can receive a base mesh, displacement, static dynamic mesh, and / or attribute map. The encoder can encode mesh-related data to generate a compressed bitstream.

[0084] FIG. 4 shows the pre-processing of V-MESH compression according to the embodiments.

[0085] Figure 4 shows the configuration and operation of the pre-processor of Figure 3.

[0086] FIG. 3 illustrates a process of performing preprocessing on an input mesh. The preprocessing process (200) may include four main steps: 1) generation of a Group of Frame (GoF), 2) mesh decimation, 3) UV parameterization, and 4) fitting subdivision surface (300). The pre-processor (200) may receive the input mesh, generate displacement and / or a base mesh, and transmit it to the encoder (201). The pre-processor (200) may transmit GoF information associated with GoF generation to the encoder (201).

[0087] Below, each step of Fig. 4 is explained.

[0088] GoF Generation: This is the process of creating a reference structure for mesh data. If the number of vertices, texture coordinates, vertex connectivity, and texture coordinate connectivity of the mesh in the previous frame and the current mesh are all identical, the previous frame can be set as the reference frame. In other words, if only the vertex coordinate values ​​differ between the current input mesh and the reference input mesh, inter-frame encoding can be performed. Otherwise, the frame undergoes intra-frame encoding.

[0089] Mesh Decimation: This is the process of simplifying the input mesh to generate a simplified mesh, or base mesh. After selecting vertices to remove from the original mesh based on user-defined criteria, the selected vertices and the triangles connected to them can be removed.

[0090] In the process of performing mesh decimation, information regarding the input mesh (voxelized), target triangle ratio (TTR), and minimum triangle component (CCCount) is passed as input, and a decimated mesh can be obtained as output. In this process, connected triangle components smaller than the set minimum triangle component (CCCount) can be removed.

[0091] UV Parameterization: This is the process of mapping 3D surfaces to a texture domain for a decimated mesh. Parameterization can be performed using a UV Atlas tool. Through this process, mapping information is generated regarding where each vertex of the decimated mesh can be mapped to on a 2D image. This mapping information is expressed and stored as texture coordinates, and the final base mesh is generated through this process.

[0092] OrthoAtlas technology is a technique that generates texture coordinates using orthographic projection. In orthoAtlas technology, the processes of patch generation and patch packing are performed sequentially. First, Connected Components (CCs) are generated by splitting adjacent triangles, and then the optimal CCs are merged using a cost function to generate a patch. The cost function can measure the cost based on the degree of distortion that occurs when orthogonally projecting the patch in each direction. Finally, texture coordinates can be calculated by packing the patch that minimizes the cost function into the texture domain. In the case of orthoAtlas technology, texture coordinates can be derived in the base mesh decoder without compressing texture coordinate and texture connection information during the base mesh encoding process.

[0093] Fitting subdivision surface: This is the process of performing subdivision on a decimated mesh. User-defined methods, such as the mid-edge method, can be applied as the subdivision method. A fitting process is performed to make the input mesh and the subdivision mesh similar to each other.

[0094] This is a process of performing fitting so that the mesh obtained by subdividing the base mesh becomes similar to the surface of the input mesh. As for the subdivision method, a user-defined method such as the mid-edge method (Fig. 5), loop method, or LS3 method may be applied.

[0095] FIG. 5 illustrates a mid-edge subdivision method according to embodiments.

[0096] Figure 5 illustrates the mid-edge method of the fitting subdivision surface described in Figure 4. Referring to Figure 5, an original mesh containing 4 vertices is subdivided to create a sub-mesh. A sub-mesh can be created by creating a new vertex at the midpoint of the edge between vertices.

[0097] When a fitted subdivided mesh (hereinafter referred to as the fitted subdivided mesh) is generated, displacement is calculated using this result and a pre-compressed and decoded base mesh (hereinafter referred to as the reconstructed base mesh). That is, the reconstructed base mesh is subdivided in the same way as the fitting subdivision surface. The positional difference between this result and the fitted subdivided mesh for each vertex becomes the displacement for each vertex. Since displacement represents a positional difference in three-dimensional space, it is also expressed as a value in the (x, y, z) space of the Cartesian coordinate system. Depending on user input parameters, (x, y, z) coordinate values ​​can be converted into (normal, tangential, bi-tangential) coordinate values ​​of the local coordinate system.

[0098] FIG. 6 illustrates a displacement generation process according to embodiments.

[0099] FIG. 6 illustrates in detail the method of calculating displacement of a fitting subdivision surface (300) as described in FIG. 5.

[0100] An encoder and / or pre-processor according to the embodiments may include 1) a subdivision unit, 2) a local coordinate system calculation unit, and 3) a displacement calculation unit. The subdivision unit may receive a restored base mesh and generate a subdivided restored base mesh. The local coordinate system calculation unit may receive a fitted subdivided mesh and a subdivided restored base mesh and convert a coordinate system relating to the mesh to a local coordinate system. The local coordinate system calculation operation may be optional. The displacement calculation unit calculates the positional difference between the fitted subdivision mesh and the subdivided restored base mesh. For example, it may generate a positional difference value between the vertices of the two input meshes. The vertex positional difference value becomes the displacement.

[0101] The method and apparatus for transmitting point cloud data according to the embodiments can encode the point cloud as follows. The point cloud data according to the embodiments (which may be referred to simply as point cloud) may refer to data including vertex coordinates and color information. Point cloud is a term that includes mesh data, and in this document, point cloud and mesh data may be used interchangeably.

[0102] The V-Mesh compression (restoration) method according to the embodiments may include intra-frame encoding (Fig. 6) and inter-frame encoding (Fig. 7).

[0103] Intra-frame encoding or inter-frame encoding is performed based on the results of the aforementioned GoF generation. In the case of intra-frame encoding, the data to be compressed may include a base mesh, displacement, and attribute map. In the case of inter-frame encoding, the data to be compressed may include displacement, attribute map, and the motion field between the reference base mesh and the current base mesh.

[0104] FIG. 7 illustrates the V-DMC encoding process according to the embodiments.

[0105] The encoding process of Fig. 7 illustrates the encoding of Figs. 1 and 2 in detail.

[0106] A pre-processor receives an input mesh and can perform the aforementioned preprocessing. Through preprocessing, a base mesh and / or a fitted subdivided mesh can be generated. A quantizer can quantize the base mesh and / or the fitted subdivided mesh. A static mesh encoder can encode a static mesh. The static mesh encoder can generate a bitstream containing the encoded base mesh. A motion encoder can encode motion vectors for the base mesh based on inter-frame motion estimation and motion compensation for inter-prediction. An atlas encoder can encode an atlas for the vertices of the base mesh. The encoded base mesh can be reconstructed and inversely quantized through an inverse quantizer. A displacement computer receives the reconstructed mesh and, based on the fitted subdivided mesh, can generate displacement, which is the position difference. A lifting transform can receive the displacement and generate lifting coefficients. The quantizer can quantize the lifting coefficients. Depending on the encoding method, the image packing unit can pack the image based on the quantized lifting coefficients. The video encoder can encode the packed image. Depending on the encoding method, it can apply interprediction to the quantized lifting coefficients and encode the predicted lifting coefficients according to an arithmetic encoding method. The mesh restoration unit restores the deformed mesh through the restored displacement and the restored base mesh. The displacement data is restored, and the deformed mesh is restored based on the restored displacement data and the restored base mesh and provided to the attribute transfer. The attribute transfer receives the input mesh and / or input attribute map and generates an attribute map based on the restored deformed mesh. Push-pull padding can pad data into the attribute map based on a push-pull method. The color space converter can convert the space of the color component that is an attribute. The video encoder can encode the attributes.A multiplexer can generate a bitstream by multiplexing a compressed base mesh, compressed displacement, and compressed attributes.

[0107] Base Mesh Encoding: Base mesh compression methods can be divided into INTRA, INTER, and SKIP types depending on the base mesh type, and encoding can be performed in different ways for each. If the base mesh is of the INTRA type, it can be encoded using a static mesh encoding method. If the base mesh is of the INTER type, the motion field between the reference base mesh and the current base mesh can be encoded. If the current base mesh is of the SKIP type, the reference base mesh can be derived into the current base mesh.

[0108] After being encoded in the encoder, the base mesh can be subdivided into a subdivided mesh through a subdivision process. Subdivision algorithms such as mid-point subdivision and loop subdivision can be used.

[0109] Static Basemesh Encoding (Intra Basemesh Encoding): When performing intra encoding on the current basemesh, the base mesh generated during the preprocessing stage can be encoded using static mesh compression technology after undergoing a quantization process. Static mesh compression utilizes MPEG EdgeBreaker (MEB) technology, and the base mesh's vertex position information, mapping information (texture coordinates), vertex connectivity information, and normals are subject to compression.

[0110] The technology for compressing connection information can be encoded based on the edgebreaker algorithm. The edgebreaker algorithm is a technique that sequentially traverses triangles according to rules, maps symbols based on the characteristics of each triangle, and then encodes the corresponding symbols.

[0111] Techniques for compressing vertex location information can calculate predicted values ​​based on prediction techniques such as multiple parallelogram prediction, and then encode the residual value, which is the difference between the current vertex and the predicted value.

[0112] A technique for compressing mapping information (texture coordinates) can calculate a predicted value based on a prediction technique such as stretching, and then encode the residual value, which is the difference between the current mapping information (texture coordinates) and the predicted value.

[0113] Normal compression techniques can obtain predicted values ​​based on prediction techniques such as delta coding, multiple parallelogram prediction, and cross product-based prediction, and then encode the residual value, which is the difference between the current normal and the predicted value.

[0114] Motion Field Encoding (Inter Basemesh Encoding): Inter basemesh encoding can be performed when a one-to-one correspondence exists between the reference mesh and the current input mesh, differing only in vertex position information. When performing inter encoding, instead of compressing the base mesh, the difference between the vertices of the reference base mesh and the current base mesh—that is, the motion field—is calculated and this information is encoded. The reference base mesh is the result of quantizing already decoded base mesh data and is determined by the reference frame index.

[0115] The motion field can be encoded as is, or the predicted motion field can be calculated by averaging the motion fields of the restored vertices among those connected to the current vertex, and the residual motion field, which is the difference between the predicted motion field value and the current vertex's motion field value, can be encoded. This value can be encoded using entropy coding.

[0116] Displacement Encoding: After encoding the base mesh, reconstruction and inverse quantization are performed to reconstruct it. Once the base mesh is generated and subdivision is performed on it, the displacement between the result and the fitted subdivided mesh can be calculated. For effective encoding, data transform processes such as wavelet transform can be applied to the displacement information, and Figure 7 shows the process of transforming displacement information using the lifting transform in V-Mesh. The transform coefficients generated through the transformation process are quantized, and the quantized transform coefficients can be compressed through a video codec or through arithmetic coding, depending on the compression method.

[0117] When compressed through a video codec, the data is packed into a 2D image as shown in Figure 8. Transform coefficients are organized into one block for every N^2 (N*N) units, and each block can be packed in z-scan order. The number of horizontal blocks is fixed at N, while the number of vertical blocks can be determined by the number of vertices in the subdivided base mesh. Within a single block, transform coefficients can be packed by aligning them using Morton code. The packed images generate a displacement video for every GoF unit, and this displacement video can be encoded using an existing video compression codec.

[0118] When compressed via arithmetic coding, cross-frame prediction can be performed on the quantized displacement vector transformation coefficients. When cross-frame prediction is performed on the current quantized displacement vector transformation coefficients, the residual value, which is the difference between the current displacement vector transformation coefficient and the reference displacement vector transformation coefficient, can be encoded, and information about the reference target can be encoded. Depending on the displacement vector type, the quantized displacement vector transformation coefficients can be arithmetic encoded if it is of the INTRA type, and the residual value if it is of the INTER type. Arithmetic coding can be performed based on Context Adaptive Binary Arithmetic Coding (CABAC). The CABAC process can first binarize the displacement vector data and map it to a bin string. The bin string can be an output binarized into 0s and 1s, where each 0 or 1 can be a bin. Each bin can be arithmetic encoded using context information selected from the context model, and a process of updating probabilities can be performed.

[0119]

[0120] FIG. 8 illustrates a lifting conversion process for displacement according to embodiments.

[0121] FIG. 9 illustrates the process of packing conversion coefficients according to embodiments into a 2D image.

[0122] FIGS. 8-9 respectively illustrate the process of converting the displacement of the encoding process of FIG. 7 and the process of packing the conversion coefficients.

[0123] The encoding method according to the embodiments includes displacement encoding.

[0124] After encoding the base mesh through base mesh encoding and / or motion field encoding, a reconstructed base mesh is generated through reconstruction and inverse quantization. Subdivision is then performed on this reconstructed base mesh, and the displacement between the result and the fitted subdivided mesh generated through the fitting subdivision surface can be calculated. For effective encoding, a data transform process such as a wavelet transform can be applied to the displacement information.

[0125] Figure 8 illustrates the process of transforming displacement information using a lifting transform in V-Mesh. The transformation coefficients generated through the transformation process are quantized and then packed into a 2D image as shown in Figure 9. The transformation coefficients are 256 (=16 16) Each unit is composed of one block, and each block can be packed in z-scan order. The number of blocks is fixed at 16, but the number of blocks can be determined by the number of vertices of the subdivided base mesh. Transform coefficients can be packed by aligning them with a Morton code within a single block. The packed images generate a displacement video for each GoF unit, and this displacement video can be encoded using a conventional video compression codec.

[0126] Referring to FIG. 8, the base mesh (original) may include vertices and edges for LoD0. A first subdivision mesh generated by dividing the base mesh includes vertices generated by further dividing the edges of the base mesh. The first subdivision mesh includes vertices for LoD0 and vertices for LoD1. LoD1 includes subdivided vertices and vertices of the base mesh (LoD0). A second subdivision mesh may be generated by dividing the first subdivision mesh. The second subdivision mesh includes LoD2. LoD2 includes base mesh vertices (LoD0), LoD1 containing vertices additionally generated from LoD0, and vertices additionally divided from LoD1. LoD is a Level of Detail indicating the degree of detail; as the index of the level increases, the distance between vertices becomes closer, and the level of detail increases. LoD N includes the vertices included in the previous LoDN-1 as they are. When vertices are further subdivided through subdivision, the mesh can be encoded based on a prediction and / or update method by considering the previous vertices v1 and v2 and the subdivided vertex v. Instead of encoding the information for the current LoD N as is, the size of the bitstream can be reduced by generating residuals between the previous LoD N-1 and encoding the mesh using these residuals. The prediction process refers to the operation of predicting the current vertex v based on the previous vertices v1 and v2. Since adjacent subdivided meshes possess similar data, efficient encoding can be achieved by utilizing this property. The current vertex position information is predicted using the residuals from the previous vertex position information, and the previous vertex position information is updated using these residuals.

[0127] Referring to FIG. 9, the vertices have coefficients generated through a lifting transformation. The coefficients of the vertices related to the lifting transformation can be encoded by packing them into an image.

[0128] FIG. 10 illustrates the attribute transfer process of the V-MESH compression method according to the embodiments.

[0129] Figure 10 shows the detailed operation of the attribute transfer of the encoding of Figure 7.

[0130] The encoding according to the embodiments includes attribute map encoding.

[0131] Information regarding the input mesh is compressed through base mesh encoding, motion field encoding, and displacement encoding. The input mesh compressed during the encoding process is restored through base mesh decoding (intra frame), motion field encoding (inter frame), and displacement video decoding. The resulting restored deformed mesh (hereinafter referred to as Recon. deformed mesh) is used to compress the input attribute map, as shown in FIGS. 6 and 7. The Recon. deformed mesh possesses vertex position information, texture coordinates, and corresponding connection information, but lacks color information corresponding to the texture coordinates. Accordingly, as shown in FIG. 10, a new attribute map having color information corresponding to the texture coordinates of the reconstructed deformed mesh is generated through the attribute transfer process in the V-Mesh compression method.

[0132] Attribute transfer first checks for every point P(u, v) in the 2D texture domain whether the point belongs to a texture triangle of the reconstructed deformed mesh, and if it does, determines the barycentric coordinates (α, of P(u, v) according to that triangle T). Calculate , γ). Then, the 3D vertex positions of triangle T and (α, Calculate the 3D coordinates M(x, y, z) of P(u, v) using , γ). Find the vertex coordinates M'(x', y', z') corresponding to the location most similar to the calculated M(x, y, z) in the input mesh domain, and find triangle T' containing this point. Then, in triangle T', find the coordinates of the centroid of M'(x', y', z') (α', Calculate , γ'). Texture coordinates corresponding to the three vertices of Triangle T' and (α', Texture coordinates (u', v') are calculated using , γ'), and color information corresponding to these coordinates is found in the input attribute map. The found color information is then assigned to the (u, v) pixel location in the new attribute map. If P(u, v) does not belong to any triangle, the pixel at that location in the new attribute map can be filled with a color value using a padding algorithm such as the push-pull algorithm.

[0133] The new attribute map generated through attribute transfer is bundled in GoF units to form an attribute map video, which is then compressed using a video codec.

[0134] Atlas Encoding: Atlas information may be transmitted during the aforementioned process. The Atlas consists of information required during the mesh decoding and / or rendering process, and may include information required during the process of performing subdivision, displacement decoding, base mesh decoding, etc., as well as tile information, patch information, etc. Atlas data may be encoded using Exp-Golomb coding, etc.

[0135] Referring to FIG. 10, the reference relationships between the input mesh, input attribute map, restored mesh, and generated attribute map can be seen.

[0136] The decoding process of Fig. 1 can perform the reverse process of the corresponding process of the encoding process of Fig. 1. The specific decoding process is as follows.

[0137] FIG. 11 illustrates a VV-DMC decoding process according to embodiments.

[0138] Figure 11 shows the configuration and operation of the decoder of the receiving device of Figure 1.

[0139] The input bitstream can be separated into a Basemesh sub-stream, a Displacement sub-stream, an Attribute map sub-stream, and an Atlas sub-stream.

[0140] The Atlas sub-stream can be decoded through Exp-Golomb coding, and as a result, information necessary for performing decoding, tile information, patch information, etc. can be obtained.

[0141] Depending on the basemesh type, if the basemesh sub-stream is of the INTRA type, it can be decoded through a static mesh decoder based on MEB (MPEG EdgeBreaker) technology, and as a result, the connectivity information, vertex geometry information, and vertex mapping information (texture coordinates) of the base mesh can be restored.

[0142] When the texture parameterization method in the encoder is orthoAtlas, the decoder can derive mapping information (texture coordinates) and attribute information (texture) connection information using vertex coordinates. The process of deriving mapping information (texture coordinates) and connection information can generate mapping information (texture coordinates) and attribute information (texture) connection information by calculating the homography transform of each face and then projecting the vertex based on it.

[0143] If the Basemesh type is INTER type, motion information can be decoded through entropy decoding and inverse prediction processes. The restored motion information is combined with the reference Basemesh that has already been restored and stored in the buffer to generate a Reconstructed quantized basemesh for the current frame. An inverse quantization process can be performed on the restored Basemesh.

[0144] If the displacement sub-stream is compressed through a video codec according to the compression method used in encoding, it is decoded into a displacement video through the video compression codec's decoder, and then the image unpacking process is performed.

[0145] When compressed through arithmetic coding, the displacement vector bitstream can decode binarized syntax elements through arithmetic decoding, and a Contextual Probability Model (CPM) can be adaptively determined according to each bin of the syntax elements, and arithmetic decoding can be performed by predicting the probability of occurrence of the bin through the CPM. The binarized syntax elements can be decoded through inverse binarization. Quantized displacement vector transformation coefficients can be derived from the decoded syntax elements. Depending on the displacement information type, if it is INTER (where inter prediction is performed), an inverse inter prediction process is performed using reference information for the quantized displacement coefficients.

[0146] The quantized displacement coefficient is restored as displacement information for each vertex through inverse quantization, inverse transform, and coordinate system transformation processes.

[0147] The restored Base mesh and the restored Displacement information are combined to generate the final Decoded mesh. The Attribute map sub-stream is decoded through the decoder of the video compression codec used in Encoding, and then restored to the final Attribute map through processes such as color format conversion.

[0148] The restored Decoded mesh and Decoded attribute map can be utilized at the receiving end as final mesh data available to the user.

[0149] The atlas decoder decodes atlas data within the bitstream.

[0150] When mesh data within the bitstream is encoded based on inter-prediction, the motion decoder derives the motion field of the basemesh of the current frame through motion estimation and compensation, based on the basemesh within the reference frame. When mesh data within the bitstream is encoded based on intra-prediction, the sectic decoder decodes the basemesh. Depending on the encoding method, it decodes displacement data by applying either arithmetic coding or video decoding. The video decoder decodes attribute data within the bitstream.

[0151] The decoding method of FIG. 11 can follow the reverse process of the encoding method according to the embodiments.

[0152] FIG. 12 illustrates a V-DMC encoding process according to embodiments.

[0153] FIG. 12 illustrates the configuration and operation of an encoder of a transmitting device such as FIG. 1 or FIG. 2. Each component of FIG. 12 corresponds to hardware, software, a processor, and / or a combination thereof.

[0154] Figure 12 shows the encoding process of V-Mesh technology.

[0155] The mesh preprocessing unit receives the original mesh as input and generates a decimated mesh. Decimation can be performed based on the number of target vertices or polygons constituting the mesh. For the decimated mesh, parameterization can be performed to generate mapping information (texture coordinates) and attribute information (texture) connection information per vertex. Additionally, floating-point mesh information can be quantized into fixed-point form. This result can be encoded as a base mesh through the static mesh encoding unit. The mesh preprocessing unit can generate additional vertices by performing mesh subdivision on the base mesh. Depending on the subdivision method, vertex connection information including the added vertices, texture coordinates, and texture coordinate connection information can be generated. The subdivided mesh can be fitted by adjusting vertex positions to resemble the original mesh, thereby generating a fitted subdivided mesh.

[0156] The base mesh generated through the mesh preprocessing unit can perform intra-encoding or inter-encoding depending on the base mesh type. If the base mesh frame undergoes intra-encoding, it can be compressed through the static mesh encoding unit. In this case, encoding can be performed on the base mesh's connectivity information, vertex geometry information, vertex texture information, normal information, etc. If the base mesh frame undergoes inter-encoding, the motion vector encoding unit is executed; using the base mesh and the reference-restored base mesh as inputs, the motion vector between the two meshes is calculated and its value encoded. The motion vector encoding unit performs connectivity-based prediction using previously encoded / decoded motion vectors as predictors and can encode the residual motion vector obtained by subtracting the predicted motion vector from the current motion vector. The base mesh bitstream generated through the base mesh encoding unit is transmitted to the multiplexer.

[0157] The encoded base mesh bitstream can generate a restored base mesh through the base mesh restoration unit.

[0158] The displacement vector calculation unit can perform mesh subdivision on the reconstructed base mesh. The displacement vector can be calculated as the difference in vertex positions between the subdivided reconstructed base mesh and the fitted subdivision mesh generated by the preprocessing unit. As a result, displacement vectors can be calculated for each vertex of the subdivision mesh. The displacement vector calculation unit can convert the displacement vector calculated in the 3D Cartesian coordinate system into a local coordinate system based on the normal vector of each vertex.

[0159] The displacement vector processing unit can transform the displacement vector for effective encoding. Depending on the embodiment, the transformation may be performed using a lifting transformation, a wavelet transformation, etc. Additionally, quantization can be performed on the transformed displacement vector values, i.e., the transformation coefficients. Different quantization parameters can be applied to each axis of the transformation coefficients, and the quantization parameters can be derived by the agreement between the encoder and decoder. The quantized displacement vector transformation coefficients calculated by the displacement vector processing unit can be encoded through the displacement vector video encoding unit or the displacement vector arithmetic encoding unit, depending on the compression method.

[0160] The displacement vector video encoding unit can pack displacement vector information that has undergone transformation and quantization into 2D images. A displacement vector video can be generated by bundling the packed 2D images for each frame, and the displacement vector video can be generated for each GoF (Group of Frame) unit of the input mesh. The generated displacement vector video can be encoded using a video compression codec. The generated displacement vector video bitstream is transmitted to the multiplexer.

[0161] In the displacement vector arithmetic encoding unit, for the quantized displacement vector transformation coefficients, if the displacement vector type is INTER type, inter-frame prediction can be performed. The inter-frame prediction process may be a process of calculating the residual value, which is the difference between the current transformation coefficient and the reference transformation coefficient. The displacement vector transformation coefficient or the residual value can be encoded through the arithmetic encoding process.

[0162] The displacement vector restored through the displacement vector restoration unit and the base mesh restored and subdivided through the base mesh restoration unit are restored through the mesh restoration unit, and the restored mesh contains restored vertices, connection information between vertices, texture coordinates, and connection information between texture coordinates.

[0163] The attribute information (texture map) of the original mesh can be regenerated into the attribute information (texture map) for the restored mesh through the attribute information (texture map) video generation unit. Vertex-specific color information contained in the original mesh's texture map can be assigned to the texture coordinates of the restored mesh. The texture maps regenerated for each frame can be grouped by GoF unit to generate a texture map video.

[0164] The generated texture map video can be encoded using a video compression codec through the texture map video encoding unit. The texture map video bitstream generated through encoding is transmitted to the multiplexer.

[0165] The atlas encoding unit can encode the atlas, which is additional information required for the mesh decoding and rendering processes. The generated atlas bitstream is transmitted to the multiplexer.

[0166] The generated base mesh bitstream, displacement vector bitstream, texture map bitstream, and atlas bitstream can be multiplexed into a single bitstream and transmitted to the receiver via the transmitter. Alternatively, the generated base mesh bitstream, displacement vector bitstream, texture map bitstream, and atlas bitstream can be generated into a file with one or more track data or encapsulated into segments and transmitted to the receiver (decoder) via the transmitter.

[0167] The data input unit can receive the original mesh and / or original texture map ('attribute'). The mesh preprocessing unit can simplify the original mesh to generate a base mesh and fit it to generate a refined mesh. If the mesh encoding method is inter-prediction, the motion vector encoding unit can generate motion vectors (motion fields) by referencing the restored base mesh within a previously processed reference frame and encode them based on motion estimation and compensation methods. If the mesh encoding method is intra-prediction, the static mesh encoding unit can encode the base mesh within the frame. The displacement vector calculation unit can calculate displacement vectors for vertices from the fitted refined mesh based on the restored base mesh. The displacement vector processing unit can process the displacement vectors into a form suitable for encoding. Depending on the encoding method for the displacement vectors, the displacement vectors can be encoded based on a video method or an arithmetic encoding method. The displacement vectors are restored and can be provided to the mesh restoration unit along with the restored base mesh. Based on the restored mesh, the attribute (texture map) video generation unit can generate a video for encoding the texture map using the original mesh and the texture map for the original mesh. The attributes are encoded based on the video method. The atlas is encoded by the atlas encoding unit.

[0168] FIG. 13 illustrates a V-DMC decoding process according to embodiments.

[0169] FIG. 13 corresponds to a decoder such as FIG. 1 to FIG. 3. Each component of FIG. 13 corresponds to hardware, software, a processor, and / or a combination thereof.

[0170] The received Mesh bitstream is demultiplexed into a compressed base mesh bitstream, displacement vector bitstream, attribute information (texture map) bitstream, and atlas bitstream after file / segment decapsulation.

[0171] If the current mesh has inter-frame encoding applied based on the frame header information, decoding can be performed on the base mesh bitstream in the motion vector decoder. The final motion vector can be restored by using the previously decoded motion vector as a predictor and adding it to the residual motion vector decoded from the bitstream. The current base mesh can be restored by adding the decoded motion vector to the reference base mesh.

[0172] If the current mesh has in-frame encoding applied based on the frame header information, the base mesh bitstream can restore the base mesh's connectivity information, vertex geometry, texture coordinates, normal information, etc., through the static mesh decoder.

[0173] In the base mesh restoration unit, a restored base mesh can be generated by performing inverse quantization on the decoded base mesh.

[0174] Depending on the encoding codec type, if the displacement vector bitstream is encoded through a video codec, it can be decoded using the video codec and then subjected to a reverse packing process. If encoded through arithmetic coding, arithmetic decoding can be performed through the displacement vector arithmetic decoding unit, and if inter-frame prediction is performed, the current displacement vector transformation coefficient can be generated by adding the residual value to the reference displacement vector transformation coefficient through inter-frame prediction.

[0175] The displacement vector restoration unit restores the displacement vector by applying the decoded displacement vector transformation coefficients to inverse quantization and inverse transformation processes. If the restored displacement vector is a value in the local coordinate system, an inverse transformation to the Cartesian coordinate system can be performed.

[0176] The mesh restoration unit can generate additional vertices by performing subdivision on the restored base mesh. Through subdivision, vertex connection information including the added vertices, texture coordinates, and connection information of the texture coordinates can be generated. The subdivided restored base mesh can be combined with the restored displacement vectors to generate the final restored mesh.

[0177] The texture map bitstream can be decoded as a video bitstream using a video codec in the texture map video decoder. The restored texture map contains color information for each vertex contained in the restored mesh, and the color value of the corresponding vertex can be retrieved from the texture map using the texture coordinates of each vertex.

[0178] The atlas bitstream can be decoded through the atlas decoding unit.

[0179] The restored mesh and texture map are displayed to the user through a rendering process using a mesh data renderer, etc.

[0180] The decoder receives the encoded bitstream and decodes the base mesh, displacement vector, attributes, and atlas within the bitstream based on the parameter information (which may be referred to as signaling information, metadata, etc.) contained within the bitstream. The decoding process may follow the inverse of the encoding process. Based on the decoded atlas, a mesh is reconstructed from the reconstructed base mesh and the reconstructed displacement mesh. Based on the reconstructed mesh and the reconstructed attributes, the mesh can be rendered.

[0181] A point cloud data encoding device and method according to the embodiments can encode mesh data and transmit a bitstream containing the encoded mesh data. A point cloud data decoding device and method according to the embodiments can receive a bitstream containing mesh data and decode the mesh data. A point cloud data encoding / decoding method / device according to the embodiments may be referred to simply as a method / device according to the embodiments. A point cloud data encoding / decoding method / device according to the embodiments may also be referred to as a mesh data encoding / decoding method / device according to the embodiments. Additionally, it may be used in this document simply as an encoding / decoding method / device.

[0182] An encoding method / device according to embodiments (e.g., encoder in FIG. 1, transmitting device (200) in FIG. 2, acquisition unit (201), encoder (202), encapsulator (203), transmitter (204), encoder (401) in FIG. 4, encoding in FIG. 7, encoding in FIG. 12, bitstream and syntax generation in FIG. 15, encoder in FIG. 16, encoding method in FIG. 18, etc.) can encode mesh data and generate a bitstream.

[0183] A decoding method / device according to embodiments (Fig. 1 decoder, Fig. 2 receiving device (210), receiving unit (211), decapsulator (212), decoder (213), renderer (214), Fig. 11, Fig. 13 decoding, Fig. 15 bitstream and syntax acquisition, Fig. 17 decoder, Fig. 19 decoding method, etc.) can decode mesh data based on parameter information contained in the bitstream.

[0184] The invention according to the embodiments relates to Video-based Dynamic Mesh Compression (V-DMC), which is a method for compressing three-dimensional dynamic mesh data using a 2D video codec. The method and apparatus according to the embodiments may include and perform a normal-based subdivision weight decoding process and a signaling method related to weight parameters according to a mesh patch type when a normal-based subdivision method is applied in V-DMC.

[0185] The method and apparatus according to the embodiments relate to Video-based Dynamic Mesh Compression (V-DMC), which is a method for compressing three-dimensional dynamic mesh data using an existing 2D video codec, and can include and perform a normal-based subdivision weight decoding process according to a mesh patch type and a signaling method related to weight parameters.

[0186] Active technical standardization of V-DMC technology has been underway since the Cfp Response in April 2022, and as of November 2024, the DIS phase is in progress, with the V-DMC version 1 standard nearing completion. In V-DMC, a single frame can be divided into multiple mesh patches, and the components of V-DMC—base mesh, displacement, texture map, and atlas tile data—can be encoded and decoded for each mesh patch and processed in parallel. When applying a normal vector-based subdivision method to V-DMC technology where there are two or more mesh patches and subdivision vertices are generated using weight parameters, the decoding process must be defined according to the mesh patch type (Skip, Merge, Intra, Inter), and the weight parameters must be signaled. Currently, the V-DMC standard (ISO / IEC 23090-29) does not define a normal-based subdivision weight decoding process based on mesh patch type.

[0187] The method and apparatus according to the embodiments may include and perform a normal-based subdivision weight decoding process for each mesh patch type (Skip, Merge, Intra, Inter). Additionally, the method and apparatus according to the embodiments may include and perform a signaling method related to a normal weight parameter.

[0188] The normal-based subdivision method according to the embodiments is a method of subdividing a base mesh based on the normal vectors of the vertices of the base mesh, and may be interpreted as other terms having the same or similar meaning within the scope of understanding by a person skilled in the art. For example, normal-based subdivision may be interpreted as normal vector-based subdivision, subdivision based on normal vectors of vertices for base mesh, etc.

[0189] The structure, operation, and other features of the present invention can be understood by the embodiments of the present invention described with reference to the accompanying drawings.

[0190] FIG. 14 illustrates an example of a normal vector-based subdivision method according to embodiments.

[0191] Referring to Fig. 14, subdivision is performed on vertices v and p. If the conventional midpoint method is applied, a vertex is generated at position q. However, if the normal-based subdivision method is applied, a vertex is generated at position q' through Equation 1 below.

[0192] Formula 1:

[0193] According to the embodiments, of Formula 1 represents the weight parameter when normal subdivision is performed at the i-th LoD (Level of Detail).

[0194] According to the embodiments, It can be derived as =pow(2, subdivisionIterationIndex - subdivisionIterationCount - 1) and can be signaled through mesh patch data unit syntax.

[0195] According to the embodiments, is a variable that indicates the degree to which the normal vector of vertex v influences vertex p. It can be calculated using a formula like this. Likewise, is a variable that indicates the degree to which the normal vector of vertex p influences vertex v. It can be calculated using a formula like this.

[0196] According to the embodiments, when applying normal-based subdivision, a normal vector is calculated for all vertices for each LoD as shown in FIG. 14, and based on this, a new vertex is generated by calculating as in Equation 1 to perform subdivision.

[0197] FIG. 15 shows the mesh patch data unit syntax according to the embodiments.

[0198] The mesh patch data unit (meshpatch_data_unit) of FIG. 15 is included in the bitstream according to the embodiments.

[0199] Clause 8 of the V-DMC standard document (ISO / IEC 23090-29) defines syntax and semantics. The syntax related to mesh patches is defined in Clause 8.3.9.3 Meshpatch data unit syntax. The mesh patch data unit syntax of FIG. 15 according to the embodiments can be configured to signal weight parameters when normal-based subdivision is applied, based on Clause 8.3.9.3 Meshpatch data unit syntax of the V-DMC standard document (ISO / IEC 23090-29).

[0200] A subdivision method present flag (mdu_subdivision_method_present_flag[ tileID ][ patchIdx ]) of 1 indicates that mdu_subdivision_method is present in the mesh patch with index patchIdx within the current atlas tile that has the same tile ID as tileID. When mdu_subdivision_method_present_flag[ tileID ][ patchIdx ] does not exist, if PatchSubdivisionCount[ tileID ][ patchIdx ] is 0, mdu_subdivision_method_present_flag[ tileID ][ patchIdx ] is inferred to be 0. Otherwise, if mdu_subdivision_iteration_count_present_flag[ tileID ][ patchIdx ] is 1 and PatchSubdivisionCount[ tileID ][ patchIdx ] is not 0, mdu_subdivision_method_present_flag[ tileID ][ patchIdx ] is inferred to be 1. Otherwise, mdu_subdivision_method_present_flag[ tileID ][ patchIdx ] is inferred to be 0.

[0201] A LoD adaptive subdivision flag (mdu_lod_adaptive_subdivision_flag[ tileID ][ patchIdx ]) of 1 indicates that the subdivision method is signaled for each subdivision iteration in the mesh patch with index patchIdx within the current atlas tile with the same tile ID as tileID. A mdu_lod_adaptive_subdivision_flag of 0 indicates that the same subdivision method is applied for all subdivision iterations in the mesh patch with index patchIdx within the current atlas tile with the same tile ID as tileID. When absent, the value of mdu_lod_adaptive_subdivision_flag is inferred to be 0.

[0202] The subdivision method (mdu_subdivision_method[ tileID ][ patchIdx ][ i ]) represents the identifier of a method for subdividing the submesh associated with the mesh patch having index patchIdx within the current atlas tile having the same tileID, into a subdivision with the same subdivision index as i. When mdu_subdivision_method[ tileID ][ patchIdx ][ i ] does not exist, the variable PatchSubdivisionMethod[ tileID ][ patchIdx ][ i ] is inferred to be the same as AfpsSubdivisionMethod[ i ]. For example, if mdu_subdivision_method[ tileID ][ patchIdx ][ i ] is 4, it may indicate that normal-based subdivision is applied. i can represent each LoD level index.

[0203] The normal subdivision weight numerator (mdu_normal_subdivision_weight_numerator[ tileID ][ patchIdx ][ i ]) represents the numerator of the weight coefficients used for the refinement vector in the i-th subdivision step of normal-based subdivision for the subpart of the mesh associated with the mesh patch with index patchIdx within the current atlas tile having the same tile ID as tileID. When mdu_normal_subdivision_weight_numerator[ tileID ][ patchIdx ][ i ] does not exist, mdu_normal_subdivision_weight_numerator[ tileID ][ patchIdx ][ i ] is the same as afve_normal_subdivision_weight_numerator[ i ].

[0204] The value plus 1 added to the normal subdivision weight denominator (mdu_normal_subdivision_weight_denominator_minus1[ tileID ][ patchIdx ][ i ]) represents the denominator of the weight coefficients used for the refinement vector at the i-th subdivision step of normal-based subdivision for the mesh subpart associated with the mesh patch with index patchIdx within the current atlas tile having the same tile ID as tileID. When mdu_normal_subdivision_weight_denominator_minus1[ tileID ][ patchIdx ][ i ] does not exist, mdu_normal_subdivision_weight_denominator_minus1[ tileID ][ patchIdx ][ i ] is the same as afve_normal_subdivision_weight_denominator_minus1[ i ].

[0205] Referring to FIG. 15, the subdivision method (mdu_subdivision_method[ tileID ][ patchIdx ][ i ]), normal subdivision weight numerator (mdu_normal_subdivision_weight_numerator[ tileID ][ patchIdx ][ i ]), and normal subdivision weight denominator (mdu_normal_subdivision_weight_denominator_minus1[ tileID ][ patchIdx ][ i ]) for each LoD level are shown as being parsed when the subdivision method present flag (mdu_subdivision_method_present_flag[ tileID ][ patchIdx ]) is 1; however, otherwise, whether information regarding the subdivision method (mdu_subdivision_method) and subdivision iteration count (mdu_subdivision_iteration_count_minus1) within the mesh patch identified by the tile ID and mesh patch index exists If the indicated subdivision override flag (mdu_subdivision_override_flag) is 1, the subdivision method (mdu_subdivision_method[ tileID ][ patchIdx ][ i ]), normal subdivision weight numerator (mdu_normal_subdivision_weight_numerator[ tileID ][ patchIdx ][ i ]), and normal subdivision weight denominator (mdu_normal_subdivision_weight_denominator_minus1[ tileID ][ patchIdx ][ i ]) for each LoD level can be parsed.

[0206] Clause 9 of the V-DMC standard document (ISO / IEC 23090-29) describes the decoding process of V-DMC. The decoding process related to mesh patches is described in detail in clause 9.2.8, "Decoding process for meshpatch data units." When a normal-based subdivision is applied to a mesh patch according to the embodiments, the process of parsing and decoding weight parameters can be performed as follows, based on clause 9.2.8, "Decoding process for meshpatch data units," of the V-DMC standard document (ISO / IEC 23090-29).

[0207] Decoding process for meshpatch data units coded in intra mode

[0208] The inputs for the decoding process of mesh patch data coded in intra mode are the current patch index p and the current tile ID, TileID. That is, the decoding process for mesh patch data coded in intra mode can be performed for each mesh patch identified by the patch index p and the tile ID TileID.

[0209] Variables related to mesh patches are assigned according to the elements parsed from the mesh patch data unit:

[0210] 1) As shown below, a variable for the submesh ID for the current mesh patch (TileMeshpatchSubmeshID[ tileID ][ p ]) can be assigned to mdu_submesh_id[ tileID ][ p ], which is a syntax element parsed from a mesh patch data unit in a bitstream according to the embodiments.

[0211] TileMeshpatchSubmeshID[ tileID ][ p ] = mdu_submesh_id[ tileID ][ p ]

[0212] 2) As shown below, a variable (TileMeshpatchLoDIdx[ tileID ][ p ]) for the LoD index for the current mesh patch can be assigned to mdu_submesh_id[ tileID ][ p ], which is a syntax element parsed from a mesh patch data unit in a bitstream according to the embodiments.

[0213] TileMeshpatchLoDIdx[ tileID ][ p ] = mdu_lod_idx[ tileID ][ p ]

[0214] The process of 3)-6) below can be performed when the value of the variable (TileMeshpatchLoDIdx[ tileID ][ p ]) regarding the LoD index for the current mesh patch is 0.

[0215] 3) As shown below, a variable (TileMeshpatchSubdivCount[ tileID ][ p ]) regarding the subdivision count for the current mesh patch can be assigned to PatchSubdivisionCount[ tileID ][ p ], which is a syntax element parsed from a mesh patch data unit in a bitstream according to the embodiments. The subdivision count according to the embodiments may represent the number of times the base mesh has been subdivided. For example, the subdivision count may be the number of levels of LoDs (Levels of Detail) minus 1.

[0216] TileMeshpatchSubdivCount[ tileID ][ p ] = PatchSubdivisionCount[ tileID ][ p ]

[0217] 4) As shown below, based on the value of the variable (TileMeshpatchSubdivCount[ tileID ][ p ]) regarding the subdivision count for the current mesh patch, for each level of LoDs, the variable (TileMeshpatchSubdivMethod[ tileID ][ p ][ i ]) regarding the subdivision method for the current mesh patch can be assigned to PatchSubdivisionMethod[ tileID ][ patchIdx ][ i ], which is a syntax element parsed from the mesh patch data unit in the bitstream according to the embodiments, and the variable (TileMeshpatchNormalSubdivisionWeight[ tileID ][ p ][ i ]) regarding the normal-based subdivision weight for the current mesh patch can be assigned to PatchNormalSubdivisionWeight[ tileID ][ patchIdx ][ i ], which is a syntax element parsed from the mesh patch data unit in the bitstream according to the embodiments.

[0218] for( i = 0; i < TileMeshpatchSubdivCount[ tileID ][ p ] ; i++ ){

[0219] TileMeshpatchSubdivMethod[tileID][p][i] =

[0220] PatchSubdivisionMethod[tileID][patchIdx][i]

[0221] TileMeshpatchNormalSubdivisionWeight[tileID][p][i] =

[0222] PatchNormalSubdivisionWeight [tileID][patchIdx][i]

[0223] }

[0224] Here, i can represent the index of each level of LoDs.

[0225] 5) As shown below, a variable regarding the displacement coordinate system for the current mesh patch (TileMeshpatchDispCoordSys[ tileID ][ p ]) can be assigned to mdu_displacement_coordinate_system[ tileID ][ p ], which is a syntax element parsed from the mesh patch data unit in the bitstream according to the embodiments.

[0226] TileMeshpatchDispCoordSys[tileID][p] =

[0227] mdu_displacement_coordinate_system[tileID][p]

[0228] 6) As shown below, a variable (TileMeshpatchTransformMethod[ tileID ][ p ]) regarding the transformation method for the current mesh patch can be assigned to mdu_transform_method[ tileID ][ p ], which is a syntax element parsed from a mesh patch data unit in a bitstream according to the embodiments.

[0229] TileMeshpatchTransformMethod[ tileID ][ p ]= mdu_transform_method[ tileID ][ p ]

[0230] Decoding process for meshpatch data units coded in skip prediction mode

[0231] The decoding process for mesh patch data coded in skip prediction mode can be performed for each mesh patch identified by the patch index p and tile ID TileID.

[0232] If the value of the variable related to the LoD index for the current mesh patch (TileMeshpatchLoDIdx[ tileID ][ p ]) is 0, the following process may be performed.

[0233] 1) As shown below, the variable for the subdivision count for the current mesh patch (TileMeshpatchSubdivCount[ tileID ][ p ]) can be assigned to the variable for the subdivision count for the reference mesh patch that the current mesh patch references (refMeshpatchSubdivCount).

[0234] TileMeshpatchSubdivCount[ tileID ][ p ] = refMeshpatchSubdivCount

[0235] 2) As shown below, based on the value of the variable (TileMeshpatchSubdivCount[ tileID ][ p ]) regarding the subdivision count for the current mesh patch, for each level of LoDs, the variable (TileMeshpatchSubdivMethod[ tileID ][ p ][ i ]) regarding the subdivision method for the current mesh patch can be assigned as the variable (refMeshpatchSubdivMethod[ i ]) regarding the subdivision method for the reference mesh patch referenced by the current mesh patch. Additionally, for each level of LoDs, the variable (TileMeshpatchNormalSubdivisionWeight[ tileID ][ p ][ i ]) regarding the normal-based subdivision weight for the current mesh patch can be assigned as the variable (refMeshpatchNormalSubdivisionWeight[ i ]) regarding the normal-based subdivision weight for the reference mesh patch referenced by the current mesh patch.

[0236] for( i = 0; i < TileMeshpatchSubdivCount[ tileID ][ p ] ; i++ ){

[0237] TileMeshpatchSubdivMethod[ tileID ][ p ][ i ] = refMeshpatchSubdivMethod[ i ]

[0238] TileMeshpatchNormalSubdivisionWeight[tileID][p][i] =

[0239] refMeshpatchNormalSubdivisionWeight [i]

[0240] }

[0241] Here, i can represent the index of each level of LoDs.

[0242] 3) As shown below, a variable regarding the displacement coordinate system for the current mesh patch (TileMeshpatchDispCoordSys[ tileID ][ p ]) can be assigned to a variable regarding the displacement coordinate system for the reference mesh patch that the current mesh patch references (refMeshpatchDispCoordSys).

[0243] TileMeshpatchDispCoordSys[ tileID ][ p ] = refMeshpatchDispCoordSys

[0244] 4) As shown below, a variable regarding the transformation method for the current mesh patch (TileMeshpatchTransformMethod[ tileID ][ p ]) can be assigned to a variable regarding the transformation method for the reference mesh patch that the current mesh patch references (refMeshpatchTransformMethod).

[0245] TileMeshpatchTransformMethod[ tileID ][ p ] = refMeshpatchTransformMethod

[0246] Decoding process for meshpatch data units coded in merge prediction mode

[0247] The decoding process for mesh patch data coded in merge prediction mode can be performed for each mesh patch identified by patch index p and tile ID TileID.

[0248] In the decoding process for mesh patch data coded in merge prediction mode, variables can be derived as follows.

[0249] 1) As shown below, a variable for the submesh ID of the current mesh patch (TileMeshpatchSubmeshID[ tileID ][ p ]) can be assigned to a variable for the submesh ID of the reference mesh patch that the current mesh patch references (refMeshpatchSubmeshID).

[0250] TileMeshpatchSubmeshID[tileID][p] = refMeshpatchSubmeshID

[0251] 2) As shown below, a variable for the LoD index for the current mesh patch (TileMeshpatchLoDIdx[ tileID ][ p ]) can be assigned to mmdu_lod_idx [ tileID ][ p ], which is a syntax element parsed from a merge mesh patch data unit in a bitstream according to the embodiments.

[0252] TileMeshpatchLoDidx[ tileID ][ p ] = mmdu_lod_idx[ tileID ][ p ]

[0253] 3) As shown below, when the value of the variable (TileMeshpatchLoDIdx[ tileID ][ p ]) for the LoD index for the current mesh patch is 0, the variable (TileMeshpatchSubdivCount[ tileID ][ p ]) for the subdivision count for the current mesh patch can be assigned to MergePatchSubdivisionCount[ tileID ][ p ], which is a syntax element parsed from the merge mesh patch data unit in the bitstream according to the embodiments.

[0254] Additionally, based on the value of the variable (TileMeshpatchSubdivCount[ tileID ][ p ]) regarding the subdivision count for the current mesh patch, for each level of LoDs, the variable (TileMeshpatchSubdivMethod[ tileID ][ p ][ i ]) regarding the subdivision method for the current mesh patch can be assigned to the variable (refMeshpatchSubdivMethod[ i ]) regarding the subdivision method for the reference mesh patch referenced by the current mesh patch, and the variable (TileMeshpatchNormalSubdivisionWeight[ tileID ][ p ][ i ]) regarding the normal-based subdivision weight for the current mesh patch can be assigned to the variable (refMeshpatchNormalSubdivisionWeight[ i ]) regarding the normal-based subdivision weight for the reference mesh patch referenced by the current mesh patch.

[0255] In addition, a variable regarding the displacement coordinate system for the current mesh patch (TileMeshpatchDispCoordSys[ tileID ][ p ]) can be assigned to a variable regarding the displacement coordinate system for the reference mesh patch that the current mesh patch references (refMeshpatchDispCoordSys).

[0256] In addition, the variable for the transformation method for the current mesh patch (TileMeshpatchTransformMethod[ tileID ][ p ]) can be assigned to the variable for the transformation method for the reference mesh patch that the current mesh patch references (refMeshpatchTransformMethod).

[0257] if(TileMeshpatchLoDIdx[ tileID ][ p ] == 0) {

[0258] TileMeshpatchSubdivCount[tileID][p]=

[0259] MergePatchSubdivisionCount[tileID][p]

[0260] for( i = 0; i < TileMeshpatchSubdivCount[ tileID ][ p ] ; i++ ){

[0261] TileMeshpatchSubdivMethod[tileID][p][i] =

[0262] refMeshpatchSubdivMethod[ i ]

[0263] TileMeshpatchNormalSubdivisionWeight[tileID][p][i] =

[0264] refMeshpatchNormalSubdivisionWeight[ i ]

[0265] }

[0266] TileMeshpatchDispCoordSys[ tileID ][ p ] = refMeshpatchDispCoordSys

[0267] TileMeshpatchTransformMethod[ tileID ][ p ] = refMeshpatchTransformMethod

[0268] }

[0269] Decoding process for meshpatch data units coded in inter prediction mode

[0270] The decoding process for mesh patch data coded in inter-predicted mode can be performed for each mesh patch identified by the patch index p and tile ID TileID.

[0271] In the decoding process for mesh patch data coded in inter-prediction mode, variables can be derived as follows.

[0272] 1) As shown below, a variable for the submesh ID of the current mesh patch (TileMeshpatchSubmeshID[ tileID ][ p ]) can be assigned to a variable for the submesh ID of the reference mesh patch that the current mesh patch references (refMeshpatchSubmeshID).

[0273] TileMeshpatchSubmeshID[tileID][p] = refMeshpatchSubmeshID

[0274] 2) As shown below, a variable for the LoD index for the current mesh patch (TileMeshpatchLoDIdx[ tileID ][ p ]) can be assigned to imdu_lod_idx[ tileID ][ p ], which is a syntax element parsed from the merge mesh patch data unit in the bitstream according to the embodiments.

[0275] TileMeshpatchLoDidx[ tileID ][ p ] = imdu_lod_idx[ tileID ][ p ]

[0276] 3) As shown below, if the value of the variable (TileMeshpatchLoDIdx[ tileID ][ p ]) for the LoD index for the current mesh patch is 0, the variable (TileMeshpatchSubdivCount[ tileID ][ p ]) for the subdivision count for the current mesh patch can be assigned to the variable (refMeshpatchSubdivCount) for the subdivision count for the reference mesh patch that the current mesh patch references.

[0277] Additionally, based on the value of the variable (TileMeshpatchSubdivCount[ tileID ][ p ]) regarding the subdivision count for the current mesh patch, for each level of LoDs, the variable (TileMeshpatchSubdivMethod[ tileID ][ p ][ i ]) regarding the subdivision method for the current mesh patch can be assigned to the variable (refMeshpatchSubdivMethod[ i ]) regarding the subdivision method for the reference mesh patch referenced by the current mesh patch, and the variable (TileMeshpatchNormalSubdivisionWeight[ tileID ][ p ][ i ]) regarding the normal-based subdivision weight for the current mesh patch can be assigned to the variable (refMeshpatchNormalSubdivisionWeight[ i ]) regarding the normal-based subdivision weight for the reference mesh patch referenced by the current mesh patch.

[0278] In addition, a variable regarding the displacement coordinate system for the current mesh patch (TileMeshpatchDispCoordSys[ tileID ][ p ]) can be assigned to a variable regarding the displacement coordinate system for the reference mesh patch that the current mesh patch references (refMeshpatchDispCoordSys).

[0279] In addition, the variable for the transformation method for the current mesh patch (TileMeshpatchTransformMethod[ tileID ][ p ]) can be assigned to the variable for the transformation method for the reference mesh patch that the current mesh patch references (refMeshpatchTransformMethod).

[0280] if(TileMeshpatchLoDIdx[ tileID ][ p ] == 0){

[0281] TileMeshpatchSubdivCount[ tileID ][ p ]= refMeshpatchSubdivCount

[0282] for( i = 0; i < TileMeshpatchSubdivCount[ tileID ][ p ] ; i++ ){

[0283] TileMeshpatchSubdivMethod[ tileID ][ p ][ i ] = refMeshpatchSubdivMethod[ i ]

[0284] TileMeshpatchNormalSubdivisionWeight [tileID][p][i] =

[0285] refMeshpatchNormalSubdivisionWeight[ i ]

[0286] }

[0287] TileMeshpatchDispCoordSys[ tileID ][ p ] = refMeshpatchDispCoordSys

[0288] TileMeshpatchTransformMethod[ tileID ][ p ] = refMeshpatchTransformMethod

[0289] }

[0290] Flow Chart

[0291] FIG. 16 shows the configuration and operation of an encoder according to embodiments.

[0292] An encoder (transmitter) according to the embodiments generates a base mesh from an original mesh to be encoded (transmitted) by passing it through a simplification unit and a parameterization unit. The generated base mesh is quantized. In the case of an inter-frame, motion vectors are calculated from the previously referenced reconstructed base mesh, and the motion vectors are encoded. In the case of an intra-frame, the base mesh is encoded through a static mesh encoding unit. A base mesh bitstream containing the encoded base mesh is output and transmitted to a decoder. Then, a displacement vector is calculated between the mesh data obtained by subdividing and fitting the simplified mesh through the mesh simplification unit and the mesh data reconstructed from the previously encoded base mesh. To efficiently encode the calculated displacement vector, the displacement vector coordinate system can be converted to a local coordinate system, and the displacement vector is converted into displacement vector coefficients and quantized in the displacement vector transformation unit to be encoded into a displacement vector bitstream.

[0293] The process of subdividing the simplified base mesh according to the embodiments and the method of performing the lifting transform / inverse transform prediction process based on normal vectors during the displacement vector encoding and decoding process are described in more detail below.

[0294] In the mesh subdivision section, the subdivision process can be performed n times using a simplified base mesh as input. The number of mesh subdivisions is determined by an agreement between the encoder and decoder, or by units such as sequence, GOF (Group of Frame), frame, or submesh. Subdivision_iteration_count information can be signaled for each subdivision level (per LoD). Additionally, the subdivision method to be performed at each subdivision level can be determined. Subdivision_method information can be signaled according to the mesh subdivision method. For example, the value of subdivision_method can be set to Mid-point subdivision method = 0, normal vector-based subdivision method (no plane constraints applied) = 1, normal vector-based subdivision method (plane constraints applied) = 2, Butterfly subdivision method = 3, Loop subdivision method = 4, and LS3 (Least Squares Subdivision Surfaces) method = 5. Alternatively, if the value of subdivision_method is 4, it can be set to represent the normal vector-based subdivision method of the vertices.

[0295] When determined by the normal vector-based subdivision method proposed in the present invention, subdivision can be performed for each vertex in the manner of Equation 1. In the case of a method that also applies planar constraints, subdivision can be performed such that the distance between two vertices and adjacent faces is minimized according to Equation 2 below.

[0296] Formula 2:

[0297]

[0298]

[0299] Once the subdivision of the base mesh is completed, a fitting process is performed to make it similar to the original mesh, and the displacement vector, which is the coordinate difference with the result of subdividing the base mesh, can be calculated. The calculated displacement vector can be encoded using a 2D video codec or an arithmetic coding method. Both methods perform displacement vector transformation, and depending on the embodiment, wavelet transformation, lifting transformation, etc., may be performed. When lifting transformation is performed, the lifting transformation prediction unit can perform predictions based on normal vectors according to the embodiments as shown in Equations 3 and 4 below.

[0300] Formula 3:

[0301] Formula 4:

[0302] Normal vector-based prediction weight information can be signaled to normal_subdivision_refine_weight_numerator and normal_subdivision_refine_weight_denominator by calculating the denominator and numerator, respectively. This information can be signaled via ASPS (Atlas Sequence Parameter Set), AFPS (Atlas Frame Parameter Set), and MDU (Meshpatch Data Unit). Subsequently, a process of updating the displacement vector of the vertex used for prediction can be performed using the residual signal generated in the lifting transformation prediction unit. Once the displacement vector transformation is performed, the generated displacement vector coefficients are quantized and encoded either through a 2D video encoder or through an arithmetic encoding method according to each encoding method to generate a displacement vector subbitstream.

[0303] The texture map generation unit generates a new texture map having color information corresponding to the texture coordinates of the restored mesh, and encodes the texture map through a 2D video encoder and transmits it as a texture bitstream.

[0304] The base mesh bitstream, displacement vector bitstream, and texture bitstream generated through the entire process above at the encoder (transmitter) are combined into a single bitstream through the multiplexer and transmitted through the transmitter.

[0305] FIG. 17 shows the configuration and operation of a decoder according to embodiments.

[0306] The decoder (receiving end) according to the embodiments receives the bitstream transmitted from the encoder (transmitting end) and performs the process of decoding the base mesh bitstream, displacement vector bitstream, and texture map bitstream, respectively, through the demultiplexer.

[0307] The base mesh bitstream is decoded through the motion vector decoding unit for inter-frames and the static mesh decoding unit for intra-frames. The decoded base mesh is then processed through the restoration unit to perform mesh subdivision.

[0308] The displacement vector bitstream decodes the displacement vector coefficients in the reverse order of encoding, performs inverse quantization and inverse transformation, and then inversely transforms the coordinate system to restore the mesh geometric information along with the base mesh data.

[0309] The execution process of the normal vector-based mesh subdivision unit in the decoder according to the embodiments and the inverse transformation process of the displacement vector coefficients in the displacement vector decoder are explained in detail below.

[0310] The decoded base mesh is subdivided through the mesh subdivision unit. By parsing the subdivision_iteration_count from the bitstream, one can determine at which level subdivision is performed, and by parsing the subdivision_method for each subdivision level (LoD), one can determine how subdivision is performed. For example, if the value is 0, the Mid-point subdivision method is performed; if the value is 1, the Normal Vector-based subdivision method (without plane constraints applied) is performed; if the value is 2, the Normal Vector-based subdivision method (with plane constraints applied) is performed; if the value is 3, the Butterfly subdivision method is performed; if the value is 4, the Loop subdivision method is performed; and if the value is 5, the LS3 (Least Squares Subdivision Surfaces) method is performed. Conversely, if the value is 4, the vertex Normal Vector-based subdivision method can be performed.

[0311] Segmentation is performed for each LoD level using the parsed method, and the result can be restored as final geometric information along with the final restored displacement vector in the mesh restoration unit.

[0312] In the displacement vector decoder, dispEncType is parsed from the displacement vector subbitstream to determine whether it was encoded using a 2D video codec or an arithmetic coding method, and profilesetIdx is parsed to determine the profile information defined between the decoder and the decoder. If dispEncType is a 2D video codec, decoding is performed through the 2D video codec, and inverse packing and inverse quantization of the displacement vector transform coefficients are performed. Subsequently, inverse displacement vector transform is performed. If dispEncType is an arithmetic coding method, arithmetic coding decoding is performed, and displacement vector prediction and restoration, along with displacement vector inverse quantization, are performed. Subsequently, inverse displacement vector transform is performed.

[0313] During the process of performing the displacement vector inverse transformation, the lifting inverse transformation is performed as many times as subdivision_iteration_count. In the lifting inverse transformation update section, the process of updating the vertex displacement vector used for prediction is performed using the parsed residual signal. Subsequently, the results of performing the prediction can be obtained by applying the parsed normal_subdivision_refine_weight_numerator and normal_subdivision_refine_weight_denominator information from the lifting inverse transformation prediction section to Equation 3.

[0314] In the mesh restoration unit, the vertex geometry of the restored mesh is calculated by adding the restoration displacement vector to the vertices generated through the subdivision process in the mesh subdivision unit, thereby restoring the final geometry. The received texture map bitstream is decoded through the texture map decoding unit, and together with the previously restored geometry, the final restored mesh is generated.

[0315] FIG. 18 illustrates a encoding method according to embodiments.

[0316] The encoding method according to the embodiments may include the step of encoding a base mesh of mesh data (S1800), the step of encoding a displacement of mesh data (S1810), and the step of encoding attributes of mesh data (S1820).

[0317] The encoding method according to the embodiments may include the method described in FIGS. 1 to 17 above. Specifically, the step of encoding the base mesh of the mesh data (S1800) may include the method described in FIGS. 1 to 5, FIG. 7, FIG. 12, FIG. 14 to 16, etc. above. The step of encoding the displacement of the mesh data (S1810) may include the method described in FIGS. 1 to 4, FIGS. 6 to 10, FIG. 12, FIG. 16, etc. above. The step of encoding the attributes of the mesh data (S1820) may include the method described in FIGS. 1 to 4, FIG. 7, FIG. 10, FIG. 12, FIG. 16, etc. above.

[0318] Referring together with FIG. 16, the encoding method according to the embodiments further includes the step of subdividing the decoded basemesh based on normal vectors of vertices for the basemesh, and the step of subdividing the basemesh based on normal vectors of vertices for the basemesh may include the step of generating a new vertex for the basemesh based on a normal vector of a first vertex of the basemesh, a normal vector of a second vertex of the basemesh, and a weight for the subdivision based on normal vectors of vertices.

[0319] Referring together with FIG. 15, the encoding method according to the embodiments further includes the step of generating information representing a subdivision method for a base mesh, and based on a first value of the information representing the subdivision method, the method may further include the step of generating information about the numerator of the weight for a vertex normal vector-based subdivision; and the step of generating information about the denominator of the weight for a vertex normal vector-based subdivision.

[0320] Referring together with FIG. 16, the encoding method according to the embodiments may include the steps of: subdividing a base mesh to generate levels of detail (LoDs) for the base mesh; generating information representing a subdivision count for the base mesh; and generating information about a subdivision method based on the information representing the subdivision count.

[0321] Referring together with FIG. 16, the encoding method according to the embodiments can encode displacements based on the subdivided basemesh.

[0322] The encoding method is performed by an encoding device. The encoding device includes a memory; and at least one processor connected to the memory; and the at least one processor may be configured to: encode base mesh data of the mesh data; encode geometry data of the mesh data; and encode attribute data of the mesh data.

[0323] The embodiments may further include a computer-readable storage medium that stores a bitstream generated by the method of FIG. 18.

[0324] The embodiments further include a method comprising the steps of: acquiring a bitstream for mesh data; generating the bitstream based on the steps of: encoding a base mesh of the mesh data; encoding a displacement of the mesh data; and encoding attributes of the mesh data; and transmitting data including the bitstream.

[0325] FIG. 19 illustrates a decoding method according to embodiments.

[0326] The decoding method according to the embodiments may include the step of decoding a base mesh in a bitstream (S1900); the step of decoding a displacement in a bitstream (S1910); and the step of decoding an attribute in a bitstream (S1920).

[0327] The decoding method according to the embodiments may include the method described in FIGS. 1 to 17 above. Specifically, the step of decoding a base mesh in a bitstream (S1900) may include the method described in FIGS. 1 to 2, FIG. 11, FIGS. 13 to 15, FIG. 17, etc. above. The step of decoding a displacement in a bitstream (S1910) may include the method described in FIGS. 1 to 2, FIG. 11, FIG. 13, FIG. 17, etc. above. The step of decoding an attribute in a bitstream (S1920) may include the method described in FIGS. 1 to 2, FIG. 11, FIG. 13, FIG. 17, etc. above.

[0328] Referring together with FIG. 17, the decoding method according to the embodiments further includes the step of subdividing the decoded basemesh based on normal vectors of vertices for the basemesh, and the step of subdividing the decoded basemesh based on normal vectors of vertices for the basemesh may include the step of generating a new vertex for the decoded basemesh based on a normal vector of a first vertex of the decoded basemesh, a normal vector of a second vertex of the decoded basemesh, and a weight for the subdivision based on normal vectors of vertices.

[0329] Referring together with FIG. 15, the decoding method according to the embodiments comprises a bitstream including a mesh patch data unit, the mesh patch data unit including information about a subdivision method, and based on a first value of the information about the subdivision method, the bitstream may further include information about the numerator of the weight for a vertex normal vector-based subdivision and information about the denominator of the weight for a vertex normal vector-based subdivision.

[0330] Referring together with FIG. 15, the decoding method according to the embodiments is based on a first value of information regarding a subdivision method, wherein the bitstream includes information regarding weights for a vertex normal vector-based subdivision, and the information regarding weights for a vertex normal vector-based subdivision can be derived based on information regarding the numerator of the weights for a vertex normal vector-based subdivision and information regarding the denominator of the weights for a vertex normal vector-based subdivision.

[0331] The decoding method according to the embodiments further includes the step of decoding a mesh patch identified by a tile ID and a patch index based on an intra mode, and the step of decoding the mesh patch based on an intra mode may include: the step of parsing subdivision count information for the mesh patch from a mesh patch data unit in a bitstream; and the step of parsing weight information for a vertex normal vector-based subdivision for the mesh patch from a mesh patch data unit based on the subdivision count information.

[0332] The decoding method according to the embodiments further includes the step of decoding a mesh patch identified by a tile ID and a patch index based on a skip prediction mode, a merge prediction mode, or an inter prediction mode, and the step of decoding the mesh patch based on a skip prediction mode, a merge prediction mode, or an inter prediction mode may include the step of assigning a value of information for a weight for the subdivision based on normal vectors of vertices for the mesh patch to a value of information for a weight for the subdivision based on normal vectors of vertices for the reference mesh patch based on a subdivision count value for the mesh patch.

[0333] The decoding method is performed by a decoding device. The decoding device includes a memory; and at least one processor connected to the memory; and the at least one processor may be configured to: decode a base mesh in a bitstream; decode a displacement in a bitstream; and decode an attribute in a bitstream.

[0334] In the current V-DMC technology, a new subdivision method utilizing the normal vectors of neighboring vertices has been added, replacing the existing midpoint subdivision. This method is a technique that uses normal vectors to ensure the subdivided mesh resembles the original mesh. When this normal vector-based subdivision method is applied, applying weights enables a more precise and detailed subdivision process. To perform this weighted normal-based subdivision method—especially to apply it to mesh patches—it is necessary to define the decoding process, syntax, and semantics for the mesh patches accordingly.

[0335] The invention according to the embodiments may include decoding processes and syntax and semantics related to weight parameters for each mesh patch type (Inter, Intra, Skip, Merge). According to the decoding processes and syntax and semantics related to weight parameters for each mesh type according to the embodiments, a subdivided mesh similar to the original mesh can be obtained through a more sophisticated and detailed mesh subdivision process, and as a result, the bit amount of displacement data, which is the difference from the original mesh data, can also be reduced. That is, according to the decoding processes and syntax and semantics related to weight parameters for each mesh type according to the embodiments, the effect of reducing the bit amount and obtaining mesh data of sophisticated quality can be achieved.

[0336] The embodiments have been described in terms of methods and / or devices, and the description of the methods and the description of the devices may be applied complementarily.

[0337] Although the drawings have been described separately for the convenience of explanation, it is also possible to design a new embodiment by combining the embodiments described in each drawing. Furthermore, designing a computer-readable recording medium containing a program for executing the previously described embodiments, as required by a person skilled in the art, falls within the scope of the embodiments. The apparatus and method according to the embodiments are not limited to the configuration and method of the embodiments described above; rather, the embodiments may be configured by selectively combining all or part of each embodiment to allow for various modifications. Although preferred embodiments have been illustrated and described, the embodiments are not limited to the specific embodiments described above. It is not only possible for a person skilled in the art to make various modifications without departing from the essence of the embodiments claimed in the claims, but such modifications should not be understood individually from the technical concept or perspective of the embodiments.

[0338] Various components of the device of the embodiments may be implemented by hardware, software, firmware, or a combination thereof. Various components of the embodiments may be implemented as a single chip, for example, a single hardware circuit. Depending on the embodiments, the components according to the embodiments may each be implemented as separate chips. Depending on the embodiments, at least one of the components of the device according to the embodiments may be composed of one or more processors capable of executing one or more programs, and one or more programs may include instructions for performing or executing any one or more of the operations / methods according to the embodiments. Executable instructions for performing the methods / operations of the device according to the embodiments may be stored in non-transient CRMs or other computer program products configured to be executed by one or more processors, or may be stored in transient CRMs or other computer program products configured to be executed by one or more processors. Additionally, memory according to the embodiments may be used as a concept that includes not only volatile memory (e.g., RAM, etc.) but also non-volatile memory, flash memory, PROM, etc. In addition, it may also include implementation in the form of carrier waves, such as transmission over the Internet. Furthermore, processor-readable recording media are distributed across networked computer systems, allowing processor-readable code to be stored and executed in a distributed manner.

[0339] In this document, " / " and "," are interpreted as "and / or." For example, "A / B" is interpreted as "A and / or B," and "A, B" is interpreted as "A and / or B." Additionally, "A / B / C" means "at least one of A, B and / or C." Also, "A, B, C" means "at least one of A, B and / or C." Additionally, in this document, "or" is interpreted as "and / or." For example, "A or B" may mean 1) "A" only, 2) "B" only, or 3) "A and B." In other words, "or" in this document may mean "additionally or alternatively."

[0340] Terms such as "first," "second," etc., may be used to describe various components of the embodiments. However, the interpretation of the various components according to the embodiments should not be limited by these terms. These terms are merely used to distinguish one component from another. For example, the first user input signal may be referred to as the second user input signal. Similarly, the second user input signal may be referred to as the first user input signal. The use of these terms should be interpreted as not departing from the scope of the various embodiments. Although the first user input signal and the second user input signal are both user input signals, they do not mean the same user input signals unless clearly indicated in the context.

[0341] The terms used to describe the embodiments are intended for the purpose of describing specific embodiments and are not intended to limit the embodiments. As used in the description of the embodiments and in the claims, the singular is intended to include the plural unless explicitly indicated in the context. Expressions of and / or are used to mean including all possible combinations between the terms. Expressions of include describe the presence of features, numbers, steps, elements, and / or components and do not imply the exclusion of additional features, numbers, steps, elements, and / or components. Conditional expressions such as "if" or "when" used to describe the embodiments are not limited to being optional. It is intended to be interpreted as "when a specific condition is satisfied," "when a related action is performed in response to a specific condition," or "when a related definition is interpreted."

[0342] Additionally, operations according to the embodiments described herein may be performed by a transmitting and receiving device including memory and / or a processor, depending on the embodiments. The memory may store programs for processing / controlling operations according to the embodiments, and the processor may control various operations described in this document. The processor may be referred to as a controller, etc. Operations in the embodiments may be performed by firmware, software, and / or a combination thereof, and the firmware, software, and / or a combination thereof may be stored in the processor or in memory.

[0343] Meanwhile, the operation according to the embodiments described above may be performed by a transmitting device and / or a receiving device according to the embodiments. The transmitting and receiving device may include a transmitting and receiving unit for transmitting and receiving media data, a memory for storing instructions (program code, algorithm, flowchart and / or data) for a process according to the embodiments, and a processor for controlling the operations of the transmitting and receiving devices.

[0344] The processor may be referred to as a controller, etc., and may correspond, for example, to hardware, software, and / or a combination thereof. The operation according to the embodiments described above may be performed by the processor. Additionally, the processor may be implemented as an encoder / decoder, etc., for the operation of the embodiments described above.

[0345]

[0346]

[0347] As described above, the relevant details have been explained in the best mode for carrying out the embodiments.

[0348]

[0349] As described above, the embodiments can be applied wholly or partially to point cloud data transmission and reception devices and systems.

[0350] Those skilled in the art may make various changes or modifications to the embodiments within the scope of the embodiments.

[0351] The embodiments may include modifications / variations, and such modifications / variations do not exceed the scope of the claims and their equivalents.

Claims

1. Step of decoding the basemesh within the bitstream; A step of decoding displacement within the bitstream; and A step of decoding attributes within the bitstream; comprising Decryption method.

2. In paragraph 1, the above method is, It further includes the step of subdividing the decoded basemesh based on normal vectors of vertices for the basemesh, and The step of subdividing the above-decoded base mesh based on the normal vectors of the vertices for the base mesh is, A method comprising the step of generating a new vertex for the decoded base mesh based on a normal vector of a first vertex of the decoded base mesh, a normal vector of a second vertex of the decoded base mesh, and a weight for the subdivision based on normal vectors of vertices. Decryption method.

3. In Paragraph 2, The above bitstream includes a mesh patch data unit, and The above mesh patch data unit includes information regarding the subdivision method, and Based on a first value of information regarding the above subdivision method, the bitstream further includes information regarding the numerator of the weight for the normal vector-based subdivision of the vertex and information regarding the denominator of the weight for the normal vector-based subdivision of the vertex. Decryption method.

4. In Paragraph 3, Based on the first value of the information regarding the above subdivision method, the bitstream includes information regarding weights for the normal vector-based subdivision of the vertex, and Information regarding the weights for the normal vector-based subdivision of the above vertex is derived based on information regarding the numerator of the weights for the normal vector-based subdivision of the above vertex and information regarding the denominator of the weights for the normal vector-based subdivision of the above vertex. Decryption method.

5. In paragraph 1, the above method is, It further includes a step of decoding mesh patches identified by tile IDs and patch indexes based on intra mode, and The step of decoding the above mesh patch based on intra mode is, A step of parsing subdivision count information for the mesh patch from the mesh patch data unit within the bitstream; and Based on the above subdivision count information, the method comprises the step of parsing weight information for a vertex normal vector-based subdivision for the mesh patch from the mesh patch data unit. Decryption method.

6. In paragraph 1, the above method is, The method further includes a step of decoding a mesh patch identified by a tile ID and a patch index based on a skip prediction mode, a merge prediction mode, or an inter prediction mode. The step of decoding the above mesh patch based on a skip prediction mode, a merge prediction mode, or an inter prediction mode is: A method comprising the step of assigning, based on a subdivision count value for the mesh patch, a weight information value for a weight for subdivision based on normal vectors of vertices for the mesh patch to a weight information value for a weight for the subdivision based on normal vectors of vertices for a reference mesh patch. Decryption method.

7. Memory; and At least one processor connected to the memory; comprising, wherein the at least one processor: Decoding the basemesh within the bitstream; Decoding displacement within the above bitstream; and Configured to decode attributes within the above bitstream, Decoding device.

8. Step of encoding the base mesh of the mesh data; A step of encoding the displacement of the above mesh data; and A step of encoding attributes of the above mesh data; comprising Encoding method.

9. In paragraph 8, the above method is, It further includes the step of subdividing the decoded basemesh based on normal vectors of vertices for the basemesh, and The step of subdividing the above base mesh based on the normal vectors of the vertices for the above base mesh is, A method comprising the step of generating a new vertex for the base mesh based on a normal vector of a first vertex of the base mesh, a normal vector of a second vertex of the base mesh, and a weight for the subdivision based on normal vectors of vertices. Encoding method.

10. In paragraph 9, the above method is, The method further includes the step of generating information indicating a subdivision method for the above base mesh, and Based on the first value of the information representing the above subdivision method, the method is, A step of generating information about the numerator of the weight for the normal vector-based subdivision of the above vertex; and A step further comprising generating information on the denominator of the weights for the normal vector-based subdivision of the above vertex. Encoding method.

11. In paragraph 9, the above method is, A step of subdividing the base mesh to generate levels of detail (LoDs) for the base mesh; A step of generating information indicating a subdivision count for the above base mesh; and A method comprising the step of generating information about the subdivision method based on information representing the subdivision count. Encoding method.

12. In Paragraph 11, The above displacement is encoded based on the subdivided basemesh, Encoding method.

13. Memory; and At least one processor connected to the memory; comprising, wherein the at least one processor: Encode the base mesh of the mesh data; Encoding the displacement of the above mesh data; and Configured to encode the attributes of the above mesh data, Encoding device.

14. A computer-readable storage medium for storing a bitstream generated by the method according to paragraph 8 (encoding).

15. Step of acquiring a bitstream for mesh data, The bitstream is generated based on the steps of: encoding the base mesh of the mesh data; encoding the displacement of the mesh data; and encoding the attributes of the mesh data; and A method comprising the step of transmitting data including the bitstream above.

Images