Mesh data encoding device, mesh data encoding method, mesh data decoding device, and mesh data decoding method

Abstract

A decoding method according to embodiments may comprise the steps of: decoding base mesh data in a bitstream; decoding displacement data in the bitstream; and decoding attribute data in the bitstream. An encoding method according to embodiments may comprise the steps of: encoding base mesh data of mesh data; encoding displacement data of the mesh data; and encoding attribute data of the mesh data.

Description

Mesh data encoding device, mesh data encoding method, mesh data decoding device and mesh data decoding method

[0001] The embodiments provide a method for providing Point Cloud content to provide various services to users, such as VR (Virtual Reality), AR (Augmented Reality), MR (Mixed Reality), and autonomous driving services.

[0002] A point cloud is a collection of points in 3D space. There is a problem in that it is difficult to generate point cloud data because there are many points in 3D space.

[0003] Mesh data refers to a form of data in which connectivity information between the vertices of a mesh is added to point cloud data.

[0004] There is a problem in that a large amount of throughput is required to transmit and receive dynamic mesh data.

[0005] The technical problem according to the embodiments is to provide a point mesh data transmission device, a transmission method, a mesh data reception device, and a reception method for efficiently transmitting and receiving mesh data in order to solve the aforementioned problems, etc.

[0006] The technical problem according to the embodiments is to provide a point mesh transmission device, a transmission method, a mesh data receiving device, and a receiving method for solving latency and encoding / decoding complexity.

[0007] However, the scope of rights of the embodiments is not limited to the technical problems described above, and may be extended to other technical problems that can be inferred by a person skilled in the art based on the entire content of this document.

[0008] To achieve the above-described purpose and other advantages, the decoding method according to the embodiments may include the step of decoding base mesh data in a bitstream; the step of decoding displacement data in a bitstream; and the step of decoding attribute data in a bitstream. The encoding method according to the embodiments may include the step of encoding base mesh data of mesh data; the step of encoding displacement data of mesh data; and the step of encoding attribute data of mesh data.

[0009] A mesh data transmission method, a transmission device, a mesh data reception method, and a reception device according to the embodiments can provide a high-quality mesh service.

[0010] A mesh data transmission method, a transmission device, a mesh data reception method, and a reception device according to the embodiments can achieve various video codec methods.

[0011] The mesh transmission method, transmission device, mesh data reception method, and reception device according to the embodiments can provide general-purpose mesh content such as autonomous driving services.

[0012] Drawings are included to further understand the embodiments, and the drawings illustrate the embodiments along with descriptions related to the embodiments. For a better understanding of the various embodiments described below, one must refer to the description of the embodiments below in relation to the following drawings, which include parts corresponding to similar reference numerals throughout the drawings.

[0013] FIG. 1 shows a V-DMC-based encoder and decoder according to embodiments.

[0014] FIG. 2 shows a system for providing dynamic mesh content according to embodiments.

[0015] FIG. 3 illustrates a V-MESH compression method according to embodiments.

[0016] FIG. 4 shows the pre-processing of V-MESH compression according to the embodiments.

[0017] FIG. 5 illustrates a mid-edge subdivision method according to embodiments.

[0018] FIG. 6 illustrates a displacement generation process according to embodiments.

[0019] FIG. 7 illustrates the V-DMC encoding process according to the embodiments.

[0020] FIG. 8 illustrates a lifting conversion process for displacement according to embodiments.

[0021] FIG. 9 illustrates the process of packing conversion coefficients according to embodiments into a 2D image.

[0022] FIG. 10 illustrates the attribute transfer process of the V-MESH compression method according to the embodiments.

[0023] FIG. 11 illustrates a V-DMC decoding process according to embodiments.

[0024] FIG. 12 illustrates a V-DMC encoding process according to embodiments.

[0025] FIG. 13 illustrates a V-DMC decoding process according to embodiments.

[0026] FIG. 14 shows an encoded dynamic bitstream structure according to embodiments.

[0027] FIG. 15 shows a general displacement sequence parameter set low byte sequence payload syntax according to embodiments.

[0028] FIG. 16 illustrates a encoding method according to embodiments.

[0029] FIG. 17 illustrates a decoding method according to embodiments.

[0030] Preferred embodiments of the embodiments are described in detail, and examples thereof are shown in the accompanying drawings. The following detailed description, with reference to the accompanying drawings, is intended to describe preferred embodiments of the embodiments rather than merely embodiments that may be implemented according to the embodiments. The following detailed description includes details to provide a thorough understanding of the embodiments. However, it is obvious to those skilled in the art that the embodiments may be practiced without these details.

[0031] Most terms used in the embodiments are selected from those commonly used in the field, but some terms are chosen at the applicant's discretion, and their meanings are described in detail in the following description as necessary. Accordingly, the embodiments should be understood based on the intended meaning of the terms, rather than their mere names or meanings.

[0032] FIG. 1 shows a V-DMC-based encoder and decoder according to embodiments.

[0033] The basic structure of the currently ongoing V-DMC (v-mesh) is shown in Fig. 1. The encoder and decoder according to Fig. 1 perform the encoding and decoding processes of media representing a dynamic mesh using V3C technology. The preprocessor converts the input dynamic mesh representation into several V3C components (base mesh, displacement set, 2D representation of attributes, and atlas). The original mesh is simplified into a base mesh. The base mesh can be encoded using any mesh codec. Displacement vectors can be represented by a profile or encoded into V3C geometric video components using any video codec via SEI messages. For example, depending on the profile, displacement vectors (displacement data) can be encoded using arithmetic coding. Attribute data may include additional attributes. For example, texture or material information may be included as additional attributes and can be encoded using any video codec. Atlas data contains information on how to perform inverse reconstruction and is provided to the V3C decoding and / or rendering system. For example, atlas data may include methods for performing subdivision of the base mesh, methods for applying displacement vectors to the vertices of the subdivided mesh, methods for applying attributes to the reconstructed mesh, etc.

[0034] The encoder may be composed of a memory and at least one processor connected to the memory. The at least one processor may be configured to perform operations such as a preprocessor, an atlas encoder, a basemesh encoder, a displacement vector encoder, a video encoder, and a multiplexer.

[0035] The atlas encoding unit encodes the atlas of the mesh data to generate an atlas bitstream. The basemesh encoding unit encodes the basemesh of the mesh data to generate a basemesh bitstream. The displacement vector encoding unit encodes the displacement vector of the mesh data to generate a displacement vector bitstream. The video encoding unit encodes the attributes of the mesh data to generate an attribute bitstream. The encoder generates parameter information (which may be referred to as signaling information, metadata, etc.) related to each encoding. The encoder can generate a bitstream containing parameter information, the atlas, the basemesh, the displacement vector, and / or attributes.

[0036] The decoder may be composed of a memory and at least one processor connected to the memory. The at least one processor may be configured to perform operations such as a demultiplexer, an atlas decoder, a basemesh decoder, a displacement vector decoder, and a video decoder.

[0037] The atlas decoder decodes the atlas within the bitstream. The basemesh decoder decodes the basemesh within the bitstream. The displacement vector decoder decodes the displacement vector within the bitstream. The video decoder decodes the attributes within the bitstream. The decoder can perform each decoding operation based on parameter information within the bitstream. The decoder can reconstruct dynamic mesh data based on the atlas, displacement vector, attributes, and basemesh.

[0038] Below, the operation of the V-DMC encoder and decoder of FIG. 1 is explained in more detail.

[0039] FIG. 2 shows a system for providing dynamic mesh content according to embodiments.

[0040] The system of FIG. 2 includes a point cloud data transmission device (100) and a point cloud data reception device (110) according to embodiments. The point cloud data transmission device may include a dynamic mesh video acquisition unit (101), a dynamic mesh video encoder (102), a file / segment encapsulator (103), and a transmitter (104). The point cloud data reception device (110) may include a receiver (111), a file / segment decapsulator (112), a dynamic mesh video decoder (113), and a renderer (114). Each component of FIG. 1 may correspond to hardware, software, a processor, and / or a combination thereof. Hereinafter, the point cloud data transmission device according to embodiments may be interpreted as a term referring to the transmission device (100) or the dynamic mesh video encoder (hereinafter, encoder) (102). The point cloud data receiving device according to the embodiments may be interpreted as a term referring to the receiving device (110) or the dynamic mesh video decoder (hereinafter, decoder) (113).

[0041] The system of Fig. 2 can perform video-based dynamic mesh compression and decompression.

[0042] With advancements in 3D capture, modeling, and rendering, users can access various forms of 3D content, such as AR, XR, the metaverse, and holograms, across multiple platforms and devices. 3D content represents objects more sophisticatedly and realistically to enable users to enjoy immersive experiences, and for this purpose, the creation and use of 3D models require a large amount of data. Among the various types of 3D content, 3D meshes are widely used for efficient data utilization and realistic object representation. The embodiments include a series of processing steps in a system that uses such mesh content.

[0043] First, the method for compressing dynamic mesh data originates from the V-PCC (Video-based point cloud compression) standard technology. Point cloud data consists of data containing color information along with vertex coordinates (X, Y, Z). Mesh data refers to data where connectivity information between vertices is added to this vertex data. Content can be created in a mesh data format from the outset when generating content. Point cloud data can be converted into mesh data and used by adding connectivity information.

[0044] Currently, the MPEG standards organization defines the data types for dynamic mesh data as the following two types: Category 1: Mesh data containing texture maps as color information. Category 2: Mesh data containing vertex colors as color information.

[0045] Mesh coding standards for Category 1 data are currently being developed, and standardization work for Category 2 data is also scheduled to proceed in the future. As shown in Fig. 1, the entire process for providing mesh content services may include an acquisition process, an encoding process, a transmission process, a decoding process, a rendering process, and / or a feedback process.

[0046] To provide mesh content services, 3D data acquired through multiple cameras or special cameras can be processed into a mesh data type through a series of processes and then generated as a video. The generated mesh video is transmitted after undergoing a series of processes, and at the receiving end, the received data can be processed back into a mesh video and rendered. Through this, the mesh video is provided to the user, and the user can use the mesh content according to their intention through interaction.

[0047] A mesh compression system may include a transmission device and a reception device. The transmission device can encode mesh video to output a bitstream and transmit it to the reception device via a digital storage medium or network in the form of a file or streaming (streaming segment). The digital storage medium may include various storage media such as USB, SD, CD, DVD, Blu-ray, HDD, and SSD.

[0048] The transmission device may schematically include a mesh video acquisition unit, a mesh video encoder, and a transmission unit. The receiving device may schematically include a receiver, a mesh video decoder, and a renderer. The encoder may be referred to as a mesh video / image / picture / frame encoding device, and the decoder may be referred to as a mesh video / image / picture / frame decoding device. The transmitter may be included in the mesh video encoder. The receiver may be included in the mesh video decoder. The renderer may include a display unit, and the renderer and / or the display unit may be composed of separate devices or external components. The transmission device and the receiving device may further include separate internal or external modules / units / components for a feedback process.

[0049] Mesh data represents the surface of an object using multiple polygons. Each polygon is defined by a vertex in 3D space and connectivity information indicating how those vertices are connected. It may also include vertex attributes such as color and normals. Mapping information, which enables the surface of the mesh to be mapped onto a 2D planar area, can also be included as an attribute of the mesh. Mapping can generally be described by a set of parametric coordinates, referred to as UV coordinates or texture coordinates, associated with the mesh vertices. The mesh contains a 2D attribute map, which can be used to store high-resolution attribute information such as textures, normals, and displacements.

[0050] The mesh video acquisition unit may include processing 3D object data acquired through a camera, etc., into a mesh data type having the attributes described above through a series of processes, and generating a video composed of such mesh data. In a mesh video, the attributes of the mesh, namely vertices, polygons, connectivity information between vertices, color, normals, etc., may change over time. A mesh video having attributes and connectivity information that change over time in this way can be described as a dynamic mesh video.

[0051] A mesh video encoder can encode input mesh video into one or more video streams. A single video may contain multiple frames, and a single frame may correspond to a still image or picture. In this document, the term "mesh video" may include mesh images, frames, or pictures, and the terms mesh video and mesh images, frames, or pictures may be used interchangeably. A mesh video encoder can perform a Video-based Dynamic Mesh (V-Mesh) Compression procedure. To improve compression and coding efficiency, a mesh video encoder can perform a series of procedures such as prediction, transformation, quantization, and entropy coding. The encoded data (encoded video / image information) can be output in the form of a bitstream.

[0052] The file / segment encapsulation module can encapsulate encoded mesh video data and / or mesh video-related metadata into a file or the like. Here, the mesh video-related metadata may be received from a metadata processing module or the like. The metadata processing module may be included in the mesh video encoder or may be configured as a separate component / module. The encapsulation module can encapsulate the data into a file format such as ISOBMFF or process it into other forms such as DASH segments. Depending on the embodiment, the encapsulation module may include mesh video-related metadata in the file format. Mesh video metadata may be included, for example, in various levels of boxes in the ISOBMFF file format or as data within a separate track in the file. Depending on the embodiment, the encapsulation module may encapsulate the mesh video-related metadata itself into a file.

[0053] The transmission processing unit can apply transmission processing to mesh video data encapsulated according to the file format. The transmission processing unit may be included in the transmission unit or may be configured as a separate component / module. The transmission processing unit can process mesh video data according to any transmission protocol. Processing for transmission may include processing for delivery via a broadcast network and processing for delivery via broadband. According to an embodiment, the transmission processing unit may receive mesh video-related metadata from the metadata processing unit in addition to mesh video data, and apply transmission processing to it.

[0054] The transmission unit can transmit encoded video / image information or data output in the form of a bitstream to the receiving unit of a receiving device via a digital storage medium or a network in the form of a file or streaming. The digital storage medium may include various storage media such as USB, SD, CD, DVD, Blu-ray, HDD, and SSD. The transmission unit may include elements for creating a media file through a predetermined file format and elements for transmission via a broadcasting / communication network. The receiving unit can extract the bitstream and transmit it to a decoding device.

[0055] The receiver can receive mesh video data transmitted by a mesh video transmission device. Depending on the transmission channel, the receiver may receive mesh video data via a broadcast network or via broadband. Alternatively, it may receive mesh video data via a digital storage medium.

[0056] The receiving processing unit can perform processing on the received mesh video data according to the transmission protocol. The receiving processing unit may be included in the receiving unit or may be configured as a separate component or module. Corresponding to the processing for transmission performed on the transmitting side, the receiving processing unit may perform the reverse process of the aforementioned transmission processing unit. The receiving processing unit may transmit the acquired mesh video data to the decapsulation processing unit and the acquired mesh video-related metadata to the metadata parser. The mesh video-related metadata acquired by the receiving processing unit may be in the form of a signaling table.

[0057] The file / segment decapsulation module can decapsulate mesh video data in file format received from the receiving module. The decapsulation module can decapsulate files based on ISOBMFF, etc., to obtain a mesh video bitstream or mesh video-related metadata (metadata bitstream). The obtained mesh video bitstream can be transmitted to a mesh video decoder, and the obtained mesh video-related metadata (metadata bitstream) can be transmitted to a metadata processing module. The mesh video bitstream may also contain metadata (metadata bitstream). The metadata processing module may be included in the mesh video decoder or configured as a separate component / module. The mesh video-related metadata obtained by the decapsulation module may be in the form of boxes or tracks within the file format. If necessary, the decapsulation module may receive metadata required for decapsulation from the metadata processing module. Mesh video-related metadata may be passed to a Mesh video decoder and used in the Mesh video decoding process, or passed to a renderer and used in the Mesh video rendering process.

[0058] The Mesh Video Decoder can decode video by receiving a bitstream as input and performing operations corresponding to those of the Mesh Video Encoder. The decoded Mesh Video can be displayed through a display unit. The user can view all or part of the rendered result through a VR / AR display or a standard display.

[0059] The feedback process may include the process of transmitting various feedback information obtainable during the rendering / display process to the transmitting side or to the decoder of the receiving side. Interactivity in mesh video consumption may be provided through the feedback process. According to an embodiment, head orientation information, viewport information indicating the area the user is currently viewing, etc., may be transmitted during the feedback process. According to an embodiment, the user may interact with elements implemented in a VR / AR / MR / autonomous driving environment, and in this case, information related to such interaction may be transmitted to the transmitting side or the service provider side during the feedback process. According to an embodiment, the feedback process may not be performed.

[0060] Head orientation information can refer to information regarding the user's head position, angle, movement, etc. Based on this information, information about the area the user is currently viewing within the mesh video—that is, viewport information—can be calculated.

[0061] Viewport information may be information about the area currently being viewed by the user within a mesh video. Through this, gaze analysis can be performed to determine how the user consumes the mesh video and to what extent they gaze at specific areas of the mesh video. Gaze analysis may be performed at the receiving end and transmitted to the transmitting end via a feedback channel. Devices such as VR / AR / MR displays can extract the viewport area based on the user's head position / orientation, the vertical or horizontal FOV supported by the device, etc.

[0062] According to an embodiment, the aforementioned feedback information may not only be transmitted to the transmitting side but may also be consumed at the receiving side. That is, decoding and rendering processes at the receiving side may be performed using the aforementioned feedback information. For example, using head orientation information and / or viewport information, only the mesh video for the area currently viewed by the user may be preferentially decoded and rendered.

[0063] This document relates to dynamic mesh video compression as described above. The methods / embodiments disclosed in this document may be applied to the MPEG (Moving Picture Experts Group) Video-based Dynamic Mesh Compression Method (V-Mesh) standard or next-generation video / image coding standards. Dynamic mesh video compression is a method for processing mesh connection information and attributes that change over time, and it can perform lossy and lossless compression for various applications such as real-time communication, storage, free-viewpoint video, and AR / VR.

[0064] The dynamic mesh video compression method described below is based on MPEG's V-Mesh method.

[0065] In this document, "picture" or "frame" generally refers to a unit representing a single image of a specific time period.

[0066] A pixel or pel may refer to the smallest unit that constitutes a picture (or image). Additionally, the term 'sample' may be used as a counterpart to pixel. A sample can generally represent a pixel or a pixel value, and may represent only the pixel / pixel value of the lumina component, only the pixel / pixel value of the chroma component, or only the pixel / pixel value of the depth component.

[0067] A unit may represent a basic unit of image processing. A unit may include at least one of a specific area of ​​a picture and information related to that area. Depending on the case, the term unit may be used interchangeably with terms such as block or area. In general, an MxN block may include samples (or sample arrays) or a set (or array) of transform coefficients consisting of M columns and N rows.

[0068] The encoding process of Fig. 2 is as follows.

[0069] Video-based dynamic mesh compression (V-Mesh) compression methods can provide a method for compressing dynamic mesh video data based on 2D video codecs such as HEVC and VVC. In the V-Mesh compression process, compression is performed by receiving the following data as input.

[0070] Input mesh: It includes the 3D coordinates (geometry) of the vertices constituting the mesh, normal information for each vertex, mapping information for mapping the mesh surface to a 2D plane, and connection information between the vertices constituting the surface. The surface of the mesh can be represented by triangles or polygons of greater size, and connection information between the vertices constituting each surface is stored according to a defined shape. The input mesh can be saved in the OBJ file format.

[0071] Attribute map: (Hereafter, Texture map is used with the same meaning): It contains information on the attributes (color, normals, displacement, etc.) of a mesh and stores data in the form of mapping the mesh surface onto a 2D image. Mapping which part of the mesh (surface or vertex) corresponds to each data point in this attribute map is based on the mapping information contained in the input mesh. Since the attribute map holds data for each frame of the mesh video, it can also be referred to as an attribute map video (or simply "attribute"). In the V-Mesh compression method, the attribute map primarily contains the mesh's color information and is stored in image file formats (PNG, BMP, etc.).

[0072] Material Library File: Contains material attribute information used in the mesh, specifically information that links the input mesh with its corresponding attribute map. This is saved in the Wavefront Material Template Library (MTL) file format.

[0073] In the V-Mesh compression method, the following data and information can be generated through the compression process.

[0074] Base Mesh: By simplifying (decimating) the input mesh through a preprocessing stage, objects from the input mesh are represented using the minimum number of vertices determined according to user criteria.

[0075] Displacement: Displacement information used to represent the input mesh as similarly as possible using the base mesh, expressed in the form of 3D coordinates.

[0076] Atlas information: This is metadata required to reconstruct a mesh using base mesh, displacement, and attribute map information. It can be generated and utilized as sub-units (sub-mesh, patch, etc.) that constitute the mesh.

[0077] Referring to FIGS. 3 to 7, a method for encoding mesh position information (vertices) is described, and referring to FIGS. 7-10 and others, a method for encoding attribute information (attribute map) by restoring mesh position information is described.

[0078] FIG. 3 illustrates a V-MESH compression method according to embodiments.

[0079] FIG. 3 illustrates the encoding process of FIG. 2, and the encoding process may include pre-processing and encoding processes. The encoder of FIG. 2 may include a pre-processor (200) and an encoder (201) as in FIG. 3. The transmitting device of FIG. 2 may be broadly referred to as an encoder, and the dynamic mesh video encoder of FIG. 2 may be referred to as an encoder. The V-Mesh compression method may include pre-processing (200) and encoding (201) processes as in FIG. 3. The pre-processor of FIG. 3 may be located in front of the encoder of FIG. 3. The pre-processor and encoder of FIG. 3 may be referred to as a single encoder.

[0080] The pre-processor can receive a static dynamic mesh and / or attribute map. The pre-processor can generate a base mesh and / or displacement through preprocessing. The pre-processor can receive feedback information from an encoder and generate a base mesh and / or displacement based on the feedback information.

[0081] The encoder can receive a base mesh, displacement, static and dynamic meshes, and / or attribute maps. The encoder can encode mesh-related data to generate a compressed bitstream.

[0082] FIG. 4 shows the pre-processing of V-MESH compression according to the embodiments.

[0083] Figure 4 shows the configuration and operation of the pre-processor of Figure 3.

[0084] FIG. 3 illustrates a process of performing preprocessing on an input mesh. The preprocessing process (200) may include four main steps: 1) generation of a Group of Frame (GoF), 2) mesh decimation, 3) UV parameterization, and 4) fitting subdivision surface (300). The pre-processor (200) may receive the input mesh, generate displacement and / or a base mesh, and transmit it to the encoder (201). The pre-processor (200) may transmit GoF information associated with GoF generation to the encoder (201).

[0085] Below, each step of Fig. 4 is explained.

[0086] GoF Generation: This is the process of creating a reference structure for mesh data. If the number of vertices, texture coordinates, vertex connectivity, and texture coordinate connectivity of the mesh in the previous frame and the current mesh are all identical, the previous frame can be set as the reference frame. In other words, if only the vertex coordinate values ​​differ between the current input mesh and the reference input mesh, inter-frame encoding can be performed. Otherwise, the frame undergoes intra-frame encoding.

[0087] Mesh Decimation: This is the process of simplifying the input mesh to generate a simplified mesh, or base mesh. After selecting vertices to remove from the original mesh based on user-defined criteria, the selected vertices and the triangles connected to them can be removed.

[0088] In the process of performing mesh decimation, information regarding the input mesh (voxelized), target triangle ratio (TTR), and minimum triangle component (CCCount) is passed as input, and a decimated mesh can be obtained as output. In this process, connected triangle components smaller than the set minimum triangle component (CCCount) can be removed.

[0089] UV Parameterization: This is the process of mapping 3D surfaces to a texture domain for a decimated mesh. Parameterization can be performed using a UV Atlas tool. Through this process, mapping information is generated regarding where each vertex of the decimated mesh can be mapped to on a 2D image. This mapping information is expressed and stored as texture coordinates, and the final base mesh is generated through this process.

[0090] OrthoAtlas technology is a technique that generates texture coordinates using orthographic projection. In orthoAtlas technology, the processes of patch generation and patch packing are performed sequentially. First, Connected Components (CCs) are generated by splitting adjacent triangles, and then the optimal CCs are merged using a cost function to generate a patch. The cost function can measure the cost based on the degree of distortion that occurs when orthogonally projecting the patch in each direction. Finally, texture coordinates can be calculated by packing the patch that minimizes the cost function into the texture domain. In the case of orthoAtlas technology, texture coordinates can be derived in the base mesh decoder without compressing texture coordinate and texture connection information during the base mesh encoding process.

[0091] Fitting subdivision surface: This is the process of performing subdivision on a decimated mesh. User-defined methods, such as the mid-edge method, can be applied as the subdivision method. A fitting process is performed to make the input mesh and the subdivision mesh similar to each other.

[0092] This is a process of performing fitting so that the mesh obtained by subdividing the base mesh becomes similar to the surface of the input mesh. As for the subdivision method, a user-defined method such as the mid-edge method (Fig. 5), loop method, or LS3 method may be applied.

[0093] FIG. 5 illustrates a mid-edge subdivision method according to embodiments.

[0094] Figure 5 illustrates the mid-edge method of the fitting subdivision surface described in Figure 4. Referring to Figure 5, an original mesh containing 4 vertices is subdivided to create a sub-mesh. A sub-mesh can be created by creating a new vertex at the midpoint of the edge between vertices.

[0095] When a fitted subdivided mesh (hereinafter referred to as the fitted subdivided mesh) is generated, displacement is calculated using this result and a pre-compressed and decoded base mesh (hereinafter referred to as the reconstructed base mesh). That is, the reconstructed base mesh is subdivided in the same way as the fitting subdivision surface. The positional difference between this result and the fitted subdivided mesh for each vertex becomes the displacement for each vertex. Since displacement represents a positional difference in three-dimensional space, it is also expressed as a value in the (x, y, z) space of the Cartesian coordinate system. Depending on the user input parameters, the (x, y, z) coordinate values ​​can be converted into (normal, tangential, bi-tangential) coordinate values ​​of the local coordinate system.

[0096] FIG. 6 illustrates a displacement generation process according to embodiments.

[0097] FIG. 6 illustrates in detail the method of calculating displacement of a fitting subdivision surface (300) as described in FIG. 5.

[0098] An encoder and / or pre-processor according to the embodiments may include 1) a subdivision unit, 2) a local coordinate system calculation unit, and 3) a displacement calculation unit. The subdivision unit may receive a restored base mesh and generate a subdivided restored base mesh. The local coordinate system calculation unit may receive a fitted subdivided mesh and a subdivided restored base mesh and convert a coordinate system relating to the mesh to a local coordinate system. The local coordinate system calculation operation may be optional. The displacement calculation unit calculates the positional difference between the fitted subdivision mesh and the subdivided restored base mesh. For example, it may generate a positional difference value between the vertices of the two input meshes. The vertex positional difference value becomes the displacement.

[0099] The method and apparatus for transmitting point cloud data according to the embodiments can encode the point cloud as follows. The point cloud data according to the embodiments (which may be referred to simply as point cloud) may refer to data including vertex coordinates and color information. Point cloud is a term that includes mesh data, and in this document, point cloud and mesh data may be used interchangeably.

[0100] The V-Mesh compression (restoration) method according to the embodiments may include intra-frame encoding (Fig. 6) and inter-frame encoding (Fig. 7).

[0101] Intra-frame encoding or inter-frame encoding is performed based on the results of the aforementioned GoF generation. In the case of intra-frame encoding, the data to be compressed may include a base mesh, displacement, and attribute map. In the case of inter-frame encoding, the data to be compressed may include displacement, attribute map, and the motion field between the reference base mesh and the current base mesh.

[0102] FIG. 7 illustrates the V-DMC encoding process according to the embodiments.

[0103] The encoding process of Fig. 7 illustrates the encoding of Figs. 1 and 2 in detail.

[0104] A pre-processor receives an input mesh and can perform the aforementioned preprocessing. Through preprocessing, a base mesh and / or a fitted subdivided mesh can be generated. A quantizer can quantize the base mesh and / or the fitted subdivided mesh. A static mesh encoder can encode a static mesh. The static mesh encoder can generate a bitstream containing the encoded base mesh. A motion encoder can encode motion vectors for the base mesh based on inter-frame motion estimation and motion compensation for inter-prediction. An atlas encoder can encode an atlas for the vertices of the base mesh. The encoded base mesh can be reconstructed and inversely quantized through an inverse quantizer. A displacement computer receives the reconstructed mesh and, based on the fitted subdivided mesh, can generate displacement, which is the position difference. A lifting transform can receive the displacement and generate lifting coefficients. The quantizer can quantize the lifting coefficients. Depending on the encoding method, the image packing unit can pack the image based on the quantized lifting coefficients. The video encoder can encode the packed image. Depending on the encoding method, it can apply interprediction to the quantized lifting coefficients and encode the predicted lifting coefficients according to an arithmetic encoding method. The mesh restoration unit restores the deformed mesh through the restored displacement and the restored base mesh. The displacement data is restored, and the deformed mesh is restored based on the restored displacement data and the restored base mesh and provided to the attribute transfer. The attribute transfer receives the input mesh and / or input attribute map and generates an attribute map based on the restored deformed mesh. Push-pull padding can pad data into the attribute map based on a push-pull method. The color space converter can convert the space of the color component that is an attribute. The video encoder can encode the attributes.A multiplexer can generate a bitstream by multiplexing a compressed base mesh, compressed displacement, and compressed attributes.

[0105] Base Mesh Encoding: Base mesh compression methods can be divided into INTRA, INTER, and SKIP types depending on the base mesh type, and encoding can be performed in different ways for each. If the base mesh is of the INTRA type, it can be encoded using a static mesh encoding method. If the base mesh is of the INTER type, the motion field between the reference base mesh and the current base mesh can be encoded. If the current base mesh is of the SKIP type, the reference base mesh can be derived into the current base mesh.

[0106] After being encoded in the encoder, the base mesh can be subdivided into a subdivided mesh through a subdivision process. Subdivision algorithms such as mid-point subdivision and loop subdivision can be used.

[0107] Static Basemesh Encoding (Intra Basemesh Encoding): When performing intra encoding on the current basemesh, the base mesh generated during the preprocessing stage can be encoded using static mesh compression technology after undergoing a quantization process. Static mesh compression utilizes MPEG EdgeBreaker (MEB) technology, and the base mesh's vertex position information, mapping information (texture coordinates), vertex connectivity information, and normals are subject to compression.

[0108] The technology for compressing connection information can be encoded based on the edgebreaker algorithm. The edgebreaker algorithm is a technique that sequentially traverses triangles according to rules, maps symbols based on the characteristics of each triangle, and then encodes the corresponding symbols.

[0109] Techniques for compressing vertex location information can calculate predicted values ​​based on prediction techniques such as multiple parallelogram prediction, and then encode the residual value, which is the difference between the current vertex and the predicted value.

[0110] A technique for compressing mapping information (texture coordinates) can calculate a predicted value based on a prediction technique such as stretching, and then encode the residual value, which is the difference between the current mapping information (texture coordinates) and the predicted value.

[0111] Normal compression techniques can obtain predicted values ​​based on prediction techniques such as delta coding, multiple parallelogram prediction, and cross product-based prediction, and then encode the residual value, which is the difference between the current normal and the predicted value.

[0112] Motion Field Encoding (Inter Basemesh Encoding): Inter basemesh encoding can be performed when a one-to-one correspondence exists between the reference mesh and the current input mesh, differing only in vertex position information. When performing inter encoding, instead of compressing the base mesh, the difference between the vertices of the reference base mesh and the current base mesh—that is, the motion field—is calculated and this information is encoded. The reference base mesh is the result of quantizing already decoded base mesh data and is determined by the reference frame index.

[0113] The motion field can be encoded as is, or the predicted motion field can be calculated by averaging the motion fields of the restored vertices among those connected to the current vertex, and the residual motion field, which is the difference between the predicted motion field value and the current vertex's motion field value, can be encoded. This value can be encoded using entropy coding.

[0114] Displacement Encoding: After encoding the base mesh, reconstruction and inverse quantization are performed to reconstruct it. Once the base mesh is generated and subdivision is performed on it, the displacement between the result and the fitted subdivided mesh can be calculated. For effective encoding, data transform processes such as wavelet transform can be applied to the displacement information, and Figure 7 shows the process of transforming displacement information using the lifting transform in V-Mesh. The transform coefficients generated through the transformation process are quantized, and the quantized transform coefficients can be compressed through a video codec or through arithmetic coding, depending on the compression method.

[0115] When compressed through a video codec, the data is packed into a 2D image as shown in Figure 8. Transform coefficients are organized into one block for every N^2 (N*N) units, and each block can be packed in z-scan order. The number of horizontal blocks is fixed at N, while the number of vertical blocks can be determined by the number of vertices in the subdivided base mesh. Within a single block, transform coefficients can be packed by aligning them using Morton code. The packed images generate a displacement video for every GoF unit, and this displacement video can be encoded using an existing video compression codec.

[0116] When compressed via arithmetic coding, cross-frame prediction can be performed on the quantized displacement vector transformation coefficients. When cross-frame prediction is performed on the current quantized displacement vector transformation coefficients, the residual value—the difference between the current coefficient and the reference coefficient—can be encoded, and information regarding the reference target can also be encoded. Depending on the displacement vector type, the quantized displacement vector transformation coefficients can be arithmetic encoded if it is of the INTRA type, and the residual value if it is of the INTER type. Arithmetic coding can be performed based on Context Adaptive Binary Arithmetic Coding (CABAC). The CABAC process can first binarize the displacement vector data and map it to a bin string. The bin string can be an output binarized into 0s and 1s, where each 0 or 1 represents a bin. Each bin can be arithmetic encoded using context information selected from the context model, and a process of updating probabilities can be performed.

[0117]

[0118] FIG. 8 illustrates a lifting conversion process for displacement according to embodiments.

[0119] FIG. 9 illustrates the process of packing conversion coefficients according to embodiments into a 2D image.

[0120] FIGS. 8-9 respectively illustrate the process of converting the displacement of the encoding process of FIG. 7 and the process of packing the conversion coefficients.

[0121] The encoding method according to the embodiments includes displacement encoding.

[0122] After encoding the base mesh through base mesh encoding and / or motion field encoding, a reconstructed base mesh is generated through reconstruction and inverse quantization. Subdivision is then performed on this reconstructed base mesh, and the displacement between the result and the fitted subdivided mesh generated through the fitting subdivision surface can be calculated. For effective encoding, a data transform process such as a wavelet transform can be applied to the displacement information.

[0123] Figure 8 illustrates the process of transforming displacement information using a lifting transform in V-Mesh. The transformation coefficients generated through the transformation process are quantized and then packed into a 2D image as shown in Figure 9. The transformation coefficients are 256 (=16 16) Each unit is composed of one block, and each block can be packed in z-scan order. The number of blocks is fixed at 16, but the number of blocks can be determined by the number of vertices of the subdivided base mesh. Transform coefficients can be packed by aligning them with a Morton code within a single block. The packed images generate a displacement video for each GoF unit, and this displacement video can be encoded using a conventional video compression codec.

[0124] Referring to FIG. 8, the base mesh (original) may include vertices and edges for LoD0. A first subdivision mesh generated by dividing the base mesh includes vertices generated by further dividing the edges of the base mesh. The first subdivision mesh includes vertices for LoD0 and vertices for LoD1. LoD1 includes subdivided vertices and vertices of the base mesh (LoD0). A second subdivision mesh may be generated by dividing the first subdivision mesh. The second subdivision mesh includes LoD2. LoD2 includes base mesh vertices (LoD0), LoD1 containing vertices additionally generated from LoD0, and vertices additionally divided from LoD1. LoD is a Level of Detail indicating the degree of detail; as the level index increases, the distance between vertices becomes closer, and the level of detail increases. LoD N includes the vertices included in the previous LoDN-1 as they are. When vertices are further subdivided through subdivision, the mesh can be encoded based on a prediction and / or update method by considering the previous vertices v1 and v2 and the subdivided vertex v. Instead of encoding the information for the current LoD N as is, the size of the bitstream can be reduced by generating residuals between the previous LoD N-1 and encoding the mesh using these residuals. The prediction process refers to the operation of predicting the current vertex v based on the previous vertices v1 and v2. Since adjacent subdivided meshes possess similar data, efficient encoding can be achieved by utilizing this property. The current vertex position information is predicted using the residuals from the previous vertex position information, and the previous vertex position information is updated using these residuals.

[0125] Referring to FIG. 9, the vertices have coefficients generated through a lifting transformation. The coefficients of the vertices related to the lifting transformation can be encoded by packing them into an image.

[0126] FIG. 10 illustrates the attribute transfer process of the V-MESH compression method according to the embodiments.

[0127] Figure 10 shows the detailed operation of the attribute transfer of the encoding of Figure 7.

[0128] The encoding according to the embodiments includes attribute map encoding.

[0129] Information regarding the input mesh is compressed through base mesh encoding, motion field encoding, and displacement encoding. The input mesh compressed during the encoding process is restored through base mesh decoding (intra frame), motion field encoding (inter frame), and displacement video decoding. The resulting restored deformed mesh (hereinafter referred to as Recon. deformed mesh) is used to compress the input attribute map as shown in FIGS. 6 and 7. The Recon. deformed mesh possesses vertex position information, texture coordinates, and corresponding connection information, but lacks color information corresponding to the texture coordinates. Accordingly, as shown in FIG. 10, a new attribute map having color information corresponding to the texture coordinates of the reconstructed deformed mesh is generated through the attribute transfer process in the V-Mesh compression method.

[0130] Attribute transfer first checks for every point P(u, v) in the 2D texture domain whether the point belongs to a texture triangle of the reconstructed deformed mesh, and if it does, determines the barycentric coordinates (α, of P(u, v) according to that triangle T). Calculate , γ). Then, the 3D vertex positions of triangle T and (α, Calculate the 3D coordinates M(x, y, z) of P(u, v) using , γ). Find the vertex coordinates M'(x', y', z') corresponding to the location most similar to the calculated M(x, y, z) in the input mesh domain, and find triangle T' containing this point. Then, in triangle T', find the coordinates of the centroid of M'(x', y', z') (α', Calculate , γ'). Texture coordinates corresponding to the three vertices of Triangle T' and (α', Texture coordinates (u', v') are calculated using , γ'), and color information corresponding to these coordinates is found in the input attribute map. The found color information is then assigned to the (u, v) pixel location in the new attribute map. If P(u, v) does not belong to any triangle, the pixel at that location in the new attribute map can be filled with a color value using a padding algorithm such as the push-pull algorithm.

[0131] The new attribute map generated through attribute transfer is bundled in GoF units to form an attribute map video, which is then compressed using a video codec.

[0132] Atlas Encoding: Atlas information may be transmitted during the aforementioned process. The Atlas consists of information required during the mesh decoding and / or rendering process, and may include information required during the process of performing subdivision, displacement decoding, base mesh decoding, etc., as well as tile information, patch information, etc. Atlas data may be encoded using Exp-Golomb coding, etc.

[0133] Referring to FIG. 10, the reference relationships between the input mesh, input attribute map, restored mesh, and generated attribute map can be seen.

[0134] The decoding process of Fig. 1 can perform the reverse process of the corresponding process of the encoding process of Fig. 1. The specific decoding process is as follows.

[0135] FIG. 11 illustrates a V-DMC decoding process according to embodiments.

[0136] Figure 11 shows the configuration and operation of the decoder of the receiving device of Figure 1.

[0137] The input bitstream can be separated into a Basemesh sub-stream, a Displacement sub-stream, an Attribute map sub-stream, and an Atlas sub-stream.

[0138] The Atlas sub-stream can be decoded through Exp-Golomb coding, and as a result, information necessary for performing decoding, tile information, patch information, etc. can be obtained.

[0139] Depending on the basemesh type, if the basemesh sub-stream is of the INTRA type, it can be decoded through a static mesh decoder based on MEB (MPEG EdgeBreaker) technology, and as a result, the connectivity information, vertex geometry information, and vertex mapping information (texture coordinates) of the base mesh can be restored.

[0140] When the texture parameterization method in the encoder is orthoAtlas, the decoder can derive mapping information (texture coordinates) and attribute information (texture) connection information using vertex coordinates. The process of deriving mapping information (texture coordinates) and connection information can generate mapping information (texture coordinates) and attribute information (texture) connection information by calculating the homography transform of each face and then projecting the vertex based on it.

[0141] If the Basemesh type is INTER type, motion information can be decoded through entropy decoding and inverse prediction processes. The restored motion information is combined with the reference Basemesh that has already been restored and stored in the buffer to generate a Reconstructed quantized basemesh for the current frame. An inverse quantization process can be performed on the restored Basemesh.

[0142] If the displacement sub-stream is compressed through a video codec according to the compression method used in encoding, it is decoded into a displacement video through the video compression codec's decoder, and then the image unpacking process is performed.

[0143] When compressed through arithmetic coding, the displacement vector bitstream can decode binarized syntax elements through arithmetic decoding, and a Contextual Probability Model (CPM) can be adaptively determined according to each bin of the syntax elements, and arithmetic decoding can be performed by predicting the probability of occurrence of the bin through the CPM. The binarized syntax elements can be decoded through inverse binarization. Quantized displacement vector transformation coefficients can be derived from the decoded syntax elements. Depending on the displacement information type, if it is INTER (where inter prediction is performed), an inverse inter prediction process is performed using reference information for the quantized displacement coefficients.

[0144] The quantized displacement coefficient is restored as displacement information for each vertex through inverse quantization, inverse transform, and coordinate system transformation processes.

[0145] The restored Base mesh and the restored Displacement information are combined to generate the final Decoded mesh. The Attribute map sub-stream is decoded through the decoder of the video compression codec used in Encoding, and then restored to the final Attribute map through processes such as color format conversion.

[0146] The restored Decoded mesh and Decoded attribute map can be utilized at the receiving end as final mesh data available to the user.

[0147] The atlas decoder decodes atlas data within the bitstream.

[0148] When mesh data within the bitstream is encoded based on inter-prediction, the motion decoder derives the motion field of the current frame's basemesh through motion estimation and compensation, based on the basemesh within the reference frame. When mesh data within the bitstream is encoded based on intra-prediction, the sectic decoder decodes the basemesh. Depending on the encoding method, it decodes displacement data by applying either arithmetic coding or video decoding. The video decoder decodes attribute data within the bitstream.

[0149] The decoding method of FIG. 11 can follow the reverse process of the encoding method according to the embodiments.

[0150] FIG. 12 illustrates a V-DMC encoding process according to embodiments.

[0151] FIG. 12 illustrates the configuration and operation of an encoder of a transmitting device such as FIG. 1 or FIG. 2. Each component of FIG. 12 corresponds to hardware, software, a processor, and / or a combination thereof.

[0152] Figure 12 shows the encoding process of V-Mesh technology.

[0153] The mesh preprocessing unit receives the original mesh as input and generates a decimated mesh. Decimation can be performed based on the number of target vertices or polygons constituting the mesh. For the decimated mesh, parameterization can be performed to generate mapping information (texture coordinates) and attribute information (texture) connection information per vertex. Additionally, floating-point mesh information can be quantized into fixed-point form. This result can be encoded as a base mesh through the static mesh encoding unit. The mesh preprocessing unit can generate additional vertices by performing mesh subdivision on the base mesh. Depending on the subdivision method, vertex connection information including the added vertices, texture coordinates, and texture coordinate connection information can be generated. The subdivided mesh can be fitted by adjusting vertex positions to resemble the original mesh, thereby generating a fitted subdivided mesh.

[0154] The base mesh generated through the mesh preprocessing unit can perform intra-encoding or inter-encoding depending on the base mesh type. If the base mesh frame undergoes intra-encoding, it can be compressed through the static mesh encoding unit. In this case, encoding can be performed on the base mesh's connectivity information, vertex geometry information, vertex texture information, normal information, etc. If the base mesh frame undergoes inter-encoding, the motion vector encoding unit is executed; using the base mesh and the reference-restored base mesh as inputs, the motion vector between the two meshes is calculated and its value encoded. The motion vector encoding unit performs connectivity-based prediction using previously encoded / decoded motion vectors as predictors and can encode the residual motion vector obtained by subtracting the predicted motion vector from the current motion vector. The base mesh bitstream generated through the base mesh encoding unit is transmitted to the multiplexer.

[0155] The encoded base mesh bitstream can generate a restored base mesh through the base mesh restoration unit.

[0156] The displacement vector calculation unit can perform mesh subdivision on the reconstructed base mesh. The displacement vector can be calculated as the difference in vertex positions between the subdivided reconstructed base mesh and the fitted subdivision mesh generated by the preprocessing unit. As a result, displacement vectors can be calculated for each vertex of the subdivision mesh. The displacement vector calculation unit can convert the displacement vector calculated in the 3D Cartesian coordinate system into a local coordinate system based on the normal vector of each vertex.

[0157] The displacement vector processing unit can transform the displacement vector for effective encoding. Depending on the embodiment, the transformation may be performed using a lifting transformation, a wavelet transformation, etc. Additionally, quantization can be performed on the transformed displacement vector values, i.e., the transformation coefficients. Different quantization parameters can be applied to each axis of the transformation coefficients, and the quantization parameters can be derived by the agreement between the encoder and decoder. The quantized displacement vector transformation coefficients calculated by the displacement vector processing unit can be encoded through the displacement vector video encoding unit or the displacement vector arithmetic encoding unit, depending on the compression method.

[0158] The displacement vector video encoding unit can pack displacement vector information that has undergone transformation and quantization into 2D images. A displacement vector video can be generated by bundling the packed 2D images for each frame, and the displacement vector video can be generated for each GoF (Group of Frame) unit of the input mesh. The generated displacement vector video can be encoded using a video compression codec. The generated displacement vector video bitstream is transmitted to the multiplexer.

[0159] In the displacement vector arithmetic encoding unit, for the quantized displacement vector transformation coefficients, if the displacement vector type is INTER type, inter-frame prediction can be performed. The inter-frame prediction process may be a process of calculating the residual value, which is the difference between the current transformation coefficient and the reference transformation coefficient. The displacement vector transformation coefficient or the residual value can be encoded through the arithmetic encoding process.

[0160] The displacement vector restored through the displacement vector restoration unit and the base mesh restored and subdivided through the base mesh restoration unit are restored through the mesh restoration unit, and the restored mesh contains restored vertices, connection information between vertices, texture coordinates, and connection information between texture coordinates.

[0161] The attribute information (texture map) of the original mesh can be regenerated into the attribute information (texture map) for the restored mesh through the attribute information (texture map) video generation unit. Vertex-specific color information contained in the original mesh's texture map can be assigned to the texture coordinates of the restored mesh. The texture maps regenerated for each frame can be grouped by GoF unit to generate a texture map video.

[0162] The generated texture map video can be encoded using a video compression codec through the texture map video encoding unit. The texture map video bitstream generated through encoding is transmitted to the multiplexer.

[0163] The atlas encoding unit can encode the atlas, which is additional information required for the mesh decoding and rendering processes. The generated atlas bitstream is transmitted to the multiplexer.

[0164] The generated base mesh bitstream, displacement vector bitstream, texture map bitstream, and atlas bitstream can be multiplexed into a single bitstream and transmitted to the receiver via the transmitter. Alternatively, the generated base mesh bitstream, displacement vector bitstream, texture map bitstream, and atlas bitstream can be generated into a file with one or more track data or encapsulated into segments and transmitted to the receiver (decoder) via the transmitter.

[0165] The data input unit can receive the original mesh and / or original texture map ('attribute'). The mesh preprocessing unit can simplify the original mesh to generate a base mesh and fit it to generate a refined mesh. If the mesh encoding method is inter-prediction, the motion vector encoding unit can generate motion vectors (motion fields) by referencing the restored base mesh within a previously processed reference frame and encode them based on motion estimation and compensation methods. If the mesh encoding method is intra-prediction, the static mesh encoding unit can encode the base mesh within the frame. The displacement vector calculation unit can calculate displacement vectors for vertices from the fitted refined mesh based on the restored base mesh. The displacement vector processing unit can process the displacement vectors into a form suitable for encoding. Depending on the encoding method for the displacement vectors, the displacement vectors can be encoded based on a video method or an arithmetic encoding method. The displacement vectors are restored and can be provided to the mesh restoration unit along with the restored base mesh. Based on the restored mesh, the attribute (texture map) video generation unit can generate a video for encoding the texture map using the original mesh and the texture map for the original mesh. The attributes are encoded based on the video method. The atlas is encoded by the atlas encoding unit.

[0166] FIG. 13 illustrates a V-DMC decoding process according to embodiments.

[0167] FIG. 13 corresponds to a decoder such as FIG. 1 to FIG. 3. Each component of FIG. 13 corresponds to hardware, software, a processor, and / or a combination thereof.

[0168] The received Mesh bitstream is demultiplexed into a compressed base mesh bitstream, displacement vector bitstream, attribute information (texture map) bitstream, and atlas bitstream after file / segment decapsulation.

[0169] If the current mesh has inter-frame encoding applied based on the frame header information, decoding can be performed on the base mesh bitstream in the motion vector decoder. The final motion vector can be restored by using the previously decoded motion vector as a predictor and adding it to the residual motion vector decoded from the bitstream. The current base mesh can be restored by adding the decoded motion vector to the reference base mesh.

[0170] If the current mesh has in-frame encoding applied based on the frame header information, the base mesh bitstream can restore the base mesh's connectivity information, vertex geometry, texture coordinates, normal information, etc., through the static mesh decoder.

[0171] In the base mesh restoration unit, a restored base mesh can be generated by performing inverse quantization on the decoded base mesh.

[0172] Depending on the encoding codec type, if the displacement vector bitstream is encoded through a video codec, it can be decoded using the video codec and then subjected to a reverse packing process. If encoded through arithmetic coding, arithmetic decoding can be performed through the displacement vector arithmetic decoding unit, and if inter-frame prediction is performed, the current displacement vector transformation coefficient can be generated by adding the residual value to the reference displacement vector transformation coefficient through inter-frame prediction.

[0173] The displacement vector restoration unit restores the displacement vector by applying the decoded displacement vector transformation coefficients to inverse quantization and inverse transformation processes. If the restored displacement vector is a value in the local coordinate system, an inverse transformation to the Cartesian coordinate system can be performed.

[0174] The mesh restoration unit can generate additional vertices by performing subdivision on the restored base mesh. Through subdivision, vertex connection information including the added vertices, texture coordinates, and connection information of the texture coordinates can be generated. The subdivided restored base mesh can be combined with the restored displacement vectors to generate the final restored mesh.

[0175] The texture map bitstream can be decoded as a video bitstream using a video codec in the texture map video decoder. The restored texture map contains color information for each vertex contained in the restored mesh, and the color value of the corresponding vertex can be retrieved from the texture map using the texture coordinates of each vertex.

[0176] The atlas bitstream can be decoded through the atlas decoding unit.

[0177] The restored mesh and texture map are displayed to the user through a rendering process using a mesh data renderer, etc.

[0178] The decoder receives the encoded bitstream and decodes the base mesh, displacement vector, attributes, and atlas within the bitstream based on the parameter information (which may be referred to as signaling information, metadata, etc.) contained within the bitstream. The decoding process may follow the inverse of the encoding process. Based on the decoded atlas, a mesh is reconstructed from the reconstructed base mesh and the reconstructed displacement mesh. Based on the reconstructed mesh and the reconstructed attributes, the mesh can be rendered.

[0179] A point cloud data encoding device and method according to the embodiments can encode mesh data and transmit a bitstream containing the encoded mesh data. A point cloud data decoding device and method according to the embodiments can receive a bitstream containing mesh data and decode the mesh data. A point cloud data encoding / decoding method / device according to the embodiments may be referred to simply as a method / device according to the embodiments. A point cloud data encoding / decoding method / device according to the embodiments may also be referred to as a mesh data encoding / decoding method / device according to the embodiments. Additionally, it may be used in this document simply as an encoding / decoding method / device.

[0180] The encoding method / device (or transmission method / device) according to the embodiments includes, and can perform the encoder of FIG. 1, the transmission device (100) of FIG. 2, the acquisition unit (101), the encoder (102), the encapsulator (103), the transmitter (104), the pre-processor (200) of FIG. 3 to 4, the encoder (201), the encoding process of FIG. 7 and FIG. 12, the generation of bitstream and parameter information (syntax elements) of FIG. 14 to 15, the encoding method of FIG. 16, etc.

[0181] The decoding method / device (or receiving method / device) according to the embodiments may include, and can perform the decoder of FIG. 1, the receiving device (110) of FIG. 2, the receiving unit (111), the decapsulator (112), the decoder (113), the renderer (114), the decoding process of FIG. 11 and FIG. 13, the acquisition of bitstream and parameter information (syntax elements) of FIG. 14 to 15, the decoding method of FIG. 17, etc.

[0182] The contents of this specification may be understood based on V-DMC standard specification documents, including ISO / IEC 23090-29, disclosed at the time of the priority date or filing date of this application.

[0183] The encoding and encoder (or encoding unit) according to the embodiments may be interpreted as having the same meaning as encoding and encoder, respectively. The decoding and decoder (or decoder) according to the embodiments may be interpreted as having the same meaning as decoding and decoder, respectively.

[0184] Embodiments according to the present invention relate to Video-based Dynamic Mesh Compression (V-DMC), which is a method for compressing three-dimensional dynamic mesh data using a conventional 2D video codec. The embodiments may include an arithmetic coded displacement sub-bitstream, which is one of the components of a V-DMC bitstream, and a method for determining the size of an output memory (or output buffer) for storing a frame decoded by an arithmetic coded displacement decoder for decoding the same.

[0185] The embodiments relate to Video-based Dynamic Mesh Compression (V-DMC), a method for compressing three-dimensional dynamic mesh data using an existing 2D video codec, and may include an arithmetic coded displacement sub-bitstream, which is one of the components of a V-DMC bitstream, a method for implementing an output buffer of an arithmetic coded displacement decoder for decoding the same, and a method for signaling a syntax element for the same.

[0186] In this specification, displacement, arithmetic coded displacement, or AC displacement may refer to the same data.

[0187] The standard document ISO / IEC 23090-29 for V-DMC entered the DIS phase as of November 2024. The standard document ISO / IEC 23090-29 for V-DMC described below may refer to the standard document that entered the DIS phase as of November 2024.

[0188] Regarding the operation of an AC displacement decoder for decoding an AC displacement sub-bitstream, the V-DMC standard requires a specific definition of a decoded displacement buffer for storing one or more decoded displacement frames after decoding is complete.

[0189] A basemesh decoder and a displacement decoder each perform decoding on a received compressed and encoded basemesh sub-bitstream or displacement sub-bitstream, and the decoded result may consist of one or more basemesh frames or displacement frames. At this time, the decoded frames can be immediately presented (or displayed or rendered), but if the presentation order and the coding order differ during the actual decoding process, there may be cases where they need to be stored in output memory for a certain period of time for reasons such as reordering to match the presentation order.

[0190] In addition, since a decoded frame must be referenced during the decoding process of another frame, there are cases where the frame must be stored in output memory and continuously maintained for this purpose. That is, the basemesh decoder and the displacement decoder each have a decoder output buffer (memory) for storing the decoded basemesh frame and the decoded displacement frame, respectively, for the purpose of performing the process described above, and the buffer devices can be referred to as the decoded basemesh buffer (DBMB) and the decoded displacement buffer (DDB), respectively.

[0191] According to the embodiments, the DBMB or DDB may store decoded basemesh frames or displacement frames to store the frames before performing a series of processing for visual output, or may store reference frames for other frames during the decoding process.

[0192] The decoded basemesh buffer (DBMB) and the decoded displacement buffer (DDB) can store the decoded basemesh frame and the decoded displacement frame, respectively.

[0193] In decoder operation, it is essential to allocate sufficient buffers to store decoded output data having a size defined according to a given level of the encoded bitstream. The maximum size of these buffers can be determined based on the number of samples per access unit (mesh vertex and / or displacement value) and the upper limit of the number of samples per access unit defined for each level / tier.

[0194] For example, in the case of a basemesh, the variable BmFrameVertCount represents the number of vertices within the basemesh frame. The upper limit of the BmFrameVertCount value is determined by the variable MaxVertsPerBmFrame, which is defined for a given level of the basemesh bitstream according to subclause H.12.4 Tiers and levels of the V-DMC standard document (ISO / IEC 23090-29). The two variables mentioned above, namely BmFrameVertCount and MaxVertsPerBmFrame, are used to determine MaxDbmbSize, a variable representing the maximum size of the DBMB.

[0195] However, in the current version of the V-DMC standard document (as of November 2024, DIS stage), there is no defined method to specify or calculate BmFrameVertCount, so it is impossible to calculate MaxDbmbSize.

[0196] While it is not impossible to derive the actual number of vertices for each basemesh frame, this is only possible during the basemesh decoding process. To ensure reliable decoder operation, the value must be available prior to the decoding process so that the output buffer or memory size for storing the decoded basemesh frames can be pre-allocated. Therefore, a means is required to define a pre-determined value for the number of vertices per basemesh frame.

[0197] Since the number of vertices can vary from frame to frame, signaling the number of vertices in each basemesh frame may not be efficient. Instead, it may be more desirable to signal the maximum number of vertices per frame at the sequence level. The value signaled at the sequence level can be regulated so that the number of vertices in basemesh frames within the sequence does not exceed that value. (This is similar to the relationship between sps_pic_width_max_in_luma_samples / sps_pic_height_max_in_luma_samples and pps_pic_width_in_luma_samples / pps_pic_height_in_luma_samples in the VVC standard).

[0198] Therefore, it is necessary to add a syntax element representing the number of vertices in the basemesh frame to the basemesh sequence parameter set (SPS).

[0199] The same problem exists for displacement data, where the number of samples is used to calculate the maximum required size of the decoded displacement frame buffer. In this case, each displacement value can be considered as a single sample; therefore, it is necessary to add a syntax element indicating the number of displacement values ​​per displacement frame and define a variable indicating the maximum number of displacement values ​​per frame at a specific level.

[0200] The embodiments relate to Video-based Dynamic Mesh Compression (V-DMC), a method for compressing three-dimensional dynamic mesh data using an existing 2D video codec. The embodiments may include a method for implementing an output buffer for storing one or more displacement frames that have been decoded, and a method for signaling syntax elements for requirements regarding such a buffer, in the operation of a displacement sub-bitstream, which is one of the components of a V-DMC bitstream, and a displacement decoder for decoding it.

[0201] In the embodiments, the displacement decoder can determine in advance the size of the memory / buffer for storing the decoded data in order to receive the corresponding displacement sub-bitstream through these syntax elements and accurately decode it.

[0202] FIG. 14 shows an encoded dynamic bitstream structure according to embodiments.

[0203] The V-DMC according to the embodiments may be referred to as a V-mesh.

[0204] Dynamic mesh content can be encoded into a bitstream structure as shown in Fig. 14. Referring to Fig. 14, dynamic mesh content can use the sample stream data unit used when encoding V3C content in the V3C standard document (ISO / IEC 23090-5).

[0205] The encoding device / method illustrated in FIGS. 1 to 4, FIGS. 7, FIGS. 12, etc., according to embodiments can encode mesh data and generate a bitstream containing the encoded mesh data.

[0206] The decoding device / method illustrated in FIG. 1, FIG. 2, FIG. 11, FIG. 13, etc., according to embodiments can acquire a bitstream of FIG. 14 and decode mesh data in FIG. 14 based on parameter information in the bitstream of FIG. 14.

[0207] Each abbreviation used in Fig. 14 means the following. Each abbreviation may be referred to by other terms within the scope of equivalent meaning.

[0208] VPS: V3C / V-DMC Parameter Set

[0209] AD: Atlas Data

[0210] BMD: Base Mesh Data

[0211] DD: Displacement Data (when displacement data is encoded using arithmetic coding)

[0212] GVD: Geometry Video Data (when displacement data is encoded using a video codec)

[0213] AVD: Attribute Video Data (Video Coding)

[0214] PVD: Packing Video Data (Video Coding)

[0215] The encoding device / method illustrated in FIGS. 1 to 4, FIGS. 7, FIGS. 12, etc., according to embodiments can encode mesh data and generate parameter information related to the encoded mesh data.

[0216] The decoding device / method illustrated in FIG. 1, FIG. 2, FIG. 11, FIG. 13, etc., according to embodiments acquires a bitstream of FIG. 14, and based on parameter information within the bitstream of FIG. 14, parameter information within FIG. 14 can be acquired.

[0217] To define the maximum buffer size for a decoded displacement frame, embodiments may include a Displacement Sequence Parameter Set (DSPS) within a displacement sub-bitstream that includes a syntax element indicating the number of displacement values ​​within the displacement frame.

[0218] Specifically, dsps_max_num_displ_vals_div_1000, a syntax element defined in the set of displacement sequence parameters, represents the number of displacement data included in a single displacement frame. According to the embodiments, dsps_max_num_displ_vals_div_1000 may represent the number of displacement samples (or displacement values) included in a single displacement frame.

[0219] According to the embodiments, the value of dsps_max_num_displ_vals_div_1000 can be defined for a displacement sequence composed of a plurality of displacement frames. The value of dsps_max_num_displ_vals_div_1000 can be specified as the largest value among the number of displacement data corresponding to each displacement frame included in the displacement sequence.

[0220] As a result, the receiver can predict in advance the size of the displacement frame constituting the input displacement sub-bitstream through the syntax element dsps_max_num_displ_vals_div_1000, and based on this, can efficiently allocate the size of the decoder output buffer.

[0221] According to embodiments, to specify a maximum buffer size for decoded displacement frames, a set of displacement sequence parameters may include a syntax element indicating the number of displacement values ​​(or displacement samples) within a displacement frame. The above syntax element may indicate the maximum value among the number of displacement values ​​(or displacement samples) included in each displacement frame associated with the set of displacement sequence parameters.

[0222] Additionally, for efficient memory usage, the maximum buffer size for decoded displacement frames can be adaptively specified based on the size of the displacement frames. To this end, the variable MaxDdbSize is introduced, and the above variable MaxDdbSize can be used to define the value range of the corresponding syntax element in the semantics of dsps_max_dec_displ_frame_buffering_minus1.

[0223] FIG. 15 shows a general displacement sequence parameter set low byte sequence payload syntax according to embodiments.

[0224] The displacement sequence parameter set raw byte sequence payload (displ_sequence_parameter_set_rbsp) of FIG. 15 may be included in the bitstream of FIG. 14.

[0225] The displacement sequence parameter set low byte sequence payload of FIG. 15 may refer to the displacement sequence parameter set described above.

[0226] The sequence parameter set ID (dsps_sequence_parameter_set_id) provides an identifier for the displacement sequence parameter set for reference by other syntax elements.

[0227] The value of dsps_max_sub_layers_minus1 plus 1 specifies the maximum number of temporal sub-layers that may exist within each coded displacement sequence (CDS) referencing the set of displacement sequence parameters (DSPS). The value of dsps_max_sub_layers_minus1 must be in the range of 0 to 6 (including 0 and 6).

[0228] The value of dsps_max_dec_displ_frame_buffering_minus1 plus 1 specifies the maximum size of the decoded displacement frame buffer required for CDS in units of displacement frame storage buffer. The value of dsps_max_dec_displ_frame_buffering_minus1 must be within the range of 0 to MaxDdbSize - 1 (including 0 and MaxDdbSize - 1).

[0229] The value obtained by adding 1 to the displacement 3d bit depth (dsps_displacement_3d_bit_depth_minus1) represents the bit depth value applied to the inverse quantization of the displacement samples. dsps_displacement_3d_bit_depth_minus1 must be within the range of 0 to 31 (including 0 and 31).

[0230] The value obtained by multiplying the maximum number of displacement values ​​(dsps_max_num_displ_vals_div_1000) by 1000 represents the number of displacement values ​​within the displacement frame.

[0231] A subdivision present flag (dsps_subdivision_present_flag) of 1 indicates that dsps_subdivision_iteration_count exists within the set of displacement sequence parameters. A dsps_subdivision_present_flag of 0 indicates that dsps_subdivision_iteration_count does not exist within the set of displacement sequence parameters.

[0232] Profiles and levels

[0233] The method by which a receiver according to the embodiments allocates the maximum size of a decoded displacement buffer (DDB) using the dsps_max_num_displ_vals_div_1000 described above is as follows.

[0234] According to the embodiments, the value of the variable MaxDdbSize may represent the number of decoded displacement frames to be stored in the output buffer.

[0235] According to the embodiments, the value of the variable MaxDisplValsPerDFrame may represent the maximum number of displacement samples (or displacement values) that a displacement frame within a displacement sub-bitstream can have, for a level representing the specification of the displacement sub-bitstream.

[0236] According to the embodiments, the variable MaxDisplValsPerDFrame may have different values ​​depending on the level value representing the specifications of the displacement sub-bitstream. According to the embodiments, the value of MaxDisplValsPerDFrame at level L can be derived from the following formula.

[0237]

[0238] The maximum number of basemesh vertices per level (MaxVertsPerBmFrame) may be defined in a table in Annex H.12.4, "Tiers and levels," of the V-DMC standard document.

[0239] The maximum number of subdivisions per level (MaxSubdivisionCount) may be defined in the table of Annex A.6.1 General tier and level limits of the V-DMC standard document.

[0240] The value of the variable DframeDisplCount may represent the maximum number of displacement values ​​belonging to each displacement frame among the displacement frames included in the displacement sequence. The value of the variable DframeDisplCount can be calculated by multiplying the value of the syntax element dsps_max_num_displ_vals_div_1000 described above by 1000.

[0241] According to the embodiments, the number of displacement frames stored in the output buffer is variably determined based on the ratio of the size of the received displacement frame to MaxDisplValsPerDFrame.

[0242] As shown below, the value of the variable MaxDdbSize according to the embodiments can be variably determined based on the value of the variable DframeDisplCount for the value of the variable MaxDisplValsPerDFrame.

[0243] The value of the syntax element dsps_max_dec_displ_frame_buffering_minus1 plus 1 may be equal to or greater than the value of the variable MaxDdbSize. The value of MaxDdbSize can be derived as follows.

[0244] if( DFrameDisplCount / MaxDisplValsPerDFrame <= 1 / 4 )

[0245] MaxDdbSize = 16

[0246] else if( DFrameDisplCount / MaxDisplValsPerDFrame <= 1 / 2 )

[0247] MaxDdbSize = 12

[0248] else if( DFrameDisplCount / MaxDisplValsPerDFrame <= 3 / 4 )

[0249] MaxDdbSize = 8

[0250] else

[0251] MaxDdbSize = 6

[0252] The value of the variable DframeDisplCount can be set to be equal to the value of the syntax element dsps_max_num_displ_vals_div_1000 multiplied by 1000, and the variable MaxDisplValsPerDFrame can represent an upper limit on the number of displacement values ​​per frame of a specific level. The value can be specified in a table in the V-DMC standard document.

[0253] The above-described embodiments explained that the value of the variable DframeDisplCount is calculated based on the value of the syntax element dsps_max_num_displ_vals_div_1000, but are not limited thereto, and the embodiments may calculate the value of the DFrameDisplCount variable using the DecSubDispPerFrameCount and DecDispValueCountPerSubdisp arrays output after AC displacement decoding.

[0254] The array (DecSubDispPerFrameCount, DecDispValueCountPerSubdisp) is an array defined as the output of section 9.10 Displacement decoding process of the V3C standard document (ISO / IEC 23090-5).

[0255] The DecSubDispPerFrameCount array is a 1D array representing the number of subdisplacements within a displacement frame. Each dimension may correspond to a decoded displacement frame index.

[0256] The DecDispValueCountPerSubdisp array is a 2D array representing the number of displacement values ​​contained in subdisplacement units within a displacement frame. Each dimension may correspond to a decoded displacement frame index and a subdisplacement index.

[0257] The process of calculating the value of the DFrameDisplCount variable can be as shown in the pseudocode below, and the value of the DFrameDisplCount variable can be obtained through the number of displacement samples of the displacement frame corresponding to the displacement frame index dfIdx.

[0258] According to the embodiments, the DFrameDisplCount variable can be obtained by repeating DecSubDispPerFrameCount[dfIdx] times and adding up all the sample counts (DecDispValueCountPerSubdisp[dfIdx][j]) of each subdisplacement and accumulating them in DFrameDisplCount.

[0259] The variable DframeDisplCount can represent the number of displacement samples within an AC displacement frame with frame index dfIdx. The variable DframeDisplCount can be calculated using the following pseudocode.

[0260] DFrameDisplCount = 0;

[0261] for ( j=0; j < DecSubDispPerFrameCount[ dfIdx ]; j++ ) {

[0262] DFrameDisplCount += DecDispValueCountPerSubdisp[ dfIdx ][ j ];

[0263] }

[0264] The variables according to the embodiments are not limited to the names described above, and other names may be used within the scope of expressing an equivalent meaning.

[0265] A receiver according to the embodiments can determine the number of displacement data included in a displacement frame through the value specified in dsps_max_num_displ_vals_div_1000, which is signaled in the input displacement sub-bitstream.

[0266] In addition, the receiver according to the embodiments can read the level value signaled in the displacement sub-bitstream and determine the corresponding MaxDisplValsPerDFrame value (the maximum value of the number of displacement data that can be included in one frame allowed at that level).

[0267] In other words, MaxDdbSize is determined through the ratio of the actual number of displacement data points per frame included in the received displacement sub-bitstream to the number of displacement data points per frame required by the maximum specification for decoding the corresponding sub-stream.

[0268] At this time, MaxDdbSize is the number of decoded displacement frames. For example, if the number of displacement data per frame allowed at the level value of the currently received displacement sub-bitstream is 100K and there are 50K displacement data in the displacement frames included in the actual bitstream, then according to the formula above, the receiving device must secure an output buffer capable of storing at least 12 displacement frames.

[0269] FIG. 16 illustrates a encoding method according to embodiments.

[0270] The encoding method according to the embodiments may include the step of encoding base mesh data of the mesh data (S1600); the step of encoding displacement data of the mesh data (S1610); and the step of encoding attribute data of the mesh data (S1620).

[0271] The encoding method according to the embodiments may include the method described in FIGS. 1 to 15 above. Specifically, the step of encoding base mesh data (S1600) may include the method described in FIGS. 1 to 7, FIG. 12, FIG. 14, etc. above. The step of encoding displacement data of mesh data (S1610) may include the method described in FIGS. 1 to 4, FIGS. 6 to 9, FIG. 12, FIGS. 14 to 15, etc. above. The step of encoding attribute data of mesh data (S1620) may include the method described in FIGS. 1 to 4, FIG. 10, FIG. 12, FIG. 14, etc. above.

[0272] The encoding method according to the embodiments includes encoded displacement data in a displacement sub-bitstream, and the method may include the step of deriving a value of a third variable for a maximum size of a displacement buffer of the displacement sub-bitstream based on a first variable for a number of displacement values ​​in a displacement frame for the displacement data and a second variable for a maximum number of displacement values ​​in a displacement frame related to a level for the displacement sub-bitstream.

[0273] Referring together with FIG. 15, the encoding method according to the embodiments includes a displacement sub-bitstream comprising a displacement sequence parameter set raw byte sequence payload (displ_sequence_parameter_set_rbsp), and the displacement sequence parameter set raw byte sequence payload comprises first information regarding the number of displacement values ​​within a displacement frame for displacement data, and the value of a first variable can be derived based on the value of the first information.

[0274] In the encoding method according to the embodiments, the value of the first variable can be derived based on the value of the variable for the number of sub-displacements within the displacement frame for the displacement data and the value of the variable for the number of displacement values ​​included in the sub-displacement unit within the displacement frame.

[0275] A encoding method according to embodiments includes encoded displacement data in a displacement sub-bitstream, and the method comprises the step of deriving a value of a third variable for a maximum size of a displacement buffer of the displacement sub-bitstream based on the ratio of a first variable for a number of displacement samples in a displacement frame for the displacement data to a second variable for a maximum number of displacement samples in a displacement frame related to a level for the displacement sub-bitstream, wherein the value of the first variable may be derived based on the value of a variable for the number of sub-displacements in a displacement frame for the displacement data and the value of a variable for the number of displacement values ​​included in a sub-displacement unit in a displacement frame.

[0276] The encoding method is performed by an encoding device. The encoding device includes a memory; and at least one processor connected to the memory; and the at least one processor may be configured to: encode base mesh data of the mesh data; encode displacement data of the mesh data; and encode attribute data of the mesh data.

[0277] The embodiments may further include a computer-readable storage medium that stores a bitstream generated by the method of FIG. 16.

[0278] The embodiments may further include a method comprising the steps of: acquiring a bitstream for mesh data; generating the bitstream based on the steps of: encoding base mesh data of the mesh data; encoding displacement data of the mesh data; and encoding attribute data of the mesh data; and transmitting data including the bitstream.

[0279] FIG. 17 illustrates a decoding method according to embodiments.

[0280] The decoding method according to the embodiments may include the step of decoding base mesh data in a bitstream (S1700); the step of decoding displacement data in a bitstream (S1710); and the step of decoding attribute data in a bitstream (S1720).

[0281] The decoding method according to the embodiments may include the method described in FIGS. 1 to 15 above. Specifically, the step of decoding basemesh data in the bitstream (S1700) may include the method described in FIGS. 1, 2, 11, 13, 14, etc. above. The step of decoding displacement data in the bitstream (S1710) may include the method described in FIGS. 1, 2, 11, 13, 14, 15, etc. above. The step of decoding attribute data in the bitstream (S1720) may include the method described in FIGS. 1, 2, 11, 13, 14, etc. above.

[0282] The decoding method according to the embodiments includes displacement data contained in a displacement sub-bitstream within a bitstream, and the step of decoding the displacement data may include deriving a value of a third variable for a maximum size of a decoded displacement buffer of the displacement sub-bitstream based on a first variable for a number of displacement values ​​in a displacement frame for the displacement data and a second variable for a maximum number of displacement values ​​in a displacement frame related to a level for the displacement sub-bitstream.

[0283] Referring together with FIG. 15, the decoding method according to the embodiments includes a displacement sub-bitstream comprising a displacement sequence parameter set raw byte sequence payload (displ_sequence_parameter_set_rbsp), and the displacement sequence parameter set raw byte sequence payload may include first information regarding the number of displacement values ​​within a displacement frame for displacement data.

[0284] In the decoding method according to the embodiments, the value of the first variable can be derived based on the value of the first information.

[0285] The decoding method according to the embodiments can derive the value of the first variable based on the value of the variable for the number of sub-displacements within the displacement frame for the displacement data and the value of the variable for the number of displacement values ​​included in the sub-displacement unit within the displacement frame.

[0286] A decoding method according to embodiments comprises, wherein displacement data is included in a displacement sub-bitstream within a bitstream, and the step of decoding the displacement data includes deriving a value of a third variable for a maximum size of a decoded displacement buffer of the displacement sub-bitstream based on the ratio of a first variable for a number of displacement samples in a displacement frame for the displacement data to a second variable for a maximum number of displacement samples in a displacement frame related to a level for the displacement sub-bitstream, and the value of the first variable may be derived based on the value of a variable for the number of sub-displacements in a displacement frame for the displacement data and the value of a variable for the number of displacement values ​​included in a sub-displacement unit in a displacement frame.

[0287] Referring to FIG. 15, the decoding method according to the embodiments includes a displacement sub-bitstream comprising a displacement sequence parameter set raw byte sequence payload (displ_sequence_parameter_set_rbsp), and the displacement sequence parameter set raw byte sequence payload comprising second information for representing a maximum required size of a decoded displacement frame buffer for a coded displacement sequence in units of displacement frame storage buffers, and the range of values ​​of the second information can be derived based on the value of a third variable.

[0288] The embodiments may include a method for allocating memory size for an output buffer for storing a decoded frame during the operation of a displacement sub-bitstream, which is one of the components of a V-DMC bitstream, and a method for signaling syntax elements therefor.

[0289] Through these syntax elements, the displacement decoder can determine in advance what level of memory specification is required to receive and accurately decode the corresponding sub-bitstream, and perform preliminary operations to secure the corresponding memory / buffer.

[0290] The embodiments have been described in terms of methods and / or devices, and the description of the methods and the description of the devices may be applied complementarily.

[0291] Although the drawings have been described separately for the convenience of explanation, it is also possible to design a new embodiment by combining the embodiments described in each drawing. Furthermore, designing a computer-readable recording medium containing a program for executing the previously described embodiments, as required by a person skilled in the art, falls within the scope of the claims of the embodiments. The apparatus and method according to the embodiments are not limited to the configuration and method of the embodiments described above; rather, the embodiments may be configured by selectively combining all or part of each embodiment to allow for various modifications. Although preferred embodiments have been illustrated and described, the embodiments are not limited to the specific embodiments described above. It is not only possible for a person skilled in the art to make various modifications without departing from the essence of the embodiments claimed in the claims, but such modifications should not be understood individually from the technical concept or perspective of the embodiments.

[0292] Various components of the device of the embodiments may be implemented by hardware, software, firmware, or a combination thereof. Various components of the embodiments may be implemented as a single chip, for example, a single hardware circuit. Depending on the embodiments, the components according to the embodiments may each be implemented as separate chips. Depending on the embodiments, at least one of the components of the device according to the embodiments may be composed of one or more processors capable of executing one or more programs, and one or more programs may include instructions for performing or executing any one or more of the operations / methods according to the embodiments. Executable instructions for performing the methods / operations of the device according to the embodiments may be stored in non-transient CRMs or other computer program products configured to be executed by one or more processors, or may be stored in transient CRMs or other computer program products configured to be executed by one or more processors. Additionally, memory according to the embodiments may be used as a concept that includes not only volatile memory (e.g., RAM, etc.) but also non-volatile memory, flash memory, PROM, etc. In addition, it may also include implementation in the form of carrier waves, such as transmission over the Internet. Furthermore, processor-readable recording media are distributed across networked computer systems, allowing processor-readable code to be stored and executed in a distributed manner.

[0293] In this document, " / " and "," are interpreted as "and / or." For example, "A / B" is interpreted as "A and / or B," and "A, B" is interpreted as "A and / or B." Additionally, "A / B / C" means "at least one of A, B and / or C." Also, "A, B, C" means "at least one of A, B and / or C." Additionally, in this document, "or" is interpreted as "and / or." For example, "A or B" may mean 1) "A" only, 2) "B" only, or 3) "A and B." In other words, "or" in this document may mean "additionally or alternatively."

[0294] Terms such as "first," "second," etc., may be used to describe various components of the embodiments. However, the interpretation of the various components according to the embodiments should not be limited by these terms. These terms are merely used to distinguish one component from another. For example, the first user input signal may be referred to as the second user input signal. Similarly, the second user input signal may be referred to as the first user input signal. The use of these terms should be interpreted as not departing from the scope of the various embodiments. Although the first user input signal and the second user input signal are both user input signals, they do not mean the same user input signals unless clearly indicated in the context.

[0295] The terms used to describe the embodiments are intended for the purpose of describing specific embodiments and are not intended to limit the embodiments. As used in the description of the embodiments and in the claims, the singular is intended to include the plural unless explicitly indicated in the context. Expressions of and / or are used to mean including all possible combinations between the terms. Expressions of include describe the presence of features, numbers, steps, elements, and / or components and do not imply the exclusion of additional features, numbers, steps, elements, and / or components. Conditional expressions such as "if" or "when" used to describe the embodiments are not limited to being optional. It is intended to be interpreted as "when a specific condition is satisfied," "when a related action is performed in response to a specific condition," or "when a related definition is interpreted."

[0296] Additionally, operations according to the embodiments described herein may be performed by a transmitting and receiving device including memory and / or a processor, depending on the embodiments. The memory may store programs for processing / controlling operations according to the embodiments, and the processor may control various operations described in this document. The processor may be referred to as a controller, etc. Operations in the embodiments may be performed by firmware, software, and / or a combination thereof, and the firmware, software, and / or a combination thereof may be stored in the processor or in memory.

[0297] Meanwhile, the operation according to the embodiments described above may be performed by a transmitting device and / or a receiving device according to the embodiments. The transmitting and receiving device may include a transmitting and receiving unit for transmitting and receiving media data, a memory for storing instructions (program code, algorithm, flowchart and / or data) for a process according to the embodiments, and a processor for controlling the operations of the transmitting and receiving devices.

[0298] The processor may be referred to as a controller, etc., and may correspond, for example, to hardware, software, and / or a combination thereof. The operation according to the embodiments described above may be performed by the processor. Additionally, the processor may be implemented as an encoder / decoder, etc., for the operation of the embodiments described above.

[0299]

[0300] As described above, the relevant details have been explained in the best mode for carrying out the embodiments.

[0301]

[0302] As described above, the embodiments may be applied wholly or partially to point cloud data transmission and reception devices and systems.

[0303] Those skilled in the art may make various changes or modifications to the embodiments within the scope of the embodiments.

[0304] The embodiments may include modifications / variations, and such modifications / variations do not exceed the scope of the claims and their equivalents.

Claims

1. A step of decoding basemesh data within a bitstream; A step of decoding displacement data within the bitstream; and A step of decoding attribute data within the bitstream; comprising Decryption method.

2. In Paragraph 1, The above displacement data is included in the displacement sub-bitstream within the bitstream, and The step of decoding the above displacement data is, A method comprising the step of deriving a value of a third variable for a maximum size of a decoded displacement buffer of the displacement sub-bitstream, based on a first variable for a number of displacement values ​​in a displacement frame for the displacement data and a second variable for a maximum number of displacement values ​​in a displacement frame related to a level for the displacement sub-bitstream. Decryption method.

3. In Paragraph 2, The above displacement sub-bitstream includes a displacement sequence parameter set raw byte sequence payload (displ_sequence_parameter_set_rbsp), and The above displacement sequence parameter set low byte sequence payload is, including first information regarding the number of displacement values ​​within the displacement frame for the above displacement data, Decryption method.

4. In Paragraph 3, Based on the value of the first information above, the value of the first variable is derived, Decryption method.

5. In Paragraph 2, The value of the first variable is derived based on the value of a variable regarding the number of sub-displacements within the displacement frame for the displacement data and the value of a variable regarding the number of displacement values ​​included in the sub-displacement unit within the displacement frame. Decryption method.

6. In Paragraph 1, The above displacement data is included in the displacement sub-bitstream within the bitstream, and The step of decoding the above displacement data is, The method includes the step of deriving a value of a third variable for a maximum size of a decoded displacement buffer of the displacement sub-bitstream, based on the ratio of a first variable for a number of displacement samples in a displacement frame for the displacement data to a second variable for a maximum number of displacement samples in a displacement frame related to a level for the displacement sub-bitstream. The value of the first variable is derived based on the value of a variable regarding the number of sub-displacements within the displacement frame for the displacement data and the value of a variable regarding the number of displacement values ​​included in the sub-displacement unit within the displacement frame. Decryption method.

7. In Paragraph 2, The above displacement sub-bitstream includes a displacement sequence parameter set raw byte sequence payload (displ_sequence_parameter_set_rbsp), and The above displacement sequence parameter set low byte sequence payload is, It includes second information for representing a maximum required size of a decoded displacement frame buffer for a coded displacement sequence in units of displacement frame storage buffers, and The range of values ​​of the second information above is derived based on the value of the third variable, Decryption method.

8. Memory; and At least one processor connected to the memory; comprising, wherein the at least one processor: Decoding basemesh data within a bitstream; Decoding displacement data within the above bitstream; and Configured to decode attribute data within the above bitstream, Decoding device.

9. Step of encoding the base mesh data of the mesh data; A step of encoding displacement data of the above mesh data; and A step of encoding attribute data of the above mesh data; comprising Encoding method.

10. In Paragraph 9, The above encoded displacement data is included in the displacement sub-bitstream, and The above method is, A method comprising the step of deriving a value of a third variable for a maximum size of a displacement buffer of the displacement sub-bitstream, based on a first variable for a number of displacement values ​​in a displacement frame for the displacement data and a second variable for a maximum number of displacement values ​​in a displacement frame related to a level for the displacement sub-bitstream. Encoding method.

11. In Paragraph 10, The above displacement sub-bitstream includes a displacement sequence parameter set raw byte sequence payload (displ_sequence_parameter_set_rbsp), and The above displacement sequence parameter set low byte sequence payload is, including first information regarding the number of displacement values ​​within the displacement frame for the above displacement data, Based on the value of the first information above, the value of the first variable is derived, Encoding method.

12. In Paragraph 10, The value of the first variable is derived based on the value of a variable regarding the number of sub-displacements within the displacement frame for the displacement data and the value of a variable regarding the number of displacement values ​​included in the sub-displacement unit within the displacement frame. Encoding method.

13. In Paragraph 9, The above encoded displacement data is included in the displacement sub-bitstream, and The above method is, The method includes the step of deriving a value of a third variable for a maximum size of a displacement buffer of the displacement sub-bitstream based on the ratio of a first variable for a number of displacement samples in a displacement frame for the displacement data to a second variable for a maximum number of displacement samples in a displacement frame related to a level for the displacement sub-bitstream. The value of the first variable is derived based on the value of a variable regarding the number of sub-displacements within the displacement frame for the displacement data and the value of a variable regarding the number of displacement values ​​included in the sub-displacement unit within the displacement frame. Encoding method.

14. Memory; and At least one processor connected to the memory; comprising, wherein the at least one processor: Encoding the base mesh data of the mesh data; Encoding the displacement data of the above mesh data; and Configured to encode the attribute data of the above mesh data; Encoding device.

15. A computer-readable storage medium for storing a bitstream generated by the method according to paragraph 9.

16. Step of acquiring a bitstream for mesh data, The bitstream is generated based on the steps of: encoding base mesh data of the mesh data; encoding displacement data of the mesh data; and encoding attribute data of the mesh data; and A method comprising the step of transmitting data including the bitstream above.

Images