Gaussian splat data encoding apparatus, gaussian splat data encoding method, gaussian splat data decoding apparatus, and gaussian splat data decoding method

Abstract

A method according to embodiments comprises the steps of: acquiring a bitstream including Gaussian splat data; and decoding the Gaussian splat data. A method according to embodiments comprises the steps of: encoding Gaussian splat data; and generating a bitstream including the Gaussian splat data.

Description

Gaussian splat data encoding device, Gaussian splat data encoding method, Gaussian splat data decoding device and Gaussian splat data decoding method

[0001] The embodiments provide a method for providing Gaussian Splat content to provide various services to users, such as VR (Virtual Reality), AR (Augmented Reality), MR (Mixed Reality), and autonomous driving services.

[0002] In the fields of 3D graphics and computer vision, scene representation and rendering technologies have been developed to digitally represent real or virtual environments and visualize them from various viewpoints. While scenes can be represented using meshes, point clouds, voxels, etc., issues such as increased data size in complex scenes, computational load and memory burden, and additional processing for high-quality rendering (e.g., surface rendering, interpolation, high-density sampling) may arise. Therefore, technology that simultaneously satisfies representation efficiency and rendering quality / speed is required.

[0003] Against this backdrop, 3D Gaussian Splatting is proposed as a technique that represents 3D space as a set of multiple small Gaussian elements and indicates spatial contribution by utilizing the Gaussian distribution characteristic where the value of each element decreases as it moves away from the center. This method can construct and render a scene by projecting Gaussian elements onto an image plane according to the camera viewpoint and cumulatively synthesizing their contributions. However, additional improvement techniques may be required to efficiently generate and optimize the parameters of the Gaussian elements, control computational load and memory usage as the number of elements increases, and achieve efficient representation and encoding from the perspective of transmission and storage.

[0004] The technical problem according to the embodiments is to provide a Gaussian splat data transmission device, a transmission method, a Gaussian splat data reception device, and a reception method for efficiently transmitting and receiving Gaussian splat data in order to solve the aforementioned problems, etc.

[0005] The technical problem according to the embodiments is to provide a Gaussian splat data transmission device, a transmission method, a Gaussian splat data reception device, and a reception method for solving latency and encoding / decoding complexity.

[0006] However, the scope of rights of the embodiments is not limited to the technical problems described above, and may be extended to other technical problems that can be inferred by a person skilled in the art based on the entire content of this document.

[0007] A method according to the embodiments comprises the steps of: acquiring a bitstream containing Gaussian splat data; and decoding the Gaussian splat data. A method according to the embodiments comprises the steps of: encoding the Gaussian splat data; and generating a bitstream containing the Gaussian splat data.

[0008] The Gaussian splat data transmission method, transmission device, Gaussian splat data reception method, and reception device according to the embodiments can provide a high-quality Gaussian splat service.

[0009] The Gaussian splat data transmission method, transmission device, Gaussian splat data reception method, and reception device according to the embodiments can achieve various video codec methods.

[0010] The Gaussian splat data transmission method, transmission device, Gaussian splat data reception method, and reception device according to the embodiments can provide general-purpose Gaussian splat content, such as autonomous driving services.

[0011] Drawings are included to further understand the embodiments, and the drawings illustrate the embodiments along with descriptions related to the embodiments.

[0012] FIG. 1 shows an example of the structure of a transmitting / receiving system for providing Point Cloud content according to embodiments.

[0013] FIG. 2 shows an example of point cloud data capture according to embodiments.

[0014] FIG. 3 shows examples of point clouds, geometry, and texture images according to embodiments.

[0015] FIG. 4 shows an example of V-PCC encoding processing according to embodiments.

[0016] FIG. 5 shows examples of a tangent plane and a normal vector of a surface according to embodiments.

[0017] FIG. 6 shows an example of a bounding box of a point cloud according to embodiments.

[0018] FIG. 7 shows an example of determining the location of individual patches of an occupancy map according to embodiments.

[0019] FIG. 8 shows an example of the relationship between the normal, tangent, and bitangent axes according to the embodiments.

[0020] FIG. 9 shows an example of the configuration of the minimum mode and maximum mode of the projection mode according to the embodiments.

[0021] FIG. 10 shows an example of an EDD code according to embodiments.

[0022] FIG. 11 shows an example of recoloring using color values ​​of adjacent points according to embodiments.

[0023] FIG. 12 shows an example of push-pull background filling according to embodiments.

[0024] FIG. 13 shows examples of possible traversal orders for a 4x4 block according to embodiments.

[0025] FIG. 14 shows an example of a best traversal order according to embodiments.

[0026] FIG. 15 shows an example of a 2D video / image encoder according to embodiments.

[0027] FIG. 16 shows an example of a V-PCC decoding process according to embodiments.

[0028] FIG. 17 shows an example of a 2D video / image decoder according to embodiments.

[0029] FIG. 18 shows an example of an operation flowchart of a transmitting device according to embodiments.

[0030] FIG. 19 shows an example of an operation flowchart of a receiving device according to embodiments.

[0031] FIG. 20 shows an example of a structure capable of interoperability with a point cloud data transmission and reception method / device according to embodiments.

[0032] FIG. 21 shows the components of 3DGS data according to embodiments.

[0033] FIG. 22 shows a V3C bitstream structure according to embodiments.

[0034] FIG. 23 illustrates an example of a V-GSC encoder structure according to embodiments.

[0035] FIG. 24 shows pre-encoding according to embodiments.

[0036] FIG. 25 shows a V-GSC decoder according to embodiments.

[0037] FIG. 26 shows a post-decoder according to embodiments.

[0038] FIG. 27 shows a graphics engine according to embodiments.

[0039] FIG. 28 illustrates a encoding method according to embodiments.

[0040] FIG. 29 illustrates a decoding method according to embodiments.

[0041] Preferred embodiments of the embodiments are described in detail, and examples thereof are shown in the accompanying drawings. The following detailed description, with reference to the accompanying drawings, is intended to describe preferred embodiments of the embodiments rather than merely embodiments that may be implemented according to the embodiments. The following detailed description includes details to provide a thorough understanding of the embodiments. However, it is obvious to those skilled in the art that the embodiments may be practiced without these details.

[0042] Most terms used in the embodiments are selected from those commonly used in the field, but some terms are chosen at the applicant's discretion, and their meanings are described in detail in the following description as necessary. Accordingly, the embodiments should be understood based on the intended meaning of the terms, rather than their mere names or meanings.

[0043] FIG. 1 shows an example of the structure of a transmitting / receiving system for providing Point Cloud content according to embodiments.

[0044] This document provides a method for providing Point Cloud content to provide various services to users, such as VR (Virtual Reality), AR (Augmented Reality), MR (Mixed Reality), and autonomous driving services. In the embodiments, Point Cloud content represents data in which objects are represented as points, and may be referred to as a point cloud, point cloud data, point cloud video data, point cloud image data, etc.

[0045] A point cloud data transmission device (Transmission device, 10000) according to embodiments includes a point cloud video acquisition unit (Point Cloud Video Acquisition, 10001), a point cloud video encoder (Point Cloud Video Encoder, 10002), a file / segment encapsulation unit (10003), and / or a transmitter (or communication module, 10004). The transmission device according to embodiments can acquire and process point cloud video (or point cloud content) and transmit it. According to embodiments, the transmission device may include a fixed station, a base transceiver system (BTS), a network, an AI (Artificial Intelligence) device and / or system, a robot, an AR / VR / XR device and / or server, etc. Additionally, according to embodiments, the transmission device (10000) may include a device, robot, vehicle, AR / VR / XR device, mobile device, home appliance, IoT (Internet of Thing) device, AI device / server, etc. that communicates with a base station and / or other wireless device using wireless access technology (e.g., 5G NR (New RAT), LTE (Long Term Evolution)).

[0046] A point cloud video acquisition unit (Point Cloud Video Acquisition, 10001) according to the embodiments acquires a point cloud video through a process such as capturing, synthesizing, or generating a point cloud video.

[0047] A point cloud video encoder (10002) according to embodiments encodes point cloud video data. According to embodiments, the point cloud video encoder (10002) may be referred to as a point cloud encoder, a point cloud data encoder, an encoder, etc. Furthermore, point cloud compression coding (encoding) according to embodiments is not limited to the embodiments described above. The point cloud video encoder may output a bitstream containing encoded point cloud video data. The bitstream may include not only the encoded point cloud video data but also signaling information related to the encoding of the point cloud video data.

[0048] The encoder according to the embodiments may support both G-PCC (Geometry-based Point Cloud Compression) encoding and / or V-PCC (Video-based Point Cloud Compression) encoding. Additionally, the encoder may encode a point cloud (referring to both point cloud data and points) and / or signaling data regarding the point cloud. The specific operation of the encoding according to the embodiments is described below.

[0049] Meanwhile, the term V-PCC used in this document refers to Video-based Point Cloud Compression (V-PCC), and the term V-PCC is synonymous with Visual Volumetric Video-based Coding (V3C) and may be referred to as mutually complementary.

[0050] A file / segment encapsulation module (10003) according to the embodiments encapsulates point cloud data in the form of a file and / or segment. A point cloud data transmission method / device according to the embodiments can transmit point cloud data in the form of a file and / or segment.

[0051] A transmitter (or communication module, 10004) according to the embodiments transmits encoded point cloud video data in the form of a bitstream. According to the embodiments, a file or segment may be transmitted to a receiving device via a network or stored on a digital storage medium (e.g., USB, SD, CD, DVD, Blu-ray, HDD, SSD, etc.). The transmitter according to the embodiments can communicate wired or wirelessly with a receiving device (or receiver) via a network such as 4G, 5G, or 6G. Additionally, the transmitter may perform necessary data processing operations according to a network system (e.g., a communication network system such as 4G, 5G, or 6G). Additionally, the transmitting device may transmit encapsulated data according to an on-demand method.

[0052] A point cloud data receiving device (Reception device, 10005) according to embodiments includes a receiver (10006), a file / segment decapsulation unit (10007), a point cloud video decoder (Point Cloud Decoder, 10008), and / or a renderer (Renderer, 10009). According to embodiments, the receiving device may include a device, robot, vehicle, AR / VR / XR device, mobile device, home appliance, IoT (Internet of Thing) device, AI device / server, etc., which communicates with a base station and / or other wireless device using a wireless access technology (e.g., 5G NR (New RAT), LTE (Long Term Evolution)).

[0053] A receiver (10006) according to the embodiments receives a bitstream containing point cloud video data. According to the embodiments, the receiver (10006) can transmit feedback information to a point cloud data transmission device (10000).

[0054] A file / segment decapsulation module (10007) decapsulates a file and / or segment containing point cloud data. The decapsulation module according to the embodiments can perform the reverse process of the encapsulation process according to the embodiments.

[0055] A point cloud video decoder (Point Cloud Decoder, 10007) decodes received point cloud video data. The decoder according to the embodiments can perform the reverse process of encoding according to the embodiments.

[0056] A renderer (Renderer, 10007) renders decoded point cloud video data. According to embodiments, the renderer (10007) may transmit feedback information obtained from the receiving end to a point cloud video decoder (10006). According to embodiments, the point cloud video data may transmit feedback information to a receiver. According to embodiments, feedback information received by the point cloud transmission device may be provided to a point cloud video encoder.

[0057] The arrows indicated by dotted lines in the drawing represent the transmission path of feedback information obtained from the receiving device (10005). The feedback information is information intended to reflect interaction with a user consuming point cloud content, and includes user information (e.g., head orientation information), viewport information, etc. In particular, if the point cloud content is content for a service requiring interaction with a user (e.g., autonomous driving service, etc.), the feedback information may be transmitted to the content transmitting side (e.g., the transmitting device (10000)) and / or the service provider. Depending on the embodiments, the feedback information may be used in the receiving device (10005) as well as the transmitting device (10000), or it may not be provided.

[0058] Head orientation information according to the embodiments is information regarding the user's head position, direction, angle, movement, etc. The receiving device (10005) according to the embodiments can calculate viewport information based on the head orientation information. Viewport information is information about the area of ​​the point cloud video that the user is looking at. The viewpoint refers to the point where the user is looking at the point cloud video, and may mean the exact center point of the viewport area. That is, the viewport is an area centered on the viewpoint, and the size and shape of the area can be determined by the FOV (Field Of View). Therefore, the receiving device (10004) can extract viewport information based on the vertical or horizontal FOV supported by the device in addition to the head orientation information. In addition, the receiving device (10005) performs gaze analysis, etc., to check the user's point cloud consumption method, the point cloud video area the user is looking at, the gaze time, etc. According to embodiments, the receiving device (10005) may transmit feedback information including gaze analysis results to the transmitting device (10000). According to embodiments, the feedback information may be obtained during the rendering and / or display process. According to embodiments, the feedback information may be obtained by one or more sensors included in the receiving device (10005). Also, according to embodiments, the feedback information may be obtained by the renderer (10009) or a separate external element (or device, component, etc.). The dotted line in FIG. 1 indicates the process of transmitting the feedback information obtained from the renderer (10009). The point cloud content providing system may process (encode / decode) point cloud data based on the feedback information. Accordingly, the point cloud video data decoder (10008) may perform a decoding operation based on the feedback information.Additionally, the receiving device (10005) can transmit feedback information to the transmitting device. The transmitting device (or point cloud video data encoder (10002)) can perform an encoding operation based on the feedback information. Thus, the point cloud content providing system can efficiently process necessary data (e.g., point cloud data corresponding to the user's head position) based on the feedback information without processing (encoding / decoding) all point cloud data, and provide point cloud content to the user.

[0059] According to embodiments, the transmission device (10000) may be referred to as an encoder, transmission device, transmitter, etc., and the receiving device (10004) may be referred to as a decoder, receiving device, receiver, etc.

[0060] Point cloud data processed in the point cloud content providing system of FIG. 1 according to embodiments (processed through a series of processes of acquisition / encoding / transmission / decoding / rendering) may be referred to as point cloud content data or point cloud video data. According to embodiments, point cloud content data may be used as a concept including metadata or signaling information related to point cloud data.

[0061] The elements of the point cloud content delivery system illustrated in FIG. 1 can be implemented using hardware, software, processors, and / or combinations thereof.

[0062] The embodiments may provide point cloud content to provide various services to users, such as VR (Virtual Reality), AR (Augmented Reality), MR (Mixed Reality), and autonomous driving services.

[0063] To provide Point Cloud content services, Point Cloud video may first be acquired. The acquired Point Cloud video is transmitted after undergoing a series of processes, and the receiving end can process the received data back into the original Point Cloud video and render it. Through this, the Point Cloud video can be provided to the user. The embodiments provide the necessary methods to effectively carry out this series of processes.

[0064] The entire process for providing Point Cloud content services (point cloud data transmission method and / or point cloud data reception method) may include an acquisition process, an encoding process, a transmission process, a decoding process, a rendering process and / or a feedback process.

[0065] According to the embodiments, the process of providing point cloud content (or point cloud data) may be referred to as a point cloud compression process. According to the embodiments, the point cloud compression process may mean a geometry-based point cloud compression process.

[0066] Each element of the point cloud data transmission device and the point cloud data receiving device according to the embodiments may mean hardware, software, a processor and / or a combination thereof, etc.

[0067] In order to provide Point Cloud content services, Point Cloud video may first be acquired. The acquired Point Cloud video is transmitted after undergoing a series of processes, and the receiving end can process the received data back into the original Point Cloud video and render it. Through this, the Point Cloud video can be provided to the user. The present invention provides a method necessary to effectively carry out this series of processes.

[0068] The entire process for providing Point Cloud content services may include an acquisition process, an encoding process, a transmission process, a decoding process, a rendering process, and / or a feedback process.

[0069] A Point Cloud Compression system may include a transmission device and a receiving device. The transmission device may encode Point Cloud video to output a bitstream and transmit it to the receiving device via a digital storage medium or network in the form of a file or streaming (streaming segment). The digital storage medium may include various storage media such as USB, SD, CD, DVD, Blu-ray, HDD, and SSD.

[0070] The transmission device may schematically include a Point Cloud video acquisition unit, a Point Cloud video encoder, a file / segment encapsulation unit, and a transmission unit. The receiving device may schematically include a receiving unit, a file / segment decapsulation unit, a Point Cloud video decoder, and a renderer. The encoder may be referred to as a Point Cloud video / image / picture / frame encoding device, and the decoder may be referred to as a Point Cloud video / image / picture / frame decoding device. The transmitter may be included in the Point Cloud video encoder. The receiver may be included in the Point Cloud video decoder. The renderer may include a display unit, and the renderer and / or the display unit may be composed of separate devices or external components. The transmission device and the receiving device may further include separate internal or external modules / units / components for the feedback process.

[0071] According to the embodiments, the operation of the receiving device may follow the reverse process of the operation of the transmitting device.

[0072] The Point Cloud video acquisition unit can perform a process of acquiring Point Cloud video through the capture, synthesis, or generation process of Point Cloud video. Through the acquisition process, 3D position (x, y, z) / attribute (color, reflectance, transparency, etc.) data for multiple points, such as PLY (Polygon File format or the Stanford Triangle format) files, may be generated. In the case of a video with multiple frames, one or more files may be acquired. Point cloud-related metadata (e.g., metadata related to the capture, etc.) may be generated during the capture process.

[0073] A point cloud data transmission device according to the embodiments may include an encoder that encodes point cloud data; and a transmitter that transmits point cloud data. Additionally, it may be transmitted in the form of a bit stream containing a point cloud.

[0074] A point cloud data receiving device according to embodiments may include a receiving unit for receiving point cloud data; a decoder for decoding point cloud data; and a renderer for rendering point cloud data.

[0075] The method / device according to the embodiments represents a point cloud data transmitting device and / or a point cloud data receiving device.

[0076] FIG. 2 shows an example of point cloud data capture according to embodiments.

[0077] Point cloud data according to the embodiments can be acquired by a camera, etc. The capture method according to the embodiments may include, for example, inward-facing and / or outward-facing.

[0078] Inward-facing according to the embodiments allows one or more cameras to photograph an object of point cloud data from the outside to the inside of the object.

[0079] Outward-facing according to the embodiments allows one or more cameras to photograph an object of point cloud data from the inside out. For example, according to the embodiments, there may be four cameras.

[0080] The point cloud data or point cloud content according to the embodiments may be a video or still image of an object / environment expressed in various forms of 3D space. According to the embodiments, the point cloud content may include video / audio / image, etc. of an object.

[0081] To capture Point Cloud content, the system may be composed of camera equipment capable of acquiring depth (a combination of an infrared pattern projector and an infrared camera) and a combination of RGB cameras capable of extracting color information corresponding to the depth information. Alternatively, depth information can be extracted using LiDAR, which utilizes a radar system that measures the position coordinates of a reflector by emitting a laser pulse and measuring the time it takes for it to reflect back. From the depth information, the shape of the geometry composed of points in 3D space can be extracted, and from the RGB information, attributes representing the color / reflection of each point can be extracted. Point Cloud content can consist of position (x, y, z) and color (YCbCr or RGB) or reflectance (r) information for the points. Point Cloud content may utilize an outward-facing method that captures the external environment and an inward-facing method that captures the central object. When constructing Point Cloud content that allows users to freely view objects (e.g., characters, players, objects, actors, etc.) in a VR / AR environment in a 360-degree manner, the capture camera configuration may use an inward-facing method. When constructing Point Cloud content of the current surrounding environment in a vehicle, such as in autonomous driving, the capture camera configuration may use an outward-facing method. Since Point Cloud content can be captured through multiple cameras, a camera calibration process may be required before capturing the content to establish a global spatial coordinate system between the cameras.

[0082] Point Cloud content may be a video or still image of an object / environment displayed in various forms of 3D space.

[0083] Additionally, regarding methods for acquiring Point Cloud content, arbitrary Point Cloud videos can be synthesized based on captured Point Cloud videos. Alternatively, if the intention is to provide Point Cloud videos for a computer-generated virtual space, capture via a physical camera may not be performed. In this case, the capture process can be substituted by simply generating the relevant data.

[0084] Captured Point Cloud videos may require post-processing to improve content quality. While maximum and minimum depth values ​​can be adjusted within the range provided by the camera equipment during the video capture process, unwanted point data may still be included; therefore, post-processing can be performed to remove unwanted areas (e.g., backgrounds) or to recognize connected spaces and fill in spatial holes. Additionally, Point Clouds extracted from cameras sharing a spatial coordinate system can be integrated into a single piece of content by converting each point to a global coordinate system based on the position coordinates of each camera obtained through a calibration process. This allows for the creation of a single, wide-ranging Point Cloud content or the acquisition of Point Cloud content with a high density of points.

[0085] A Point Cloud video encoder can encode input Point Cloud video into one or more video streams. A single video may contain multiple frames, and a single frame may correspond to a still image or picture. In this document, the term "Point Cloud video" may include Point Cloud images, frames, pictures, video, audio, images, etc., and the terms Point Cloud video may be used interchangeably with Point Cloud images, frames, or pictures. A Point Cloud video encoder can perform Video-based Point Cloud Compression (V-PCC) procedures. To improve compression and coding efficiency, a Point Cloud video encoder can perform a series of procedures such as prediction, transformation, quantization, and entropy coding. The encoded data (encoded video / video information) can be output in the form of a bitstream. Based on the V-PCC procedure, the Point Cloud video encoder can encode the Point Cloud video by dividing it into geometry video, attribute video, occupancy map video, and auxiliary information as described below. The geometry video may include geometry images, the attribute video may include attribute images, and the occupancy map video may include occupancy map images. The auxiliary information may include auxiliary patch information. The attribute video / image may include texture videos / images.

[0086] The encapsulation processing unit (file / segment encapsulation module, 10003) can encapsulate encoded point cloud video data and / or point cloud video-related metadata into a file or the like. Here, the point cloud video-related metadata may be received from a metadata processing unit or the like. The metadata processing unit may be included in the point cloud video encoder or may be configured as a separate component / module. The encapsulation processing unit can encapsulate the data into a file format such as ISOBMFF or process it into other forms such as DASH segments. According to the embodiment, the encapsulation processing unit may include point cloud video-related metadata in the file format. Point cloud video metadata may be included, for example, in boxes at various levels within the ISOBMFF file format or as data within a separate track within the file. According to the embodiment, the encapsulation processing unit may encapsulate the point cloud video-related metadata itself into a file. The transmission processing unit may apply processing for transmission to the point cloud video data encapsulated according to the file format. The transmission processing unit may be included in the transmission unit or may be configured as a separate component / module. The transmission processing unit may process point cloud video data according to any transmission protocol. Processing for transmission may include processing for delivery via a broadcast network and processing for delivery via broadband. According to an embodiment, the transmission processing unit may receive point cloud video-related metadata from the metadata processing unit in addition to point cloud video data, and apply processing for transmission to it.

[0087] The transmission unit (10004) can transmit encoded video / image information or data output in the form of a bitstream to a receiving unit of a receiving device via a digital storage medium or network in the form of a file or streaming. The digital storage medium may include various storage media such as USB, SD, CD, DVD, Blu-ray, HDD, SSD, etc. The transmission unit may include elements for creating a media file through a predetermined file format and elements for transmission via a broadcasting / communication network. The receiving unit may extract the bitstream and transmit it to a decoding device.

[0088] The receiver (10003) can receive point cloud video data transmitted by the point cloud video transmission device according to the present invention. Depending on the transmission channel, the receiver may receive point cloud video data through a broadcasting network or through broadband. Alternatively, it may receive point cloud video data through a digital storage medium.

[0089] The receiving processing unit can perform processing on the received point cloud video data according to the transmission protocol. The receiving processing unit may be included in the receiving unit or may be configured as a separate component or module. Corresponding to the processing for transmission performed on the transmitting side, the receiving processing unit may perform the reverse process of the aforementioned transmission processing unit. The receiving processing unit may transmit the acquired point cloud video data to the decapsulation processing unit and the acquired point cloud video-related metadata to the metadata parser. The point cloud video-related metadata acquired by the receiving processing unit may be in the form of a signaling table.

[0090] The decapsulation processing unit (file / segment decapsulation module, 10007) can decapsulate point cloud video data in file form received from the receiving processing unit. The decapsulation processing unit can decapsulate files according to ISOBMFF, etc., to obtain a point cloud video bitstream or point cloud video-related metadata (metadata bitstream). The obtained point cloud video bitstream can be transmitted to a point cloud video decoder, and the obtained point cloud video-related metadata (metadata bitstream) can be transmitted to a metadata processing unit. The point cloud video bitstream may include metadata (metadata bitstream). The metadata processing unit may be included in the point cloud video decoder or may be configured as a separate component / module. The point cloud video-related metadata obtained by the decapsulation processing unit may be in the form of boxes or tracks within the file format. If necessary, the decapsulation processing unit may receive metadata required for decapsulation from the metadata processing unit. Point cloud video-related metadata may be passed to a point cloud video decoder and used in the point cloud video decoding process, or passed to a renderer and used in the point cloud video rendering process.

[0091] A Point Cloud video decoder can receive a bitstream as input and perform an operation corresponding to the operation of a Point Cloud video encoder to decode video / images. In this case, the Point Cloud video decoder can decode the Point Cloud video by dividing it into geometry video, attribute video, occupancy map video, and auxiliary information as described below. Geometry video may include geometry images, attribute video may include attribute images, and occupancy map video may include occupancy map images. Auxiliary information may include auxiliary patch information. Attribute video / image may include texture video / image.

[0092] 3D geometry is restored using the decoded geometry image, occupancy map, and additional patch information, and can subsequently undergo a smoothing process. A color point cloud image / picture can be restored by assigning color values ​​to the smoothed 3D geometry using a texture image. The renderer can render the restored geometry and the color point cloud image / picture. The rendered video / image can be displayed through a display unit. The user can view all or part of the rendered result through a VR / AR display or a standard display.

[0093] The feedback process may include the process of transmitting various feedback information, which can be obtained during the rendering / display process, to the transmitting side or to the decoder of the receiving side. Interactivity in Point Cloud video consumption may be provided through the feedback process. According to an embodiment, head orientation information, viewport information indicating the area the user is currently viewing, etc., may be transmitted during the feedback process. According to an embodiment, the user may interact with elements implemented in a VR / AR / MR / autonomous driving environment, and in this case, information related to such interaction may be transmitted to the transmitting side or the service provider side during the feedback process. According to an embodiment, the feedback process may not be performed.

[0094] Head orientation information can refer to information regarding the user's head position, angle, movement, etc. Based on this information, viewport information—that is, information about the area the user is currently viewing within the Point Cloud video—can be calculated.

[0095] Viewport information may be information about the area currently being viewed by the user in the Point Cloud video. Through this, gaze analysis can be performed to determine how the user consumes the Point Cloud video and which areas of the video they gaze at for how long. Gaze analysis may be performed at the receiving end and transmitted to the transmitting end via a feedback channel. Devices such as VR / AR / MR displays can extract the viewport area based on the user's head position / orientation, the vertical or horizontal FOV supported by the device, etc.

[0096] According to an embodiment, the aforementioned feedback information may not only be transmitted to the transmitting side but may also be consumed at the receiving side. That is, decoding and rendering processes at the receiving side may be performed using the aforementioned feedback information. For example, using head orientation information and / or viewport information, only the Point Cloud video of the area currently viewed by the user may be preferentially decoded and rendered.

[0097] Here, the viewport or viewport area may refer to the area that the user is viewing in the Point Cloud video. The viewpoint is the point that the user is viewing in the Point Cloud video, and may refer to the exact center point of the viewport area. In other words, the viewport is an area centered on the viewpoint, and the size and shape of that area can be determined by the Field of View (FOV).

[0098] This document relates to Point Cloud video compression as described above. For example, the methods / executions disclosed in this document may be applied to the MPEG (Moving Picture Experts Group) PCC (point cloud compression or point cloud coding) standard or next-generation video / image coding standards.

[0099] In this document, "picture" or "frame" generally refers to a unit representing a single image of a specific time period.

[0100] A pixel or pel may refer to the smallest unit that constitutes a picture (or image). Additionally, the term 'sample' may be used as a counterpart to pixel. A sample can generally represent a pixel or a pixel value, and may represent only the pixel / pixel value of the lumina component, only the pixel / pixel value of the chroma component, or only the pixel / pixel value of the depth component.

[0101] A unit may represent a basic unit of image processing. A unit may include at least one of a specific area of ​​a picture and information related to that area. Depending on the case, the term unit may be used interchangeably with terms such as block or area. In general, an MxN block may include samples (or sample arrays) or a set (or array) of transform coefficients consisting of M columns and N rows.

[0102] FIG. 3 shows examples of point clouds, geometry, and texture images according to embodiments.

[0103] The point cloud according to the embodiments can be input into the V-PCC encoding process of FIG. 4, which will be described later, to generate a geometry image and a texture image. According to the embodiments, the point cloud can be used with the same meaning as point cloud data.

[0104] As shown in the drawing, the left side is a point cloud, representing a point cloud where an object is located in 3D space and can be represented by a bounding box, etc. The middle side represents geometry, and the right side represents a texture image (non-padding).

[0105] Video-based Point Cloud Compression (V-PCC) can provide a method for compressing 3D point cloud data based on 2D video codecs such as HEVC and VVC. The following data and information can be generated during the V-PCC compression process.

[0106] Occupancy map: Represents a binary map that indicates whether data exists at a corresponding location on a 2D plane with a value of 0 or 1 when dividing the points forming a point cloud into patches and mapping them onto a 2D plane. The occupancy map represents a 2D array corresponding to an atlas, and the value of the occupancy map can indicate whether each sample position within the atlas corresponds to a 3D point.

[0107] An atlas is a set of 2D bounding boxes located in rectangular frames corresponding to 3D bounding boxes in the 3D space where volumetric data is rendered, and related information.

[0108] An atlas bitstream is a bitstream of one or more atlas frames and associated data that make up an atlas.

[0109] An atlas frame is a 2D rectangular array of atlas samples onto which patches are projected.

[0110] An atlas sample is the position of a rectangular frame onto which patches associated with the atlas are projected.

[0111] An atlas frame can be divided into tiles. A tile is a unit that divides a 2D frame. In other words, a tile is a unit that divides the signaling information of point cloud data called an atlas.

[0112] Patch: A set of points that make up a point cloud, where points belonging to the same patch are adjacent to each other in 3D space and are mapped in the same direction among the 6 bounding box planes during the mapping process to a 2D image.

[0113] Geometry image: Represents an image in the form of a depth map that expresses the positional information (geometry) of each point forming a point cloud in patch units. A geometry image can consist of 1-channel pixel values. Geometry represents a set of coordinates associated with a point cloud frame.

[0114] Texture image: Represents an image that expresses color information of each point forming a point cloud in patch units. The texture image may consist of pixel values ​​of multiple channels (e.g., 3 channels R, G, B). The texture is included in the attribute. Depending on the embodiments, the texture and / or attribute may be interpreted as having the same object and / or inclusion relationship.

[0115] Auxiliary patch info: Represents metadata necessary to reconstruct a point cloud from individual patches. Auxiliary patch info may include information about the location, size, etc. of the patch in 2D / 3D space.

[0116] Point cloud data according to the embodiments, for example, V-PCC components may include atlases, accusation maps, geometry, attributes, etc.

[0117] An atlas represents a set of 2D bounding boxes. It can be patches, for example, patches projected onto a rectangular frame. It can also correspond to 3D bounding boxes in 3D space and represent a subset of a point cloud.

[0118] An attribute represents a scalar or vector associated with each point in the point cloud, and may include, for example, color, reflectance, surface normal, time stamps, material ID, etc.

[0119] The point cloud data according to the embodiments represents PCC data based on the V-PCC (Video-based Point Cloud Compression) method. The point cloud data may include a plurality of components. For example, it may include an accusation map, a patch, geometry and / or texture, etc.

[0120] FIG. 4 shows an example of V-PCC encoding processing according to embodiments.

[0121] The drawing illustrates a V-PCC encoding process for generating and compressing an occupancy map, a geometry image, a texture image, and auxiliary patch information. The V-PCC encoding process of FIG. 4 can be processed by the point cloud video encoder (10002) of FIG. 1. Each component of FIG. 4 can be performed by software, hardware, a processor, and / or a combination thereof.

[0122] A patch generation unit (40000) or patch generator receives a point cloud frame (which may be in the form of a bitstream containing point cloud data). The patch generation unit (40000) generates a patch from the point cloud data. It also generates patch info containing information regarding patch generation.

[0123] Patch packing (40001) or a patch packer packs patches for point cloud data. For example, one or more patches may be packed. Additionally, an accusation map containing information about the patch packing is generated.

[0124] Geometry image generation (40002) or a geometry image generator generates a geometry image based on point cloud data, patches, and / or packed patches. A geometry image refers to data containing geometry related to point cloud data.

[0125] Texture image generation (40003) or texture image generator generates a texture image based on point cloud data, patches, and / or packed patches. Additionally, a texture image can be generated based on smoothed geometry created by smoothing (number) a reconstructed geometry image based on patch info.

[0126] Smoothing (40004) or a smoother can mitigate or remove errors contained in image data. For example, a reconstructed geometry image can be smoothed by filtering out parts that may cause errors between data based on patch info to generate smoothed geometry.

[0127] Auxiliary patch info compression (40005) or an oscillary patch information compressor compresses additional patch information related to patch information generated during the patch generation process. Additionally, the compressed oscillary patch info is transmitted to a multiplexer, and geometry image generation (40002) can also use the oscillary patch information.

[0128] Image padding (image padding, 40006, 40007) or image feather can pad a geometry image and a texture image, respectively. Padding data can be padded to a geometry image and a texture image.

[0129] Group dilation (40008) or group delighter can add data to a texture image similar to image padding. The added data can be inserted into the texture image.

[0130] Video compression (40009, 40010, 40011) or a video compressor can compress padded geometry images, padded texture images, and / or accusation maps, respectively. Compression can encode geometry information, texture information, accusation information, etc.

[0131] Entropy compression (40012) or an entropy compressor can compress (e.g., encode) an accusation map based on an entropy method.

[0132] According to the embodiments, depending on whether the point cloud data is lossless and / or lossy, entropy compression and / or video compression may be performed, respectively.

[0133] The multiplexer (40013) multiplexes the compressed geometry image, the compressed texture image, and the compressed accusation map into a bitstream.

[0134] The detailed operation of each process of FIG. 4 according to the embodiments is as follows.

[0135] Patch generation (40000)

[0136] The patch generation process refers to the process of dividing a point cloud into patches, which are the units for mapping, in order to map the point cloud onto a 2D image. The patch generation process can be divided into three stages as follows: normal value calculation, segmentation, and patch division.

[0137] Referring to Fig. 5, the process of calculating the normal value is explained in detail.

[0138] FIG. 5 shows examples of a tangent plane and a normal vector of a surface according to embodiments.

[0139] The surface of FIG. 5 is used as follows in the patch generation process (40000) of the V-PCC encoding process of FIG. 4.

[0140] Normal calculation regarding patch generation:

[0141] Each point (e.g., a point) forming a point cloud has a unique orientation, which is represented by a 3D vector called a normal. By utilizing the neighbors of each point obtained using tools such as a KD tree, the tangent plane and normal vector of each point forming the surface of the point cloud, as shown in the drawing, can be determined. The search range during the process of finding neighbors can be defined by the user.

[0142] Tangent plane: Represents a plane that passes through a point on a surface and completely contains the tangent line to a curve on the surface.

[0143] FIG. 6 shows an example of a bounding box of a point cloud according to embodiments.

[0144] According to the embodiments, a method / device, for example, may use a bounding box in the process of patch generation generating a patch from point cloud data.

[0145] A bounding box according to the embodiments refers to a unit box that divides point cloud data based on a cuboid in 3D space.

[0146] A bounding box can be used in the process of projecting an object that is the target of point cloud data onto the plane of each cuboid based on a cuboid in 3D space. The bounding box can be generated and processed by the point cloud video acquisition unit (10000) and the point cloud video encoder (10002) of FIG. 1. Additionally, based on the bounding box, patch generation (40000), patch packing (40001), geometry image generation (40002), and texture image generation (40003) of the V-PCC encoding process of FIG. 2 can be performed.

[0147] Segmentation regarding patch generation

[0148] Segmentation consists of two processes: initial segmentation and refine segmentation.

[0149] A point cloud encoder (10002) according to the embodiments projects points onto one side of a bounding box. Specifically, each point forming the point cloud is projected onto one of the six sides of a bounding box that encloses the point cloud as shown in the drawing, and initial segmentation is a process of determining one of the planes of the bounding box on which each point will be projected.

[0150] The normal values ​​corresponding to each of the 6 planes is defined as follows.

[0151] (1.0, 0.0, 0.0), (0.0, 1.0, 0.0), (0.0, 0.0, 1.0), (-1.0, 0.0, 0.0), (0.0, -1.0, 0.0), (0.0, 0.0, -1.0).

[0152] The normal values ​​of each point obtained in the previous normal value calculation process as shown in the following formula ( )class The plane with the maximum dot product is determined as the projection plane of that plane. That is, the plane having a normal in the direction most similar to the point's normal is determined as the projection plane of that point.

[0153]

[0154] The determined plane can be identified by a value in the form of an index (cluster index) of 0 to 5.

[0155] Refine segmentation is the process of improving the projection plane of each point forming the point cloud, determined in the preceding initial segmentation process, by considering the projection planes of adjacent points. In this process, a score normal—which represents the degree of similarity between the normal of each point considered for determining the projection plane in the initial segmentation process and the normal value of each plane of the bounding box—can be considered simultaneously with a score smooth—which indicates the degree of agreement between the projection plane of the current point and the projection planes of adjacent points.

[0156] Score smoothing can be considered by assigning weights to the score normal, and the weight values ​​can be defined by the user. Refine segmentation can be performed iteratively, and the number of iterations can also be defined by the user.

[0157] Patch division (segment patches) regarding patch generation

[0158] Patch segmentation is the process of dividing the entire point cloud into patches, which are sets of adjacent points, based on the projection plane information of each point forming the point cloud obtained during the preceding initial / refine segmentation process. Patch segmentation can consist of the following steps.

[0159] ① Calculate the adjacent points of each point forming the point cloud using a KD tree, etc. The maximum number of adjacent points can be defined by the user.

[0160] ② If adjacent points are projected onto the same plane as the current point (i.e., have the same cluster index value), the current point and those adjacent points are extracted as a single patch.

[0161] ③ Calculate the geometry values ​​of the extracted patch. The detailed process is explained below.

[0162] ④ Repeat steps ② and ④ until the unextracted points are gone.

[0163] Through the patch partitioning process, the size of each patch and the occupancy map, geometry image, texture image, etc., for each patch are determined.

[0164] FIG. 7 shows an example of determining the location of individual patches of an occupancy map according to embodiments.

[0165] The point cloud encoder (10002) according to the embodiments can generate patch packing and accupan maps.

[0166] Patch packing and occupancy map generation (40001)

[0167] This process determines the location of individual patches within a 2D image in order to map the previously divided patches onto a single 2D image. An occupancy map is a type of 2D image that is a binary map indicating whether data exists at a given location using a value of 0 or 1. An occupancy map consists of blocks, and its resolution can be determined by the block size; for example, if the block size is 1x1, it has a resolution in pixel units. The block size (occupancy packing block size) can be determined by the user.

[0168] The process of determining the location of individual patches within an occupancy map can be structured as follows.

[0169] ① Set all values ​​in the entire occupancy map to 0.

[0170] ② Place the patch at point (u, v) on the occupancy map plane, where the horizontal coordinates are within the range [0, occupancySizeU - patch.sizeU0) and the vertical coordinates are within the range [0, occupancySizeV - patch.sizeV0).

[0171] ③ Set the current point (x, y) that exists on the patch plane and has horizontal coordinates in the range [0, patch.sizeU0) and vertical coordinates in the range [0, patch.sizeV0).

[0172] ④ For point (x, y), if the (x, y) coordinate value of the patch occupancy map is 1 (data exists at the corresponding point within the patch) and the (u+x, v+y) coordinate value of the entire occupancy map is 1 (the occupancy map is filled by the previous patch), change the (x, y) position in raster order and repeat steps ③ and ④. Otherwise, perform step ⑥.

[0173] ⑤ Repeat the process of ③⑤ by changing the (u, v) position in raster order.

[0174] ⑥ Determine (u, v) as the location of the corresponding patch, and assign (copy) the patch's occupancy map data to the corresponding part of the entire occupancy map.

[0175] ⑦ Repeat the process of ②⑦ for the next patch.

[0176] Occupancy Size U: Represents the width of the occupancy map, and the unit is the occupancy packing block size.

[0177] Occupancy Size V: Represents the height of the occupancy map, and the unit is the occupancy packing block size.

[0178] Patch size U0 (patch.sizeU0): Represents the width of the occupancy map, and the unit is the occupancy packing block size.

[0179] Patch size V0 (patch.sizeV0): Represents the height of the occupancy map, and the unit is the occupancy packing block size.

[0180] For example, as shown in FIG. 7, there exists a box corresponding to a patch having a patch size within a box corresponding to an accu-pan packing size block, and a point (x, y) can be located within the box.

[0181] FIG. 8 shows an example of the relationship between the normal, tangent, and bitangent axes according to the embodiments.

[0182] A point cloud encoder (10002) according to the embodiments can generate a geometry image. A geometry image refers to image data containing geometry information of a point cloud. The geometry image generation process can utilize the three axes (normal, tangent, and bitangent) of the patch of FIG. 8.

[0183] Geometry image generation (40002)

[0184] In this process, depth values ​​constituting the geometry image of individual patches are determined, and the overall geometry image is generated based on the patch positions determined in the previous patch packing process. The process of determining the depth values ​​constituting the geometry image of individual patches can be structured as follows.

[0185] ① Parameters related to the location and size of individual patches are calculated. The parameters may include the following information.

[0186] Index representing the normal axis: The normal is obtained during the patch generation process mentioned earlier; the tangent axis is the axis perpendicular to the normal that coincides with the horizontal (u) axis of the patch image; and the bitangent axis is the axis perpendicular to the normal that coincides with the vertical (v) axis of the patch image; the three axes can be represented as shown in the diagram.

[0187] FIG. 9 shows an example of the configuration of the minimum mode and maximum mode of the projection mode according to the embodiments.

[0188] The point cloud encoder (10002) according to the embodiments can perform patch-based projection to generate a geometry image, and the projection modes according to the embodiments include a minimum mode and a maximum mode.

[0189] 3D spatial coordinates of the patch: These can be calculated through a bounding box of the smallest size that encloses the patch. For example, they may include the minimum value in the patch's tangent direction (patch 3d shift tangent axis), the minimum value in the patch's bitangent direction (patch 3d shift bitangent axis), the minimum value in the patch's normal direction (patch 3d shift normal axis), etc.

[0190] 2D size of a patch: Represents the horizontal and vertical dimensions when the patch is packed into a 2D image. The horizontal size (patch 2d size u) can be calculated as the difference between the maximum and minimum values ​​of the bounding box's tangent direction, and the vertical size (patch 2d size v) can be calculated as the difference between the maximum and minimum values ​​of the bounding box's bitangent direction.

[0191] ② Determine the projection mode of the patch. The projection mode can be either a min mode or a max mode. The geometry information of the patch is represented by depth values; when projecting each point forming the patch along the normal direction of the patch, two layers of images can be generated: one composed of the maximum depth value and the other composed of the minimum depth value.

[0192] When generating two layers of images d0 and d1, in min mode, the minimum depth is configured in d0 as shown in the drawing, and the maximum depth within the surface thickness from the minimum depth can be configured in d1.

[0193] For example, when a point cloud is located in 2D as shown in the drawing, there may be multiple patches containing multiple points. As shown in the drawing, points marked with the same style of shading indicate that they may belong to the same patch. The drawing illustrates the process of projecting a patch of points marked as blank.

[0194] When projecting points marked with blank spaces to the left / right, numbers for calculating the depth of the points can be displayed to the right by increasing the depth by 1 from the left side, such as 0, 1, 2, ..6, 7, 8, 9.

[0195] The projection mode can be customized so that the same method is applied to all point clouds, or applied differently per frame or patch. If different projection modes are applied per frame or patch, a projection mode that can increase compression efficiency or minimize missing points can be adaptively selected.

[0196] ③ Calculate the depth values ​​of the individual points.

[0197] In Min mode, the d0 image is constructed using depth0, which is the value obtained by subtracting the patch's normal direction minimum value (patch 3d shift normal axis) calculated in step ① from the patch's normal direction minimum value (patch 3d shift normal axis) at the normal axis minimum value of each point. If another depth value exists at the same location within the range of depth0 and surface thickness, this value is set as depth1. If it does not exist, the value of depth0 is assigned to depth1 as well. The d1 image is constructed using the depth1 value.

[0198] For example, when determining the depth of the points of d0, the minimum value may be calculated (4 2 4 4 0 6 0 0 9 9 0 8 0). Also, when determining the depth of the points of d1, the larger value among two or more points may be calculated, or if there is only one point, that value may be calculated (4 4 4 4 6 6 6 8 9 9 8 8 9). Furthermore, some points may be lost during the process of encoding and reconstructing the patch points (for example, 8 points were lost in the drawing).

[0199] In Max mode, the d0 image is constructed using depth0, which is the value obtained by subtracting the minimum normal direction value of the patch (patch 3d shift normal axis) calculated in step ① from the minimum normal direction value of the patch (patch 3d shift normal axis) at the maximum normal axis value of each point. If another depth value exists at the same location within the range of depth0 and surface thickness, this value is set as depth1. If it does not exist, the value of depth0 is assigned to depth1 as well. The d1 image is constructed using the depth1 value.

[0200] For example, when determining the depth of the points of d0, the maximum value may be calculated (4 4 4 4 6 6 6 8 9 9 8 8 9). Also, when determining the depth of the points of d1, the smaller value among two or more points may be calculated, or if there is only one point, that value may be calculated (4 2 4 4 5 6 0 6 9 9 0 8 0). Additionally, some points may be lost during the process of encoding and reconstructing the patch points (for example, 6 points were lost in the drawing).

[0201] The entire geometry image can be generated by placing the geometry images of individual patches generated through the above process onto the entire geometry image using the patch position information determined in the previous patch packing process.

[0202] The d1 layer of the generated entire geometry image can be encoded in several ways. The first is to encode the depth values ​​of the previously generated d1 image as they are (absolute d1 method). The second is to encode the difference between the depth values ​​of the previously generated d1 image and the depth values ​​of the d0 image (differential method).

[0203] Since the encoding method using depth values ​​of two layers, d0 and d1, loses geometry information of points when other points exist between the two depths during the encoding process, Enhanced-Delta-Depth (EDD) code may be used for lossless coding.

[0204] Referring to Fig. 10, the EDD code is explained in detail.

[0205] FIG. 10 shows an example of an EDD code according to embodiments.

[0206] A point cloud encoder (10002) and / or part / all of the V-PCC encoding process (e.g., video compression (40009)) can encode geometric information of points based on EOD codes.

[0207] The EDD code is a method of binary encoding the locations of all points within the surface thickness range, including d1, as shown in the drawing. For example, in the case of points included in the second column from the left of the drawing, points exist at the first and fourth positions above D0, while the second and third positions are empty, so they can be represented by the EDD code 0b1001 (=9). If the EDD code is encoded and sent along with D0, the receiving end can restore the geometry information of all points without loss.

[0208] For example, if a point exists above a reference point, it is 1, and if no point exists, it is 0, so the code can be represented based on 4 bits.

[0209] Smoothing (Smoothing, 40004)

[0210] Smoothing is a process to eliminate discontinuities that may occur at patch boundaries due to image quality degradation during the compression process, and can be performed by a point cloud encoder or a smoother.

[0211] ① Reconstruct the point cloud from the geometry image. This process can be considered the reverse process of the geometry image generation described earlier. For example, the reverse process of encoding can be reconstruction.

[0212] ② Calculate the adjacent points of each point constituting the regenerated point cloud using a KD tree, etc.

[0213] ③ For each point, determine whether the point is located on the patch boundary. For example, if there exists an adjacent point with a different projection plane (cluster index) from the current point, it can be determined that the point is located on the patch boundary.

[0214] ④ If it exists on the patch boundary, move the point to the centroid of the adjacent points (located at the average x, y, z coordinates of the adjacent points). In other words, change the geometry value. Otherwise, maintain the previous geometry value.

[0215] FIG. 11 shows an example of recoloring using color values ​​of adjacent points according to embodiments.

[0216] The point cloud encoder or texture image generator (40003) according to the embodiments can generate a texture image based on recoloring.

[0217] Texture image generation (Texture image generation, 40003)

[0218] The texture image generation process, similar to the geometry image generation process described earlier, consists of generating texture images for individual patches and placing them at determined locations to create the overall texture image. However, in the process of generating the texture images for individual patches, an image containing the color values ​​(e.g., R, G, B) of the points constituting the point cloud corresponding to that location is generated, instead of the depth values ​​used for geometry generation.

[0219] In the process of determining the color values ​​of each point constituting the point cloud, the geometry that has undergone the smoothing process mentioned earlier may be used. Since the smoothed point cloud may have shifted the positions of some points compared to the original point cloud, a recoloring process may be necessary to find colors suitable for the changed locations. Recoloring can be performed using the color values ​​of adjacent points. For example, as shown in the drawing, a new color value can be calculated by considering the color value of the nearest point and the color values ​​of adjacent points.

[0220] For example, referring to the drawing, recoloring can calculate a suitable color value for the changed location based on the average of the attribute information of the nearest original points to the point and / or the average of the attribute information of the nearest original location to the point.

[0221] Texture images can also be generated with two layers of t0 / t1, just like geometry images generated with two layers of d0 / d1.

[0222]

[0223] Auxiliary patch info compression (40005)

[0224] A point cloud encoder or oscillary patch information compressor according to the embodiments can compress oscillary patch information (additional information regarding the point cloud).

[0225] The oscillary patch information compressor compresses additional patch information generated during the previously described patch generation, patch packing, and geometry generation processes. The additional patch information may include the following parameters:

[0226] An index (cluster index) that identifies the projection plane (normal)

[0227] Patch's 3D spatial position: Patch's tangent minimum (patch 3d shift tangent axis), Patch's bitangent minimum (patch 3d shift bitangent axis), Patch's normal minimum (patch 3d shift normal axis)

[0228] Patch's 2D spatial position, size: horizontal size (patch 2d size u), vertical size (patch 2d size v), horizontal minimum (patch 2d shift u), vertical minimum (patch 2d shift u)

[0229] Mapping information for each block and patch: candidate index (when patches are positioned in order based on the 2D spatial location and size information of the patch above, multiple patches may be mapped to a single block. The patches being mapped constitute a candidate list, and this index indicates which patch's data exists in the corresponding block), local patch index (an index pointing to one of the total patches existing in the frame). Table X is a pseudo code representing the block and patch matching process using the candidate list and local patch index.

[0230] The maximum number of candidate lists can be defined by the user.

[0231] for(i=0; i <BlockCount; i++) { if(candidatePatches[i].size() == 1) { blockToPatch[i] = candidatePatches[i][0]} else { candidate_index if(candidate_index == max_candidate_count){ blockToPatch[i] = local_patch_index} else { blockToPatch[i] = candidatePatches[i][candidate_index]}}}

[0232] FIG. 12 shows an example of push-pull background filling according to embodiments. Image padding and group dilation (40006, 40007, 40008)

[0233] The image feather according to the embodiments can fill the space outside the patch area with meaningless additional data based on a push-pull background filling method.

[0234] Image padding is a process of filling the space outside the patch area with meaningless data to improve compression efficiency. For image padding, a method can be used in which pixel values ​​from columns or rows corresponding to the boundary of the patch are copied to fill the empty space. Alternatively, as in a drawing, a push-pull background filling method can be used in which the empty space is filled with pixel values ​​from a low-resolution image by gradually reducing the resolution of an unpadded image and then increasing it again.

[0235] Group delay is a method for filling empty spaces in a geometry and texture image consisting of two layers, d0 / d1 and t0 / t1, and is a process of filling the values ​​of the empty spaces in the two layers calculated through image padding with the average value of the values ​​for the same location in the two layers.

[0236] FIG. 13 shows examples of possible traversal orders for a 4x4 block according to embodiments.

[0237] Occupancy map compression (40012, 40011)

[0238] The occupancy map compressor according to the embodiments can compress the previously generated occupancy map. Specifically, there may be two methods: video compression for lossy compression and entropy compression for lossless compression. Video compression is described below.

[0239] The entropy compression process can be performed as follows.

[0240] ① For each block constituting the occupancy map, if the block is completely filled, encode 1 and repeat the same process for the next block. Otherwise, encode 0 and perform the process in ② and ⑤.

[0241] ② Determine the best traversal order for performing run-length coding on the filled pixels of the block. The figure shows four possible traversal orders as an example for a 4x4 block.

[0242] FIG. 14 shows an example of a best traversal order according to embodiments.

[0243] As described above, the entropy compressor according to the embodiments can code blocks based on a traversal order method as shown in the drawing.

[0244] For example, among the possible traversal orders, the best traversal order having the minimum number of runs is selected and its index is encoded. For example, the drawing shows the case where the third traversal order of the preceding Fig. 13 is selected, and since the number of runs can be minimized to 2 in this case, it can be selected as the best traversal order.

[0245] At this time, the number of runs is encoded. In the example of Fig. 14, since there are 2 runs, 2 is encoded.

[0246] ④ Encode the occupancy of the first run. In the example of Fig. 14, since the first run corresponds to unfilled pixels, 0 is encoded.

[0247] ⑤ Encode the length (as many as the number of runs) for each individual run. In the example of FIG. 14, the lengths of the first and second runs, 6 and 10, are encoded sequentially.

[0248] Video compression(Video compression, 40009, 40010, 40011)

[0249] A video compressor according to the embodiments uses a 2D video codec such as HEVC, VVC, etc. to encode a sequence of geometry images, texture images, occupancy map images, etc., generated by the process described above.

[0250] FIG. 15 shows an example of a 2D video / image encoder according to embodiments.

[0251] The drawing shows a schematic block diagram of a 2D video / image encoder (15000) in which encoding of a video / image signal is performed, as an embodiment of the video compression (40009, 40010, 40011) or video compressor described above. The 2D video / image encoder (15000) may be included in the point cloud video encoder described above, or may be composed of internal / external components. Each component of FIG. 15 may correspond to software, hardware, a processor, and / or a combination thereof.

[0252] Here, the input image may include the geometry image, texture image (attribute(s) image), occupancy map image, etc. described above. The output bitstream of the point cloud video encoder (i.e., point cloud video / image bitstream) may include output bitstreams for each input image (geometry image, texture image (attribute(s) image), occupancy map image, etc.).

[0253] The inter prediction unit (15090) and the intra prediction unit (15100) may be collectively referred to as the prediction unit. That is, the prediction unit may include the inter prediction unit (15090) and the intra prediction unit (15100). The conversion unit (15030), the quantization unit (15040), the inverse quantization unit (15050), and the inverse conversion unit (15060) may be included in the residual processing unit. The residual processing unit may further include a subtraction unit (15020). The above-described image segmentation unit (15010), subtraction unit (15020), transformation unit (15030), quantization unit (15040), inverse quantization unit (150), inverse transformation unit (15060), addition unit (155), filtering unit (15070), inter prediction unit (15090), intra prediction unit (15100), and entropy encoding unit (15110) may be configured by a single hardware component (e.g., an encoder or a processor) according to an embodiment. Additionally, the memory (15080) may include a decoded picture buffer (DPB) and may be configured by a digital storage medium.

[0254] The image segmentation unit (15010) can divide an input image (or picture, frame) input to an encoding device (15000) into one or more processing units. For example, a processing unit may be called a coding unit (CU). In this case, the coding unit may be recursively divided from a coding tree unit (CTU) or a largest coding unit (LCU) according to a QTBT (Quad-tree binary-tree) structure. For example, a single coding unit may be divided into multiple coding units of a deeper depth based on a quad-tree structure and / or a binary tree structure. In this case, for example, the quad-tree structure may be applied first and the binary tree structure may be applied later. Or, the binary tree structure may be applied first. A coding procedure according to the present invention may be performed based on the final coding unit that is no longer divided. In this case, based on coding efficiency according to image characteristics, the maximum coding unit may be used directly as the final coding unit, or, if necessary, the coding unit may be recursively divided into lower-depth coding units so that a coding unit of the optimal size is used as the final coding unit. Here, the term "coding procedure" may include procedures such as prediction, transformation, and restoration described below. As another example, the processing unit may further include a prediction unit (PU) or a transformation unit (TU). In this case, the prediction unit and the transformation unit may each be divided or partitioned from the aforementioned final coding unit. The prediction unit may be a unit for sample prediction, and the transformation unit may be a unit for deriving transformation coefficients and / or a unit for deriving a residual signal from transformation coefficients.

[0255] The term "unit" may be used interchangeably with terms such as "block" or "area" depending on the context. In general, an MxN block may represent a set of samples or transform coefficients consisting of M columns and N rows. A sample can generally represent a pixel or a pixel value, and may represent only the pixel / pixel value of the luminance component or only the pixel / pixel value of the chroma component. A sample may be used to refer to a single picture (or image) as a term corresponding to a pixel or pel.

[0256] The encoding device (15000) can generate a residual signal (residual block, residual sample array) by subtracting a prediction signal (predicted block, prediction sample array) output from an inter prediction unit (15090) or an intra prediction unit (15100) from an input image signal (original block, original sample array), and the generated residual signal is transmitted to a conversion unit (15030). In this case, as illustrated, the unit that subtracts the prediction signal (predicted block, prediction sample array) from the input image signal (original block, original sample array) within the encoder (15000) may be called a subtraction unit (15020). The prediction unit performs a prediction for a block to be processed (hereinafter referred to as the current block) and can generate a predicted block containing prediction samples for the current block. The prediction unit can determine whether intra prediction is applied or inter prediction is applied at the current block or CU level. The prediction unit can generate various information regarding prediction, such as prediction mode information, as described below in the description of each prediction mode, and transmit it to the entropy encoding unit (15110). The information regarding prediction can be encoded in the entropy encoding unit (15110) and output in the form of a bitstream.

[0257] The intra prediction unit (15100) can predict the current block by referencing samples within the current picture. Depending on the prediction mode, the referenced samples may be located near the current block or away from it. In intra prediction, the prediction modes may include a plurality of non-directional modes and a plurality of directional modes. The non-directional modes may include, for example, a DC mode and a Planar mode. The directional modes may include, for example, 33 directional prediction modes or 65 directional prediction modes depending on the degree of fineness of the prediction direction. However, this is merely an example, and depending on the settings, more or fewer directional prediction modes may be used. The intra prediction unit (15100) may also determine the prediction mode applied to the current block by using the prediction mode applied to the surrounding blocks.

[0258] The inter prediction unit (15090) can derive a predicted block for the current block based on a reference block (reference sample array) specified by a motion vector on a reference picture. At this time, to reduce the amount of motion information transmitted in the inter prediction mode, motion information can be predicted in blocks, sub-blocks, or samples based on the correlation of motion information between neighboring blocks and the current block. Motion information may include motion vectors and reference picture indices. Motion information may further include information on inter prediction directions (L0 prediction, L1 prediction, Bi prediction, etc.). In the case of inter prediction, neighboring blocks may include spatial neighboring blocks existing within the current picture and temporal neighboring blocks existing in the reference picture. The reference picture containing the reference blocks and the reference picture containing the temporal neighboring blocks may be the same or different. Temporal surrounding blocks may be referred to by names such as collocated reference block, collocated CU (colCU), etc., and a reference picture containing temporal surrounding blocks may be referred to as a collocated picture (colPic). For example, the inter prediction unit (15090) may construct a list of motion information candidates based on surrounding blocks and generate information indicating which candidate is used to derive the motion vector and / or reference picture index of the current block. Inter prediction may be performed based on various prediction modes, for example, in the case of skip mode and merge mode, the inter prediction unit (15090) may use the motion information of surrounding blocks as the motion information of the current block. In the case of skip mode, unlike merge mode, a residual signal may not be transmitted.In the motion vector prediction (MVP) mode, the motion vector of surrounding blocks is used as a motion vector predictor, and the motion vector of the current block can be indicated by signaling the motion vector difference.

[0259] The prediction signal generated through the inter prediction unit (15090) and the intra prediction unit (15100) can be used to generate a restoration signal or to generate a residual signal.

[0260] The transformation unit (15030) can generate transform coefficients by applying a transformation technique to a residual signal. For example, the transformation technique may include at least one of a Discrete Cosine Transform (DCT), a Discrete Sine Transform (DST), a Karhunen-Loeve Transform (KLT), a Graph-Based Transform (GBT), or a Conditionally Non-linear Transform (CNT). Here, GBT refers to a transformation obtained from a graph when the relationship information between pixels is represented as a graph. CNT refers to a transformation obtained based on a prediction signal generated using all previously reconstructed pixels. Additionally, the transformation process may be applied to a pixel block of the same size in a square, or to a block of variable size that is not square.

[0261] The quantization unit (15040) quantizes the transformation coefficients and transmits them to the entropy encoding unit (15110), and the entropy encoding unit (15110) can encode the quantized signal (information regarding the quantized transformation coefficients) and output it as a bitstream. The information regarding the quantized transformation coefficients may be called residual information. The quantization unit (15040) can rearrange the block-shaped quantized transformation coefficients into a one-dimensional vector form based on the coefficient scan order, and can also generate information regarding the quantized transformation coefficients based on the one-dimensional vector-shaped quantized transformation coefficients. The entropy encoding unit (15110) can perform various encoding methods such as, for example, exponential Golomb, CAVLC (context-adaptive variable length coding), CABAC (context-adaptive binary arithmetic coding), etc. The entropy encoding unit (15110) may encode information necessary for video / image restoration (e.g., values ​​of syntax elements) together or separately, in addition to quantized transformation coefficients. The encoded information (e.g., encoded video / image information) may be transmitted or stored in the form of a bitstream in units of NAL (network abstraction layer) units. The bitstream may be transmitted through a network or stored in a digital storage medium. Here, the network may include a broadcasting network and / or a communication network, and the digital storage medium may include various storage media such as USB, SD, CD, DVD, Blu-ray, HDD, SSD, etc. A transmission unit (not shown) that transmits the signal output from the entropy encoding unit (15110) and / or a storage unit (not shown) that stores it may be configured as internal / external elements of the encoding device (15000), or the transmission unit may be included in the entropy encoding unit (15110).

[0262] Quantized transform coefficients output from the quantization unit (15040) can be used to generate a prediction signal. For example, a residual signal (residual block or residual samples) can be restored by applying inverse quantization and inverse transformation to the quantized transform coefficients through the inverse quantization unit (15040) and the inverse transformation unit (15060). An adder (155) can generate a reconstructed signal (reconstructed picture, reconstructed block, reconstructed sample array) by adding the restored residual signal to the prediction signal output from the inter-prediction unit (15090) or the intra-prediction unit (15100). In cases where there is no residual for the block to be processed, such as when a skip mode is applied, the predicted block can be used as the reconstructed block. The adder (155) may be called a reconstruction unit or a reconstruction block generation unit. The generated restoration signal can be used for intra prediction of the next processing target block within the current picture, and can also be used for inter prediction of the next picture after filtering as described below.

[0263] The filtering unit (15070) can improve subjective / objective image quality by applying filtering to the restored signal. For example, the filtering unit (15070) can generate a modified restored picture by applying various filtering methods to the restored picture, and can store the modified restored picture in memory (15080), specifically in the DPB of memory (15080). Various filtering methods may include, for example, deblocking filtering, sample adaptive offset, adaptive loop filter, bilateral filter, etc. The filtering unit (15070) can generate various information regarding filtering and transmit it to the entropy encoding unit (15110), as described below in the description of each filtering method. The information regarding filtering can be encoded in the entropy encoding unit (15110) and output in the form of a bitstream.

[0264] The modified restored picture transmitted to the memory (15080) can be used as a reference picture in the inter-prediction unit (15090). Through this, when inter-prediction is applied, the encoding device can avoid prediction mismatches between the encoding device (15000) and the decoding device, and can also improve encoding efficiency.

[0265] The memory (15080) DPB can store the modified restored picture to be used as a reference picture in the inter-prediction unit (15090). The memory (15080) can store motion information of blocks from which motion information is derived (or encoded) within the current picture and / or motion information of blocks within the picture that have already been restored. The stored motion information can be transmitted to the inter-prediction unit (15090) to be used as motion information of spatially surrounding blocks or motion information of temporally surrounding blocks. The memory (15080) can store restoration samples of the blocks restored within the current picture and transmit them to the intra-prediction unit (15100).

[0266] Meanwhile, at least one of the aforementioned prediction, transformation, and quantization procedures may be omitted. For example, for a block to which pulse coding mode (PCM) is applied, the prediction, transformation, and quantization procedures may be omitted, and the value of the original sample may be encoded as is and output as a bitstream.

[0267] FIG. 16 shows an example of a V-PCC decoding process according to embodiments.

[0268] The V-PCC decoding process or V-PCC decoder may follow the reverse process of the V-PCC encoding process (or encoder) of FIG. 4. Each component of FIG. 16 may correspond to software, hardware, a processor, and / or a combination thereof.

[0269] The demultiplexer (16000) demultiplexes the compressed bitstream to output a compressed texture image, a compressed geometry image, a compressed occupancy map, and compressed assimilation patch information.

[0270] Video decompression (16001, 16002) or video decompressor decompresses (or decodes) each of the compressed texture image and the compressed geometry image.

[0271] Occupancy map decompression (16003) or occupancy map decompresses a compressed occupancy map.

[0272] Auxiliary patch info decompression (16004) or auxiliary patch information decompressor decompresses auxiliary patch information.

[0273] Geometry reconstruction (16005) or geometry reconstructor restores (reconstructs) geometry information based on a decompressed geometry image, a decompressed accusation map, and / or decompressed assiliary patch information. For example, geometry that was changed during the encoding process can be reconstructed.

[0274] Smoothing (16006) or a smoother can apply smoothing to reconstructed geometry. For example, smoothing filtering can be applied.

[0275] Texture reconstruction (16007) or a texture reconstructor reconstructs a texture from a decompressed texture image and / or smoothed geometry.

[0276] Color smoothing (16008) or a color smoother smooths color values ​​from a reconstructed texture. For example, smoothing filler can be applied.

[0277] As a result, reconstructed point cloud data can be generated.

[0278] The drawing illustrates the decoding process of V-PCC for reconstructing a point cloud by decoding a compressed occupancy map, geometry image, texture image, and auxiliary path information. The operation of each process according to the embodiments is as follows.

[0279] Video decompression (Video decompression, 16001, 16002)

[0280] As the reverse process of the video compression described earlier, this is a process of decoding compressed bitstreams such as geometry images, texture images, and occupancy map images generated by the process described earlier using 2D video codecs such as HEVC and VVC.

[0281] FIG. 17 shows an example of a 2D video / image decoder according to embodiments.

[0282] The 2D video / image decoder can follow the inverse process of the 2D video / image encoder of FIG. 15.

[0283] The 2D video / image decoder of FIG. 17 is an embodiment of the video decompression or video decompressor of FIG. 16, and represents a schematic block diagram of a 2D video / image decoder (17000) in which decoding of a video / image signal is performed. The 2D video / image decoder (17000) may be included in the point cloud video decoder of FIG. 1, or may be composed of internal / external components. Each component of FIG. 17 may correspond to software, hardware, a processor, and / or a combination thereof.

[0284] Here, the input bitstream may include a bitstream for the geometry image, texture image (attribute(s) image), occupancy map image, etc. described above. The reconstructed image (or output image, decoded image) may represent a reconstructed image for the geometry image, texture image (attribute(s) image), and occupancy map image described above.

[0285] Referring to the drawings, the inter prediction unit (17070) and the intra prediction unit (17080) may be collectively referred to as the prediction unit. That is, the prediction unit may include the inter prediction unit (180) and the intra prediction unit (185). The inverse quantization unit (17020) and the inverse transform unit (17030) may be collectively referred to as the residual processing unit. That is, the residual processing unit may include the inverse quantization unit (17020) and the inverse transform unit (17030). The above-described entropy decoding unit (17010), inverse quantization unit (17020), inverse transform unit (17030), addition unit (17040), filtering unit (17050), inter prediction unit (17070), and intra prediction unit (17080) may be configured by a single hardware component (e.g., a decoder or a processor) according to the embodiment. In addition, the memory (170) may include a DPB (decoded picture buffer) and may be configured by a digital storage medium.

[0286] When a bitstream containing video / image information is input, the decoding device (17000) can restore the image in correspondence with the process in which the video / image information is processed in the encoding device of FIG. 0.2-1. For example, the decoding device (17000) can perform decoding using a processing unit applied in the encoding device. Thus, the processing unit for decoding may be, for example, a coding unit, and the coding unit may be divided along a quad tree structure and / or a binary tree structure from a coding tree unit or a maximum coding unit. And, the restored image signal decoded and output through the decoding device (17000) can be played back through a playback device.

[0287] The decoding device (17000) can receive a signal output from the encoding device in the form of a bitstream, and the received signal can be decoded through the entropy decoding unit (17010). For example, the entropy decoding unit (17010) can parse the bitstream to derive information (e.g., video / image information) required for image restoration (or picture restoration). For example, the entropy decoding unit (17010) can decode information within the bitstream based on coding methods such as exponential coding, CAVLC, or CABAC, and output values ​​of syntax elements required for image restoration and quantized values ​​of transformation coefficients regarding residuals. More specifically, the CABAC entropy decoding method receives a bin corresponding to each syntactic element in a bitstream, determines a context model using information on the syntactic element to be decoded, decoding information of surrounding and target blocks, or information on symbols / bins decoded in the previous step, predicts the probability of bin occurrence according to the determined context model, and performs arithmetic decoding of the bin to generate a symbol corresponding to the value of each syntactic element. At this time, after determining the context model, the CABAC entropy decoding method can update the context model using information on the decoded symbols / bins for the context model of the next symbol / bin. Information regarding prediction among the information decoded in the entropy decoding unit (17010) is provided to the prediction unit (inter prediction unit (17070) and intra prediction unit (265)), and residual values, i.e., quantized transformation coefficients and related parameter information, from which entropy decoding is performed in the entropy decoding unit (17010) can be input to the inverse quantization unit (17020). Additionally, information regarding filtering among the information decoded in the entropy decoding unit (17010) can be provided to the filtering unit (17050).Meanwhile, a receiving unit (not shown) that receives a signal output from an encoding device may be further configured as an internal / external element of a decoding device (17000), or the receiving unit may be a component of an entropy decoding unit (17010).

[0288] In the inverse quantization unit (17020), the quantized transform coefficients can be inversely quantized to output transform coefficients. The inverse quantization unit (17020) can rearrange the quantized transform coefficients into a two-dimensional block form. In this case, the rearrangement can be performed based on the coefficient scan order performed by the encoding device. The inverse quantization unit (17020) can perform inverse quantization on the quantized transform coefficients using quantization parameters (e.g., quantization step size information) and obtain transform coefficients.

[0289] In the inverse conversion unit (17030), the conversion coefficients are inversely converted to obtain a residual signal (residual block, residual sample array).

[0290] The prediction unit performs a prediction for the current block and can generate a predicted block containing prediction samples for the current block. Based on the prediction information output from the entropy decoding unit (17010), the prediction unit can determine whether an intra prediction or an inter prediction is applied to the current block and can determine a specific intra / inter prediction mode.

[0291] The intra prediction unit (265) can predict the current block by referring to samples within the current picture. The referenced samples may be located near the current block or away from it, depending on the prediction mode. In intra prediction, the prediction modes may include a plurality of non-directional modes and a plurality of directional modes. The intra prediction unit (265) may determine the prediction mode applied to the current block by using the prediction mode applied to the surrounding blocks.

[0292] The inter prediction unit (17070) can derive a predicted block for the current block based on a reference block (reference sample array) specified by a motion vector on a reference picture. At this time, to reduce the amount of motion information transmitted in the inter prediction mode, motion information can be predicted in blocks, sub-blocks, or samples based on the correlation of motion information between neighboring blocks and the current block. Motion information may include a motion vector and a reference picture index. Motion information may further include information on the inter prediction direction (L0 prediction, L1 prediction, Bi prediction, etc.). In the case of inter prediction, neighboring blocks may include spatial neighboring blocks existing within the current picture and temporal neighboring blocks existing in the reference picture. For example, the inter prediction unit (17070) can construct a motion information candidate list based on the neighboring blocks and derive the motion vector and / or reference picture index of the current block based on the received candidate selection information. Inter-prediction can be performed based on various prediction modes, and information regarding the prediction may include information indicating the mode of inter-prediction for the current block.

[0293] The adder (17040) can generate a restoration signal (restored picture, restored block, restored sample array) by adding the acquired residual signal to the prediction signal (predicted block, predicted sample array) output from the inter prediction unit (17070) or the intra prediction unit (265). In cases where there is no residual for the block to be processed, such as when a skip mode is applied, the predicted block can be used as the restoration block.

[0294] The addition unit (17040) may be called a restoration unit or a restoration block generation unit. The generated restoration signal may be used for intra-predicting the next block to be processed within the current picture, and may also be used for inter-predicting the next picture after filtering as described below.

[0295] The filtering unit (17050) can improve subjective / objective image quality by applying filtering to the restored signal. For example, the filtering unit (17050) can generate a modified restored picture by applying various filtering methods to the restored picture, and can transmit the modified restored picture to memory (17060), specifically to the DPB of memory (17060). Various filtering methods may include, for example, deblocking filtering, sample adaptive offset, adaptive loop filter, bilateral filter, etc.

[0296] The (modified) restored picture stored in the DPB of the memory (17060) can be used as a reference picture in the inter-prediction unit (17070). The memory (17060) can store motion information of blocks from which motion information within the current picture has been derived (or decoded) and / or motion information of blocks within the picture that have already been restored. The stored motion information can be transmitted to the inter-prediction unit (17070) to be used as motion information of spatially surrounding blocks or motion information of temporally surrounding blocks. The memory (170) can store restoration samples of blocks restored within the current picture and transmit them to the intra-prediction unit (17080).

[0297] In this specification, the embodiments described in the filtering unit (160), inter prediction unit (180), and intra prediction unit (185) of the encoding device (100) may be applied to the filtering unit (17050), inter prediction unit (17070), and intra prediction unit (17080) of the decoding device (17000), respectively, in the same or corresponding manner.

[0298] Meanwhile, at least one of the aforementioned prediction, transformation, and quantization procedures may be omitted. For example, for blocks to which pulse coding mode (PCM) is applied, the prediction, transformation, and quantization procedures may be omitted, and the decoded sample values ​​may be used as samples of the reconstructed image.

[0299] Occupancy map decompression (16003)

[0300] This is the reverse process of the previously explained occupancy map compression, a process to restore the occupancy map by decoding the compressed occupancy map bitstream.

[0301] Auxiliary patch info decompression (16004)

[0302] The auxiliary patch info can be restored by performing the reverse process of the auxiliary patch info compression described earlier and decoding the compressed auxiliary patch info bitstream.

[0303] Geometry Reconstruction (16005)

[0304] This is the inverse process of the geometry image generation described earlier. First, patches are extracted from the geometry image using the restored occupancy map, the 2D position / size information of the patches included in the auxiliary patch info, and the mapping information between blocks and patches. Subsequently, a point cloud is restored in 3D space using the geometry image of the extracted patches and the 3D position information of the patches included in the auxiliary patch info. Let g(u, v) be the geometry value corresponding to an arbitrary point (u, v) existing within a single patch, and let (d0, s0, r0) be the normal, tangent, and bitangent axis coordinate values ​​of the patch's position in 3D space. Then, the normal, tangent, and bitangent axis coordinate values ​​d(u, v), s(u, v), and r(u, v) of the 3D space position mapped to point (u, v) can be expressed as follows.

[0305] d(u, v) = d0 + g(u, v)

[0306] s(u, v) = s0 + u

[0307] r(u, v) = r0 + v

[0308] Smoothing (16006)

[0309] It is identical to the smoothing described earlier in the encoding process and is a process designed to eliminate discontinuities that may occur at patch boundaries due to image quality degradation during compression.

[0310] Texture Reconstruction (16007)

[0311] This is a process of restoring a color point cloud by assigning color values ​​to each point constituting the smoothed point cloud. This can be performed by using the mapping information between the geometry image and the point cloud from the geometry reconstruction process described in 2.4, and assigning color values ​​corresponding to texture image pixels at the same location in the geometry image in 2D space to points in the point cloud at the same location in 3D space.

[0312] Color smoothing (16008)

[0313] Similar to the geometry smoothing process described earlier, this is a task designed to eliminate discontinuities in color values ​​that may occur at patch boundaries due to image quality degradation during compression. It can be performed through the following process.

[0314] ① Calculate the adjacent points of each point constituting the reconstructed color point cloud using a KD tree, etc. Alternatively, the adjacent point information calculated during the geometry smoothing process described in Section 2.5 can be used as is.

[0315] ② For each point, determine whether the point is located on the patch boundary. The boundary information calculated during the geometry smoothing process described in Section 2.5 may be used as is.

[0316] ③ For points adjacent to a point on the boundary surface, the distribution of color values ​​is examined to determine whether smoothing is required. For example, if the luminance entropy is below the threshold local entry (i.e., if there are many similar luminance values), the area is determined not to be an edge, and smoothing can be performed. Smoothing methods may include changing the color value of the point to the average of its adjacent points.

[0317] FIG. 18 shows an example of an operation flowchart of a transmitting device according to embodiments.

[0318] A transmitting device according to the embodiments may correspond to the transmitting device of FIG. 1, the encoding process of FIG. 4, and the 2D video / image encoder of FIG. 15, or may perform some or all of their operations. Each component of the transmitting device may correspond to software, hardware, a processor, and / or a combination thereof.

[0319] The operation process of the transmitting end for compression and transmission of point cloud data using V-PCC can be as shown in the diagram.

[0320] The point cloud data transmission device according to the embodiments may be referred to as a transmission device, etc.

[0321] Regarding the patch generation unit (18000), first, a patch for 2D image mapping of a point cloud is generated. Additional patch information is generated as a result of the patch generation, and this information can be used in the geometry image generation, texture image generation, and geometry restoration process for smoothing.

[0322] Regarding the patch packing section (18001), the generated patches undergo a patch packing process in which they are mapped into a 2D image. An occupancy map can be generated as a result of the patch packing, and the occupancy map can be used in the geometry image generation, texture image generation, and geometry restoration process for smoothing.

[0323] The geometry image generation unit (18002) generates a geometry image using additional patch information and an occupancy map, and the generated geometry image is encoded into a single bitstream through video encoding.

[0324] The encoding preprocessing (18003) may include an image padding procedure. The generated geometry image or the geometry image regenerated by decoding the encoded geometry bitstream can be used for 3D geometry restoration and can then undergo a smoothing process.

[0325] The texture image generation unit (18004) can generate a texture image using (smoothed) 3D geometry, a point cloud, additional patch information, and an occupancy map. The generated texture image can be encoded into a single video bitstream.

[0326] The metadata encoding unit (18005) can encode additional patch information into a single metadata bitstream.

[0327] The video encoding unit (18006) can encode the occupancy map into a single video bitstream.

[0328] The multiplexer (18007) multiplexes the video bitstream of the generated geometry, texture image, and occupancy map, and the additional patch information metadata bitstream into a single bitstream.

[0329] The transmitter (18008) can transmit the bitstream to the receiver. Alternatively, the video bitstream of the generated geometry, texture image, and Occupancy map and the additional patch information metadata bitstream can be created as a file with one or more track data or encapsulated into segments and transmitted to the receiver through the transmitter.

[0330] FIG. 19 shows an example of an operation flowchart of a receiving device according to embodiments.

[0331] A receiving device according to the embodiments may correspond to the receiving device of FIG. 1, the decoding process of FIG. 16, and the 2D video / image encoder of FIG. 17, or may perform some or all of their operations. Each component of the receiving device may correspond to software, hardware, a processor, and / or a combination thereof.

[0332] The operation process of the receiving end for receiving and restoring point cloud data using V-PCC can be as shown in the figure. The operation of the V-PCC receiving end can follow the inverse process of the operation of the V-PCC transmitting end in Fig. 18.

[0333] The point cloud data receiving device according to the embodiments may be referred to as a receiving device, etc.

[0334] The bitstream of the received point cloud is demultiplexed by the demultiplexer (19000) into video bitstreams of a compressed geometry image, texture image, and occupancy map, and additional patch information metadata bitstream after file / segment decapsulation. The video decoding unit (19001) and the metadata decoding unit (19002) decode the demultiplexed video bitstreams and metadata bitstreams. The geometry image, occupancy map, and additional patch information decoded by the geometry restoration unit (19003) are used to restore the 3D geometry, and then undergo a smoothing process by the smoother (19004). By assigning color values ​​to the smoothed 3D geometry using the texture image, the color point cloud image / picture can be restored by the texture restoration unit (19005). Subsequently, a color smoothing process may be additionally performed to improve objective / subjective visual quality, and the modified point cloud image / picture derived therefrom is displayed to the user through a rendering process (e.g., by a point cloud renderer). Meanwhile, the color smoothing process may be omitted depending on the case.

[0335] FIG. 20 shows an example of a structure capable of interoperability with a point cloud data transmission and reception method / device according to embodiments.

[0336] In the structure according to the embodiments, at least one of a server (2360), a robot (2010), an autonomous vehicle (2020), an XR device (2030), a smartphone (2040), a home appliance (2050), and / or an HMD (2070) is connected to a cloud network (2010). Here, the robot (2010), the autonomous vehicle (2020), the XR device (2030), the smartphone (2040), or the home appliance (2050) may be referred to as a device. Additionally, the XR device (2030) may correspond to or be linked with a point cloud data (PCC) device according to the embodiments.

[0337] The cloud network (2000) may refer to a network that constitutes part of the cloud computing infrastructure or exists within the cloud computing infrastructure. Here, the cloud network (2000) may be configured using a 3G network, a 4G or LTE (Long Term Evolution) network, or a 5G network, etc.

[0338] The server (2360) is connected to at least one of a robot (2010), an autonomous vehicle (2020), an XR device (2030), a smartphone (2040), a home appliance (2050) and / or an HMD (2070) via a cloud network (2000) and can assist in at least some of the processing of the connected devices (2010 to 2070).

[0339] The HMD (Head-Mount Display) (2070) represents one of the types in which an XR device and / or PCC device according to the embodiments may be implemented. A device of the HMD type according to the embodiments includes a communication unit, a control unit, a memory unit, an I / O unit, a sensor unit, and a power supply unit, etc.

[0340] Hereinafter, various embodiments of devices (2010 to 2070) to which the above-described technology is applied are described. Here, the devices (2000 to 2700) illustrated in FIG. 20 can be linked / coupled with point cloud data transmission and reception devices according to the above-described embodiments.

[0341] <PCC+XR> The XR / PCC device (2030) may be implemented as a Head-Mount Display (HMD), a Head-Up Display (HUD) equipped in a vehicle, a television, a mobile phone, a smartphone, a computer, a wearable device, a home appliance, digital signage, a vehicle, a stationary robot, or a mobile robot by applying PCC and / or XR (AR+VR) technology.

[0342] The XR / PCC device (2030) can obtain information about surrounding space or real objects by analyzing 3D point cloud data or image data obtained through various sensors or from an external device to generate location data and attribute data for 3D points, and can render and output an XR object to be output. For example, the XR / PCC device (2030) can output an XR object containing additional information about a recognized object by associating it with the recognized object.

[0343] <PCC+자율주행+XR> The autonomous vehicle (2020) can be implemented as a mobile robot, vehicle, unmanned aerial vehicle, etc. by applying PCC technology and XR technology.

[0344] An autonomous vehicle (2020) equipped with XR / PCC technology may refer to an autonomous vehicle equipped with means for providing XR images, or an autonomous vehicle that is the subject of control / interaction within the XR images. In particular, the autonomous vehicle (2020) that is the subject of control / interaction within the XR images is distinguished from the XR device (2030) and can be interconnected with it.

[0345] An autonomous vehicle (2020) equipped with means for providing XR / PCC images can acquire sensor information from sensors including cameras and output XR / PCC images generated based on the acquired sensor information. For example, the autonomous vehicle can provide an XR / PCC object corresponding to a real object or an object in the screen to the occupant by providing an XR / PCC object by outputting an XR / PCC image with a HUD.

[0346] In this case, when an XR / PCC object is displayed on a HUD, at least a portion of the XR / PCC object may be displayed so as to overlap with the actual object to which the occupant's gaze is directed. Conversely, when an XR / PCC object is displayed on a display installed inside the autonomous vehicle, at least a portion of the XR / PCC object may be displayed so as to overlap with an object on the screen. For example, the autonomous vehicle may display XR / PCC objects corresponding to objects such as lanes, other vehicles, traffic lights, traffic signs, motorcycles, pedestrians, and buildings.

[0347] VR (Virtual Reality) technology, AR (Augmented Reality) technology, MR (Mixed Reality) technology and / or PCC (Point Cloud Compression) technology according to the embodiments can be applied to various devices.

[0348] In other words, VR technology is a display technology that provides real-world objects or backgrounds solely as CG images. On the other hand, AR technology refers to a technology that displays virtual CG images alongside images of real objects. Furthermore, MR technology is similar to the aforementioned AR technology in that it mixes and combines virtual objects with the real world. However, it is distinguished from AR technology in that while AR technology maintains a clear distinction between real-world objects and virtual objects created from CG images, using virtual objects to complement real-world objects, MR technology regards virtual objects as having the same nature as real-world objects. To give a more specific example, the aforementioned MR technology is applied in hologram services.

[0349] However, recently, rather than clearly distinguishing between VR, AR, and MR technologies, they are also referred to as XR (extended Reality) technology. Therefore, embodiments of the present invention are applicable to all VR, AR, MR, and XR technologies. Such technology may utilize encoding / decoding based on PCC, V-PCC, and G-PCC technologies.

[0350] The PCC method / device according to the embodiments can be applied to a vehicle providing autonomous driving services.

[0351] Vehicles providing autonomous driving services are connected to PCC devices to enable wired / wireless communication.

[0352] When a point cloud data (PCC) transceiver according to the embodiments is connected to a vehicle for wired or wireless communication, it can receive and process content data related to AR / VR / PCC services that can be provided along with an autonomous driving service, and transmit it to the vehicle. Additionally, when the point cloud data transceiver is mounted on a vehicle, the point cloud transceiver can receive and process content data related to AR / VR / PCC services according to a user input signal received through a user interface device and provide it to the user. A vehicle or a user interface device according to the embodiments can receive a user input signal. The user input signal according to the embodiments may include a signal indicating an autonomous driving service.

[0353] The encoding method and apparatus according to the embodiments may include and perform the transmission device (10000) of FIG. 1, the encoder of FIG. 4, the encoder of FIG. 15, the transmission device of FIG. 18, each device of FIG. 20, the encoders of FIG. 23 to 24, the method of FIG. 28, etc.

[0354] The decoding method and device according to the embodiments may include and perform the receiving device (10002) of FIG. 1, the decoder of FIG. 16, the decoder of FIG. 17, the receiving device of FIG. 19, each device of FIG. 20, the decoders of FIG. 25 to 26, the decoder (renderer) of FIG. 27, the method of FIG. 29, etc.

[0355] Point cloud data can be defined as a set of multiple points distributed in three-dimensional space, and each point may include color (RGB), reflectance, normal, or other attribute information along with three-dimensional coordinates (e.g., x, y, z). Gaussian splat data can be defined as a set of multiple Gaussian elements defined corresponding to each point or local region of points in the point cloud, and each Gaussian element may include a center coordinate (μ), variance / covariance (Σ) or scale / rotation defining the spatial distribution, opacity or weight, and color / appearance parameters. Accordingly, Gaussian splat data can be understood as a data structure that extends discrete samples of a point cloud into a continuous distribution-based representation.

[0356] The association between point cloud data and Gaussian splat data can be extended and defined as a mutually convertible mapping relationship. For example, μi can be set as the coordinates of point Pi so that Gaussian Gi corresponds 1:1 to point Pi, Σi is derived from the k-nearest neighbors (kNN) or local patch distribution around Pi, and color / attribute parameters can be determined by Pi's attribute values ​​or aggregate values ​​of surrounding points. Conversely, an extended point cloud can be defined from Gaussian splat data to include Gaussian parameters such as Σ, scale / rotation, and opacity as additional attributes in addition to the point coordinates and color / attributes. If necessary, the definition can be extended to allow 1:N (point to multiple Gaussians) or N:1 (multiple points to a single Gaussian) correspondences and the configuration of parameter sets per LoD.

[0357] The method and apparatus according to the embodiments may include and perform a V3C-based multiple data support method for 3D Gaussian Splatting data encoding / decoding.

[0358] Recently, 3D Gaussian Splatting (GS) technology has been actively researched. 3D Gaussian Splatting is a technique that represents 3D space in the form of small Gaussians. It utilizes a Gaussian distribution, which has the characteristic that values ​​decrease as they move away from the center, to represent 3D space and subsequently construct and render scenes. It possesses more efficient characteristics compared to previously studied NeRF technology. Since complex 3D scenes can be represented as Gaussian splatters to effectively display details, it can achieve a similar level of performance with relatively less data compared to NeRF. Furthermore, due to the characteristics of the Gaussian distribution, elements naturally overlap to generate smooth and realistic images, providing more natural video performance.

[0359] Such 3D GS is also receiving attention in MPEG standards, and discussions regarding compression methods for 3D GS data are ongoing. In terms of utilizing points in three-dimensional space, it shares many similarities with the characteristics of a point cloud, and in fact, there are cases where the same attributes are used. Accordingly, MPEG is making efforts to interpret 3D GS data as a kind of extended form of a point cloud and to establish a compression standard. Among the point cloud compression methods, G-PCC and V-PCC (V3C), the embodiments propose a method for compressing 3D GS based on V-PCC (V3C) and extending it. That is, the embodiments relate to a method for compressing 4D GS, which is a time-axis extended version of 3D GS data representing three-dimensional space using an existing 2D video codec, namely the V-GSC (Video-based Gaussian Splat Coding) method.

[0360] When compressing 3D GS based on V3C, depending on the type of data, geometry data can be encoded as the GVD type among v3C units, and attribute data can be encoded as the AVD type among v3C units. However, currently, V3C only considers geometry data corresponding to position information for the GVD type. On the other hand, since the geometry data of Gaussian splat data exists in various forms such as rotation and scale in addition to position information, if the data is encoded as the GVD type, a problem may arise in that it is difficult to identify the type of decoded geometry data during the decoding process. In addition, there is also a problem in that support is difficult because SH data and other attribute data are data types that are not handled by V3C. In the embodiments, a method is proposed to encode / decode based on V3C by considering multiple geometry data of 3D GS and new attribute types of 3D GS.

[0361] The embodiments relate to an encoding method and a decoding method including 3D Gaussian Splatting (3D GS) processing in V-PCC encoding / decoding, and propose a method for efficiently compressing attributes and data constituting 3D Gaussian Splatting (3D GS) based on V-PCC. Additionally, the embodiments relate to an encoding / decoding method based on V3C for processing and compressing 3D Gaussian Splatting (3D GS) data types that are not defined within V3C. In the embodiments, the range of data to be compressed may include not only 3D Gaussian splatting (3D GS) but also 4D Gaussian splatting (4D GS), which is a time-axis extension concept of 3D Gaussian splatting (3D GS), and 4D Gaussian splatting (4D GS) may mean a data sequence in which each frame consists of 3D Gaussian splatting (3D GS) data.

[0362] In this regard, from the perspective of a video codec, 3D Gaussian splatting (3D GS) can be understood as a concept corresponding to a single video frame, and 4D Gaussian splatting (4D GS) can be understood as a concept corresponding to a video sequence composed of multiple frames. Additionally, in the embodiments, 3D Gaussian splatting (3D GS) may refer to a still volumetric image frame that represents a three-dimensional space of a specific time period and is composed of multiple Gaussian splats, and 4D Gaussian splatting (4D GS) may refer to a dynamic visual volumetric data sequence composed of a set of dynamic volumetric frames that change along the time axis. Unless otherwise specified, the functions described in this document can be applied to both 3D Gaussian Splatting (3D GS) and 4D Gaussian Splatting (4D GS), and V3C is merely one example for utilizing a video codec and is not limited thereto, and can be described in the same or similar ways based on a general video codec.

[0363] FIG. 21 shows the components of 3DGS data according to embodiments.

[0364] FIG. 21 illustrates the components of 3DGS type point cloud data encoded and decoded by an encoding and decoding method / device according to embodiments (a transmitting device (10000) of FIG. 1, an encoder of FIG. 4, an encoder of FIG. 15, a transmitting device of FIG. 18, each device of FIG. 20, encoders of FIG. 23 to 24, a method of FIG. 28, a receiving device (10002) of FIG. 1, a decoder of FIG. 16, a decoder of FIG. 17, a receiving device of FIG. 19, each device of FIG. 20, decoders of FIG. 25 to 26, a decoder (renderer) of FIG. 27, a method of FIG. 29, etc.). The point cloud according to embodiments may be configured as 3DGS (Fig. 21).

[0365] Existing V-PCC encoding / decoding standards have been developed to efficiently encode and decode point cloud data. This point cloud data has a structure in which attribute information is stored along with vertices, and the types of attributes are also practically developed to primarily support color information. On the other hand, Gaussian Splatting (GS), which is currently under research, is characterized by containing more components (e.g., coefficients and parameters) compared to existing point cloud data. Figure 21 illustrates the components of Gaussian Splatting (GS) data as an example.

[0366] As illustrated in FIG. 21, 3D Gaussian splatting (3DGS) data can be broadly divided into geometry and attributes. In the geometry portion, similar to the point cloud encoding standard V-PCC / V3C, 3 dimensions (x, y, z) can be used to represent the positions of vertices, 4 dimensions (x, y, z, w) can be used to represent rotation information, and 3 dimensions can be used to represent scale information. Additionally, in the attributes portion, 1 dimension for opacity, 3 dimensions for DC, and 45 dimensions for spherical harmonic coefficients (SH coeff.) can be used.

[0367] Accordingly, the embodiments propose an encoding / decoding method based on a 2D video codec, centered on a configuration that extends V-PCC (V3C) to compress 3D Gaussian Splatting (3DGS) and 4D Gaussian Splatting (4DGS) data. For example, when compressing 3D Gaussian Splatting (3DGS) data into a V3C bitstream structure, information such as position, rotation, and scale can be encoded into geometry video data (GVD) among V3C units. However, since the geometry type of the geometry video data (GVD) of the V3C is defined as representing only position information, a problem may arise in that it is difficult to distinguish various types of geometry data other than position during the decoding process. In addition, there is a problem that data types not defined in V3C, such as multiple geometry data and spherical harmonic coefficients (SH coefficient), are difficult to directly support in the existing structure. Therefore, the embodiments propose a method to extend V3C to enable encoding / decoding based on V3C by considering multiple geometry data and new attribute types of 3D Gaussian splatting (3DGS).

[0368] FIG. 22 shows a V3C bitstream structure according to embodiments.

[0369] The encoding and decoding method / device according to the embodiments (transmitting device (10000) of FIG. 1, encoder of FIG. 4, encoder of FIG. 15, transmitting device of FIG. 18, each device of FIG. 20, encoders of FIG. 23 to 24, method of FIG. 28, receiving device (10002) of FIG. 1, decoder of FIG. 16, decoder of FIG. 17, receiving device of FIG. 19, each device of FIG. 20, decoders of FIG. 25 to 26, decoder (renderer) of FIG. 27, method of FIG. 29, etc.) can encode point cloud data (Gaussian splat data) to generate a V3C bitstream such as FIG. 22, and decode point cloud data (Gaussian splat data) from the V3C bitstream.

[0370] The term V-PCC (Video-based Point Cloud Compression) used in this document may be used with the same meaning as V3C (Visual Volumetric Video-based Coding), and the two terms may be used interchangeably. Therefore, in this document, the term V-PCC may be interpreted as the term V3C. Additionally, the term 3D Gaussian Splatting (3DGS) used in this document may be used with the same meaning as Gaussian splatting, Gsplat, GS, etc., and these terms may be used interchangeably.

[0371] The term 4DGS (four-dimensional Gaussian splatting) used in this document is a concept of 3DGS (three-dimensional Gaussian splatting) with the addition of time (time t) or temporal concepts, and can be used to refer to video data constructed from a 3DGS dataset in which one or more parameters change over time. Accordingly, dynamic 3DGS or moving 3DGS can also be used with the same meaning as 4DGS. Meanwhile, Gaussian splatting content can be encoded into a V3C bitstream structure, and FIG. 222 illustrates an exemplary V3C bitstream structure used when encoding V3C content in the V3C codec document ISO / IEC 23090-5.

[0372] FIG. 22 illustrates an exemplary V3C bitstream structure and shows the hierarchical encapsulation relationship of the V3C Sample Stream, NAL Sample Stream, and Atlas Tile Layer. FIG. 22 illustrates the V3C Sample Stream, which may be composed of a series of V3C units, along with stream precision and stream size fields.

[0373] For example, V3C parameter sets (V3C_VPS, v3c_parameter_set()), units containing atlas sub-bitstreams (atlas_sub_bitstream()) (e.g., V3C_AD, V3C_CAD, V3C_PVD), units containing video sub-bitstreams (video_sub_bitstream()) (e.g., V3C_OVD, V3C_GVD, V3C_AVD), and units containing basemesh / displacement sub-bitstreams (basemesh_sub_bitstream(), displacementsub_bitstream()) (e.g., V3C_BMD, V3C_ADD) are illustrated (Fig. 22, V3C Sample Stream area).

[0374] Referring to FIG. 22, a NAL Sample Stream is shown and may be composed of a series of NAL units combined with a NAL unit header (nal_unit_header()) and a payload, along with NAL Unit Size Precision and NAL Unit Size fields. FIG. 22 illustrates an NAL unit type (ACL NAL unit type) corresponding to an atlas sequence parameter set (NAL_ASPS, atlas_sequence_parameter_set_rbsp()), an atlas adaptation parameter set (NAL_AAPS, atlas_adaptation_parameter_set_rbsp()), an atlas frame parameter set (NAL_AFPS, atlas_frame_parameter_set_rbsp()), prefix / suffix extension auxiliary information (NAL_PREFIX_ESEI, NAL_SUFFIX_ESEI, sei_rbsp()), and an atlas tile layer (atlas_tile_layer_rbsp()) (Fig. 22, NAL Sample Stream area). FIG. 22 illustrates an atlas tile layer, which may include a tile header (atlas_tile_header()) and a tile data unit (atlas_tile_data_unit()).For example, an i-tile (I_TILE) may be composed of an intra patch data unit (I_INTRA, patch_data_unit()), a raw patch data unit (I_RAW, raw_patch_data_unit()), a termination marker unit (I_EOM, eom_patch_data_unit()), and a termination (I_END), and a p-tile (P_TILE) may be composed of a skip (P_SKIP, skip_patch_data_unit()), a merge (P_MERGE, merge_patch_data_unit()), an inter (P_INTER, inter_patch_data_unit()), an intra (P_INTRA, patch_data_unit()), a raw (P_RAW), a termination marker (P_EOM), and a termination (P_END) unit (Fig. 22, Atlas Tile Layer area).

[0375] A V3C unit may consist of a V3C unit header and a payload, and may include V3C VPS (VPCC / VGSC parameter set), V3C AD (atlas data), V3C GVD (geometry video data), V3C AVD (attribute video data), V3C OVD (accumulation video data), etc. as an embodiment configuring a V3C unit type. V3C VPS may include V3C and V-GSC parameter information as a V3C / V-GSC Parameter Set. V3C AD may include atlas information of V3C as atlas data. V3C GVD may include geometry video sub-bitstream and related information as geometry video data. A V3C AVD may include an attribute video sub-bitstream and related information as attribute video data. A V3C OVD may include an occupancy video sub-bitstream and related information as occupancy video data.

[0376] In the embodiments, as illustrated in FIG. 23, the encoder structure of the existing V3C can be extended to support Gaussian splatting data. The V-GSC encoder performs a pre-encoding process on the four-dimensional Gaussian splatting (4DGS) parameters provided as input, and the video frame generated as a result of the pre-encoding process can be encoded through a video encoder. Subsequently, the units encapsulated into V3C units can be configured into a V3C-based bitstream structure through a multiplexer.

[0377] The pre-encoding function may include a series of data processing steps executed before performing video coding on input GS parameters, such as, for example, linear / non-linear transforms on GS parameter values, bit-depth transformations, rotation parameter conversions, clipping, quantization, and packing into 2D frames. Additionally, while three existing video encoders (e.g., HEVC or VVC, etc.) are currently used in V3C, the video-based GS compression method proposed in the embodiments allows for a configuration utilizing six video encoders.

[0378] Meanwhile, V3C's atlas data follows the existing method and may not be extended separately. The atlas data can be used as metadata to decode the V-GSC bitstream and restore it to GS.

[0379] FIG. 23 illustrates an example of a V-GSC encoder structure according to embodiments.

[0380] FIG. 23 corresponds to an encoding method / device (Fig. 1 transmitting device (10000), Fig. 4 encoder, Fig. 15 encoder, Fig. 18 transmitting device, Fig. 20 each device, Fig. 23 to Fig. 24 encoders, Fig. 28 method).

[0381] In FIG. 23, the input data may be four-dimensional Gaussian splatting (4DGS) parameters, such as, for example, a Gaussian distribution center position (x, y, z), opacity (o), a Gaussian scale (Sx, Sy, Sz), a rotation parameter (Vx, Vy, Vz, r), SH DC coefficients (r, g, b), and SH AC coefficients (shr(i), shg(i), shb(i), i=1..15). A pre-encoding function performs processing to convert these input parameters into video frames, and the converted video frames can be encoded into Video NALUs by a plurality of video encoders (Video Encoder #1 to #6). Additionally, an Atlas Encoder can generate an Atlas NALU.

[0382] In the example of FIG. 23, NALUs can be encapsulated according to the V3C unit type. For example, the Atlas NALU can be encapsulated as V3C_AD, the output of the video encoders (Video Encoder #1, #3, #4) can be encapsulated as V3C_GVD or V3C_AVD, and the output of the video encoder (Video Encoder #2) can be encapsulated as V3C_OVD or V3C_AVD. Additionally, the output of the video encoders (Video Encoder #5, #6) can be encapsulated as V3C_AVD. A multiplexer can multiplex the encapsulated multiple V3C units to generate a V-GSC bitstream.

[0383] FIG. 24 shows pre-encoding according to embodiments.

[0384] The encoding and decoding method / device according to the embodiments (transmitting device (10000) of FIG. 1, encoder of FIG. 4, encoder of FIG. 15, transmitting device of FIG. 18, each device of FIG. 20, encoders of FIG. 23 to 24, method of FIG. 28, etc.) can pre-encode point cloud data (Gaussian splat data) as shown in FIG. 24.

[0385] FIG. 24 illustrates an example of a preprocessing step that can be performed before video coding four-dimensional Gaussian splatting (4DGS) parameters in the embodiments. According to the embodiments, for each 4DGS parameter, rotation conversion, GSTF, 2D frame packing, and video encoder steps may be performed, and some of these steps may be omitted or added depending on the characteristics of the parameter. For example, the 4DGS parameters may be provided as a set of values ​​having multiple dimensions, and in the preprocessing step, each parameter may be converted into a form suitable for video codec-based coding, then packed into 2D frame units, and subsequently compressed into video.

[0386] In the embodiments, the position data of the GS (e.g., (X, Y, Z) of the center positions) can be compressed using the GVD or AVD of the V3C. In this case, additional projection or transformation may be performed to efficiently utilize the video codec, for example, the 3D data of (X, Y, Z) can be transformed into RGB or YUV to match the video input format. Additionally, it can be compressed into video after 2D frame packing, and a mapping process to represent the position values ​​as integers for video compression efficiency may be performed before video compression.

[0387] In the embodiments, the opacity data of the GS can be compressed using the OVD or AVD of the V3C. Since opacity is one-dimensional data, it can be configured to compress video using the OVD or to process it using entropy coding with arithmetic coding. When using the OVD, it can be compressed into video after 2D frame packing, and a mapping process to represent the opacity value as an integer can be performed before video compression.

[0388] In the embodiments, the scale data of the GS can be compressed using the V3C's GVD or AVD as 3D data. At this time, additional projection or transformation may be performed to efficiently utilize the video codec, and, for example, the 3D data (e.g., (X, Y, Z) components) may be converted to RGB or YUV, and then compressed into video after 2D frame packing. Additionally, for video compression efficiency, a mapping process to represent the scale value as an integer may be performed before video compression.

[0389] In the embodiments, the rotation data of the GS can be configured to convert the rotation representation method through a rotation conversion unit. For example, rotation data expressed in a quaternion format can be converted and expressed in Euler angles. The input may be rotation data of the GS expressed in a quaternion and may be configured in four dimensions, and the output may be rotation data of the GS expressed in Euler angles and may be configured in three dimensions. An embodiment of converting rotation data in a quaternion format into Euler angles format may be as shown in Equation 1 below.

[0390]

[0391]

[0392]

[0393]

[0394] In the embodiments, q_w, q_x, q_y, and q_z may represent rotation parameters expressed in quaternion form, and α, , may refer to a rotation parameter expressed in the form of Euler angles. The rotation data of GS is 4-dimensional data and can be compressed using V3C's GVD or AVD. Generally, since video encoders use 3-channel data in RGB or YUV format as input, in the embodiments, the rotation data can be converted to match the 3-channel input format and input into the video encoder. For example, rotation data ( It can be configured to convert the data into the form and input each coefficient into a respective video channel, and accordingly, the rotation data converted into 3D can be used as a GVD type input by mapping it one-to-one to RGB or YUV to input it into a video codec. In addition, for video compression efficiency, a mapping process to express the rotation values ​​as integers can be performed before video compression.

[0395] In addition, in the embodiments, it is also possible to configure 4-channel data in quaternion format to be input directly into a video codec without conversion. For example, during the process of constructing a 2D frame, rotation parameters belonging to each GS included in the frame can be processed by sequentially packing them, and such sequential packing configuration may be possible when using the YUV 4:2:2 format.

[0396] for (i=0; i <Fmax; i++) {

[0397] Y(2i) = Qw(i);

[0398] Y(2i+1) = Qx(i);

[0399] U(i) = Qy(i);

[0400] V(i) = Qz(i);

[0401] }

[0402] In the embodiments, to frame pack rotation parameters Qw(i), Qx(i), Qy(i), and Qz(i) in quaternion form into a YUV format for video codec input, the following mapping can be performed. For example, for indices from i=0 to i=Fmax-1, Y(2i)=Qw(i) and Y(2i+1)=Qx(i) can be set so that Qw(i) and Qx(i) are stored alternately in Y, which is the luminance component, and U(i)=Qy(i) and V(i)=Qz(i) can be set so that Qy(i) and Qz(i) are stored in U and V, which are the chroma components, respectively. Accordingly, 4-channel rotation parameter data can be configured into 2D frames using a 4:2:2 based YUV sample arrangement and can be packed so that they can be input into a video encoder without conversion.

[0403] In the embodiments, Y(i), U(i), and V(i) may each represent the Y, U, and V components constituting a two-dimensional frame (2D frame) for video encoder input, Fmax may represent the total number of pixels in the frame (based on Y*2), and i may represent the order when raster scanning the frame. Additionally, Qw(i), Qx(i), Qy(i), and Qz(i) may each represent integer values ​​converted to 10-bit for the q_w(i), q_x(i), q_y(i), and q_z(i) components.

[0404] In the embodiments, the SH DC coefficients data of the GS can be compressed using the AVD of the V3C as 3D data. For video utilization, a transformation to RGB or YUV can be performed, and the data can be compressed into video after 2D frame packing. Since the SH DC coefficients are related to the color information of the GS and are not significantly different from the form of color information in the V3C, the AVD can be configured to be utilized as is. Additionally, for video compression efficiency, a mapping process to represent the SH DC values ​​as integers can be performed before video compression.

[0405] In the embodiments, the SH AC coefficients data of the GS can be compressed using the AVD of the V3C as 15x3 dimension data. For example, by considering the 15x3 dimension characteristics, it is possible to expand the AVD to utilize multiple dimensions (e.g., 15 layers). In addition, when the k-th SH AC coefficient of the j-th GS is expressed as (shr(j, k), shg(j, k), shb(j, k)), the 15 coefficients from r(j, 0) to r(j, 14) that constitute the AC coefficients of spherical harmonics for the R color signal are data corresponding to R among the R, G, and B elements that constitute the frame input to the video encoder, and can be configured to be mapped to positions from (j*15) to (j*15+14) in raster scan order. SH AC coefficient information can be skipped depending on importance, and if it belongs to data of low importance within the GS data, there is a possibility that it can be optimized through future research. Even when using AVD, a transformation to RGB or YUV may be performed for video utilization, and the video may be compressed after 2D frame packing. Additionally, a mapping process to represent SH AC values ​​as integers may be performed before video compression to improve video compression efficiency.

[0406] In addition, in the embodiments, a Gaussian Splat Transfer Function (GSTF) may be used as a mapping function as an example of the mapping process for GS parameters. The GSTF may be used to map GS parameter values ​​expressed as real numbers to symbol values ​​expressed as integers. The GSTF may be a monotonically increasing function, and as an example of the GSTF, the GSTF may be expressed as a linear function as shown in Equation 2.

[0407]

[0408] In the embodiments, the Gaussian Splat Transfer Function (GSTF) represents a GS parameter a expressed in the real number domain as an integer symbol value corresponding to bit-depth bd_a. It can be defined as a function for normalization and mapping. For example, if the minimum value a_min and maximum value a_max of parameter a are set, the mapped value can be defined as shown in Equation 2 below. At this time can be configured to be scaled to a value in the range from 0 to (2^(bd_a)-1), and a_min and a_max can be used as values ​​to limit the valid range of parameter a.

[0409] Mathematical formula 2

[0410]

[0411] In the embodiments, in Equation 2 may be a real number value of any one of the GS parameters: position, opacity, scale, rotation, SH DC, and SH AC. Additionally, if a GS parameter is expressed as a vector with a length of 1 or more, such as position, may mean a value corresponding to one element of that vector. of Equation 2 can be a Gaussian parameter value expressed as an integer.

[0412] In the embodiments, a_max, a_min, and bd_a may represent the maximum value of, the minimum value of, and the bit depth for representing as an integer, respectively. The parameters of the GSTF (e.g., a_max, a_min, bd_a) may be determined by input from a user or through analysis of the GS input to the encoder. Additionally, the GSTF parameters may be included in a bitstream and transmitted to a decoder. The GSTF parameters may have different values ​​for each GS parameter, and if a set containing one or more GS parameters is defined as a GS parameter group, the GSTF may be configured to have different GSTF parameters for each GS parameter group. Also, in the embodiments, a may be clipped using a_max and a_min before being input into the GSTF.

[0413] FIG. 24 illustrates an example of the configuration of a pre-encoding function according to embodiments. In FIG. 24, the input data may include a Gaussian distribution center position (x, y, z), opacity (o), a Gaussian scale (Sx, Sy, Sz), a rotation parameter (Vx, Vy, Vz, r), SH DC coefficients (r, g, b), and SH AC coefficients (shr(i), shg(i), shb(i), i=1..15), etc. The pre-encoding function may be configured to apply GSTF to each input parameter to map real-valued parameters to integer symbol values, and then perform 2D frame packing to generate a frame that can be provided as input to a video encoder.

[0414] In the example of FIG. 24, the Gaussian distribution center position, opacity, Gaussian scale, SH DC coefficient, and SH AC coefficient can be transmitted to a video encoder after sequentially passing through GSTF and 2D frame packing, respectively. Additionally, the rotation parameter can be configured to be transmitted to a video encoder after the rotation representation method is converted through rotation conversion, and then passing through GSTF and 2D frame packing. Accordingly, in the embodiments, multiple types of GS parameters can be preprocessed into a form suitable for video encoding-based compression, and the preprocessed results can be configured in frame units to be encoded in a subsequent video encoder.

[0415] Data generated by encoding 3D Gaussian splatting (3D GS) data through a video codec can be encapsulated into a V3C unit such as GVD or AVD depending on the data type. For example, geometry data of 3D GS (e.g., position, scale, rotation) can be encapsulated into a GVD type V3C unit according to the embodiments, and it is possible to configure each geometry data to be encapsulated into a separate GVD V3C unit.

[0416] In Example 1, a map index can be utilized to distinguish the geometry type for each GVD unit. Specifically, the geometry type can be specified according to the map index included in the GVD unit, and the geometry type can be implicitly derived accordingly. For example, as shown in Table 1 (Example of geometry type information mapped to a map index), geometry type information mapped to a specific map index can be predefined, and the encoder and decoder can be configured to use the same mapping rule. Table 1 is an example defining the geometry type mapped to a map index, and the types of geometry types can be added, and the map index values ​​mapped to the geometry types can be changed. In addition, the syntax representing the map index within the V3C unit header may be vuh_map_index, which may be described in section 8.3.2.2 "V3C unit header syntax" of the V3C codec specification document (ISO / IEC 23090-5(4E):2025) (ISO / IEC 23090-5(4E):2025, Table of Contents: 8.3.2.2). Thus, by the map index signaled through the V3C unit header, it is possible to induce that the geometry type of the corresponding GVD unit is position when the map index is 0, scale when the map index is 1, and rotation when the map index is 2.

[0417] Map indexGeometry type0Position1Scale2rotation

[0418] Attribute data of 3D Gaussian Splatting (3D GS) (e.g., SH DC, SH AC, etc.) may be encapsulated into AVD type V3C units according to the embodiments. When each attribute data is encapsulated into a separate AVD V3C unit, the attribute type may be configured to be distinguished using a map index to distinguish attribute data per AVD unit. For example, for data among 3D GS attribute data such as SH DC and SH AC for which the attribute type is not defined in the V3C, the attribute type may be implicitly derived by specifying the attribute type according to the map index of the AVD. Specifically, attribute type information mapped to a specific map index can be predefined as in Table 2 (an example of attribute type information mapped to a map index), and the encoder and decoder can be made to use the same mapping rule, and the types of attribute types in Table 2 can be added, and the map index values ​​mapped to the attribute types can be changed. In addition, by means of a map index signaled through a V3C unit header, if the map index is 0, it can be derived that the attribute type of the AVD unit is SH DC, and if the map index is 1, it can be derived that the attribute type of the AVD unit is SH AC.

[0419] In addition, in the embodiments, mapping information between a map index and geometry or attribute type information may be implicitly derived by the same agreement between the encoder and the decoder, or may be configured to include the mapping information in a bitstream for signaling.

[0420] Map indexGeometry type0SH DC1SH AC

[0421] In Example 2, a configuration is proposed to distinguish 3D GS data types by encapsulating 3D Gaussian Splatting (3D GS) data into an AVD type among V3C units. For example, attribute data and geometry data of GS data can be encapsulated into an AVD. Since AVD, unlike GVD, can signal type information of attribute data through VPS (V3C Parameter Set), Example 2 can be configured to distinguish data types by adding types that are not defined as attribute types among the 3D GS data, or to distinguish data types through a custom attribute SEI message.

[0422] Example 2-1 proposes a configuration that extends the attribute type. When GS data is encoded into the AVD type, the VPS can distinguish the data type by signaling the attribute type syntax ai_attribute_type_id through the attribute information syntax. The corresponding syntax ai_attribute_type_id may be described in section 8.3.4.5 "Attribute information syntax" of the V3C codec specification document ISO / IEC 23090-5(4E):2025 (ISO / IEC 23090-5(4E):2025, Table of Contents: 8.3.4.5). In addition, for data that is not defined as an attribute type, the system can be configured to add new attribute types for GS data as shown in Table 3 (V3C attribute types), for example, scale, rotation, SH DC, SH AC, etc., as new types. In the examples of Table 3, new attribute types may be added or some may be deleted.

[0423] The ai_attribute_type_id signaled through the VPS can be used as information representing the ID of an attribute type, and the mapping information between each ID and attribute type can be configured to be implicitly determined by an agreement between the encoder and the decoder. Additionally, according to the embodiments, opacity data can be determined by an existing transparency attribute type (ATTR_TRANSPARENCY) and encapsulated into an AVD, and position information can be encapsulated into a GVD. Spherical Harmonics (SH) may be represented as a single attribute type according to the embodiments, or may be represented as two attribute types by distinguishing SH AC and SH DC.

[0424] ai_attribute_type_id[ j ][ i ]IdentifierAttribute type0ATTR_TEXTURETexture1ATTR_MATERIAL_IDMaterial ID2ATTR_TRANSPARENCYTransparency3ATTR_REFLECTANCEReflectance4ATTR_NORMALNormals5ATTR_SCALESscale6ATTR_ROTATIONRotation7ATTR_SH_DCSpherical Harmonics DC8ATTR_SH ACSpherical Harmonics AC9..14ATTR_RESERVEDReserved15ATTR_CUSTOMCustom

[0425] Table 3 illustrates an example of an attribute type identifier (attribute type id) signaled via a VPS (V3C Parameter Set) in V3C. Specifically, an identifier and an attribute type can be mapped according to an AI attribute type ID (ai_attribute_type_id), for example, 0 can be defined as ATTR_TEXTURE (Texture), 1 as ATTR_MATERIAL_ID (Material-ID), 2 as ATTR_TRANSPARENCY (Transparency), 3 as ATTR_REFLECTANCE (Reflectance), and 4 as ATTR_NORMAL (Normals). In addition, Table 3 includes examples of extending attribute types for processing 3D Gaussian splatting (3D GS) data in the embodiments, for example, 5 can be defined as ATTR_SCALE (Scale), 6 as ATTR_ROTATION (Rotation), 7 as ATTR_SH_DC (Spherical Harmonics DC), and 8 as ATTR_SH_AC (Spherical Harmonics AC). Furthermore, 5, 9 to 14 can be defined as ATTR_RESERVED (Reserved) and 15 as ATTR_CUSTOM (Custom), and such AI attribute type ID (ai_attribute_type_id) related signaling may be described in section 8.3.4.5 "Attribute information syntax" of the V3C codec specification document ISO / IEC 23090-5(4E):2025.

[0426] In Example 2-2, a configuration is proposed for distinguishing 3D Gaussian Splatting (3D GS) data types using custom attributes. For example, when encoding GS data into AVD type, the attribute type can be specified as a custom attribute type for encoding. According to the embodiments, for data where no attribute type is specified (e.g., scale, rotation, SH DC, SH AC, etc.), a custom attribute type can be specified, and the attribute count, attribute dimension, etc. can be signaled through a Custom attribute information SEI message among V3C SEI messages. The syntax for custom attribute information SEI messages may be described in section F.2.15 "Custom attribute information SEI payload syntax" of the V3C codec specification document ISO / IEC 23090-5(4E):2025 (ISO / IEC 23090-5(4E):2025, Table of Contents: F.2.15).

[0427] In addition, the embodiments may be configured to determine index information mapped to each 3D GS data type by an agreement between the encoder and the decoder and to signal it. The mapping information between the signaled index information and the 3D GS data type may be derived by an agreement between the encoder and the decoder, or the mapping information itself may be configured to be signaled. For example, Table 4 (an example of a 3D GS type mapped to an attribute index within a Custom attribute information SEI message) is an example of a 3D GS type mapped to an attribute index within a Custom attribute information SEI message, wherein the attribute index may be determined as 0 for the Scale type, 1 for the Rotation type, 2 for the SH DC type, and 3 for the SH AC type. Additionally, attribute index information can be included in and signaled in custom attribute information SEI messages.

[0428] Attribute index3D GS type0Scale1Rotation2SH DC3SH AC

[0429] In the embodiments, each compressed GS data may be multiplexed by a multiplexer to form a V-GSC bitstream, and the formed V-GSC bitstream may be stored in a storage medium or transmitted over a network.

[0430] FIG. 25 shows a V-GSC decoder according to embodiments.

[0431] Fig. 25 can perform the reverse process of the V-GSC encoder of Fig. 23.

[0432] FIG. 25 corresponds to a decoding method / device (FIG. 1 receiving device (10002), FIG. 16 decoder, FIG. 17 decoder, FIG. 19 receiving device, FIG. 20 each device, FIG. 25 to FIG. 26 decoder, FIG. 27 decoder (renderer), FIG. 29 method, etc.).

[0433] FIG. 25 illustrates a block diagram of the decoding process of a V-GSC system according to embodiments. In the embodiments, the V-GSC bitstream can be separated into individual element bitstreams through a demultiplexer after being received, and the separated bitstreams can be reconstructed through a post-decoding process after undergoing video decoding. The reconstructed GS data can be stored in a GS data file format and, depending on the embodiment, may consist of position data, opacity, scale, rotation, SH DC / AC coefficients, etc., and some components may be omitted or added. Additionally, the components of the 3D Gaussian stored in the GS data file format may be stored in a listed form corresponding to each component constituting the 3D Gaussian. Meanwhile, in the embodiments, the data type (e.g., position, opacity, scale, rotation, SH DC, SH AC, etc.) of the V3C unit, in which 3D Gaussian splatting (3D GS) data is encapsulated, can be derived.

[0434] In Example 1, 3D GS data types can be distinguished by utilizing map index information. For example, a V3C unit encapsulated as a GVD type can distinguish geometry types through a map index, and geometry type information mapped to the map index can be determined by the same agreement between the encoder and decoder, or derived by parsing the corresponding mapping information. Accordingly, the map index can be parsed from the unit header of each V3C unit using the mapping information between the map index and the geometry type, and geometry information mapped to the parsed map index can be derived. For example, if mapping information between a map index and a geometry type is defined as in Table 1, if the parsed map index is 0, it can be inferred that the geometry type is position; if the parsed map index is 1, it can be inferred that it is scale; and if the parsed map index is 2, it can be inferred that it is rotation. Additionally, a V3C unit encapsulated in an AVD type can distinguish attribute types through the map index, and attribute type information mapped to the map index can be determined by the same convention or derived by parsing the mapping information. For example, if mapping information between a map index and an attribute type is defined as in Table 2, the map index of the V3C unit header can be parsed so that if the parsed map index is 0, it can be inferred that the attribute type is SH DC, and if the parsed map index is 1, it can be inferred that the attribute type is SH AC.

[0435] In Example 2, 3D GS data can be encapsulated into an AVD type to distinguish the 3D GS data type, and the V3C unit encapsulated into the AVD type can distinguish the geometry or / and the attribute type through an attribute type or a custom attribute SEI message.

[0436] In Example 2-1, when GS data is encoded as an AVD type, the data type can be distinguished by parsing the attribute type syntax through the attribute information syntax of the VPS, and as shown in Table 3, data that is not defined as an attribute type in V3C, such as scale, rotation, SH DC, and SH AC, can be added as a new attribute type and expanded, and mapping information between the attribute type and the attribute index can be derived by an agreement between the encoder and the decoder.

[0437] In Example 2-2, when GS data is encapsulated as an AVD type and the attribute type is a custom attribute, the attribute count, attribute dimension, attribute index, etc. can be obtained by parsing the custom attribute SEI message, and the mapping information between the parsed index and each 3D GS data type can be derived by a convention or determined by parsing the mapping information. For example, when using a mapping as shown in Table 4, if the attribute index parsed from the SEI message is 0, it can be derived that it is a scale type; if it is 1, it can be derived that it is a rotation type; if it is 2, it can be derived that it is an SH DC type; and if it is 3, it can be derived that it is an SH AC type.

[0438] In addition, in the embodiments, an Atlas decoder may be utilized for the AD of the V3C. Position data encoded using GVD or AVD can be reconstructed through nominal format conversion after passing through a video decoder (e.g., HEVC or VVC, etc.). Opacity encoded using OVD or AVD can be reconstructed through nominal format conversion after passing through a video decoder, and if arithmetic coding is applied, it can be converted through an arithmetic decoder. Scale data and rotation data encoded using GVD or AVD can also be reconstructed through nominal format conversion after passing through a video decoder. Additionally, SH DC data and SH AC data encoded using AVD can be reconstructed through nominal format conversion after passing through a video decoder. Meanwhile, compared to the current V3C which uses three video decoders, the video-based GS compression method proposed in the embodiments allows for a configuration in which six video decoders are used.

[0439] FIG. 25 illustrates an overview of a V-GSC decoder according to embodiments. In FIG. 25, the V-GSC bitstream can be separated into multiple NALU streams by a demultiplexer, and the NALUs can be divided into an atlas NALU and a video NALU according to the V3C unit type and provided to an atlas decoder and multiple video decoders (Video decoders #1 to #6), respectively. For example, an atlas NALU encapsulated in V3C_AD can be transmitted to an atlas decoder to decode atlas-related information, and a video NALU can be encapsulated in V3C_GVD, V3C_AVD, or V3C_OVD and transmitted to each video decoder.

[0440] In the example of FIG. 25, the video decoder (Video decoder #1) can output the result of decoding the GS center position from V3C_GVD or V3C_AVD, and the video decoder (Video decoder #2) can output the result of decoding the GS opacity from V3C_OVD or V3C_AVD. Additionally, the video decoder (Video decoder #3) can output the result of decoding the GS scale, the video decoder (Video decoder #4) can output the result of decoding the GS rotation, the video decoder (Video decoder #5) can output the result of decoding the GS SH DC coefficient (color), and the video decoder (Video decoder #6) can output the result of decoding the GS SH AC coefficient (AC coeff). A post-decoding function can generate reconstructed four-dimensional Gaussian splatting (4DGS) data by receiving these decoding results and performing combination, transformation, and reconstruction processes, and the generated 4DGS data can be provided to a graphic engine and used for rendering or subsequent processing.

[0441] FIG. 26 shows a post-decoder according to embodiments.

[0442] As shown in FIG. 26, the decoding method / device (FIG. 1 receiving device (10002), FIG. 16 decoder, FIG. 17 decoder, FIG. 19 receiving device, FIG. 20 each device, FIG. 25 to FIG. 26 decoder, FIG. 27 decoder (renderer), FIG. 29 method, etc.) can perform a post-decoding process.

[0443] FIG. 26 may illustrate an example of a post-decoding process for restoring 4DGS data from a result decoded by a video decoder in a V-GSC decoding process according to embodiments. For example, the input to the restoration process may be a 2D video (or 2D frame sequence) decoded through a video decoder (not shown), and the output to the post-decoding process may be restored 4DGS data (e.g., 4D Gaussian Splatting data).

[0444] The post-decoding process according to the embodiments may be configured to include a frame inverse packing unit (2D frame unpacking), an inverse GSTF, a rotation conversion unit, a composer, etc., to recover 4DGS data from a decoded video. In this case, depending on the characteristics of the GS data (e.g., presence of specific parameters, encoding method, compression configuration, etc.), some steps of the post-decoding process may be omitted, or additional steps (e.g., filtering, normalization, post-processing correction, etc.) may be included.

[0445] Additionally, according to the embodiments, the post-decoding process may be performed independently on each component (or parameter) of the 4DGS data, or it may be performed in combination (e.g., by considering interdependencies) on all or some of the 4DGS components. For example, some components may be restored and transformed together by considering the combined relationship between position / scale / rotation or the combined relationship between color coefficients (SH coefficients).

[0446] According to an embodiment, GS data decoded through a 2D video codec can undergo a 2D frame unpacking process. For example, unpacking of decoded GS components can be performed by defining the i-th GS parameter provided as input to the renderer in the decoding result as decoded_GS(i), and mapping the decoded_GS(i) to a parameter in the Gaussian splat unit on the image domain (x, y).

[0447] Specifically, according to the embodiments, a plurality of GS parameters included in the same Gaussian splat may be configured to correspond to the same image domain coordinates (x, y). For example, position(x, y) may represent position data decoded at (x, y), opacity(x, y) may represent opacity data decoded at (x, y), scale(x, y) may represent scale data decoded at (x, y), rotation(x, y) may represent rotation data decoded at (x, y), SH DC coeff(x, y) may represent color (SH DC coefficient) data decoded at (x, y), and SH AC coeff(x, y) may represent color (SH AC coefficient) data decoded at (x, y).

[0448] In addition, according to the embodiments, multiple GS parameters such as position(x, y), opacity(x, y), scale(x, y), rotation(x, y), SH DC coeff(x, y), and SH AC coeff(x, y) can be arranged according to the same packing rule for the image domain (x, y). For example, if each parameter is packed according to the same rule where the index is defined as x + frame_width·y, as in the example of the following formula, the parameters corresponding to the same (x, y) are aligned to form the same Gaussian splat, so unpacking and subsequent restoration processing can be performed without additional work to synchronize between the parameters.

[0449] decoded_GS(i) = {position (x + frame_width * y), opacity (x + frame_width * y), scale (x + frame_width * y), rotation (x + frame_width * y), SH DC (x + frame_width * y) SH AC (x + frame_width *y)}

[0450] If necessary, in the post-decoding process according to the embodiments, additional parameters or data (e.g., parameter identifier, component index / offset information, timestamp, block / frame unit mapping information, packing rule identification information, etc.) may be used to synchronize in order to ensure alignment between each GS parameter. For example, if different packing rules, resolutions, sampling structures, or encoding configurations are applied for each component, a sync parameter may be referenced to accurately align parameters corresponding to the same Gaussian splat.

[0451] Additionally, the post-decoding process according to the embodiments may include a rotation conversion unit, and the rotation conversion unit may be a process for converting the representation method of a rotation parameter. For example, for rotation data to which unpacking and inverse GSTF processes have been performed, the rotation conversion unit may convert the rotation data from a first rotation representation format to a second rotation representation format.

[0452] Specifically, when rotation data is expressed in the form of Euler angles, the rotation transformer according to the embodiments may convert the rotation data in the form of Euler angles into a quaternion format. For example, the input of the rotation transformer may be rotation data of GS expressed in Euler angles, and the input rotation data may be a 3D vector (e.g., roll, pitch, yaw, or It can be composed of ). In addition, the output of the rotation transformation unit may be rotation data of GS expressed as a quaternion, and the output rotation data may be composed of a 4-dimensional vector (e.g., qx, qy, qz, qw or (w, x, y, z)).

[0453] According to the embodiments, the process of converting rotation data from Euler angle form to quaternion form can be performed as follows. For example, input Euler angle ( For ), the rotation transformation unit can apply a trigonometric function-based transformation to calculate a quaternion component and output the calculated quaternion component as a rotation parameter of GS.

[0454]

[0455]

[0456]

[0457]

[0458] According to the embodiments, q_w, q_x, q_y, and q_z may represent rotation parameters expressed in quaternion form, and can refer to a rotation parameter expressed in Euler angle form. For example, the rotation transform part is a rotation parameter in Euler angle form ( It can be configured to receive ) as input and output rotation parameters (q_w, q_x, q_y, q_z) in quaternion format.

[0459] Additionally, the post-decoding process according to the embodiments may include a composer unit, and the composer unit may be a process of merging decoded 4DGS parameters. For example, the composer unit may perform a process of aligning a plurality of components corresponding to the decoded Gaussian splatting. According to the embodiment, the composer unit may align the elements constituting each 3D Gaussian (e.g., position, opacity, scale, rotation, SH DC coefficients, SH AC coefficients, etc.) according to a predetermined order of elements.

[0460] In addition, according to the embodiment, the composer unit can align 3D Gaussian elements packed at co-located pixel locations within the frame so that they correspond to each other by considering packing information. Through this, parameters constituting the same Gaussian splat can be accurately matched and combined into a 4DGS data structure.

[0461] Additionally, the post-decoding process according to the embodiments may include an inverse GSTF section, and the inverse GSTF section may be an inverse mapping process for the recovered GS parameters. For example, if the GS parameters recovered by decoding from the bitstream are represented as integers (or quantized values), the inverse GSTF section may convert the integer data into real numbers (or non-quantized values) for rendering. The inverse GSTF section may perform a process of inversely converting the recovered GS parameter values ​​to a numerical range and precision suitable for renderer input.

[0462] As an example, the inverse GSTF part may perform the inverse mapping using a linear function as shown in the following formula. For example, the inverse GSTF part may apply scaling and an offset to the restored integer value to calculate a real value, and output the calculated real value as the restored value of each GS parameter.

[0463]

[0464] According to the embodiments, in the above formula and can mean the GS parameter restored as an integer and the GS parameter restored as a floating-point number, respectively. For example, the GS parameter decoded from the bitstream is As shown, it can be restored in the integer domain (or quantization domain), and the inverse GSTF part Receives as input a real domain suitable for rendering It can be inversely mapped to.

[0465] Additionally, according to the embodiments, a_max, a_min, and bd_a may represent the maximum value, minimum value, and bit depth of a expressed as an integer, respectively, of the GS parameter a. For example, a_max, a_min, and bd_a may be included in a bitstream and transmitted from an encoder, and a decoder may parse a_max, a_min, and bd_a and use them for an inverse mapping operation of the inverse GSTF part.

[0466] Additionally, according to the embodiments, the inverse GSTF portion is a real-valued restoration parameter calculated as an inverse mapping result. To prevent it from going outside the predefined valid range, clipping using a_max and a_min can be performed. For example, the inverse GSTF part If it exceeds a_max, limit it to a_max, and By limiting to a_min when it is less than a_min, numerical stability in the rendering process and the validity of GS parameters can be guaranteed.

[0467] FIG. 26 illustrates an example of a post-decoding function. In FIG. 26, the decoded Gaussian distribution center position, decoded opacity, decoded Gaussian scale, decoded rotation parameter, decoded SH DC coefficients, and decoded SH AC coefficients can each perform 2D frame unpacking and inverse GSTF processes.

[0468] In FIG. 26, the decoded rotation parameter can further perform a rotation conversion process after 2D frame unpacking and Inverse GSTF. The output of each path can be input into a Composer and merged, and a Reconstructed 4DGS can be generated as the output of the Composer.

[0469] FIG. 27 shows a graphics engine according to embodiments.

[0470] FIG. 27 shows a graphics engine for rendering operation of a decoding method / device (FIG. 1 receiving device (10002), FIG. 16 decoder, FIG. 17 decoder, FIG. 19 receiving device, FIG. 20 each device, FIG. 25 to 26 decoder, FIG. 27 decoder (renderer), FIG. 29 method, etc.).

[0471] FIG. 27 may illustrate an example of a graphic engine for rendering 4DGS.

[0472] According to the embodiments, the image plane tiling process can generate an image plane using camera parameters for rendering. At this time, for parallel processing of the graphics device, the image plane may be divided into one or more regions, and each divided region may be referred to as a tile.

[0473] According to the embodiments, in the Gaussian splats culling process, among a number of Gaussian splats (GS), only the Gaussian splats to be included in a specific divided tile can be kept, and the remaining Gaussian splats can be removed.

[0474] According to the embodiments, the process of projecting 3D Gaussian splats to 2D can project the Gaussian splats contained in the tile onto a 2D image plane.

[0475] According to the embodiments, the process of calculating the 2D covariance matrix calculates the 3D covariance matrix using the restored scale and rotation parameters, and can approximate the 3D variance matrix to the 2D covariance matrix using camera parameters.

[0476] According to the embodiments, the Spherical Harmonics (SH) Convert SH to Color process can convert the restored SH DC and SH AC values ​​into a color (e.g., RGB or YUV). In this case, the center position value of a two-dimensional Gaussian splat and camera parameters may be used to convert SH to a color.

[0477] According to the embodiments, the alpha blending process can calculate pixel values ​​of an image to be rendered using a two-dimensionally projected Gaussian splat, a two-dimensional covariance matrix, and a transformed color. At this time, multiple Gaussian splats may exist at specific locations among the two-dimensionally projected Gaussian splats, and in such cases, before calculating pixel values, the distance can be calculated using camera parameters and the center position of the Gaussian splats, and the Gaussian splats can be aligned based on the distance. Subsequently, color information of the pixel values ​​can be calculated by multiplying the color and opacity in order of shortest distance from the camera and accumulating them. Additionally, since the total sum of opacity cannot exceed 1, accumulation can be continued only when the accumulated sum of opacity is less than 1.

[0478] According to the embodiments, the rendered image can finally be scaled by a graphics device and provided to a user through a display device.

[0479] FIG. 28 illustrates a encoding method according to embodiments.

[0480] The encoding method / device (transmitting device (10000) in FIG. 1, encoder in FIG. 4, encoder in FIG. 15, transmitting device in FIG. 18, each device in FIG. 20, encoders in FIG. 23 to 24, method in FIG. 28) can perform the method as in FIG. 28.

[0481] The method according to the embodiments may include the step of encoding Gaussian splat data (S2800); and / or the step of generating a bitstream containing Gaussian splat data (S2810); etc.

[0482] With respect to the encoder and GS configuration of FIG. 23 and FIG. 21, the step of encoding Gaussian splat data (S2800) may include: the step of encoding atlas data in a bitstream; the step of encoding position information of Gaussian splat data in a bitstream; the step of encoding opacity of Gaussian splat data; the step of encoding scale information of Gaussian splat data; the step of encoding rotation information of Gaussian splat data; and the step of encoding SH coefficients of Gaussian splat data.

[0483] The bitstream includes a V3C unit, and the V3C unit includes at least one of a GVD or an AVD. Geometry data including position information, scale information, and rotation information is included in the GVD within the V3C unit, and the type of geometry data is identified based on a map index, and the first value of the map index represents position information, the second value of the map index represents scale information, and the third value of the map index represents rotation information.

[0484] The bitstream includes a V3C unit, the V3C unit includes at least one of GVD or AVD, and Gaussian splat data including scale information, rotation information and SH coefficients is identified by a VPS within the bitstream, the VPS includes attribute information syntax, the attribute information syntax includes an attribute type ID, the first value of the attribute type ID represents scale information, the second value of the attribute type ID represents rotation information, and the third value of the attribute type ID may represent SH coefficients.

[0485] When Gaussian splat data is encoded as Attribute Video Data (AVD) type, the attribute type ID is indicated as a custom attribute type, and the attribute count and attribute dimension for the data indicated as the custom attribute type are indicated based on the Custom attribute information SEI message among the V3C SEI messages in the bitstream.

[0486] The method of FIG. 28 can be performed by a device. A device according to embodiments includes a memory; and at least one processor connected to the memory; and the at least one processor may be configured to: encode Gaussian splat data; and generate a bitstream containing Gaussian splat data.

[0487] The embodiments further include a computer-readable storage medium for storing a bitstream generated by the method according to FIG. 28.

[0488] The embodiments further include a method comprising the steps of: obtaining a bitstream for Gaussian splat data; encoding the bitstream for Gaussian splat data; generating a bitstream containing the Gaussian splat data; and transmitting data containing the bitstream.

[0489] FIG. 29 illustrates a decoding method according to embodiments.

[0490] The decoding method / device (receiving device (10002) of FIG. 1, decoder of FIG. 16, decoder of FIG. 17, receiving device of FIG. 19, each device of FIG. 20, decoders of FIG. 25 to 26, decoder (renderer) of FIG. 27, method of FIG. 29, etc.) can perform a method such as that of FIG. 29.

[0491] The method in Fig. 29 corresponds to the method in Fig. 28 as the inverse process.

[0492] The method according to the embodiments may include the step of acquiring a bitstream containing Gaussian splat data (S2900); and / or the step of decoding the Gaussian splat data (S2910); etc.

[0493] With respect to FIG. 25, FIG. 21, decoder and GS configuration, etc., the step of decoding Gaussian splat data (S2910) may include: a step of decoding atlas data in a bitstream; a step of decoding position information of Gaussian splat data in a bitstream; a step of decoding opacity of Gaussian splat data; a step of decoding scale information of Gaussian splat data; a step of decoding rotation information of Gaussian splat data; and a step of decoding SH coefficients of Gaussian splat data.

[0494] In Example 1, regarding a method for distinguishing 3D GS data types using map index information, the bitstream includes a V3C unit, and the V3C unit includes at least one of a GVD or an AVD. Geometry data including position information, scale information, and rotation information is included in the GVD within the V3C unit, and the type of geometry data is identified based on a map index. A first value of the map index represents position information, a second value of the map index represents scale information, and a third value of the map index represents rotation information.

[0495] In relation to Example 2, a method for distinguishing 3D GS data types by encapsulating 3D GS data into AVD type and Example 2-1) an example of extending attribute types, the bitstream includes a V3C unit, the V3C unit includes at least one of GVD or AVD, and Gaussian splat data including scale information, rotation information and SH coefficients is identified by a VPS within the bitstream, the VPS includes attribute information syntax, the attribute information syntax includes an attribute type ID, the first value of the attribute type ID represents scale information, the second value of the attribute type ID represents rotation information, and the third value of the attribute type ID represents SH coefficients.

[0496] In relation to Example 2-2, when Gaussian splat data is encoded as Attribute Video Data (AVD) type, the attribute type ID is indicated as a custom attribute type, and the attribute count and attribute dimension for the data indicated as the custom attribute type can be indicated based on the Custom attribute information SEI message among the V3C SEI messages in the bitstream.

[0497] In relation to Example 2-2, the first value of the attribute index of the custom attribute information SEI message may represent scale information of the Gaussian splat data, the second value of the attribute index may represent rotation information of the Gaussian splat data, and the third value of the attribute index may represent SH coefficient information of the Gaussian splat data.

[0498] The method of FIG. 29 can be performed by a device. A device according to embodiments includes a memory; and at least one processor connected to the memory; and the at least one processor may be configured to: acquire a bitstream containing Gaussian splat data; and decode the Gaussian splat data.

[0499] The embodiments support 3DGS data compression using the V3C structure by supporting data types of 3D Gaussian splat (3D GS) that are not considered in V3C based on the V3C bitstream structure. The proposed method has the advantage of being able to support various data types of 3D GS without additional bits, as it can distinguish various data types of 3D GS by utilizing the syntax signaled in the existing V3C bitstream without adding new syntax.

[0500] In addition, the embodiments can provide a method for effectively compressing 3DGS data, which is currently being actively researched, using the existing V-PCC (V3C) method. For example, through the video-based Gaussian Splat compression method (V-GSC), 3DGS technology can be efficiently compressed using V3C, which is a stereoscopic data compression technology based on existing video codecs. Furthermore, by utilizing the existing video codec infrastructure as is, the embodiments can reduce the cost associated with introducing new Gaussian Splat technology and shorten the technology development time, thereby enabling the rapid commercial provision of Gaussian Splat services.

[0501] Operations according to the embodiments may be described in combination with the point cloud data transmission and reception device and method according to the embodiments described below. Operations according to the embodiments described herein may be performed by a transmission and reception device including a memory and / or a processor according to the embodiments. The memory may store a program for processing and / or controlling operations according to the embodiments, and the processor may control various operations described herein and may be referred to as a controller, etc. Additionally, operations according to the embodiments may be performed by firmware, software and / or a combination thereof, and the firmware, software and / or a combination thereof may be stored in the processor or memory.

[0502] The embodiments have been described in terms of methods and / or devices, and the description of the methods and the description of the devices may be applied complementarily.

[0503] Although the drawings have been described separately for the convenience of explanation, it is also possible to design a new embodiment by combining the embodiments described in each drawing. Furthermore, designing a computer-readable recording medium containing a program for executing the previously described embodiments, as required by a person skilled in the art, falls within the scope of the embodiments. The apparatus and method according to the embodiments are not limited to the configuration and method of the embodiments described above; rather, the embodiments may be configured by selectively combining all or part of each embodiment to allow for various modifications. Although preferred embodiments have been illustrated and described, the embodiments are not limited to the specific embodiments described above. It is not only possible for a person skilled in the art to make various modifications without departing from the essence of the embodiments claimed in the claims, but such modifications should not be understood individually from the technical concept or perspective of the embodiments.

[0504] Various components of the device of the embodiments may be implemented by hardware, software, firmware, or a combination thereof. Various components of the embodiments may be implemented as a single chip, for example, a single hardware circuit. Depending on the embodiments, the components according to the embodiments may each be implemented as separate chips. Depending on the embodiments, at least one of the components of the device according to the embodiments may be composed of one or more processors capable of executing one or more programs, and one or more programs may include instructions for performing or executing any one or more of the operations / methods according to the embodiments. Executable instructions for performing the methods / operations of the device according to the embodiments may be stored in non-transient CRMs or other computer program products configured to be executed by one or more processors, or may be stored in transient CRMs or other computer program products configured to be executed by one or more processors. Additionally, memory according to the embodiments may be used as a concept that includes not only volatile memory (e.g., RAM, etc.) but also non-volatile memory, flash memory, PROM, etc. In addition, it may also include implementation in the form of carrier waves, such as transmission over the Internet. Furthermore, processor-readable recording media are distributed across networked computer systems, allowing processor-readable code to be stored and executed in a distributed manner.

[0505] In this document, “ / ” and “,” are interpreted as “and / or.” For example, “A / B” is interpreted as “A and / or B,” and “A, B” is interpreted as “A and / or B.” Additionally, “A / B / C” means “at least one of A, B and / or C.” Also, “A, B, C” means “at least one of A, B and / or C.” Additionally, in this document, “or” is interpreted as “and / or.” For example, “A or B” may mean 1) “A” alone, 2) “B” alone, or 3) “A and B.” In other words, “or” in this document may mean “additionally or alternatively.”

[0506] Terms such as "first," "second," etc., may be used to describe various components of the embodiments. However, the interpretation of the various components according to the embodiments should not be limited by these terms. These terms are merely used to distinguish one component from another. For example, the first user input signal may be referred to as the second user input signal. Similarly, the second user input signal may be referred to as the first user input signal. The use of these terms should be interpreted as not departing from the scope of the various embodiments. Although the first user input signal and the second user input signal are both user input signals, they do not imply the same user input signals unless clearly indicated in the context.

[0507] The terms used to describe the embodiments are intended for the purpose of describing specific embodiments and are not intended to limit the embodiments. As used in the description of the embodiments and in the claims, the singular is intended to include the plural unless explicitly indicated in the context. Expressions of and / or are used to mean including all possible combinations between the terms. Expressions of include describe the presence of features, numbers, steps, elements, and / or components and do not imply the exclusion of additional features, numbers, steps, elements, and / or components. Conditional expressions such as "if" or "when" used to describe the embodiments are not limited to being optional. It is intended to be interpreted as "when a specific condition is satisfied," "when a related action is performed in response to a specific condition," or "when a related definition is interpreted."

[0508] Additionally, operations according to the embodiments described herein may be performed by a transmitting and receiving device including memory and / or a processor, depending on the embodiments. The memory may store programs for processing / controlling operations according to the embodiments, and the processor may control various operations described in this document. The processor may be referred to as a controller, etc. Operations in the embodiments may be performed by firmware, software, and / or a combination thereof, and the firmware, software, and / or a combination thereof may be stored in the processor or in memory.

[0509] As described above, the relevant details have been explained in the best mode for carrying out the embodiments.

[0510] As described above, the embodiments may be applied wholly or partially to point cloud data transmission and reception devices and systems.

[0511] Those skilled in the art may make various changes or modifications to the embodiments within the scope of the embodiments.

[0512] The embodiments may include modifications / variations, and such modifications / variations do not exceed the scope of the claims and their equivalents.

Claims

A step of acquiring a bitstream containing Gaussian splat data; and The step of decoding the above Gaussian splat data; comprising, method. In paragraph 1, The step of decoding the above Gaussian splat data is: A step of decoding atlas data within the bitstream above; A step of decoding position information of the Gaussian splat data within the bitstream; A step of decoding the opacity of the above Gaussian splat data; A step of decoding the scale information of the above Gaussian splat data; A step of decoding rotation information of the above Gaussian splat data; and The step of decoding the SH coefficients of the above Gaussian splat data; comprising method. In paragraph 2, The above bitstream includes a V3C unit, and the V3C unit includes at least one of a GVD or an AVD, and Geometry data including the above position information, the above scale information and the above rotation information is included in the GVD within the V3C unit, and The type of the above geometry data is identified based on the map index, and The first value of the above map index represents the above location information, and The second value of the above map index represents the above scale information, and The third value of the above map index represents the rotation information, method. In paragraph 2, The above bitstream includes a V3C unit, and the V3C unit includes at least one of a GVD or an AVD, and Gaussian splat data including the scale information, the rotation information, and the SH coefficient is identified by the VPS within the bitstream, and The above VPS includes attribute information syntax, and The above attribute information syntax includes an attribute type ID, and The first value of the above attribute type ID represents the above scale information, and The second value of the above attribute type ID represents the rotation information, and The third value of the above attribute type ID represents the above SH coefficient, method. In paragraph 4, When the above Gaussian splat data is encoded as an Attribute Video Data (AVD) type, the attribute type ID is indicated as a custom attribute type, and The attribute count and attribute dimension for the data indicated by the above custom attribute type are indicated based on the Custom attribute information SEI message among the V3C SEI messages within the bitstream, method. In paragraph 5, of the above custom attribute information SEI message The first value of the attribute index represents the scale information of the Gaussian splat data, and The second value of the above attribute index represents rotation information of the Gaussian splat data, and The third value of the above attribute index represents the SH coefficient information of the above Gaussian splat data, method. Memory; and At least one processor connected to the memory; comprising, wherein the at least one processor: Acquire a bitstream containing Gaussian splat data; and Configured to decode the above Gaussian splat data, device. Step of encoding Gaussian splat data; and A step of generating a bitstream including the above Gaussian splat data; comprising, method. In paragraph 8, The step of encoding the above Gaussian splat data is: A step of encoding atlas data within the above bitstream; A step of encoding position information of the Gaussian splat data within the bitstream; A step of encoding the opacity of the above Gaussian splat data; A step of encoding scale information of the above Gaussian splat data; A step of encoding rotation information of the above Gaussian splat data; and The step of encoding the SH coefficients of the above Gaussian splat data; comprising method. In Paragraph 9, The above bitstream includes a V3C unit, and the V3C unit includes at least one of a GVD or an AVD, and Geometry data including the above position information, the above scale information and the above rotation information is included in the GVD within the V3C unit, and The type of the above geometry data is identified based on the map index, and The first value of the above map index represents the above location information, and The second value of the above map index represents the above scale information, and The third value of the above map index represents the rotation information, method. In Paragraph 9, The above bitstream includes a V3C unit, and the V3C unit includes at least one of a GVD or an AVD, and Gaussian splat data including the scale information, the rotation information, and the SH coefficient is identified by the VPS within the bitstream, and The above VPS includes attribute information syntax, and The above attribute information syntax includes an attribute type ID, and The first value of the above attribute type ID represents the above scale information, and The second value of the above attribute type ID represents the rotation information, and The third value of the above attribute type ID represents the above SH coefficient, method. In Paragraph 11, When the above Gaussian splat data is encoded as an Attribute Video Data (AVD) type, the attribute type ID is indicated as a custom attribute type, and The attribute count and attribute dimension for the data indicated by the above custom attribute type are indicated based on the Custom attribute information SEI message among the V3C SEI messages within the bitstream, method. Memory; and At least one processor connected to the memory; comprising, wherein the at least one processor: Encoding Gaussian splat data; and Configured to generate a bitstream including the above Gaussian splat data, device. A computer-readable storage medium for storing a bitstream generated by the method according to paragraph 8. Step of acquiring a bitstream for Gaussian splat data, The bitstream is generated based on the steps of: encoding Gaussian splat data; and generating a bitstream containing the Gaussian splat data; and A method comprising the step of transmitting data including the bitstream above.

Images