Encoding method, decoding method, encoding device, and decoding device

Abstract

An encoding method according to the present invention includes: generating encoded data by hierarchically encoding a plurality of first components included in Gaussian data (S301); and generating a bit stream that includes the encoded data and hierarchy information related to a hierarchical structure in the hierarchization (S302). For example, during the hierarchization, the plurality of first components is assigned to a base hierarchy that can be used alone or an extended hierarchy that can be used together with the base hierarchy, and the hierarchy information may indicate whether each of the plurality of first components belongs to the base hierarchy or the extended hierarchy.

Description

Encoding method, decoding method, encoding device, and decoding device

[0001] The present disclosure relates to an encoding method, a decoding method, an encoding device, and a decoding device.

[0002] The present disclosure relates to a three-dimensional data encoding method, a three-dimensional data decoding method, a three-dimensional data encoding device, and a three-dimensional data decoding device.

[0003] In a wide range of fields such as computer vision for autonomous operation of automobiles or robots, map information, surveillance, infrastructure inspection, and video distribution, the spread of devices or services utilizing three-dimensional data is expected in the future. Three-dimensional data is acquired by various methods such as a distance sensor like a range finder, a stereo camera, or a combination of a plurality of monocular cameras.

[0004] As one of the methods for expressing three-dimensional data, there is an expression method called a point cloud that represents the shape of a three-dimensional structure using a point group in a three-dimensional space. In the point cloud, the positions and colors of the point group are stored. The point cloud is expected to become a mainstream method for expressing three-dimensional data, but the data volume of the point group is extremely large. Therefore, in the storage or transmission of three-dimensional data, similar to two-dimensional moving images (examples include MPEG-4 AVC (Advanced Video Coding) or HEVC (High Efficiency Video Coding) standardized by MPEG), compression of the data volume by encoding is essential.

[0005] In addition, point cloud compression is partially supported by public libraries that perform point cloud-related processing, such as the Point Cloud Library.

[0006] Further, a technique is known that uses three-dimensional map data to search for and display facilities located around a vehicle (see, for example, Patent Document 1).

[0007] International Publication No. 2014 / 020663

[0008] In such encoding and decoding of three-dimensional data, reduction of processing amount or speeding up of processing is desired.

[0009] This disclosure aims to provide an encoding method, a decoding method, an encoding device, or a decoding device that can reduce the amount of processing or speed up processing.

[0010] An encoding method according to one aspect of the present disclosure generates encoded data by encoding a plurality of first components contained in Gaussian data in a hierarchical manner, and generates a bitstream including hierarchical information relating to the hierarchical structure in the hierarchical encoding and the encoded data.

[0011] A decoding method according to one aspect of the present disclosure obtains encoded data generated by encoding a plurality of first components contained in Gaussian data in a hierarchical manner from a bitstream, and hierarchical information relating to the hierarchical structure in the hierarchical encoding, and decodes the encoded data using the hierarchical information.

[0012] This disclosure provides an encoding method, a decoding method, an encoding device, or a decoding device that can reduce the amount of processing or speed up processing.

[0013] Figure 1 is a diagram showing an example configuration of a three-dimensional data encoding and decoding system according to an embodiment. Figure 2 is a diagram showing an example configuration of point cloud data according to an embodiment. Figure 3 is a diagram showing an example configuration of a data file describing point cloud data information according to an embodiment. Figure 4 is a diagram showing an example configuration of mesh data according to an embodiment. Figure 5 is a diagram showing an example configuration of a data file describing mesh data information according to an embodiment. Figure 6 is a diagram showing the types of three-dimensional data according to an embodiment. Figure 7 is a diagram showing the configuration of the first encoding unit according to an embodiment. Figure 8 is a block diagram of the first encoding unit according to an embodiment. Figure 9 is a diagram showing the configuration of the first decoding unit according to an embodiment. Figure 10 is a block diagram of the first decoding unit according to an embodiment. Figure 11 is a diagram showing the configuration of the second encoding unit according to an embodiment. Figure 12 is a block diagram of the second encoding unit according to an embodiment. Figure 13 is a diagram showing the configuration of the second decoding unit according to an embodiment. Figure 14 is a block diagram of the second decoding unit according to an embodiment. Figure 15 is a block diagram of the position information encoding unit according to an embodiment. Figure 16 is a block diagram of the position information decoding unit according to an embodiment. Figure 17 is a block diagram of the octree coding unit according to the embodiment. Figure 18 is a diagram showing an example of location information according to the embodiment. Figure 19 is a diagram showing an example of the octree representation of location information according to the embodiment. Figure 20 is a block diagram of the octree decoding unit according to the embodiment. Figure 21 is a block diagram of the attribute information coding unit according to the embodiment. Figure 22 is a block diagram of the attribute information decoding unit according to the embodiment. Figure 23 is a block diagram of the attribute information coding unit according to the embodiment. Figure 24 is a block diagram of the attribute information decoding unit according to the embodiment. Figure 25 is a block diagram of the three-dimensional data coding device according to the embodiment. Figure 26 is a block diagram of the three-dimensional data decoding device according to the embodiment. Figure 27 is a diagram showing the relationship between tiles and slices according to the embodiment. Figure 28 is a diagram showing an example of the configuration of a bitstream according to the embodiment. Figure 29 is a block diagram showing an example of the configuration of a three-dimensional data coding device according to the embodiment. Figure 30 is a diagram showing an example of the configuration of coded data and a NAL unit according to the embodiment.Figure 31 is a diagram showing an example of the semantics of pcc_nal_unit_type according to the embodiment. Figure 32 is a block diagram showing an example of a device that generates format data from data according to the embodiment. Figure 33 is a block diagram showing an example of a device that restores the original data from the format data according to the embodiment. Figure 34 is a conceptual diagram illustrating an example of how an encoded data bitstream is stored in a system format according to the embodiment. Figure 35 is a diagram showing an example of the box structure of ISOBMFF according to the embodiment. Figure 36 is a diagram showing a part of the processing in a three-dimensional data generation system according to the embodiment. Figure 37 is a diagram showing the components of Gaussian data according to the embodiment. Figure 38 is a diagram showing an example of 2DGS data according to the embodiment. Figure 39 is a diagram showing an example of 3DVS data according to the embodiment. Figure 40 is a diagram showing an example of STG data according to the embodiment. Figure 41 is a diagram showing an example of input and output of a spherical harmonic function according to the embodiment. Figure 42 is a diagram showing an example of the SH coefficient of a spherical harmonic function according to the embodiment. Figure 43 is a diagram showing an example of the SH coefficient of a spherical harmonic function according to the embodiment. Figure 44 is a diagram showing the rendering process of Gaussian data according to the embodiment. Figure 45 is a diagram showing the rendering process according to the embodiment. Figure 46 is a block diagram of the three-dimensional data generation unit according to the embodiment. Figure 47 is a block diagram of the encoding device according to the embodiment. Figure 48 is a block diagram of the decoding device according to the embodiment. Figure 49 is a block diagram of the encoding device according to the embodiment. Figure 50 is a block diagram of the decoding device according to the embodiment. Figure 51 is a diagram showing an example of Gaussian data mapping according to the embodiment. Figure 52 is a diagram showing an example of mapping information according to the embodiment. Figure 53 is a diagram showing an example of the configuration of conversion information according to the embodiment. Figure 54 is a diagram showing an example of the configuration of encoded data according to the embodiment. Figure 55 is a diagram showing an example of the G-PCC attribute type of Gaussian data according to the embodiment. Figure 56 is a flowchart of the processing by the decoding device according to the embodiment. Figure 57 is a block diagram of the data processing system according to Embodiment 1. Figure 58 is a flowchart of the encoding process according to Embodiment 1.Figure 59 is a diagram showing an example of the hierarchical structure of SH coefficients according to Embodiment 1. Figure 60 is a diagram showing an example of the hierarchical structure of MLP parameters in the STG format according to Embodiment 1. Figure 61 is a diagram showing an example of the hierarchical structure of time components in the STG format according to Embodiment 1. Figure 62 is a diagram showing a table of information regarding attribute components included in the GS data of the 3DGS format according to Embodiment 1. Figure 63 is a diagram showing the syntax of the hierarchical structure metadata according to Embodiment 1. Figure 64 is a block diagram of the decoding unit according to Embodiment 1. Figure 65 is a flowchart of the decoding process according to Embodiment 1. Figure 66 is a diagram showing an example of a table of information regarding GS components according to Embodiment 1. Figure 67 is a diagram showing the syntax of the hierarchical structure metadata according to Embodiment 1. Figure 68 is a diagram showing an example of GS component information according to Embodiment 1. Figure 69 is a diagram showing an example of the hierarchical structure metadata according to Embodiment 1. Figure 70 is a block diagram of the decoding unit according to Embodiment 2. Figure 71 is a diagram showing an example of a table showing the correspondence between G-PCC components and G-PCC_component_layer_id according to Embodiment 2. Figure 72 is a diagram showing an example of hierarchical metadata according to Embodiment 2. Figure 73 is a diagram showing an example of hierarchical metadata according to Embodiment 2. Figure 74 is a flowchart showing the partial decoding process according to Embodiment 2. Figure 75 is a diagram showing an example of hierarchical metadata according to Embodiment 2. Figure 76 is a diagram showing the configuration of a TLV unit according to Embodiment 2. Figure 77 is a diagram showing an example of the process of extracting some ADUs from a bitstream according to Embodiment 2. Figure 78 is a diagram showing an example of hierarchical metadata in this case according to Embodiment 2. Figure 79 is a diagram showing an example of hierarchical metadata in this case according to Embodiment 2. Figure 80 is a diagram showing mapping information and a hierarchical table according to Embodiment 3. Figure 81 is a diagram showing an example of mapping information according to Embodiment 3. Figure 82 is a diagram showing an example of mapping information according to Embodiment 3. Figure 83 is a flowchart of the processing by the encoding device according to the embodiment. Figure 84 is a block diagram of the encoding device according to the embodiment.Figure 85 is a flowchart of the processing performed by the decoding device according to the embodiment. Figure 86 is a block diagram of the decoding device according to the embodiment.

[0014] [Introduction] An encoding method according to one aspect of the present disclosure generates encoded data by encoding a plurality of first components contained in Gaussian data in a hierarchical structure, and generates a bitstream including hierarchical information relating to the hierarchical structure in the hierarchical structure and the encoded data. According to this, a decoding device that decodes the bitstream can selectively decode some or all of the data from the hierarchical encoded data depending on the application. Therefore, it is possible to reduce the processing load of the decoding device or to speed up processing.

[0015] For example, in the hierarchical arrangement, the plurality of first components may be assigned to a base hierarchical level that can be used independently and an extended hierarchical level that can be used in combination with the base hierarchical level, and the hierarchical information may indicate whether each of the plurality of first components belongs to the base hierarchical level or the extended hierarchical level. With this arrangement, the decoding device can selectively decode only the base hierarchical level or both the base hierarchical level and the extended hierarchical level depending on the application. Thus, it is possible to reduce the processing load of the decoding device or to speed up processing.

[0016] For example, the base layer may include information that is independent of the viewing direction among the multiple first components, and the extended layer may include information that is dependent on the viewing direction among the multiple first components. This allows the decoding device to reduce processing load by decoding only the base layer when information dependent on the viewing direction is not needed. Furthermore, by decoding the extended layer, the decoding device can recover high-precision Gaussian data that includes information dependent on the viewing direction.

[0017] For example, the information independent of the line of sight may include a Level 0 SH (Spherical Harmonics) coefficient, while the information dependent on the line of sight may include a Level 1 or higher SH coefficient.

[0018] For example, the information independent of the line of sight may include a base color feature in the STG (Space-time Splatting) format, and the information dependent on the line of sight may include a view feature in the STG format.

[0019] For example, the base layer may include time-independent information from among the multiple first components, and the extended layer may include time-dependent information from among the multiple first components. This allows the decoder to reduce processing load by decoding only the base layer when time-dependent information is not required. Furthermore, the decoder can recover high-precision Gaussian data containing time-dependent information by decoding the extended layer.

[0020] For example, the time-independent information may include position, rotation, scale, and transmittance, while the time-dependent information may include information showing the time change of at least one of the position, rotation, scale, and transmittance.

[0021] For example, the time-independent information may include a base color feature in the STG (Space-time Splatting) format, and the time-dependent information may include at least one of Polynomial Trajectory, Polynomial Rotation, Temporal Scale, and Temporal Optics in the STG format.

[0022] For example, in the encoding process, the plurality of first components may be converted into a plurality of second components based on an encoding scheme, the converted plurality of second components may be encoded using the encoding scheme, and the hierarchical information may indicate the correspondence between the plurality of first components and the plurality of hierarchies in the hierarchical structure. According to this, the decoding device can use the hierarchical information to determine which hierarchy the plurality of first components belong to.

[0023] For example, in the encoding process, the plurality of first components may be converted into a plurality of second components based on an encoding scheme, the converted plurality of second components may be encoded using the encoding scheme, and the hierarchical information may indicate the correspondence between the plurality of second components and the plurality of hierarchies in the hierarchical structure. According to this, the decoding device can use the hierarchical information to determine which hierarchy the plurality of second components belong to. Therefore, the decoding device can determine the necessary data before converting the second components into first components.

[0024] For example, in the encoding process, the plurality of first components are converted into a plurality of second components based on an encoding scheme, the converted plurality of second components are encoded using the encoding scheme, and the hierarchical information may indicate the correspondence between the plurality of first components, the plurality of second components, and the plurality of hierarchies in the hierarchical structure. The decoding device can use the hierarchical information to determine which hierarchy each of the plurality of first components belongs to, and which hierarchy each of the plurality of second components belongs to. Therefore, the decoding device can determine the necessary data before converting the second components back into first components.

[0025] A decoding method according to one aspect of this disclosure obtains encoded data generated by hierarchically encoding a plurality of first components contained in Gaussian data from a bitstream, and hierarchical information relating to the hierarchical structure in the hierarchical encoding, and decodes the encoded data using the hierarchical information. With this, the decoding device can selectively decode some or all of the data from the hierarchically encoded data depending on the application. Therefore, it is possible to reduce the processing load of the decoding device or to speed up processing.

[0026] For example, in the hierarchical arrangement, the plurality of first components are assigned to a base hierarchical level that can be used independently and an extended hierarchical level that can be used in combination with the base hierarchical level. The hierarchical information indicates whether each of the plurality of first components belongs to the base hierarchical level or the extended hierarchical level. In the decoding process, the hierarchical information may be used to decode only the base hierarchical level, or to decode both the base hierarchical level and the extended hierarchical level. This allows the decoding device to selectively decode only the base hierarchical level or both the base hierarchical level and the extended hierarchical level, depending on the application. Thus, the processing load of the decoding device can be reduced or the processing speed can be increased.

[0027] For example, the base layer may include components of the plurality of first components that do not depend on the direction of gaze, and the extended layer may include components of the plurality of first components that do depend on the direction of gaze. In this case, the decoding device can reduce the processing load by decoding only the base layer when information that depends on the direction of gaze is not needed. Furthermore, the decoding device can recover high-precision Gaussian data that includes information that depends on the direction of gaze by decoding the extended layer.

[0028] For example, the information independent of the line of sight may include a Level 0 SH (Spherical Harmonics) coefficient, while the information dependent on the line of sight may include a Level 1 or higher SH coefficient.

[0029] For example, the information independent of the line of sight may include a base color feature in the STG (Space-time Splatting) format, and the information dependent on the line of sight may include a view feature in the STG format.

[0030] For example, the base layer may include time-independent information from among the multiple first components, and the extended layer may include time-dependent information from among the multiple first components. This allows the decoder to reduce processing load by decoding only the base layer when time-dependent information is not required. Furthermore, the decoder can recover high-precision Gaussian data containing time-dependent information by decoding the extended layer.

[0031] For example, the time-independent information may include position, rotation, scale, and transmittance, while the time-dependent information may include information showing the time change of at least one of the position, rotation, scale, and transmittance.

[0032] For example, the time-independent information may include a base color feature in the STG (Space-time Splatting) format, and the time-dependent information may include at least one of Polynomial Trajectory, Polynomial Rotation, Temporal Scale, and Temporal Optics in the STG format.

[0033] For example, in the decoding process, the encoded data is decoded using a decoding method to generate a plurality of second components based on the decoding method, the plurality of second components are converted into a plurality of first components, and the hierarchical information indicates the correspondence between the plurality of first components and the plurality of hierarchies in the hierarchical structure. According to this, the decoding device can use the hierarchical information to determine which hierarchy the plurality of first components belong to.

[0034] For example, in the decoding process, the encoded data is decoded using a decoding method to generate a plurality of second components based on the decoding method, the plurality of second components are converted into a plurality of first components, and the hierarchical information indicates the correspondence between the plurality of second components and the plurality of hierarchies in the hierarchical structure. According to this, the decoding device can use the hierarchical information to determine which hierarchy the plurality of second components belong to. Therefore, the decoding device can determine the necessary data before converting the second components into first components.

[0035] For example, in the decoding process, the encoded data is decoded using a decoding method to generate a plurality of second components based on the decoding method, the plurality of second components are converted into a plurality of first components, and the hierarchical information may indicate the correspondence between the plurality of first components, the plurality of second components, and the plurality of hierarchies in the hierarchical structure. The decoding device can use the hierarchical information to determine which hierarchy each of the plurality of first components belongs to, and which hierarchy each of the plurality of second components belongs to. Therefore, the decoding device can determine the necessary data before converting the second components into first components.

[0036] An encoding device according to one aspect of this disclosure comprises a processor and a memory, wherein the processor generates encoded data by encoding a plurality of first components contained in Gaussian data in a hierarchical manner using the memory, and generates a bitstream including hierarchical information relating to the hierarchical structure in the hierarchical encoding and the encoded data.Accordingly, a decoding device that decodes the bitstream can selectively decode some or all of the data from the hierarchical encoded data depending on the application.Therefore, it is possible to reduce the processing load of the decoding device or to speed up processing.

[0037] A decoding device according to one aspect of this disclosure comprises a processor and a memory, wherein the processor uses the memory to obtain encoded data generated by hierarchically encoding a plurality of first components contained in Gaussian data from a bitstream, and hierarchical information relating to the hierarchical structure in the hierarchical encoding, and decodes the encoded data using the hierarchical information. As a result, the decoding device can selectively decode some or all of the data from the hierarchically encoded data depending on the application. Therefore, it is possible to reduce the processing load of the decoding device or to speed up processing.

[0038] These comprehensive or specific embodiments may be implemented as a system, method, integrated circuit, computer program, or recording medium such as a computer-readable CD-ROM, or as any combination of a system, method, integrated circuit, computer program, and recording medium.

[0039] The embodiments will be described in detail below with reference to the drawings. Note that the embodiments described below are all specific examples of this disclosure. The numerical values, shapes, materials, components, arrangement and connection configurations of components, steps, and the order of steps shown in the following embodiments are examples only and are not intended to limit this disclosure. Furthermore, components in the following embodiments that are not described in an independent claim will be described as optional components.

[0040] (Embodiment) [Three-Dimensional Data Encoding and Decoding System] First, the configuration of the three-dimensional data encoding and decoding system according to this embodiment will be described. Figure 1 is a diagram showing an example of the configuration of the three-dimensional data encoding and decoding system according to this embodiment. As shown in Figure 1, the three-dimensional data encoding and decoding system includes a three-dimensional data encoding system 101, a three-dimensional data decoding system 102, a sensor terminal 103, and an external connection unit 104.

[0041] The three-dimensional data encoding system 101 generates encoded data or multiplexed data by encoding three-dimensional point cloud data, three-dimensional mesh data, or the like, which are three-dimensional data. The three-dimensional data encoding system 101 may be a three-dimensional data encoding apparatus implemented by a single device, or may be a system implemented by a plurality of devices. Further, the three-dimensional data encoding apparatus may include a part of a plurality of processing units included in the three-dimensional data encoding system 101.

[0042] The three-dimensional data encoding system 101 includes a three-dimensional data generation system 111, a presentation unit 112, an encoding unit 113, a multiplexing unit 114, an input / output unit 115, and a control unit 116. The three-dimensional data generation system 111 includes a sensor information acquisition unit 117 and a three-dimensional data generation unit 118.

[0043] The sensor information acquisition unit 117 acquires sensor information (sensor signal) from the sensor terminal 103, and outputs the sensor information to the three-dimensional data generation unit 118. The three-dimensional data generation unit 118 generates three-dimensional data from the sensor information, and outputs the three-dimensional data to the encoding unit 113.

[0044] The presentation unit 112 presents the sensor information or the three-dimensional data to the user. For example, the presentation unit 112 displays information or an image based on the sensor information or the three-dimensional data.

[0045] The encoding unit 113 encodes (compresses) the three-dimensional data, and outputs the obtained encoded data, the control information obtained in the encoding process, and other additional information to the multiplexing unit 114. The additional information includes, for example, sensor information.

[0046] The multiplexing unit 114 generates multiplexed data by multiplexing the encoded data, the control information, and the additional information input from the encoding unit 113. The format of the multiplexed data is, for example, a file format for storage or a packet format for transmission.

[0047] The input / output unit 115 (for example, a communication unit or an interface) outputs the multiplexed data to the outside. Alternatively, the multiplexed data is stored in a storage unit such as an internal memory. The control unit 116 (or an application execution unit) controls each processing unit. That is, the control unit 116 performs control such as encoding and multiplexing.

[0048] Note that the sensor information may be input to the encoding unit 113 or the multiplexing unit 114. Further, the input / output unit 115 may output the three-dimensional data or the encoded data as it is to the outside.

[0049] The transmission signal (multiplexed data) output from the three-dimensional data encoding system 101 is input to the three-dimensional data decoding system 102 via the external connection unit 104.

[0050] The three-dimensional data decoding system 102 decodes the encoded data or the multiplexed data to generate three-dimensional point cloud data, three-dimensional mesh data, or the like, which are three-dimensional data. Note that the three-dimensional data decoding system 102 may be a three-dimensional data decoding apparatus implemented by a single device, or may be a system implemented by a plurality of devices. Further, the three-dimensional data decoding apparatus may include a part of the plurality of processing units included in the three-dimensional data decoding system 102.

[0051] The three-dimensional data decoding system 102 includes a sensor information acquisition unit 121, an input / output unit 122, a demultiplexing unit 123, a decoding unit 124, a presentation unit 125, a user interface 126, and a control unit 127.

[0052] The sensor information acquisition unit 121 acquires sensor information (sensor signal) from the sensor terminal 103.

[0053] The input / output unit 122 acquires a transmission signal, decodes multiplexed data (file format or packet) from the transmission signal, and outputs the multiplexed data to the demultiplexing unit 123.

[0054] The demultiplexing unit 123 acquires encoded data, control information, and additional information from the multiplexed data, and outputs the encoded data, the control information, and the additional information to the decoding unit 124.

[0055] The decoding unit 124 reconstructs the three-dimensional data by decoding the encoded data.

[0056] The presentation unit 125 presents three-dimensional data to the user. For example, the presentation unit 125 displays information or images based on the three-dimensional data. The user interface 126 acquires instructions based on user operations. The control unit 127 (or application execution unit) controls each processing unit. In other words, the control unit 127 performs control such as demultiplexing, decoding, and presentation.

[0057] The input / output unit 122 may acquire three-dimensional data or encoded data directly from an external source. The presentation unit 125 may acquire additional information such as sensor information and present information based on that additional information. The presentation unit 125 may also make presentations based on user instructions acquired through the user interface 126.

[0058] The sensor terminal 103 generates sensor information, which is information obtained from the sensor. The sensor terminal 103 is a terminal equipped with a sensor or camera, and may be, for example, a mobile object such as an automobile, an aerial object such as an airplane, a mobile terminal, or a camera.

[0059] The sensor information that can be acquired by the sensor terminal 103 includes, for example, (1) the distance between the sensor terminal 103 and the object, or the reflectivity of the object, obtained from a LiDAR, millimeter-wave radar, or infrared sensor, and (2) the distance between the camera and the object, or the reflectivity of the object, obtained from multiple monocular camera images or stereo camera images. The sensor information may also include the sensor's attitude, orientation, gyroscope (angular velocity), position (GPS information or altitude), speed, or acceleration. The sensor information may also include temperature, atmospheric pressure, humidity, or magnetism.

[0060] The external connection unit 104 is implemented by an integrated circuit (LSI or IC), an external storage unit, communication with a cloud server via the internet, or broadcasting, etc.

[0061] Next, we will explain three-dimensional point cloud data (hereinafter also referred to as point cloud data). Figure 2 is a diagram showing the structure of point cloud data. Figure 3 is a diagram showing an example of the structure of a data file containing information about point cloud data.

[0062] Point cloud data contains data for multiple points. Each point's data includes location information (three-dimensional coordinates) and attribute information related to that location. A collection of these points is called a point cloud. For example, a point cloud represents the three-dimensional shape of an object.

[0063] Position information, such as three-dimensional coordinates, is sometimes called geometry. Furthermore, the data for each point may include attribute information of multiple attribute types. Attribute types include, for example, color or reflectance.

[0064] One location information may be associated with one attribute information, or multiple attribute information of different attribute types may be associated with one location information. Furthermore, multiple attribute information of the same attribute type may be associated with one location information.

[0065] The example data file structure shown in Figure 3 represents a case where location information and attribute information correspond one-to-one, and it shows the location information and attribute information of the N points that make up the point cloud data.

[0066] Location information includes, for example, information for the three axes: x, y, and z. Attribute information includes, for example, RGB color information. A typical data file is a ply file.

[0067] Next, we will explain three-dimensional mesh data (hereinafter also referred to as mesh data). Figure 4 shows an example of the structure of mesh data. Figure 5 shows an example of the structure of a data file containing mesh data information.

[0068] Mesh data is a data format used in computer graphics (CG). Mesh data represents the three-dimensional shape of an object through a collection of surface information. Surface information consists of polygons such as triangles or quadrilaterals, and is also referred to as a polygon or polygon mesh.

[0069] The components of mesh data include a three-dimensional point cloud (a set of points that have three-dimensional positional information and attribute information corresponding to that positional information), as well as a set of three-dimensional points as vertices, edges connecting two vertices, and faces enclosed by edges.

[0070] Vertices (also expressed as vertex or position) may have attribute information such as color information, reflectivity, or normal vectors. Information indicating the relationships between vertices that constitute an edge or face is also called connectivity. The direction of the normal vector to a point can represent the front and back of a face. Mesh data may also contain attribute information for faces.

[0071] One example of a mesh data file format is an object file. The data file contains the position information G(1) to G(N) and vertex attribute information A(1) to A(N) for the N vertices that make up the mesh. Note that the data file does not have to include attribute information. Also, the attribute information does not have to correspond one-to-one with the vertices. Figure 5 shows an example where the data file has 1 to M attribute information A2.

[0072] Face information is represented by a combination of vertex indices. n[1, 3, 4] indicates that the face is a triangular face composed of three vertices n=1, n=3, and n=4. m[2, 4, 6] indicates that attribute information m=1, m=4, and m=6 correspond to the three vertices, respectively.

[0073] Furthermore, attribute information may be recorded in a separate file from the data file, with the data file indicating its pointer information. For example, attribute information may be stored in a two-dimensional attribute map file, and the file name of the attribute map and its two-dimensional coordinates may be recorded in attribute information A2 of the data file. In any of these methods, it is possible to specify attribute information for a point.

[0074] Next, we will explain the types of three-dimensional data (point cloud data or mesh data). Figure 6 is a diagram showing the types of three-dimensional data. As shown in Figure 6, three-dimensional data includes static objects and dynamic objects.

[0075] A static object is three-dimensional data for any given time (a specific moment). A dynamic object is three-dimensional data that changes over time. Hereafter, three-dimensional point cloud data for a given time will be referred to as a PCC (Point Cloud Compression) frame, or simply a frame. Similarly, three-dimensional mesh data for a given time will be referred to as a mesh frame, or simply a frame.

[0076] The multiple points that make up an object may have limitations in terms of area, pixels, or the number of points, similar to regular video data. Alternatively, the multiple points that make up an object may not have any area limitations, similar to map information.

[0077] Furthermore, there may be point cloud data or mesh data of various densities, including both sparse and dense point cloud data or mesh data.

[0078] The details of each processing unit are described below. Sensor information is acquired by various methods, such as distance sensors like LIDAR or rangefinders, stereo cameras, or combinations of multiple monocular cameras. The three-dimensional data generation unit 118 generates three-dimensional data based on the sensor information obtained by the sensor information acquisition unit 117. The three-dimensional data generation unit 118 generates position information as three-dimensional data and adds attribute information to the position information.

[0079] The three-dimensional data generation unit 118 may process the three-dimensional data when generating position information or adding attribute information. For example, the three-dimensional data generation unit 118 may reduce the amount of data by deleting point clouds with overlapping positions. The three-dimensional data generation unit 118 may also transform the position information (e.g., position shift, rotation, or normalization). Furthermore, the three-dimensional data generation unit 118 may generate mesh data from the point cloud data. Finally, the three-dimensional data generation unit 118 may render attribute information.

[0080] In Figure 1, the three-dimensional data generation system 111 is included in the three-dimensional data encoding system 101, but it may also be provided independently outside of the three-dimensional data encoding system 101.

[0081] The encoding unit 113 generates encoded data by encoding three-dimensional data. There are the following encoding methods. The first is an encoding method using positional information, which will be referred to as the first encoding method hereafter. The second is an encoding method using a video codec, which will be referred to as the second encoding method hereafter.

[0082] The decoding unit 124 decodes the point cloud data by decoding the encoded data. The multiplexing unit 114 generates multiplexed data by multiplexing the encoded data using an existing multiplexing method. The generated multiplexed data is transmitted or stored. In addition to the PCC encoded data, the multiplexing unit 114 multiplexes other media such as video, audio, subtitles, applications, files, or reference time information. Furthermore, the multiplexing unit 114 may also multiplex sensor information or attribute information related to the three-dimensional data.

[0083] Multiplexing methods or file formats include ISOBMFF, ISOBMFF-based transmission methods such as MPEG-DASH, MMT, MPEG-2 TS Systems, and RTP.

[0084] The demultiplexing unit 123 extracts encoded data, other media, and time information from the multiplexed data.

[0085] The input / output unit 115 transmits the multiplexed data using a method appropriate to the transmission medium or storage medium, such as broadcasting or communication. The input / output unit 115 may communicate with other devices via the Internet, or with storage units such as cloud servers.

[0086] Communication protocols such as HTTP, FTP, TCP, or UDP can be used. A pull-type communication method or a push-type communication method may be used.

[0087] Either wired or wireless transmission may be used. Wired transmission methods include Ethernet®, USB, RS-232C, HDMI®, or coaxial cable. Wireless transmission methods include wireless LAN, Wi-Fi®, Bluetooth®, or millimeter wave.

[0088] Furthermore, broadcasting formats such as DVB-T2, DVB-S2, DVB-C2, ATSC3.0, or ISDB-S3 may be used.

[0089] [First Encoding Method] Hereafter, encoding and decoding methods for three-dimensional point clouds or three-dimensional meshes will be described. When the apparatus, processing, or syntax in this disclosure relates to the encoding or decoding of point cloud data, it can also be applied to the encoding or decoding of vertices in a three-dimensional mesh. Furthermore, disclosures relating to the encoding or decoding of vertices in a three-dimensional mesh can also be applied to the encoding or decoding of point cloud data. In addition, disclosures relating to the encoding or decoding of attribute information of point cloud data may be applied to the encoding or decoding of face information, or attribute information for faces or vertices in a three-dimensional mesh. Moreover, the processing may be unified between point cloud encoding and mesh encoding. By unifying the processing, it may be possible to reduce the size of the circuit or software.

[0090] Figure 7 shows the configuration of a first encoding unit 130, which is an example of an encoding unit 113 that performs encoding using the first encoding method. Figure 8 is a block diagram of the first encoding unit 130. The first encoding unit 130 generates encoded data (encoded stream) by encoding point cloud data using the first encoding method. This first encoding unit 130 includes a location information encoding unit 131, an attribute information encoding unit 132, an additional information encoding unit 133, and a multiplexing unit 134.

[0091] The first encoding unit 130 is characterized by performing encoding while being aware of the three-dimensional structure. Furthermore, the first encoding unit 130 is characterized in that the attribute information encoding unit 132 performs encoding using information obtained from the position information encoding unit 131. The first encoding method is also called G-PCC (Geometry-based PCC).

[0092] The point cloud data is PCC point cloud data such as a PLY file, or PCC point cloud data generated from sensor information, and includes position information, attribute information, and other additional information (MetaData). The position information is input to the position information encoding unit 131, the attribute information is input to the attribute information encoding unit 132, and the additional information is input to the additional information encoding unit 133.

[0093] When mesh data is encoded, the position information of the vertex information is input to the position information encoding unit 131, and the face information, or attribute information for faces and vertices, is input to the attribute information encoding unit 132.

[0094] The location information encoding unit 131 generates encoded location information (Compressed Geometry), which is encoded data, by encoding the location information. For example, the location information encoding unit 131 encodes the location information using an N-tree structure such as an octree. Specifically, in an octree, the target space is divided into eight nodes (subspaces), and eight bits of information (occupancy code) are generated to indicate whether or not a point cloud is contained in each node. Furthermore, nodes containing point clouds are further divided into eight nodes, and eight bits of information are generated to indicate whether or not a point cloud is contained in each of these eight nodes. This process is repeated until the number of point clouds contained in a predetermined hierarchy or node falls below a threshold.

[0095] The attribute information encoding unit 132 generates encoded attribute information (Compressed Attribute), which is encoded data, by encoding it using the configuration information generated by the location information encoding unit 131. For example, the attribute information encoding unit 132 determines the reference point (reference node) to be referenced in encoding the target point (target node) to be processed, based on the octave tree structure generated by the location information encoding unit 131. For example, the attribute information encoding unit 132 references a surrounding node or adjacent node whose parent node in the octave tree is the same as the target node. However, the method for determining the reference relationship is not limited to this.

[0096] Furthermore, the attribute information encoding process may include at least one of the following: quantization, prediction, and arithmetic encoding. In this case, a reference means using a reference node to calculate the predicted value of the attribute information, or using the state of a reference node (for example, occupancy information indicating whether or not the reference node contains a point cloud) to determine the encoding parameters. For example, encoding parameters may be quantization parameters in the quantization process, or context in arithmetic encoding.

[0097] The additional information encoding unit 133 generates encoded additional information (Compressed MetaData), which is encoded data, by encoding the compressible data among the additional information.

[0098] The multiplexing unit 134 generates a compressed stream, which is encoded data, by multiplexing encoded position information, encoded attribute information, encoded additional information, and other additional information. The generated compressed stream is output to a processing unit of the system layer (not shown).

[0099] Next, we will describe a first decoding unit 140, which is an example of a decoding unit 124 that performs decoding of the first encoding method. Figure 9 is a diagram showing the configuration of the first decoding unit 140. Figure 10 is a block diagram of the first decoding unit 140. The first decoding unit 140 generates point cloud data by decoding the encoded data (encoded stream) encoded by the first encoding method using the first encoding method. This first decoding unit 140 includes a demultiplexing unit 141, a location information decoding unit 142, an attribute information decoding unit 143, and an additional information decoding unit 144.

[0100] A encoded stream, which is encoded data, is input to the first decoding unit 140 from a processing unit of the system layer (not shown).

[0101] The demultiplexing unit 141 separates encoded location information (Compressed Geometry), encoded attribute information (Compressed Attribute), encoded additional information (Compressed MetaData), and other additional information from the encoded data.

[0102] The location information decoding unit 142 generates location information by decoding the encoded location information. For example, the location information decoding unit 142 reconstructs the location information of a point cloud represented by three-dimensional coordinates from encoded location information represented by an N-tree structure such as an octree.

[0103] The attribute information decoding unit 143 decodes the encoded attribute information based on the configuration information generated by the location information decoding unit 142. For example, the attribute information decoding unit 143 determines the reference point (reference node) to be referenced in the decoding of the target point (target node) to be processed, based on the octave tree structure obtained by the location information decoding unit 142. For example, the attribute information decoding unit 143 references a surrounding node or adjacent node whose parent node in the octave tree is the same as the target node. Note that the method for determining the reference relationship is not limited to this.

[0104] Furthermore, the attribute information decoding process may include at least one of the following: inverse quantization, prediction, and arithmetic decoding. In this case, "reference" means using a reference node to calculate the predicted value of the attribute information, or using the state of the reference node (for example, occupancy information indicating whether or not the reference node contains a point cloud) to determine the decoding parameters. For example, decoding parameters may be quantization parameters in the inverse quantization process, or context in arithmetic decoding.

[0105] The additional information decoding unit 144 generates additional information by decoding the encoded additional information. The first decoding unit 140 uses the additional information necessary for decoding location information and attribute information during decoding and outputs the additional information necessary for the application to the outside.

[0106] [Second Encoding Method] Next, a second encoding unit 150, which is an example of an encoding unit 113 that performs encoding using the second encoding method, will be described. Figure 11 is a diagram showing the configuration of the second encoding unit 150. Figure 12 is a block diagram of the second encoding unit 150.

[0107] The second encoding unit 150 generates encoded data (encoded stream) by encoding the point cloud data using a second encoding method. This second encoding unit 150 includes an additional information generation unit 151, a position image generation unit 152, an attribute image generation unit 153, a video encoding unit 154, an additional information encoding unit 155, and a multiplexing unit 156.

[0108] The second encoding unit 150 is characterized by generating a position image and an attribute image by projecting a three-dimensional structure onto a two-dimensional image, and encoding the generated position image and attribute image using an existing video encoding scheme. The second encoding method is also called VPCC (Video-based PCC).

[0109] The point cloud data is PCC point cloud data such as a PLY file, or PCC point cloud data generated from sensor information, and includes position information, attribute information, and other additional information (MetaData).

[0110] The additional information generation unit 151 generates map information for multiple two-dimensional images by projecting a three-dimensional structure onto a two-dimensional image.

[0111] The location image generation unit 152 generates a location image (Geometry Image) based on location information and map information generated by the additional information generation unit 151. This location image is, for example, a depth image in which the distance is indicated as a pixel value. This depth image may be an image of multiple point clouds viewed from one viewpoint (an image of multiple point clouds projected onto a single two-dimensional plane), or multiple images of multiple point clouds viewed from multiple viewpoints, or a single image formed by integrating these multiple images.

[0112] The attribute image generation unit 153 generates an attribute image based on attribute information and map information generated by the additional information generation unit 151. This attribute image is, for example, an image in which attribute information (e.g., color (RGB)) is shown as pixel values. This image may be an image of multiple point clouds viewed from one viewpoint (an image of multiple point clouds projected onto a single two-dimensional plane), or multiple images of multiple point clouds viewed from multiple viewpoints, or a single image formed by integrating these multiple images.

[0113] When mesh data is encoded, the position information of the vertex information is input to the position image generation unit 152, and the face information, or attribute information for faces and vertices, is input to the attribute image generation unit 153.

[0114] The video encoding unit 154 generates encoded data, namely a encoded geometry image and an encoded attribute image, by encoding the position image and attribute image using a video encoding scheme. Any known encoding method may be used as the video encoding scheme. For example, the video encoding scheme may be AVC or HEVC.

[0115] The additional information encoding unit 155 generates encoded additional information (Compressed MetaData) by encoding additional information and map information included in the point cloud data.

[0116] The multiplexing unit 156 generates a compressed stream, which is encoded data, by multiplexing the encoded position image, encoded attribute image, encoded additional information, and other additional information. The generated compressed stream is output to a processing unit of a system layer (not shown).

[0117] Next, we will describe a second decoding unit 160, which is an example of a decoding unit 124 that performs decoding of the second encoding method. Figure 13 is a diagram showing the configuration of the second decoding unit 160. Figure 14 is a block diagram of the second decoding unit 160. The second decoding unit 160 generates point cloud data by decoding the encoded data (encoded stream) encoded by the second encoding method using the second encoding method. This second decoding unit 160 includes a demultiplexing unit 161, a video decoding unit 162, an additional information decoding unit 163, a location information generation unit 164, and an attribute information generation unit 165.

[0118] A encoded stream (Compressed Stream), which is encoded data, is input to the second decoding unit 160 from a processing unit of the system layer (not shown).

[0119] The demultiplexing unit 161 separates the encoded location image (Compressed Geometry Image), encoded attribute image (Compressed Attribute Image), encoded additional information (Compressed MetaData), and other additional information from the encoded data.

[0120] The video decoding unit 162 generates a position image and an attribute image by decoding the encoded position image and the encoded attribute image using a video encoding scheme. Any known encoding scheme may be used as the video encoding scheme. For example, the video encoding scheme may be AVC or HEVC.

[0121] The additional information decoding unit 163 generates additional information, including map information, by decoding the encoded additional information.

[0122] The location information generation unit 164 generates location information using the location image and map information. The attribute information generation unit 165 generates attribute information using the attribute image and map information.

[0123] The second decoding unit 160 uses the additional information necessary for decoding during decoding and outputs the additional information necessary for the application to the outside.

[0124] [Location Information Coding in the First Coding Method] Figure 15 is a block diagram showing an example configuration of the location information coding unit 131. The location information coding unit 131 comprises an octree coding unit 171 and a prediction tree coding unit 172. The octree coding unit 171 generates coded location information and metadata by coding location information using an octree coding method (octree coding). The prediction tree coding unit 172 generates coded location information and metadata by coding location information using a prediction tree coding method (prediction tree coding).

[0125] The location information encoding unit 131 encodes location information using either octree encoding or predictive tree encoding, or both. The location information encoding unit 131 may switch between these two methods, use an encoding method other than these two, or include the location information in the bitstream as raw data without encoding it. Information indicating the encoding method of the encoded data is stored in metadata and notified to the decoding device.

[0126] Figure 16 is a block diagram showing an example configuration of the location information decoding unit 142. The location information decoding unit 142 comprises an octree decoding unit 173 and a predictive tree decoding unit 174. The octree decoding unit 173 generates location information by decoding the encoded location information using an octree decoding method (octree decoding). The predictive tree decoding unit 174 generates location information by decoding the encoded location information using a predictive tree decoding method (predictive tree decoding). The location information decoding unit 142 also performs decoding using the encoding method notified in the metadata.

[0127] Next, an example of the configuration of the position information coding unit will be described. Figure 17 is a block diagram of the octree coding unit 171 according to this embodiment. The octree coding unit 171 comprises an octree generation unit 181, a geometric information calculation unit 182, a coding table selection unit 183, and an entropy coding unit 184.

[0128] The octree generation unit 181 generates an octree from the input position information and generates occupancy codes for each node in the octree. The geometric information calculation unit 182 obtains information indicating whether the adjacent nodes of the target node are occupied nodes or not. For example, the geometric information calculation unit 182 calculates the occupancy information of adjacent nodes (information indicating whether the adjacent node is an occupied node or not) from the occupancy code of the parent node to which the target node belongs. Alternatively, the geometric information calculation unit 182 may store the encoded nodes in a list and search for adjacent nodes from that list. The geometric information calculation unit 182 may also switch adjacent nodes depending on the position of the target node within the parent node.

[0129] The coding table selection unit 183 selects a coding table to be used for entropy coding of the target node using the occupancy information of adjacent nodes calculated by the geometric information calculation unit 182. For example, the coding table selection unit 183 may generate a bit sequence using the occupancy information of adjacent nodes and select a coding table with an index number generated from that bit sequence.

[0130] The entropy coding unit 184 generates coded location information and metadata by performing entropy coding on the occupancy code of the target node using the coding table of the selected index number. The entropy coding unit 184 may also add information indicating the selected coding table to the coded location information.

[0131] The following describes the octree representation and the scanning order of location information. Location information (location data) is converted into an octree structure (octreeization) and then encoded. An octree structure consists of nodes and leaves. Each node has eight nodes or leaves, and each leaf has voxel (VXL) information. Figure 18 shows an example of the structure of location information containing multiple voxels. Figure 19 shows an example of the location information shown in Figure 18 converted into an octree structure. Here, among the leaves shown in Figure 19, leaves 1, 2, and 3 represent the voxels VXL1, VXL2, and VXL3 shown in Figure 18, respectively, and represent a VXL containing a point cloud (hereinafter referred to as effective VXL).

[0132] Specifically, node 1 corresponds to the overall space encompassing the positional information in Figure 18. The overall space corresponding to node 1 is divided into eight nodes, and of these eight nodes, the node containing the valid VXL is further divided into eight nodes or leaves, and this process is repeated for each level of the tree structure. Here, each node corresponds to a subspace and holds information (occupancy code) indicating the position of the next node or leaf after the division as node information. In addition, the lowest-level block is set as a leaf, and leaf information such as the number of points contained within the leaf is held.

[0133] Next, an example of the configuration of the location information decoding unit will be described. Figure 20 is a block diagram of the octree decoding unit 173 according to this embodiment. The octree decoding unit 173 comprises an octree generation unit 191, a geometric information calculation unit 192, an encoding table selection unit 193, and an entropy decoding unit 194.

[0134] The octane tree generation unit 191 generates an octane tree of a certain space (node) using the header information or metadata of the bitstream. For example, the octane tree generation unit 191 generates a large space (root node) using the size of the x, y, and z axis directions of a certain space attached to the header information, and generates an octane tree by dividing that space into two in each direction of the x, y, and z axes to generate eight small spaces A (nodes A0 to A7). Also, nodes A0 to A7 are set in order as the target nodes.

[0135] The geometric information calculation unit 192 obtains occupancy information indicating whether an adjacent node to the target node is an occupied node. For example, the geometric information calculation unit 192 calculates the occupancy information of an adjacent node from the occupancy code of the parent node to which the target node belongs. Alternatively, the geometric information calculation unit 192 may store the decoded nodes in a list and search for adjacent nodes from that list. The geometric information calculation unit 192 may also switch adjacent nodes depending on the position of the target node within its parent node.

[0136] The coding table selection unit 193 selects a coding table (decoding table) to be used for entropy decoding of the target node using the occupancy information of adjacent nodes calculated by the geometric information calculation unit 192. For example, the coding table selection unit 193 may generate a bit sequence using the occupancy information of adjacent nodes and select a coding table with an index number generated from that bit sequence.

[0137] The entropy decoding unit 194 generates location information by entropy decoding the occupancy code of the target node using the selected coding table. Alternatively, the entropy decoding unit 194 may decode and obtain information from the bitstream regarding the selected coding table, and then entropy decode the occupancy code of the target node using the coding table indicated by this information.

[0138] [Attribute Information Encoding in the First Encoding Scheme] The configuration of the attribute information encoding unit and the attribute information decoding unit will be described below. Figure 21 is a block diagram showing an example configuration of the attribute information encoding unit 132. The attribute information encoding unit may include multiple encoding units that perform different encoding methods. For example, the attribute information encoding unit may switch between the following two methods depending on the use case.

[0139] The attribute information encoding unit 132 includes an LoD attribute information encoding unit 201 and a conversion attribute information encoding unit 202. The LoD attribute information encoding unit 201 uses the positional information of the three-dimensional points to classify each three-dimensional point into multiple layers, predicts the attribute information of the three-dimensional points belonging to each layer, and encodes the predicted residual. Here, each classified layer is called LoD (Level of Detail).

[0140] The attribute information encoding unit 202 encodes attribute information using RAHT (Region Adaptive Hierarchical Transform). Specifically, the attribute information encoding unit 202 generates high-frequency and low-frequency components of each layer by applying RAHT or Haar transform to each attribute information based on the position information of three-dimensional points, and encodes these values ​​using quantization and entropy coding, etc.

[0141] Figure 22 is a block diagram showing an example configuration of the attribute information decoding unit 143. The attribute information decoding unit may include multiple decoding units that perform different decoding methods. For example, the attribute information decoding unit may decode by switching between the following two methods based on the information contained in the header and metadata.

[0142] The attribute information decoding unit 143 includes an LoD attribute information decoding unit 203 and a converted attribute information decoding unit 204. The LoD attribute information decoding unit 203 classifies each three-dimensional point into multiple layers using the position information of the three-dimensional points, and decodes the attribute values ​​while predicting the attribute information of the three-dimensional points belonging to each layer.

[0143] The attribute information decoding unit 204 decodes attribute information using RAHT (Region Adaptive Hierarchical Transform). Specifically, the attribute information decoding unit 204 decodes attribute values ​​by applying inverse RAHT or inverse Haar transform to the high-frequency and low-frequency components of each attribute value based on the position information of the three-dimensional points.

[0144] Figure 23 is a block diagram of a transformation attribute information encoding unit 202, which is an example of the transformation attribute information encoding unit 202. The transformation attribute information encoding unit 202 comprises a sorting unit 211, a Haar transformation unit 212, a quantization unit 213, an inverse quantization unit 214, an inverse Haar transformation unit 215, a memory 216, and an arithmetic encoding unit 217.

[0145] The sorting unit 211 generates Morton codes using the position information of three-dimensional points and sorts the multiple three-dimensional points in Morton code order. The Haar transform unit 212 generates coding coefficients by applying the Haar transform to the attribute information. The quantization unit 213 quantizes the coding coefficients of the attribute information.

[0146] The inverse quantization unit 214 inversely quantizes the encoded coefficients after quantization. The inverse Haar transform unit 215 applies the inverse Haar transform to the encoded coefficients. The memory 216 stores the attribute information values ​​of the decoded three-dimensional points. For example, the attribute information of the decoded three-dimensional points stored in the memory 216 may be used for predicting unencoded three-dimensional points.

[0147] The arithmetic coding unit 217 calculates ZeroCnt from the quantized coding coefficients and arithmetically codes ZeroCnt. The arithmetic coding unit 217 also arithmetically codes the non-zero coding coefficients after quantization. The arithmetic coding unit 217 may also binarize the coding coefficients before arithmetic coding. The arithmetic coding unit 217 may also generate and code various header information.

[0148] Figure 24 is a block diagram of a conversion attribute information decoding unit 204, which is an example of a conversion attribute information decoding unit 204. The conversion attribute information decoding unit 204 comprises an arithmetic decoding unit 221, an inverse quantization unit 222, an inverse Haar conversion unit 223, and a memory 224.

[0149] The arithmetic decoding unit 221 arithmetically decodes the ZeroCnt and coding coefficients contained in the bitstream. The arithmetic decoding unit 221 may also decode various header information.

[0150] The inverse quantization unit 222 inversely quantizes the arithmetic-decoded coding coefficients. The inverse Haar transform unit 223 applies the inverse Haar transform to the coding coefficients after inverse quantization. The memory 224 stores the attribute information values ​​of the multiple decoded three-dimensional points. For example, the attribute information of the decoded three-dimensional points stored in the memory 224 may be used to predict the undecoded three-dimensional points.

[0151] [Slicing] The encoding device may divide the three-dimensional data into one or more three-dimensional data points and encode them. A three-dimensional point cloud divided into multiple parts is called a slice. A slice is a set of points that have geometry information and attribute information.

[0152] Figure 25 is a block diagram showing an example configuration of a three-dimensional data encoding device in this case. The three-dimensional data encoding device comprises a data division unit 231 and an encoding unit 232.

[0153] The data division unit 231 generates multiple divided three-dimensional data by dividing the three-dimensional data. Each divided three-dimensional data corresponds to a slice. The encoding unit 232 generates encoded data by encoding the positional information and attribute information of each of the multiple divided three-dimensional data (slices).

[0154] Figure 26 is a block diagram showing an example configuration of a three-dimensional data decoding device in this case. The three-dimensional data decoding device comprises a decoding unit 233 and a data merging unit 234. The decoding unit 233 generates multiple divided three-dimensional data by decoding encoded data. Each divided three-dimensional data includes slice position information and attribute information. The data merging unit 234 restores the three-dimensional data by merging the multiple divided three-dimensional data.

[0155] In encoding, there may or may not be dependencies between slices. If there are no dependencies, slices can be encoded or decoded independently. Therefore, processing time can be reduced by processing multiple slices in parallel. Also, the amount of processing can be reduced by partial decoding, which decodes only some of the slices.

[0156] If dependencies exist, an identifier indicating the dependency is stored in the bitstream, and encoding or decoding is performed sequentially, starting with the dependent data (the referenced data).

[0157] The number of divisions and the division method can be any method. The 3D data encoding device may determine the shape of an object and divide the point cloud for each object, or it may divide it based on the number of points included in a slice. Alternatively, the 3D data encoding device may use map information or location information to divide the point cloud based on whether or not it is included in 3D space (tile information).

[0158] Figure 27 shows the relationship between tiles and slices. As shown in Figure 27, tiles correspond to three-dimensional space, and slices correspond to divided three-dimensional point clouds. Multiple tiles may overlap. This enables parallel processing in adaptive encoding or decoding depending on the content or object, improving the flexibility of the point cloud encoding or decoding system.

[0159] [Configuration of Encoded Data] Positional and attribute information for each slice are encoded. At least a portion of the resulting encoded data is stored in the data unit's payload. A header is also added to the payload.

[0160] If multiple attribute pieces of information exist for each point, each of the multiple attribute pieces of information is stored in a data unit. Figure 28 shows an example of the structure of a bitstream (encoded data). Figure 28 shows encoded data of a point cloud of two frames with two types of attribute pieces of information. The bitstream has a geometry (position information) data unit (Geometry Data Unit: Geom) and two attribute (attribute information) data units (Attribute Data Unit: Attr(0), Attr(1)) for each slice. For example, if each point has two attribute pieces of information, color and reflectance, the encoded color data is stored in Attr(0) and the encoded reflectance data is stored in Attr(1). The header of the geometry data unit stores the slice identifier (slice_id). The header of the attribute data unit shows the slice_id of the corresponding (referenced) geometry data unit. Note that data units are sometimes called slices, and data unit headers are sometimes called slice headers.

[0161] Furthermore, metadata related to location information coding is stored in the Geometry Parameter Set (GPS). Metadata related to attribute information coding is stored in the Attribute Parameter Set (APS). Metadata common to multiple PCC frames (PCC sequences) is stored in the Sequence Parameter Set (SPS).

[0162] Each data unit and parameter set is converted to either a NAL (Network Access Layer) unit or a TLV (Type Length Value) unit, and the data stream of the unit is output.

[0163] Figure 29 is a block diagram showing an example configuration of a three-dimensional data encoding device that outputs a stream of TLV units. The three-dimensional data encoding device comprises an encoding unit 241 and a TLV storage unit 242. The encoding unit 241 generates encoded data by encoding point cloud data. The TLV storage unit 242 generates a TLV stream (a single stream of TLV units) by storing (encapsulating) the encoded data in multiple TLV units.

[0164] A unit includes a Type field indicating the data type, a Length field indicating length information, and a Value field that stores the data indicated by the Type field. Note that a unit may also be in a different format, such as one that does not include length information.

[0165] Figure 28 shows a sequence of units that store data units and parameter sets. The arrows in Figure 28 indicate dependencies related to the decoding of encoded data. The source of an arrow depends on the data at the end of the arrow, and the header of the source data shows the identifier of the data at the end of the arrow (referenced). The three-dimensional data decoding device decodes the data at the end of the arrow and uses that decoded data to decode the source data of the arrow. For example, the geometry data unit (Geom) shows the GPS ID and SPS ID corresponding to that geometry data unit. The attribute data unit (Attr) shows the APS ID corresponding to that attribute data unit.

[0166] GPS and APS may be provided one per frame. Alternatively, if the encoding scheme changes for each slice, GPS and APS may be provided for each slice. Furthermore, GPS and APS may be shared across multiple frames (sequences).

[0167] Furthermore, APS may be shared across multiple attribute information sets. The parameter set of the referenced object is sent before the referenced object.

[0168] When interframe prediction is used, or when there are dependencies between data units between frames, a Group of Frames (GOF) containing multiple frames may be constructed. A GOF is a random access unit, and the first slice of a GOF is independent of dependencies and is the slice where decoding begins. A stream may have delimiters that indicate frame boundaries or GOF boundaries, or TLV units that indicate boundaries.

[0169] Next, we will explain the structure of the encoded data and how to store the encoded data in the NAL unit.

[0170] For example, a data format is defined for each type of encoded data. Figure 30 shows an example of encoded data and a NAL unit.

[0171] For example, as shown in Figure 30, the encoded data includes a header and a payload. The encoded data may also include length information indicating the length (data volume) of the encoded data, header, or payload. The encoded data may also be in TLV format including length information. Furthermore, the encoded data does not necessarily have to include a header.

[0172] The header includes, for example, identification information to identify the data. This identification information may indicate, for example, the data type or frame number.

[0173] The header contains, for example, identification information indicating a reference relationship. This identification information is stored in the header when there is a dependency between data, and it is information used to reference the referenced data from the source. For example, the header of the referenced data contains identification information to identify that data. The header of the referenced data contains identification information indicating the referenced data.

[0174] Furthermore, if the referenced or source can be identified or derived from other information, the identifying information for identifying the data or identifying information indicating the reference relationship may be omitted.

[0175] The three-dimensional data encoding device stores the encoded data in the payload of the NAL unit. The NAL unit header contains pcc_nal_unit_type, which is the identification information for the encoded data. Figure 31 shows an example of the semantics of pcc_nal_unit_type.

[0176] As shown in Figure 31, when pcc_codec_type is codec 1 (Codec1: first encoding method), the values ​​of pcc_nal_unit_type from 0 to 10 are the encoded position data (Geometry), encoded attribute X data (AttributeX), encoded attribute Y data (AttributeY), position PS (Geom.PS), attribute XPS (AttriX.PS), attribute YPS (AttriY.PS), position SPS (Geometry Sequence PS), attribute XSPS (AttributeX Sequence PS), attribute YSPS (AttributeY Sequence PS), AU header (AU Header), GOF header (GOF It is assigned to the Header. Values ​​11 and above are also assigned to the backup of Codec 1.

[0177] The following describes modifications of this embodiment. PS has levels, such as frame-level PS, sequence-level PS, and PCC sequence-level PS. If the PCC sequence level is considered a higher level and the frame level a lower level, the following method may be used to store the parameters.

[0178] The default PS value is shown in the higher-level PS. If the value of the lower-level PS differs from the value of the higher-level PS, the PS value is shown in the lower-level PS. Alternatively, the PS value is not listed in the higher-level PS, but is listed in the lower-level PS. Alternatively, information on whether the PS value is shown in the lower-level PS, the higher-level PS, or both is shown in either the lower-level PS or the higher-level PS, or both. Alternatively, the lower-level PS may be merged with the higher-level PS. Alternatively, if the lower-level PS and the higher-level PS overlap, the 3D data encoding device may omit the transmission of one of them.

[0179] The three-dimensional data encoding device may divide the data into slices or tiles and send out the divided data. The divided data contains information for identifying the divided data, and the parameters used for decoding the divided data are included in the parameter set. In this case, pcc_nal_unit_type is defined as an identifier indicating that it is data that stores data or parameters related to tiles or slices.

[0180] Figure 32 is a block diagram showing an example of a device that generates format data from data. This device includes an encoding unit 1143 and a formatting unit 1144.

[0181] The encoding unit 1143 encodes data consisting of moving images, audio, three-dimensional data (point cloud, mesh, 3D Gaussian splatting, NeRF), neural network models, metadata, SEI, etc., according to a predetermined encoding scheme and generates encoded data. The encoding unit 1143 outputs the generated encoded data as an encoded data bitstream to the formatting unit 1144. In addition to encoded video and audio data, the encoded data may also include metadata such as parameter sets, control information, and SEI. The encoded data may be stored in an encoding unit, which is treated as a processing unit within the encoded data bitstream.

[0182] The formatting unit 1144 formats the input encoded data according to a predetermined system format and outputs it as format data. The formatting unit 1144 multiplexes the encoded data according to, for example, a file format for storage compliant with ISOBMFF or a packet format for transmission compliant with RTP, and can store multiple types of media and related metadata, such as video, audio, three-dimensional data (point cloud, mesh, 3D Gaussian splatting, NeRF), subtitles, application data, file information, reference time information, sensor information, and camera information, in the same format data as needed. The format data may be transmitted externally via input / output means equipped with a communication interface or user interface, or it may be stored in internal memory or storage means.

[0183] Figure 33 is a block diagram showing an example of a device that restores the original data from formatted data.

[0184] The reverse formatting unit 1145 analyzes the input format data according to a predetermined system format, extracts the encoded data stored in the format data, and outputs it as encoded data. The reverse formatting unit 1145 may also extract encoded data according to formats such as ISOBMFF, MPEG-DASH, MMT, AVI, MPEG-2 TS Systems, RTP, glTF, and USD.

[0185] The decoding unit 1146 decodes the encoded data input from the inverse formatting unit 1145 according to a predetermined encoding scheme and outputs data consisting of video, audio, three-dimensional data (point cloud, mesh, 3D Gaussian splatting, NeRF), neural network models, metadata, etc. The output data may be used by applications.

[0186] Figure 34 is a conceptual diagram illustrating an example of how an encoded data bitstream is stored in a system format.

[0187] As shown in this figure, the encoded data bitstream includes not only encoded video and audio data, but also supplementary information such as parameter sets and SEI. The encoded data, parameter sets, and SEI are grouped together as encoding units and treated as NAL units or data units. Examples of these encoding units include the Video NAL unit in video encoding, the TLV unit in G-PCC, and the V3C unit for storing video-based volumetric media, but other types of units and data units may also be used. These encoding units are stored in a system format compliant with ISOBMFF or a transmission format compliant with RTP. Furthermore, the metadata (parameter sets, control information, SEI) of the encoded data described in this embodiment, a part of the metadata, the syntax structure, and the header of the encoded data may be defined as a box structure. These boxes may be contained in the Movie box "moov" or in the Media Data box "mdat". If the metadata is per frame, a new metadata track may be defined and the metadata may be stored in sample entries within the metadata track. Furthermore, the encoded video and three-dimensional data described in this embodiment may also be defined as a box structure and may be stored in an mdat box as a sample or subsample. The header portion may be stored in a moov box and the payload portion in an mdat box. In addition, the SEI described in this embodiment may also be defined as a box structure and may be contained within a moov box or an mdat box. The encoded data may be stored in units, or parts of units may be extracted and stored, or the syntax structure may be modified as needed. Note that the above is an example of storing encoded data in ISOBMFF, and the same approach can be applied when storing encoded data in other formats.

[0188] Figure 35 shows an example of the box structure of ISOBMFF.

[0189] ISOBMFF is an ISO-based media file format specified in ISO / IEC 14496-12, and is a media-independent file format standard for multiplexing and storing various media such as video, audio, and text. In ISOBMFF, a file is composed of multiple boxes, each box consisting of type, length, and data. This embodiment shows an example in which a File type box "ftyp", a Movie box "moov" that stores metadata such as control information, and a Media Data box "mdat" that stores media data such as encoded data are used. The ftyp box indicates the file brand using 4CC and shows compatibility, while the moov box contains a track box that shows media-specific information, including an hdlr box indicating the media type, an stsd box indicating decoding parameters, a tref box indicating reference relationships, and an stco box indicating the data storage location. The mdat box stores samples and subsamples of encoded data corresponding to each media track in the moov box. A sample is a unit of encoded data corresponding to an access unit or a frame at the same time, and a subsample is segmented data corresponding to a part of an access unit or frame. Note that the method of storing each media in ISOBMFF is specified separately; for example, the storage method for AVC video and HEVC video is specified in ISO / IEC 14496-15, and the storage method for three-dimensional data is specified in ISO / IEC 23090-10 and ISO / IEC 23090-18.

[0190] [Gaussian Data] The following describes Gaussian data (Gaussian Splatting 3D data). First, we will explain the configuration of a system that encodes or decodes three-dimensional data generated using the Gaussian Splatting method.

[0191] Figure 36 shows a part of the processing in the three-dimensional data generation system shown in Figure 1. The sensor information input unit 301 and the three-dimensional data generation unit 302 shown in Figure 36 correspond, for example, to the sensor information acquisition unit 117 and the three-dimensional data generation unit 118 shown in Figure 1.

[0192] The sensor information input unit 301 acquires point cloud data (three-dimensional point cloud data) and outputs the acquired point cloud data to the three-dimensional data generation unit 302. The three-dimensional data generation unit 302 generates Gaussian data (Gaussian Splatting data) from the point cloud data and outputs the generated Gaussian data.

[0193] Specifically, the three-dimensional data generation unit 302 first generates a grid according to the density of the input point cloud data. Each point in the point cloud data is projected onto the nearest grid. At this time, attribute information such as color, reflectance, or normal associated with the point is also linked to the grid. The three-dimensional data generation unit 302 derives parameters such as the mean and variance of the Gaussian function from the information projected onto the grid.

[0194] The method for generating Gaussian data is not limited to the above. For example, Gaussian data may be generated by machine learning using point cloud data generated from multiple two-dimensional images using Structure from Motion (SfM), etc.

[0195] Gaussian data is a collection of multiple 3D Gaussian (ellipsoid) data. Each 3D Gaussian contains data such as three-dimensional point coordinates, a 3x3 covariance matrix, color, and transparency. It is also possible to represent the above data of a 3D Gaussian by converting it into components described in the PLY format.

[0196] Figure 37 shows the components of Gaussian data described in PLY format. The Gaussian data shown in Figure 37 includes position, orientation, scale, SH coefficient, and transmittance. Transmittance is also called transparency, clarity, or opacity (Opacity or Alpha). This format is also called the 3DGS format.

[0197] Position is the three-dimensional coordinate of a three-dimensional point. Scale and Orientation are parameters of the three-dimensional covariance matrix. Furthermore, the SH coefficient is the coefficient of the spherical harmonics when color information is represented using spherical harmonics.

[0198] In other words, Gaussian data includes a reference three-dimensional coordinate (position) and information associated with that three-dimensional coordinate (rotation, scale, SH coefficient, transparency) for each 3D Gaussian.

[0199] Note that the structure of Gaussian data is not limited to the above. For example, Gaussian data does not have to include a covariance matrix. For example, Gaussian data may include a position (Position) indicating the three-dimensional coordinates of a three-dimensional point, a scale (Scale) indicating the scale along each axis, a color (Color) indicating RGB, and a transmittance (Transparency).

[0200] Furthermore, Gaussian data may include multiple cubes (voxels) that divide 3D space, instead of multiple ellipsoids. Each voxel has attribute information. For example, each voxel may include a position indicating the three-dimensional coordinates of the voxel's center point, attribute information such as color, reflectivity, or material, and a scale indicating the size of the voxel.

[0201] Furthermore, the position may be expressed not in three-dimensional coordinates, but in two-dimensional point coordinates (x, y). In this case, the scale may indicate the size or shape of the ellipsoid on a two-dimensional plane. Specifically, the Gaussian data may include a position indicating the location on a two-dimensional plane, a scale vector indicating the scale (Su, Sv) along each axis, and rotation information indicating the orientation of the ellipsoid.

[0202] Note that the format for describing Gaussian data is not limited to the PLY format. For example, the SPZ format or other formats can also be used to describe Gaussian data.

[0203] The Gaussian data format may be 2DGS format, 3DVS (Voxel Splating) format, or STG (Space-Time Splating) format.

[0204] Figure 38 shows an example of 2DGS data. 2DGS data (Gaussian data in 2DGS format) is GS data with a two-dimensional scale value. In other words, a two-dimensional ellipse is represented by the Gaussian data. 2DGS data is particularly suitable for representing planes.

[0205] Figure 39 shows an example of 3DVS data. While 3DGS data (Gaussian data in 3DGS format) represents multiple ellipsoids, 3DVS data (Gaussian data in 3DVS format) represents multiple cubes (voxels) that divide 3D space. Furthermore, while 3DGS data represents ellipsoids using scale and orientation with position as the center, 3DVS data represents multiple cubes (voxels) that divide 3D space with position as the center.

[0206] 3DVS data includes position, SH coefficient, opacity, and octree level. The octree level indicates the hierarchy when the 3D space is divided into octrees. In other words, the voxel size can be represented using the octree level. For example, Position is represented as an octree. The maximum number of levels, which is the number of usable octree division hierarchies, is max_octree_level. The octree level indicates the level from the maximum level. Here, the maximum level is the top level of the octree and corresponds to the largest space (cube). For example, the maximum level is represented by octree_level = 0, and the minimum level (the level of the smallest cube) is represented by octree_level = max_octree_level.

[0207] In this case, the size of the voxel can be calculated using the formula Voxel size = 2^(max_octree_level - octree level). The size of the voxel represents, for example, the length of one side of a cube.

[0208] Figure 40 shows an example of STG data. STG data (Gaussian data in STG format) represents GS data from multiple frames as GS data for one frame. Like 3DGS data, STG data includes position, orientation, scale, and opacity. Furthermore, STG data includes, in addition to the elements of 3DGS data, polynomial trajectory, polynomial rotation, temporal scale, temporal opacity, base color feature, view feature, and time feature.

[0209] Polynomial trajectories show the trajectory (change) of motion over time. Polynomial rotations show the trajectory (change) of rotation over time. Time scales show the visibility width of the spacetime Gaussian over time. Time transmittance shows the transmittance (opacity) of features represented by the spacetime Gaussian over time. Basic color features show the basic colors of the spacetime Gaussian, which are independent of time. Viewpoint features show the effect of line of sight direction on the radiance of the spacetime Gaussian. Time features show the time-dependent change in the radiance of the spacetime Gaussian.

[0210] Here, the polynomial trajectory, polynomial rotation, time scale, and time transmittance are motion information representing the motion components of position, rotation, scale, and transmittance. By treating this motion information as attribute information, GS data from multiple frames can be represented in a single frame. Furthermore, the basic color features, viewpoint features, and time features are parameters that represent the color information of multiple viewpoints across multiple frames. These parameters are similar to the SH coefficients of the spherical harmonics across multiple frames. In addition, the basic color features, viewpoint features, and time features are parameters of the MLP (Multilayer Perceptron).

[0211] By combining MLP with basic color features, viewpoint features, and time features, color information for any time and viewpoint can be output. Therefore, the configuration information of the MLP is also necessary.

[0212] [Spherical Harmonics] By representing color using spherical harmonics, it is possible to represent three-dimensional color information. Figure 41 shows an example of input and output of spherical harmonics. For example, as shown in Figure 41, by inputting viewpoint information into the spherical harmonic function 304 of the color, the color information (R, G, B) as seen from that viewpoint is output.

[0213] Furthermore, increasing the level of spherical harmonics allows for a more detailed representation of color changes when the direction is altered. In other words, the resolution of the representation in various directions is improved.

[0214] Figures 42 and 43 show examples of SH coefficients for spherical harmonics. The number of elements (number of SH coefficients) in spherical harmonics is determined according to the level of resolution they represent. For example, a higher level (a larger numerical value of the level) indicates higher resolution. For example, higher levels add more information about high-frequency components.

[0215] For example, to represent level 2, the spherical harmonics have a total of nine elements (SH coefficients) from levels 0 to 2. For example, to represent level 3, the spherical harmonics have a total of sixteen elements (SH coefficients) from levels 0 to 3.

[0216] For example, the examples shown in Figures 42 and 43 illustrate the spherical harmonics of a level 3 color. Since each of the 16 elements contains the three elements of color (R, G, B), the spherical harmonics have 16 × 3 = 48 elements. It is also possible to define levels 4 and above. The elements of level 0 are sometimes called DC components, and the elements of level 1 and above are sometimes called AC components.

[0217] Furthermore, other attribute information (such as reflectance or infrared information) may be represented using spherical harmonics, not just color. In that case, the Gaussian data will include the SH coefficients of the spherical harmonics for each attribute information. Also, the level may be changed depending on the attribute information or resolution.

[0218] [Rendering Gaussian Data] Next, we will explain the rendering process for Gaussian data. By rendering Gaussian data, a 3D model or a 2D image is generated. The rendered 2D image can be displayed in applications such as viewers. In addition, rendering is sometimes used to evaluate Gaussian data when generating it. Rendering is the process of converting an entire 3D scene into a 2D image, and rendering three-dimensional data is called volume rendering.

[0219] Rendering techniques include, for example, ray tracing and rasterizing. Hereafter, rendering may also be referred to as rasterizing or rasterization.

[0220] Figure 44 shows the rendering process of Gaussian data. As shown in Figure 44, the rendering unit 303 generates a 3D model by rendering the Gaussian data. This makes it possible to display the generated 3D model in applications such as 3D display devices. A 3D display device is, for example, a device that displays a three-dimensional image in space, such as a hologram, or a glasses-free 3D display device.

[0221] Figure 45 shows the rendering process. As shown in Figure 45, the rendering unit 305 generates a 3D model or 2D image from a specified viewpoint based on Gaussian data and the specified viewpoint. The generated 3D model can be displayed in applications such as VR display. The generated 2D image can also be displayed in applications such as 2D image display devices.

[0222] [3DGS Learning and Generation] Figure 46 is a block diagram of the three-dimensional data generation unit 302. The three-dimensional data generation unit 302 includes an SFM unit 401, an initialization unit 402, a GS update unit 403, a rendering unit 404, a correct value selection unit 405, an error calculation unit 406, and a GS parameter optimization unit 407.

[0223] The SFM unit 401 generates point cloud data using Structure from Motion (SfM) based on a plurality of input 2D images. The initialization unit 402 sets the position information of the generated point cloud data as the initial value of the position coordinates of the Gaussian data. The initialization unit 402 also determines the initial values ​​of other data elements (scale, orientation, SH coefficient, and opacity) in a predetermined manner. This generates the initial Gaussian data.

[0224] The rendering unit 404 generates multiple rendered images by rendering based on multiple viewpoint information using the generated Gaussian data. The ground truth selection unit 405 selects the 2D ground truth from multiple 2D images. The error calculation unit 406 generates error information by comparing multiple rendered images with multiple 2D images using multiple 2D images, the ground truth, and multiple rendered images. The GS parameter optimization unit 407 optimizes multiple Gaussian parameters by changing the values ​​of multiple parameters in the Gaussian data based on the calculated error information. The GS update unit 403 updates the Gaussian data based on the optimized Gaussian parameters. In addition to parameter optimization, at least one of the following may also be performed: adding, deleting, merging, and splitting the Gaussian data.

[0225] Through repeated optimization of parameters and the addition and deletion of Gaussian data, the final Gaussian data is generated. The three-dimensional data generation unit may also be called the Gaussian data generation unit (GS generation unit).

[0226] [Gaussian Data Encoding and Decoding] Next, the encoding and decoding processes for Gaussian data will be described. Figure 47 is a block diagram of the encoding device (encoding system) according to this embodiment. This encoding device encodes and multiplexes Gaussian data. Figure 48 is a block diagram of the decoding device (decoding system) according to this embodiment. This decoding device decodes the Gaussian data and presents a 3D model or 2D image in the application. Note that the encoding and decoding devices shown in Figures 47 and 48 are included, for example, in the three-dimensional data generation system shown in Figure 1.

[0227] The encoding device shown in Figure 47 includes a Gaussian data generation unit 310, an encoding unit 311, and a multiplexing unit 312. The Gaussian data generation unit 310 generates Gaussian data. The encoding unit 311 generates encoded data (bitstream) by encoding the input Gaussian data using a predetermined encoding method, and outputs the generated encoded data to the multiplexing unit 312.

[0228] The multiplexing unit 312 generates multiplexed data by multiplexing the input encoded data using a predetermined multiplexing method, and outputs the generated multiplexed data. This multiplexed data is stored or transmitted.

[0229] The decoding device shown in Figure 48 includes a demultiplexing unit 321, a decoding unit 322, and an application unit 323. The demultiplexing unit 321 generates encoded data by demultiplexing the input multiplexed data using a predetermined multiplexing method, and outputs the generated encoded data to the decoding unit 322.

[0230] The decoding unit 322 generates Gaussian data by decoding the input encoded data using a predetermined encoding method (decoding method), and outputs the generated Gaussian data to the application unit 323.

[0231] The application unit 323 includes an input interface unit 324, a rendering unit 325, and a presentation unit 326. The input interface unit 324 acquires user operations. For example, based on user operations, the input interface unit 324 generates viewpoint information indicating the viewpoint entered by the user.

[0232] The rendering unit 325 generates a three-dimensional model or a two-dimensional image by rendering the input Gaussian data. For example, the rendering unit 325 generates a 3D model or a 2D image viewed from a viewpoint entered by the user, indicated by the viewpoint information. The rendering unit 325 may also generate a 3D model or a 2D image viewed from a predetermined viewpoint.

[0233] The presentation unit 326 presents (displays) the three-dimensional model or two-dimensional image generated by the rendering unit 325. Whether to present a 3D model or a 2D image may be determined by, for example, the following method. For example, the rendering unit 325 first generates a three-dimensional model and generates point coordinate information, as shown in Figure 44. After viewpoint information is input, the rendering unit 325 generates a 2D image viewed from a specified viewpoint. At that time, the rendering unit 325 may also accept information specifying a region along with the specified viewpoint. In this way, by generating a 3D model first and then generating a 2D image of the specified region, the amount of processing can be reduced compared to generating a 2D image of all regions. Therefore, the rendering speed can be increased. Alternatively, whether to present a 3D display (3D model) or a 2D image may be determined according to the processing power of the device performing the rendering. For example, in the case of a device with low processing power, the device may not generate a 2D image and may present a 3D model.

[0234] [Gaussian Data Encoding Using G-PCC] Figure 49 is a block diagram showing an example configuration of the encoding device 330 (three-dimensional data encoding device) according to this embodiment. The encoding device 330 generates a bitstream (encoded data) by encoding the first Gaussian data.

[0235] The encoding device 330 includes a first preprocessing unit 331, a second preprocessing unit 332, and a G-PCC encoding unit 333.

[0236] The first Gaussian data includes, for example, a covariance matrix. The first preprocessing unit 331 converts the first Gaussian data into second Gaussian data composed of components of a ply format. For example, the components of a ply file include three-dimensional coordinates, scale, rotation, SH coefficients of spherical harmonics, and transmittance, as shown in Figure 37.

[0237] In this example, first Gaussian data is input to the encoding device 330, and the encoding device 330 converts the first Gaussian data into second Gaussian data. However, second Gaussian data may also be input to the encoding device 330. In this case, the encoding device 330 does not need to include the first preprocessing unit 331.

[0238] The second preprocessing unit 332 converts the multiple components included in the second Gaussian data into a format that can be encoded using the encoding scheme corresponding to each component, and outputs the converted multiple components to the G-PCC encoding unit 333.

[0239] The G-PCC coding unit 333 encodes the three-dimensional coordinates using a geometry (position information) coding method (such as an octave coding method, a predictive tree coding method, or a TriSoup method) within the G-PCC coding method (Geometry-based PCC). The G-PCC coding unit 333 encodes information associated with the three-dimensional coordinates, such as rotation, scale, SH coefficient, and transmittance, using an attribute coding method (such as an LoD-based coding method (LoD attribute information coding method) or a Transform-based coding method (transformation attribute information coding method)) within the G-PCC coding method.

[0240] The G-PCC coding unit 333 includes a position information coding unit 341, a rotation coding unit 342, a scale coding unit 343, an SH coefficient coding unit 344, a transmittance coding unit 345, a metadata coding unit 346, and a multiplexing unit 347.

[0241] The position information encoding unit 341 encodes three-dimensional coordinates. These three-dimensional coordinates are reference or representative three-dimensional coordinates for the Gaussian data. For example, the three-dimensional coordinates may be the center coordinates of an ellipse that constitutes the Gaussian data, or they may be other coordinates. For example, the three-dimensional coordinates may be the origin coordinates of the grid used to generate the Gaussian data.

[0242] Furthermore, the three-dimensional coordinates are not processed by the first preprocessing unit 331, but are directly input to the second preprocessing unit 332. The second preprocessing unit 332 converts the input three-dimensional coordinates into positive integer information (three-dimensional coordinates). The scale value and offset value used during the conversion are stored in metadata such as SPS and notified to the decoding device.

[0243] The position information encoding unit 341 generates encoded data by encoding the three-dimensional coordinates output from the second preprocessing unit 332 using the G-PCC geometry encoding method. This encoded data is stored in a GDU (Geometry Data Unit) and output. The metadata necessary for decoding the GDU is stored in a GPS (Geometry Parameter Set) and output.

[0244] Furthermore, if the multiple three-dimensional coordinates of the multiple Gaussian data are sparse (density is below a predetermined threshold), the location information coding unit 341 may use a predictive tree coding scheme suitable for coding sparse three-dimensional points. Also, if the multiple three-dimensional coordinates of the multiple Gaussian data are dense (density is above a predetermined threshold), the location information coding unit 341 may use an octave tree coding scheme suitable for coding dense three-dimensional points.

[0245] The first preprocessor 331 converts the covariance matrix contained in the first Gaussian data into rotation values ​​and scale values. The first preprocessor 331 converts the color information contained in the first Gaussian data into spherical harmonics and outputs the SH coefficients in the spherical harmonics.

[0246] The second preprocessor 332 converts the data type of each data contained in the second Gaussian data. Specifically, the second preprocessor 332 converts each data into a positive integer type in order to encode it using the G-PCC coding scheme. For example, if the data is of type float, the second preprocessor 332 converts the data into a positive integer type by performing scaling and offset processing on the data. At this time, the scale value and offset value for each attribute information used in the conversion are stored in metadata such as SPS and notified to the decoding device.

[0247] Furthermore, the second preprocessing unit 332 performs format conversion of each data and mapping processing to map each data to attribute components. For example, if the maximum number of elements (dimensions) per attribute component that can be encoded with the G-PCC attribute coding scheme is 3, the second preprocessing unit 332 converts the input components into attribute components that have three dimensions (three elements) per attribute component. For example, since Scale has three elements, it is output as a single attribute component. Orientation has four elements, so it is converted into two two-dimensional attribute components. The number of elements in the SH coefficient changes for each level. For example, SH coefficients with a level exceeding 3 are converted into multiple attribute components.

[0248] In this process, the second preprocessing unit 332 generates mapping information as metadata for each attribute piece of information, indicating which attribute component each element is mapped to.

[0249] The rotation coding unit 342, scale coding unit 343, SH coefficient coding unit 344, and transmittance coding unit 345 each generate multiple coded data by coding rotation, scale, SH coefficient, and transmittance using the attribute coding method in the G-PCC coding scheme. Each of the generated coded data is stored in an ADU (Attribute Data Unit) and output. The metadata necessary for decoding the ADU is stored in an APS (Attribute Parameter Set) and output. In addition, attribute information is coded using three-dimensional coordinates.

[0250] The metadata encoding unit 346 stores conversion information, including conversion parameters used by the first preprocessing unit 331 or the second preprocessing unit 332, in metadata such as SPS or SEI. The metadata encoding unit 346 may also encode the conversion information before storing it in the metadata. The metadata may also include information on the structure of the Gaussian data, or information indicating how the Gaussian data is transmitted using G-PCC encoding.

[0251] The multiplexing unit 347 stores the encoded data (ADU and GDU) and parameter sets (APS, GPS, and SPS) in the TLV unit and transmits them as a bitstream (encoded data). The bitstream may be multiplexed using a predetermined multiplexing scheme. The bitstream may also be formatted.

[0252] [Decoding Gaussian Data Using G-PCC] Figure 50 is a block diagram showing an example configuration of a decoding device 350 (three-dimensional data decoding device) according to this embodiment. The decoding device 350 generates first Gaussian data or second Gaussian data by decoding a bitstream (encoded data). For example, the decoding device 350 decodes a bitstream generated by the encoding device 330 shown in Figure 49.

[0253] The decoding device 350 includes a G-PCC decoding unit 351, a second post-processing unit 352, and a first post-processing unit 353.

[0254] The G-PCC decoding unit 351 generates three-dimensional coordinates, rotation, scale, SH coefficient, and transmittance by decoding the bitstream using the G-PCC coding scheme. The G-PCC decoding unit 351 comprises a demultiplexing unit 361, a position information decoding unit 362, a rotation decoding unit 363, a scale decoding unit 364, an SH coefficient decoding unit 365, a transmittance decoding unit 366, and a metadata decoding unit 367.

[0255] The demultiplexing unit 361 analyzes multiple TLV units contained in the input bitstream and generates data units of encoded data such as GDU, ADU, SPS, GPS, and APS. The position information decoding unit 362, rotation decoding unit 363, scale decoding unit 364, SH coefficient decoding unit 365, transmittance decoding unit 366, and metadata decoding unit 367 decode these data units using the information in the parameter set, with each attribute component using its respective encoding scheme.

[0256] The position information decoding unit 362 decodes three-dimensional coordinates from the GDU. The rotation decoding unit 363, scale decoding unit 364, SH coefficient decoding unit 365, and transmittance decoding unit 366 decode attribute information (rotation, scale, SH coefficient, and transmittance) from the ADU. The attribute information is decoded using three-dimensional coordinates. The metadata decoding unit 367 obtains conversion information, etc., from metadata such as SPS or SEI. The metadata decoding unit 367 may also decode the conversion information from the metadata. The metadata may also include information on the composition of the Gaussian data, or information indicating the correspondence of how the Gaussian data is transmitted using G-PCC coding.

[0257] The second post-processing unit 352 generates second Gaussian data by performing an inverse transformation on the decoded data. Here, the inverse transformation is the reverse process of the transformation process performed in the second pre-processing unit 332, and is performed based on the transformation information (transformation parameters) contained in the metadata.

[0258] The first post-processing unit 353 generates the first Gaussian data by inversely transforming the second Gaussian data. Here, the inverse transformation is the reverse process of the transformation process performed by the first pre-processing unit 331. At least one of the first Gaussian data and the second Gaussian data is output. Note that the decoding device 350 does not necessarily have to include the first post-processing unit 353.

[0259] [Mapping Process] The second preprocessing unit 332 maps the Gaussian data to the attribute components of the G-PCC. The method of this mapping and the mapping information generated at that time will be explained below.

[0260] The following examples illustrate the mapping of rotation, scale, and SH coefficient, omitting the explanation of transmittance. Similar processing can be applied when other attribute information is added.

[0261] Figure 51 shows an example of mapping multiple attribute components contained in Gaussian data to attribute components of the coding scheme G-PCC. Here, we show an example where the maximum number of dimensions (number of subcomponents) that can be coded in the coding of attribute components in G-PCC is 3.

[0262] In the following, attribute components of Gaussian data may be referred to as Gaussian components, and attribute components of G-PCC may be referred to as G-PCC components. Additionally, attribute components may sometimes be simply referred to as components.

[0263] For example, the Gaussian component of the scale has three subcomponents (B1 to B3) with three dimensions, and is mapped to a single G-PCC component that has three-dimensional subcomponents. In this case, the mapping information indicates that the identifier of the Gaussian component of the scale corresponds to the identifier of the G-PCC component.

[0264] Furthermore, the Gaussian component of rotation has four subcomponents (A1 to A4) with a dimension of 4, and cannot be mapped to a single G-PCC component that has three-dimensional subcomponents. Therefore, the four subcomponents of rotation (A1 to A4) are mapped to two G-PCC components that have two-dimensional subcomponents. In this case, the mapping information indicates that the identifier of the Gaussian component of rotation and the identifier indicating the subcomponent number contained in that attribute component correspond to the identifier of the G-PCC component and the identifier indicating the subcomponent number contained in that G-PCC component.

[0265] Since there are 16 SH coefficients for each level, corresponding to three-dimensional (RGB) elements, the SH coefficients have a total of 48 dimensions (C1 to C48). These 48 dimensions are mapped to 16 G-PCC components, each having three-dimensional subcomponents.

[0266] In Figure 51, "ID" is the attribute component ID, which is an identification number used to identify the attribute component of the encoded data. "ND" is the number of dimensions, which indicates the number of elements (number of subcomponents) per attribute component.

[0267] Furthermore, the method for partitioning and mapping Gaussian data is not limited to the examples above; any combination of mapping method and number of dimensions may be used.

[0268] The second preprocessor unit 332 generates mapping information in the mapping process that indicates the G-PCC component corresponding to the Gaussian component. This mapping information is stored as metadata in the header or SEI.

[0269] The decoding device decodes mapping information from the bitstream. The second post-processing unit 352 reconstructs the Gaussian data by remapping the decoded data for each G-PCC component to Gaussian data based on the mapping information.

[0270] Figure 52 shows an example of mapping information. In Figure 52, the mapping information is stored in SEI (Gaussian Data Attribute Mapping Information SEI). The mapping information includes the number of Gaussian data components. The number of Gaussian data components indicates the number of attribute information included in the Gaussian data. For example, in the example shown in Figure 51, rotation, scale, and SH coefficient are present, and the number of Gaussian data components is 3.

[0271] Furthermore, the mapping information includes, for each Gaussian component, the Gaussian data component ID, the Gaussian data type, and the number of Gaussian data dimensions.

[0272] The Gaussian component ID is an identifier used to uniquely identify a Gaussian component. The Gaussian data type indicates the type of Gaussian component. For example, a value of 0 indicates "rotation," a value of 1 indicates "scale," a value of 2 indicates "SH coefficient," and a value of 3 indicates "transparency." Note that the combinations of values ​​and types shown here are just examples, and the combinations are not limited to these.

[0273] The Gaussian data dimension indicates the number of dimensions (subcomponents) of the Gaussian components. For example, in the example shown in Figure 51, the Gaussian data dimension for rotation is 4, the Gaussian data dimension for scale is 3, and the Gaussian data dimension for SH coefficients is 48.

[0274] Furthermore, the mapping information includes G-PCC component IDs, G-PCC dimension IDs, and transformation information as information for each dimension of the Gaussian component. The G-PCC subcomponent ID corresponds to the "ID" shown in Figure 51. The G-PCC dimension ID is an identifier indicating the subcomponent number. For example, C1 in the SH coefficient shown in Figure 51 has a G-PCC dimension ID of 0, C2 has a G-PCC dimension ID of 1, and C3 has a G-PCC dimension ID of 2.

[0275] Thus, the mapping information shows the ID and dimension ID of the G-PCC component corresponding to each dimension of each component of the Gaussian data. For example, for subcomponent C46 shown in Figure 51, the G-PCC component ID = 18 and the G-PCC dimension ID = 2 are shown.

[0276] In Figure 52, an example of mapping information showing the subcomponents (dimensions) of the G-PCC corresponding to each subcomponent (dimension) of the Gaussian data is shown. However, mapping information showing the subcomponents of the Gaussian data corresponding to each subcomponent of the G-PCC may also be used.

[0277] Furthermore, while Figure 52 shows an example of mapping information indicating the correspondence between subcomponents of Gaussian data and subcomponents of G-PCC, if there are no constraints on the number of subcomponents in the G-PCC component, mapping information indicating the correspondence between the Gaussian component and the G-PCC component may be used. In other words, it is not necessary to show the correspondence between dimensions.

[0278] The conversion information shows the parameters (conversion parameters) used when converting the component data. Figure 53 shows an example of the configuration of the conversion information. For example, if a scale and an offset were used in the conversion, the conversion information would include the scale value and the offset value.

[0279] For example, the transformation is performed using the scale value and offset value according to the following formula.

[0280] Converted data = Original data × Scale value + Offset

[0281] The formula used for the conversion may be a predefined formula, or multiple conversion formulas may be defined and the formula used may be switched according to predetermined conditions. Furthermore, information indicating the used conversion formula may be stored in the bitstream.

[0282] In Figure 52, transformation information is provided for each subcomponent (dimension) of the Gaussian data, but transformation information may also be provided for each component of the Gaussian data. In other words, the transformation method may be changed on a component-by-component basis, or it may be changed on a subcomponent basis.

[0283] Note that some of the syntax shown in Figure 52 may be omitted. For example, if the order of listing information for rotation, scale, and SH coefficients, and the number of dimensions are predetermined, the Gaussian data component ID, Gaussian data type, and Gaussian data dimension may be omitted.

[0284] Furthermore, each attribute piece of information is encoded based on the mapping information to the attribute components determined by the second preprocessing unit 332. Figure 54 shows an example of the structure of encoded data when Gaussian data is encoded using G-PCC encoding.

[0285] The SPS contains, for each G-PCC component, a G-PCC attribute component ID (attr_id), a G-PCC attribute type (attribute_type), and the number of dimensions of the G-PCC attribute component (num_dimension). Here, a new G-PCC attribute type may be defined to indicate that each element of the Gaussian data is encoded.

[0286] Figure 55 shows an example of G-PCC attribute types for Gaussian data. As shown in Figure 55, for example, G-PCC attribute types may be defined that represent the rotation, scale, SH coefficient, and transmittance of Gaussian data, respectively.

[0287] The ADU header is assigned a G-PCC attribute component ID (attr_id). The decryption device can use the G-PCC attribute component ID contained in the ADU header and the information contained in the SPS to identify which G-PCC attribute type each ADU belongs to.

[0288] Furthermore, mapping information (Gaussian data attribute mapping information SEI) may be transmitted as SEI or included in SPS. For example, SPS may include mapping information indicating the Gaussian component corresponding to the G-PCC component.

[0289] Figure 56 is a flowchart of the processing performed by the decoding device according to this embodiment. First, the decoding device acquires multiple TLV units from the bitstream, and acquires multiple data units (GDU, ADU) and multiple parameter sets (SPS, GPS, APS) from the multiple TLV units (S101).

[0290] Next, the decoding device decodes the three-dimensional coordinates using SPS, GPS, and GDU (S102). Next, the decoding device decodes the multiple attribute components described in SPS using APS, ADU, and the three-dimensional coordinates (S103).

[0291] Next, the decoding device obtains mapping information by analyzing the Gaussian data attribute mapping information SEI (S104). Next, the decoding device reconstructs the second Gaussian data from the decoded attribute components and mapping information (S105). Next, the decoding device obtains transformation information and generates the first Gaussian data by inversely transforming the second Gaussian data using the transformation information (S106).

[0292] [Hierarchical Encoding] GS data includes various attribute information such as scale and rotation, which indicate the shape of the ellipse, the SH (Spherical Harmonics) coefficients of the spherical harmonics, and transparency.

[0293] While the SH coefficients of spherical harmonics can represent colors in various directions, they have a large number of components and a large amount of data. This presents a challenge in that decoding, processing, and rendering the SH coefficients takes a considerable amount of time.

[0294] Furthermore, GS data has the advantage of rendering photorealistic images in applications that display various colors by changing the orientation. On the other hand, even in applications that do not require high quality, such as applications that use GS data to display thumbnails without changing the orientation, decoding, processing, and rendering can be time-consuming.

[0295] In this embodiment, the attribute information components of GS data (hereinafter also referred to as attribute components) are layered into a base layer (basic layer) and an extended layer (extended layer). The base layer (Layer 0) is a group of attribute components for achieving minimum quality. The extended layers (Layer 1 and above) are groups of attribute components for achieving high quality by being added to lower layers. The data generation device generates metadata (metadata for partial decoding of attribute components) that shows the hierarchical structure of the attribute components, encodes the metadata, and transmits it.

[0296] In Embodiment 1, described later, the data generation device stores hierarchical information of attribute components in metadata. The decoding unit decodes the GS data and outputs a portion of the hierarchical GS data based on the metadata indicating the hierarchical structure of the attribute components. This reduces the amount of rendering processing required and shortens the rendering time.

[0297] Furthermore, in embodiments 2 and 3, the data generation device stores hierarchical information of attribute components and attribute component IDs for encoding (G-PCC) in the metadata. The decoding unit obtains encoded data of the desired layer from the encoded data based on the metadata indicating the hierarchical structure of attribute components, and decodes the obtained encoded data. In other words, the decoding unit partially decodes a portion of the GS data. The decoding unit outputs the decoded portion of the GS data, and this portion of the GS data is rendered. This makes partial decoding possible, where only a portion of the GS data is decoded. In addition, the processing load of the decoding unit and the rendering unit can be reduced, and the decoding time and rendering time can be shortened.

[0298] (Embodiment 1) Figure 57 is a block diagram showing the configuration of a data processing system according to this embodiment. The data processing system shown in Figure 57 includes a hierarchical structure determination unit 501, an encoding unit 502, a decoding unit 503, and a rendering unit 504. The hierarchical structure determination unit 501 and the encoding unit 502 are included, for example, in an encoding device (data generation device) that encodes GS data. The decoding unit 503 and the rendering unit 504 are included in a data processing device that processes GS data. Note that the decoding unit 503 and the rendering unit 504 may be included in different devices. For example, the decoding unit 503 may be included in a decoding device, and the rendering unit 504 may be included in a rendering device.

[0299] Figure 58 is a flowchart of the encoding process (hierarchical processing of attribute components) in the encoding device. First, the encoding device acquires GS data (S201). The hierarchical structure determination unit 501 hierarchizes the GS data (S202). In other words, the hierarchical structure determination unit 501 determines the hierarchical level. The hierarchical structure determination unit 501 also generates hierarchical structure metadata (S203). The encoding unit 502 generates encoded data by encoding the GS data and hierarchical structure metadata (S204).

[0300] The decoding unit 503 generates GS data and hierarchical structure metadata by decoding the encoded data. The rendering unit 504 generates a rendered image by rendering using part or all of the decoded GS data.

[0301] Next, the hierarchical structure of attribute components will be explained. The hierarchical structure determination unit 501 hierarchically structures multiple attribute components (scale, rotation, SH coefficient, transmittance, etc.) included in the GS data into multiple layers.

[0302] First, let's explain the hierarchical structure of spherical harmonics. The hierarchical structure determination unit 501 performs hierarchical structure by grouping multiple levels of the SH coefficients of the spherical harmonics. In other words, multiple levels are classified into multiple groups (hierarchical levels). The levels included in one hierarchical level are one or more consecutive levels. Hierarchical level 0 includes the SH coefficients of level 0. Hierarchical level 0 is also called the base layer. Grouping is performed sequentially from level 0 to form the hierarchical levels. Hierarchical levels that do not include the SH coefficients of level 0 are called extended layers.

[0303] Figure 59 shows an example of hierarchical structuring of SH coefficients. In this example, two hierarchical levels are set: a base layer and one extension layer. Hierarchical level 0 includes the SH coefficients of level 0. Hierarchical level 1 includes the SH coefficients of levels 1 to 3. In this case, a data processing device (rendering device, etc.) can represent the color resolution up to level 0 by using the SH coefficients up to hierarchical level 0. Furthermore, the data processing device can represent the color resolution from levels 0 to 3 by using the SH coefficients of hierarchical level 0 and hierarchical level 1. In other words, color is represented by combining hierarchical level 1 with the lower level, hierarchical level 0.

[0304] In this case, if color is represented using only level 0, the color is represented by three elements: f_dc, f_dc1, and f_dc2. This is equivalent to representing color with normal monochromatic RGB. Furthermore, a Gaussian (ellipsoid) has the same color regardless of the viewing direction, and there is no change in color depending on the direction. In other words, the level 0 elements can be said to be the DC component in terms of directional variation.

[0305] On the other hand, when color is represented using levels 0 through 3, the color changes with changes in direction. Therefore, it can be said that the resolution to changes in direction is high. In this way, the resolution to direction can be changed by changing the levels used in the SH coefficient. Furthermore, hierarchical structuring generates data with different resolutions to direction.

[0306] Note that the number of layers is not limited to two; it can be three or more. For example, layers 0, 1, and 2 may be set. In this case, the SH coefficient of layer 0 can represent a color on its own. Color is represented when the SH coefficient of layer 1 and the SH coefficient of layer 0 are used together. In other words, layer 1 depends on layer 0. Color is represented when the SH coefficient of layer 2, the SH coefficient of layer 1, and the SH coefficient of layer 0 are used together. In other words, layer 2 depends on layer 1.

[0307] In this example, we show a case where hierarchy level 0 is the base layer containing level 0, but the base layer does not necessarily have to be the 0th hierarchy level. In this case, information indicating the base layer may be included in the encoded data. Also, a flag indicating whether or not the hierarchy level contains level 0 (i.e., is the base layer) may be included in the encoded data.

[0308] Furthermore, an extended layer may be defined as a group of levels that include lower hierarchy levels. That is, level 0 may be set to hierarchy level 0, and levels 0 to 3 may be set to hierarchy level 1. Note that the following explanation describes an example where the three elements of the SH coefficient are RGB, but YUV (4:4:4, 4:2:2, 4:2:0, etc.) may be used instead of RGB.

[0309] Next, we will explain the hierarchical structure of the STG format. First, we will explain an example of the hierarchical structure of attribute components related to color. Figure 60 shows an example of the hierarchical structure of MLP parameters in the STG format. As shown in Figure 60, for example, the Base color feature is included in the base layer at hierarchy level 0. The View feature is included in the extended layer at hierarchy level 1, and the Time feature is included in the extended layer at hierarchy level 2.

[0310] Here, the data processing device can represent the base color by using the Base color feature alone. The data processing device can represent the color of an arbitrary viewpoint by using the Base color feature and the View feature. The data processing device can represent the color of an arbitrary time by using the Base color feature and the Time feature. The data processing device can represent the color of an arbitrary viewpoint at an arbitrary time by using the Base color feature, the View feature, and the Time feature.

[0311] Here, the Base color feature, which is at hierarchy level 0, can represent colors independently. On the other hand, hierarchy levels 1 and 2 cannot be represented independently and are used in combination with the base layer. That is, the attribute components of hierarchy levels 1 and 2 each depend on the attribute components of hierarchy level 0.

[0312] Next, we will explain an example of the hierarchical structure of other attribute components. In the STG format, Polynomial Trajectory represents the time trajectory of position (position coordinates), Polynomial Rotation represents the time trajectory of rotation, Temporal Scale represents the time change of scale, and Temporal Opacity represents the time change of opacity.

[0313] Figure 61 shows an example of the hierarchical arrangement of temporal components in the STG format. As shown in Figure 61, for example, position, orientation, scale, and opacity are included in the base layer at hierarchy level 0. Polynomial Trajectory, Polynomial Rotation, Temporal Scale, and Temporal Opacity are included in the extended layer at hierarchy level 1.

[0314] The data processing device can represent GS data at a given time using position, orientation, scale, and opacity. Furthermore, the data processing device can represent GS data that has changed over time from the above time by using Polynomial Trajectory, Polynomial Rotation, Temporal Scale, and Temporal Opacity.

[0315] To rephrase the above examples of hierarchical structures, in GS data, position (location coordinates) and the element of its temporal change are hierarchically structured. Also, components indicating the shape corresponding to the location coordinates (scale and rotation) and their temporal change elements are hierarchically structured. Attribute information such as color and its temporal change elements are hierarchically structured. Attribute information such as color and its viewpoint change elements are hierarchically structured. Position and components indicating the shape corresponding to the location coordinates (scale and rotation) are hierarchically structured.

[0316] Furthermore, a hierarchical structure that includes both time-dependent and viewpoint-dependent changes may be implemented. For example, one extended layer may include elements representing the change in the SH coefficient for each viewpoint, as well as elements representing the time-dependent change of the elements for each viewpoint.

[0317] The base layer (hierarchy level 0) can constitute GS data on its own. On the other hand, the extension layer (hierarchy level 1) cannot represent GS data independently and is used in combination with the base layer. In other words, one or more attribute components of hierarchy level 1 each depend on attribute components of hierarchy level 0.

[0318] Next, we will describe an example of generating hierarchical metadata that shows the hierarchical structure of attribute components. Hierarchical metadata contains metadata about the hierarchical structure of attribute components in GS data.

[0319] Figure 62 is a table showing information about attribute components included in GS data in 3DGS format. Here, the attribute components of the GS data (hereinafter also referred to as GS components) include scale, rotation, opacity, and SH coefficient. The SH coefficient includes the GS component of the SH coefficient for each level. Note that position may also be included in this table.

[0320] Furthermore, each GS component is assigned a gaussian_component_layer_id that indicates its layer in the hierarchical structure. In the example shown in Figure 62, the level 3 SH coefficient is included in the extended layer, and the level 3 SH coefficient is assigned a gaussian_component_layer_id of 1. The other GS components are included in the base layer, and these GS components are assigned a gaussian_component_layer_id of 0.

[0321] Figure 63 shows the syntax of the hierarchical metadata. This syntax represents the information in the table shown in Figure 62. The hierarchical metadata includes GS_component_info, which is information about the GS components. GS_component_info includes component_layer_present_flag, num_gaussian_component, gaussian_component_id, gaussian_component_type, and gaussian_component_layer_id.

[0322] The `component_layer_present_flag` flag indicates whether or not hierarchical information (`gaussian_component_layer_id`) exists. For example, a value of 1 indicates that hierarchical information exists, and a value of 0 indicates that hierarchical information does not exist.

[0323] `num_gaussian_component` indicates the number of GS components. `gaussian_component_id` indicates the ID (identifier) ​​of the GS component. Note that `gaussian_component_id` does not necessarily have to be included in the hierarchical structure metadata. In this case, the loop order (i) of `num_gaussian_component` may be used as the ID.

[0324] `gaussian_component_type` indicates the type of GS component. For example, a value of 0 indicates scale, a value of 1 indicates rotation, a value of 2 indicates opacity, a value of 3 indicates SH (Level 0), a value of 4 indicates SH (Level 2), and a value of 5 indicates SH (Level 3). In other words, the type of GS component is defined for each hierarchical group.

[0325] `gaussian_component_layer_id` indicates the hierarchical level of a GS component. Note that `gaussian_component_layer_id` may be assigned to all GS components, or to specific GS components (e.g., only the SH coefficient). If there are two levels, a binary flag may be used instead of an ID. For example, `gaussian_component_layer_id` is included in the hierarchical structure metadata when `component_layer_present_flag=1`, and not included when `layer_present_flag=0`.

[0326] While this example shows a hierarchy level assigned to each GS component, a hierarchy level may also be assigned to each element of the GS component. Furthermore, the names of hierarchy levels are not limited to these. A hierarchy level may also be called a layer group or component group, which groups together multiple layers. In that case, a syntax name such as layer_group_id or component_group_id may be used instead of gaussian_component_layer_id.

[0327] Next, the decoding process for decoding the encoded data generated as described above will be explained. Figure 64 is a block diagram of the decoding unit 503. The decoding unit 503 shown in Figure 64 comprises a decoding processing unit 511, a GS data restoration unit 512, and a hierarchical extraction unit 513.

[0328] Figure 65 is a flowchart of the decoding process performed by the decoding unit 503. First, the decoding processing unit 511 decodes the encoded data of the GS data to obtain the attribute components of multiple G-PCCs (hereinafter also referred to as G-PCC components), attribute mapping information, and hierarchical structure metadata. Note that the attribute mapping information and hierarchical structure metadata may be obtained by decoding metadata (such as SEI) instead of the encoded data.

[0329] Here, the G-PCC component is generated by encoding the GS component. The mapping information shows the correspondence between multiple G-PCC components and multiple GS components.

[0330] The GS data recovery unit 512 recovers the GS data by converting multiple G-PCC components into multiple GS components based on attribute mapping information (S212).

[0331] The hierarchical extraction unit 513 extracts GS data of the desired hierarchical level from the restored GS data based on the hierarchical structure metadata (S213). For example, when the hierarchical structure metadata shown in Figure 63 is used, if the application or decoding unit 503 determines to extract only hierarchical level 0, the hierarchical extraction unit 513 extracts scale, rotation, opacity, SH coefficient (Level 0), SH coefficient (Level 1), and SH coefficient (Level 2) from among the multiple GS components included in the GS data, but does not extract SH coefficient (Level 3). The hierarchical extraction unit 513 also outputs GS data containing the extracted multiple GS components (some of the GS components) (S214).

[0332] This allows the decoding unit 503 to acquire GS components at some hierarchical levels, reducing the processing time and time required for output to subsequent stages and rendering.

[0333] Next, we will explain an example of showing hierarchical level dependencies (references). The syntax may include information indicating hierarchical level dependencies. Figure 66 is a diagram showing an example of a table for GS component information. The table shown in Figure 66 has the same hierarchical structure as the table shown in Figure 62. The SH (Level 3) component of the extension layer, which is hierarchy level 1, depends on the SH (Level 2) component.

[0334] Furthermore, the table shown in Figure 66 has `gaussian_component_id` and `reference_component_id` added to the table shown in Figure 62. `gaussian_component_id` indicates the ID (identifier) ​​of the GS component.

[0335] Figure 67 shows the syntax of the hierarchical metadata. This syntax represents the information in the table shown in Figure 66. The hierarchical metadata shown in Figure 67 includes GS_component_info, which is information about the GS component. GS_component_info further includes reference_component_id, in addition to the hierarchical metadata shown in Figure 63.

[0336] The reference_component_id indicates the ID (gaussian_component_id) of the GS component that the extension layer's GS component depends on (references). In the example shown in Figure 66, the reference_component_id of SH (Level 3) is 5, which is the GS component ID of SH (Level 2).

[0337] For example, `reference_component_id` is assigned to each GS component. Furthermore, `reference_component_id` is assigned to GS components in the extension layer if the GS component's ID (`gaussian_component_id`) is not 0. Conversely, `reference_component_id` is not assigned to GS components in the base layer if the GS component's ID is 0.

[0338] Next, we will explain an example of using the above technology. In some cases, data arriving at the decoding unit may be incomplete due to packet loss during communication, or the decoding process may fail, resulting in no data being output from the decoding unit. For example, SH(Level 3) depends on the GS component of SH(Level 2). In other words, without SH(Level 2), data cannot be correctly generated using SH(Level 3). Therefore, if the GS component of SH(Level 2) with GS component ID = 5 is not output correctly, SH(Level 3) is not necessary.

[0339] When the data processing device decides whether or not to extract the SH (Level 3) hierarchy, it checks whether or not a GS component corresponding to reference_component_id exists, which is output from the decoding unit. If a GS component corresponding to reference_component_id does not exist, the data processing device may decide not to extract the SH (Level 3) hierarchy.

[0340] In this example, the reference_component_id, which indicates the ID of the referenced GS component, is used. However, the dependency (referenced object) may also be indicated by the ID of the referenced hierarchical level (gaussian_component_layer_id).

[0341] The following describes some variations. In the syntax shown in Figure 67, hierarchical information is stored in a table of information about GS components, but the information about GS components and the hierarchical information may be stored in separate metadata.

[0342] Figure 68 shows an example of GS component information in this case. Figure 69 shows an example of hierarchical metadata. The GS component information (GS_component_info) shown in Figure 68 includes information indicating the configuration of the GS component. Specifically, the GS component information includes gaussian_component_id and gaussian_component_type corresponding to the GS component. The meaning of each piece of information is the same as in Figure 67.

[0343] Furthermore, as shown in Figure 69, the hierarchical structure metadata includes GS_layer_info, which is information about the hierarchy of GS data. GS_layer_info includes gaussian_component_id, which identifies the GS component for each layer. The two metadata (GS component information and hierarchical structure metadata) are linked by gaussian_component_id. This yields the same method and effect as the syntax shown in Figure 67.

[0344] (Embodiment 2) This embodiment describes a method for partially decoding attribute components of a certain hierarchical level from hierarchical GS data. The encoding unit stores information in the hierarchical structure metadata (hierarchical structure information) that shows the correspondence between the hierarchical level and the G-PCC component ID, which is the ID of the G-PCC component. The encoding unit generates encoded data by encoding the GS data. The encoding unit also stores the G-PCC component ID in the header of the unit that stores the encoded data.

[0345] The decoding unit generates GS data by decoding the encoded data. Figure 70 is a block diagram of the decoding unit 503A according to Embodiment 2. The decoding unit 503A includes a metadata decoding unit 521, a partial decoding unit 522, and a GS data restoration unit 523.

[0346] The metadata decoding unit 521 decodes the metadata contained in the encoded data to obtain attribute mapping information and hierarchical structure metadata. The partial decoding unit 522 analyzes the hierarchical structure based on the hierarchical structure metadata. The partial decoding unit 522 determines the hierarchical level to be decoded and obtains the G-PCC component ID (G-PCC_attr_id) belonging to that hierarchical level. The partial decoding unit 522 extracts some encoded G-PCC components corresponding to the desired hierarchical level from a plurality of encoded G-PCC components contained in the encoded data, based on the G-PCC component ID. The partial decoding unit 522 generates multiple G-PCC components by decoding the extracted subset of encoded G-PCC components. The partial decoding unit 522 does not decode encoded G-PCC components other than those of the desired hierarchical level.

[0347] The GS data recovery unit 523 recovers a portion of the GS data by converting multiple G-PCC components (some of the G-PCC components) generated by the partial decoding unit 522 into multiple GS components based on attribute mapping information.

[0348] Next, an example of generating hierarchical structure metadata showing the hierarchical structure of attribute components in this embodiment will be described. In this embodiment, the hierarchical structure metadata includes metadata relating to the hierarchical structure of G-PCC components. The hierarchical structure metadata shows the correspondence between G-PCC components (G-PCC_attri_id) and G-PCC_component_layer_id.

[0349] Figure 71 is an example of a table showing the correspondence between G-PCC components and G-PCC_component_layer_id. As shown in Figure 71, G-PCC components (G-PCC_attr_id (G-PCC_attribute_id)) are associated with G-PCC_component_layer_id.

[0350] Figure 72 shows an example of hierarchical metadata that includes information showing the correspondence between G-PCC components and G-PCC_component_layer_id. The hierarchical metadata includes GPCC_component_info, which is information about the G-PCC components. GPCC_component_info includes attribute_layer_present_flag, num_G-PCC_attribute_component, G-PCC_attr_id, attribute_type, num_dimension, and G-PCC_component_layer_id.

[0351] The attribute_layer_present_flag is a flag that indicates whether or not hierarchical information (G-PCC_component_layer_id) exists. For example, a value of 1 indicates that hierarchical information exists, and a value of 0 indicates that hierarchical information does not exist.

[0352] num_G-PCC_attribute_component indicates the number of G-PCC components. G-PCC_attr_id indicates the ID (identifier) ​​of the G-PCC component.

[0353] `attribute_type` indicates the type of G-PCC component (Scale, Rotation0, Rotation1, Optics, etc.). `num_dimension` indicates the number of elements (dimensions) of the G-PCC component. `G-PCC_component_layer_id` indicates the hierarchy level of the G-PCC component.

[0354] Note that information indicating the correspondence between G-PCC components and G-PCC_component_layer_id may be stored in hierarchical information separate from the G-PCC component information. Figure 73 shows an example of hierarchical metadata in this case. Hierarchical metadata includes hierarchical information (GPCC-layer_info). GPCC-layer_info includes num_layer, num_G-PCC_component, and G-PCC_attr_id.

[0355] `num_layer` indicates the number of hierarchical levels. `num_G-PCC_component` is provided for each hierarchical level and indicates the number of G-PCC components included in that level. `G-PCC_attr_id` is provided for each hierarchical level (one or more) and indicates the G-PCC component ID included in that level. In this way, `G-PCC_attr_id` indicates the correspondence between hierarchical levels and G-PCC components.

[0356] The hierarchical metadata may be sequence-based (common to multiple frames) or frame-based. It may be stored in a parameter set such as SPS or APS, or included in SEI (Supplemental Enhancement Information). Alternatively, it may be stored in a data unit.

[0357] Next, we will describe a partial decoding process that extracts and decodes some of the hierarchical components of the attribute information. Figure 74 is a flowchart of the partial decoding process performed by the decoding unit 503A. First, the decoding unit 503A decodes the metadata contained in the encoded data to obtain attribute mapping information and hierarchical structure metadata, and analyzes the obtained hierarchical structure metadata (S221).

[0358] Next, the decoding unit 503A determines the layer to decode. In this example, the decoding unit 503A decides to partially decode the layer where G-PCC_component_layer_id=0 (S222). The decoding unit 503A determines the value of G-PCC_attribute_id corresponding to G-PCC_component_layer_id=0 from the hierarchical structure metadata (S223).

[0359] Figure 75 shows an example of hierarchical metadata (G-PCC Component Information). In this example, the value of G-PCC attr_id corresponding to G-PCC_component_layer_id=0 is between 0 and 12.

[0360] Next, the decoding unit 503A extracts the data unit corresponding to G-PCC_component_layer_id=0 from the bitstream (encoded data). Figure 76 shows the configuration of the TLV unit. As shown in Figure 76, the attribute data unit (ADU) includes an ADU header and a payload. The ADU header includes G-PCC_attr_id. The ADU is also stored in the payload of the TLV unit. The header of the TLV unit (TVL header) stores an identifier (TLV_unit_type) indicating that the data stored in the payload is an ADU.

[0361] The decoding unit 503A analyzes the headers of multiple TLV units contained in the bitstream and extracts ADUs (TLV units) from among the multiple TLV units where TLV_unit_type is ADU and G-PCC_attr_id contained in the ADU header is one of 0 to 12. Other ADUs are not extracted (S224).

[0362] Figure 77 shows an example of the process of extracting some ADUs (TLV units) from a bitstream. As shown in Figure 77, for example, ADUs (TLV units) with G-PCC_attr_id values ​​from 0 to 12 are extracted. Note that although Figure 77 shows multiple ADUs, a bitstream actually contains multiple TLV units.

[0363] Next, the decoding unit 503A decodes the extracted ADUs, but does not decode the other ADUs (S225). As a result, the decoding unit 503A can partially decode the data corresponding to G-PCC_component_layer_id=0.

[0364] In this way, partial decoding becomes possible by using hierarchical structure metadata that shows the correspondence between G-PCC_attr_id included in the ADU header and the hierarchical level. Partial decoding can reduce the decoding process or decoding time. Furthermore, it becomes possible to decode each hierarchical level, enabling hierarchical decoding that allows selection such as decoding only the base layer or decoding the base layer and extension layers. Note that if G-PCC_component_layer_id = X and the lower hierarchical level depends on the upper hierarchical level, the decoding unit 503A extracts and decodes the attribute components corresponding to the hierarchical level where G-PCC_component_layer_id <= X. In this way, the decoding unit 503A can separate ADUs for each hierarchical level by analyzing the header. Therefore, the decoding unit 503A can decode only a portion of the ADUs.

[0365] In partial decoding, the position may always be decoded. For example, if the position is set as the base layer, and extension information for the position is encoded as attribute information, then the position is decoded as the base layer, and whether or not to decode the extension information is determined based on the hierarchical structure.

[0366] The following describes some variations. G-PCC_component_layer_id may be stored in attribute_parameter(). Figures 78 and 79 show examples of hierarchical metadata in this case. As shown in Figure 78, the hierarchical metadata includes GPCC_component_info, which is information about the G-PCC component. GPCC_component_info includes attribute_parameter. The other parameters are the same as those shown in Figure 72.

[0367] As shown in Figure 79, attribute_parameter includes num_attribute_parameter and attribute_parameter_type. num_attribute_parameter indicates the number of attribute parameters. attribute_parameter_type is provided for each attribute parameter and indicates the type of attribute parameter.

[0368] For example, attribute_parameter_type=1 indicates the hierarchy level (G-PCC_Component_layer_id). When attribute_parameter_type=1, G-PCC_Component_layer_id is stored in attribute_parameter.

[0369] (Embodiment 3) This embodiment describes an example in which information on the hierarchical structure is stored in mapping information that shows the correspondence between GS components and G-PCC components.

[0370] Figure 80 shows a mapping information table and a hierarchical structure illustrating the correspondence between GS components and G-PCC components. Figure 81 shows an example of the syntax for the mapping information that stores the information shown in Figure 80. This syntax adds hierarchical structure information to the syntax shown in Figure 52.

[0371] The GS-GPCC_mapping_info shown in Figure 81 includes num_gaussian_component, attribute_layer_present_flag, gaussian_component_id, gaussian_component_type, attribute_layer_id, num_gaussian_dimension, G-PCC_component_id, G-PCC_dimension_id, and transform_parameters.

[0372] num_gaussian_component indicates the number of GS components. attribute_layer_present_flag is a flag indicating whether or not hierarchical information (attribute_layer_id) exists.

[0373] gaussian_component_id, gaussian_component_type, and attribute_layer_id are provided for each GS component. gaussian_component_id is the identifier of the GS component. gaussian_component_type indicates the type of the GS component.

[0374] `attribute_layer_id` indicates the hierarchy level of the GS component. `attribute_layer_id` corresponds to `gaussian_component_layer_id` shown in Figure 80.

[0375] `num_gaussian_dimension` indicates the number of dimensions of the Gaussian data (G-PCC component). `G-PCC_component_id`, `G-PCC_dimension_id`, and `transform_parameters` are provided for each dimension of the Gaussian data.

[0376] G-PCC_component_id is the identifier of the G-PCC component and corresponds to G-PCC_attr_id shown in Figure 80. G-PCC_dimension_id is the identifier of the dimension of the G-PCC component. transform_parameters are parameters used for the transformation process for Gaussian data, and include, for example, at least one of the scale value and the offset value.

[0377] In this way, by including hierarchical information in the mapping information (GS-GPCC_mapping_info), multiple syntaxes can be unified into a single syntax, thereby reducing the amount of data transmitted.

[0378] Furthermore, it is possible to represent the relationship between GS components and G-PCC components, the relationship with the hierarchy of GS components, and the relationship with the hierarchy of G-PCC components using a single syntax.

[0379] By using this syntax structure, the decoding device can achieve partial decoding using the same method as the partial decoding process described in Embodiment 2. In other words, the decoding device can obtain the G-PCC_attribute_id corresponding to the hierarchy based on the metadata. Furthermore, the decoding device can analyze the data unit header (TLV header and ADU header) and analyze the type of data stored in the TLV unit and the G-PCC_attribute_id to extract the data unit of the desired hierarchy and decode the extracted data unit. As a result, unnecessary data units do not need to be decoded, reducing the amount of processing and shortening the processing time.

[0380] In Figure 81, the hierarchy information (attribute_layer_id) is shown within the loop for the number of GS components (num_gaussian_component), but the hierarchy information may also be shown within the loop for the number of dimensions of the G-PCC components (num_gaussian_dimension).

[0381] The following describes variations and other modifications. A flag indicating that a hierarchical structure is used may be stored in the bitstream. The decoding device may determine that a hierarchical structure is used in the encoded data and determine that partial decoding is possible if metadata indicating a hierarchical structure is received or if syntax indicating a hierarchical structure is valid. A flag indicating whether or not a hierarchical structure is used may be stored in a parameter set such as SPS. If the flag indicating whether or not a hierarchical structure is used is used to determine partial decoding, the flag may be transmitted before the encoded data. This allows the decoding device to decide whether or not to decode the encoded data (data unit) before processing the data.

[0382] Furthermore, restrictions may be placed on references between attribute components. For example, when partial decoding is used, the encoding device encodes multiple G-PCC components independently. In other words, when partial decoding is used, multiple G-PCC components must be able to be decoded independently. Therefore, if a tool that performs encoding using references between G-PCC components is used, partial decoding may not be achievable. Thus, when an encoding device supports partial decoding of G-PCC components (when a hierarchical structure is used), it may decide not to use a tool that performs encoding using references between G-PCC components. Alternatively, the standard may stipulate that when partial decoding of G-PCC components is supported, a tool that performs encoding using references between G-PCC components should not be used. Alternatively, if a decoding device uses a tool that performs encoding using references between G-PCC components to encode the encoded data, it may determine that partial decoding is impossible and may choose not to perform partial decoding.

[0383] Furthermore, in hierarchical data structures, lower levels depend on higher levels. That is, data in the following hierarchical levels are used together with data in higher levels. Therefore, even when partial decoding is performed, the encoding device may encode the G-PCC components of lower levels by referring to the G-PCC components of higher levels.

[0384] The above technique may use progressive decoding instead of partial decoding. That is, the decoding device may decode data from multiple layers sequentially, starting from layer level 0, and output the decoded data. The application retrieves the data in hierarchical order and processes it using the retrieved data. This reduces initial latency.

[0385] For example, the application first renders using only the SH coefficient at level 0. Then, the application renders using both the SH coefficient at level 0 and level 1. This process can reduce initial latency.

[0386] Hierarchical metadata may be stored in a parameter set such as SPS or APS, or in SEI. SEI is sometimes called user data or inventory. Hierarchical metadata may be metadata common to a sequence (multiple frames) or metadata specific to each frame. If the hierarchical structure is switched between frames or sequences, the hierarchical metadata may be updated. A flag indicating whether the hierarchical structure has been updated may also be included in the bitstream.

[0387] Furthermore, although the above explanation used the hierarchical structure of attribute components as an example, the above-described hierarchical structure may be combined with at least one of temporal hierarchical structure and spatial hierarchical structure. In other words, the above-described hierarchical structure metadata may be used as metadata indicating a temporal hierarchy or metadata indicating a spatial hierarchy. In that case, attribute_layer_id may be replaced with temporal_scalable_id, spatial_scalable_id, or tile_id.

[0388] Furthermore, the hierarchical structure metadata may be metadata common to various hierarchies. In that case, the hierarchical structure metadata may include information indicating the type of hierarchy, and the ID used may be switched depending on the type of hierarchy.

[0389] Furthermore, while the above explanation mainly described examples of encoding and transmitting various parameters of 3D data as attribute information, some parameters may also be encoded and transmitted as positional information. For example, scale, covariance matrix, line-of-sight dependent information (high-dimensional spherical harmonics, view feature, etc.), or time-dependent information (Polynomial Trajectory, Polynomial Rotation, Temporal Scale, Temporal Optics, and motion vectors, etc.) may be encoded and transmitted as positional information of 3D data. One advantage of encoding and transmitting these parameters as positional information of 3D data is that this information directly represents geometric features such as the shape, deformation, or rotation of the object. In other words, properties such as local extent or continuity of motion in space can be effectively compressed by techniques such as geometric predictive coding and differential coding, similar to conventional single coordinate information. Furthermore, integrating these parameters with positional information enables a unified and continuous representation of the overall shape of the object. This improves the consistency of the geometry during reconstruction.

[0390] Furthermore, while the above explanation described an example of encoding and transmitting three-dimensional coordinates and shape information as location information of 3D data, three-dimensional coordinates or shape information may also be encoded and transmitted as attribute information. One advantage of encoding and transmitting three-dimensional coordinates and shape information as attribute information is that it emphasizes that location information is not limited to mere geometric coordinates, but also contains semantic and visual information such as the shape and orientation of an object, and even its state of motion. By reinterpreting this information as attribute information, learning-based nonlinear compression and the addition of semantic information become possible. This can, for example, improve visual quality and reconstruction accuracy. In addition, flexible data transmission can be achieved in cooperation with higher-level tasks such as segmentation. In other words, by capturing the semantic aspects contained in location information, it may be possible to apply a useful framework for attribute information compression in addition to conventional geometric encoding methods.

[0391] Furthermore, the application may use images rendered using only the information from the base layer, or using the information from the base layer and some extension layers, as thumbnails, initial displays, or default displays. This eliminates the need for the application to individually select these images, reducing the burden associated with the image selection process for display.

[0392] Furthermore, metadata such as the hierarchical structure metadata described above may be stored in the parameter set (SPS), control information, or SEI. In addition, some of the metadata, syntax structure, and encoded data headers may be stored in the box structure of the ISOBMFF file. The boxes may be contained in moov or in mdat.

[0393] The representation_type, which contains information about the format of the GS data, is indicated by the track box in the moov (movie box), or the media type (hdlr, stsd, tref) in the track box. The GS data is stored in the mdat file, and the GS data stored in the mdat file is associated with the format information described above.

[0394] Note that metadata may include metadata other than hierarchical metadata. For example, rendering metadata (RnderingMetadata) may include viewpoint information indicating the viewpoint used for training.

[0395] Furthermore, although the above explanation used G-PCC as an example, the above technology is applicable not only to G-PCC but also to encoding methods with interfaces for specific formats, particularly V-PCC (Video-based Point Cloud Compression), V-DMC (Video-based Dynamic Mesh Coding), MIV (MPEG Impressive Video), or V3C (Visual Volumetric Video-based Coding) which use video encoding such as HEVC (High Efficiency Video Coding) or VVC (Versatile Video Coding).

[0396] For example, when mapping 3DGS data to V3C (or V-PCC, MIV, or V-DMC) encoded data, the encoded data or metadata includes conversion information for converting each element of the 3DGS data into video, mapping information indicating which V3C component corresponds to the video of each element, and conversion parameters.

[0397] Figure 82 shows an example of this mapping information. As shown in Figure 82, the mapping information includes, for example, an identifier for each video, atlas_id, and the ID of the layer corresponding to atlas_id (attribute_layer_id).

[0398] (Summary) As described above, the encoding device layers the position information component and the attribute information component (attribute component) of the 3D data, and encodes the position information component and the attribute information component using G-PCC. For example, the attribute component is attribute information that includes information from multiple viewpoints. For example, the attribute component is multiple SH coefficients, and one layer includes some of the SH coefficients among the multiple SH coefficients. For example, the attribute component includes feature parameters of MLP.

[0399] For example, the base layer includes components independent of viewpoint. For example, components independent of viewpoint include the SH coefficient at level 0, or the base feature in STG. For example, the extended layer includes components related to viewpoint changes. For example, components related to viewpoint changes include the SH coefficient at level 1 or higher, or the view feature.

[0400] For example, attribute components are attribute information that includes information from multiple time points (multiple frames). For example, the base layer includes time-independent components. For example, the viewpoint-independent components include at least one of base feature, position, rotation, and scale in STG. For example, the extension layer includes time-varying components. For example, the time-varying components include at least one of Polynomial Trajectory, Polynomial Rotation, Temporal Scale, and Temporal Opacity in STG.

[0401] For example, as described in Embodiment 1, attribute components are components of GS data (GS components), and hierarchical levels are assigned to GS components. For example, information showing the correspondence between multiple GS components and multiple hierarchical levels is stored in metadata.

[0402] For example, as described in Embodiment 2, the attribute component is a component of the encoded data (G-PCC component), and a hierarchy level is set for the encoded data component. For example, information showing the correspondence between multiple components of the encoded data and multiple hierarchy levels is stored in the metadata.

[0403] For example, as described in Embodiment 3, the information of the components of the encoded data includes identifiers of attribute components included in attribute units (e.g., ADUs). For example, information showing the correspondence between multiple GS components, multiple components of the encoded data, and multiple hierarchical levels is stored in metadata.

[0404] As described above, the encoding device according to the embodiment performs the processing shown in Figure 83. Figure 83 is a flowchart of the processing performed by the encoding device. The encoding device includes, for example, a hierarchical structure determination unit 501 and an encoding unit 502 as shown in Figure 57.

[0405] The encoding device generates encoded data by hierarchically encoding multiple first components (e.g., GS components) contained in the Gaussian data (S301), and generates a bitstream that includes hierarchical information (e.g., hierarchical structure metadata) regarding the hierarchical structure in the hierarchical encoding and the encoded data (S302). As a result, the decoding device that decodes the bitstream can selectively decode some or all of the data from the hierarchically encoded data depending on the application. Therefore, it is possible to reduce the processing load of the decoding device or to speed up processing.

[0406] For example, in a hierarchical encoding system, the encoding device assigns multiple first components to a base hierarchical layer that can be used independently and to an extended hierarchical layer that can be used in conjunction with the base hierarchical layer. The hierarchical information indicates whether each of the multiple first components belongs to the base hierarchical layer or the extended hierarchical layer. As a result, the decoding device can selectively decode only the base hierarchical layer or both the base hierarchical layer and the extended hierarchical layer, depending on the application. Therefore, it is possible to reduce the processing load on the decoding device or speed up processing.

[0407] For example, the base layer contains information that is independent of the gaze direction from among multiple first components, while the extended layer contains information that is dependent on the gaze direction from among multiple first components. This allows the decoder to reduce processing load by decoding only the base layer when information dependent on the gaze direction is not needed. Furthermore, by decoding the extended layer, the decoder can recover high-precision Gaussian data that includes information dependent on the gaze direction.

[0408] For example, information independent of the line of sight includes a Level 0 SH (Spherical Harmonics) coefficient, while information dependent on the line of sight includes an SH coefficient of Level 1 or higher.

[0409] For example, information independent of the line of sight includes the base color feature in the STG (Space-Time Splatting) format, while information dependent on the line of sight includes the view feature in the STG format.

[0410] For example, the base layer contains time-independent information from among multiple first components, while the extended layer contains time-dependent information from among multiple first components. This allows the decoder to reduce processing load by decoding only the base layer when time-dependent information is not needed. Furthermore, by decoding the extended layer, the decoder can recover highly accurate Gaussian data containing time-dependent information.

[0411] For example, time-independent information includes position, rotation, scale, and transmittance, while time-dependent information includes information indicating the time change of at least one of the position, rotation, scale, and transmittance.

[0412] For example, time-independent information includes base color feature in STG (Space-time Splatting) format, while time-dependent information includes at least one of Polynomial Trage, Polynomial Rotation, Temporary Scale, and Temporary Optics in STG format.

[0413] For example, in encoding, the encoding device converts a plurality of first components (e.g., GS components) into a plurality of second components (e.g., G-PCC components) based on an encoding scheme (e.g., G-PCC), and then encodes the converted plurality of second components using the encoding scheme. For example, as described in Embodiment 1, the hierarchical information indicates the correspondence between a plurality of first components and a plurality of hierarchies in the hierarchical structure. According to this, the decoding device can use the hierarchical information to determine which hierarchy the plurality of first components belong to.

[0414] For example, in encoding, the encoding device converts a plurality of first components (e.g., GS components) into a plurality of second components (e.g., G-PCC components) based on an encoding scheme (e.g., G-PCC), and then encodes the converted plurality of second components using the encoding scheme. For example, as described in Embodiment 2, the hierarchical information indicates the correspondence between the plurality of second components and the plurality of hierarchies in the hierarchical structure. According to this, the decoding device can use the hierarchical information to determine which hierarchy the plurality of second components belong to. Therefore, the decoding device can determine the necessary data before converting the second components into first components.

[0415] For example, in encoding, the encoding device converts a plurality of first components (e.g., GS components) into a plurality of second components (e.g., G-PCC components) based on an encoding scheme (e.g., G-PCC), and then encodes the converted plurality of second components using the encoding scheme. For example, as described in Embodiment 3, the hierarchical information indicates the correspondence between the plurality of first components, the plurality of second components, and the plurality of hierarchies in the hierarchical structure. The decoding device can use the hierarchical information to determine which hierarchy each plurality of first components belongs to, and which hierarchy each plurality of second components belongs to. Therefore, the decoding device can determine the necessary data before converting the second components back into first components.

[0416] Figure 84 is a block diagram of the encoding device 10. For example, the encoding device 10 comprises a processor 11 and a memory 12, and the processor 11 performs the above processing using the memory 12.

[0417] Furthermore, the decoding device according to the embodiment performs the processing shown in Figure 85. Figure 85 is a flowchart of the processing performed by the decoding device. The decoding device includes, for example, the decoding unit 503 shown in Figure 64 or the decoding unit 503A shown in Figure 70.

[0418] The decoding device obtains encoded data generated by hierarchically encoding multiple first components (e.g., GS components) contained in Gaussian data from the bitstream, and hierarchical information (e.g., hierarchical structure metadata) relating to the hierarchical structure in the hierarchical encoding (S311), and decodes the encoded data using the hierarchical information (S312). As a result, the decoding device can selectively decode some or all of the data from the hierarchically encoded data depending on the application. Therefore, it is possible to reduce the processing load of the decoding device or speed up processing.

[0419] For example, in hierarchical structures, multiple first components are assigned to a base hierarchical structure that can be used independently and an extended hierarchical structure that can be used in conjunction with the base hierarchical structure. The hierarchical information indicates whether each of the multiple first components belongs to the base hierarchical structure or the extended hierarchical structure. The decoding device then uses the hierarchical information to decode only the base hierarchical structure, or to decode both the base hierarchical structure and the extended hierarchical structure. This allows the decoding device to selectively decode only the base hierarchical structure or both the base hierarchical structure and the extended hierarchical structure, depending on the application. Therefore, it is possible to reduce the processing load on the decoding device or to speed up processing.

[0420] For example, the base layer contains information that is independent of the gaze direction from among multiple first components, while the extended layer contains information that is dependent on the gaze direction from among multiple first components. This allows the decoder to reduce processing load by decoding only the base layer when information dependent on the gaze direction is not needed. Furthermore, by decoding the extended layer, the decoder can recover high-precision Gaussian data that includes information dependent on the gaze direction.

[0421] For example, information independent of the line of sight includes a Level 0 SH (Spherical Harmonics) coefficient, while information dependent on the line of sight includes an SH coefficient of Level 1 or higher.

[0422] For example, information independent of the line of sight includes the base color feature in the STG (Space-Time Splatting) format, while information dependent on the line of sight includes the view feature in the STG format.

[0423] For example, the base layer contains time-independent information from among multiple first components, while the extended layer contains time-dependent information from among multiple first components. This allows the decoder to reduce processing load by decoding only the base layer when time-dependent information is not needed. Furthermore, by decoding the extended layer, the decoder can recover highly accurate Gaussian data containing time-dependent information.

[0424] For example, time-independent information includes position, rotation, scale, and transmittance, while time-dependent information includes information indicating the time change of at least one of the position, rotation, scale, and transmittance.

[0425] For example, time-independent information includes base color feature in STG (Space-time Splatting) format, while time-dependent information includes at least one of Polynomial Trage, Polynomial Rotation, Temporary Scale, and Temporary Optics in STG format.

[0426] For example, in decoding, the decoding device decodes the encoded data using a decoding method (e.g., G-PCC), generating multiple second components (e.g., G-PCC components) based on the decoding method, and converts the multiple second components into multiple first components (e.g., GS components). For example, as described in Embodiment 1, the hierarchical information indicates the correspondence between multiple first components and multiple hierarchies in the hierarchical structure. According to this, the decoding device can use the hierarchical information to determine which hierarchy each of the multiple first components belongs to.

[0427] For example, in decoding, the decoding device decodes the encoded data using a decoding method (e.g., G-PCC), generating multiple second components (e.g., G-PCC components) based on the decoding method, and converts these multiple second components into multiple first components (e.g., GS components). For example, as described in Embodiment 2, the hierarchical information indicates the correspondence between the multiple second components and the multiple hierarchies in the hierarchical structure. According to this, the decoding device can use the hierarchical information to determine which hierarchy each of the multiple second components belongs to. Therefore, the decoding device can determine the necessary data before converting the second components into first components.

[0428] For example, in decoding, the decoding device decodes the encoded data using a decoding method (e.g., G-PCC), generating multiple second components (e.g., G-PCC components) based on the decoding method, and converts the multiple second components into multiple first components (e.g., GS components). For example, as described in Embodiment 3, the hierarchical information indicates the correspondence between multiple first components, multiple second components, and multiple hierarchies in the hierarchical structure. The decoding device can use the hierarchical information to determine which hierarchy each of the multiple first components belongs to, and which hierarchy each of the multiple second components belongs to. Therefore, the decoding device can determine the necessary data before converting the second components into first components.

[0429] Figure 86 is a block diagram of the decoding device 20. For example, the decoding device 20 comprises a processor 21 and a memory 22, and the processor 21 performs the above processing using the memory 22.

[0430] The above describes the encoding device (three-dimensional data encoding device) and decoding device (three-dimensional data decoding device), etc., according to embodiments and modifications of the present disclosure. However, the present disclosure is not limited to these embodiments.

[0431] Furthermore, each processing unit included in the encoding device and decoding device, etc., according to the above embodiment is typically implemented as an LSI, which is an integrated circuit. These may be individually integrated into a single chip, or some or all of them may be integrated into a single chip.

[0432] Furthermore, integrated circuit implementation is not limited to LSIs; it may also be achieved using dedicated circuits or general-purpose processors. Alternatively, an FPGA (Field Programmable Gate Array), which can be programmed after LSI manufacturing, or a reconfigurable processor capable of reconfiguring the connections and settings of circuit cells within the LSI, may be used.

[0433] Furthermore, in each of the above embodiments, each component may be implemented by being composed of dedicated hardware or by executing a software program suitable for each component. Each component may also be implemented by a program execution unit such as a CPU or processor reading and executing a software program recorded on a recording medium such as a hard disk or semiconductor memory.

[0434] Furthermore, this disclosure may be implemented as an encoding method (three-dimensional data encoding method) or a decoding method (three-dimensional data decoding method), etc., performed by an encoding device (three-dimensional data encoding device) and a decoding device (three-dimensional data decoding device), etc.

[0435] Furthermore, this disclosure may be implemented as a program that causes a computer, processor, or device to execute the above-described encoding or decoding method. Alternatively, this disclosure may be implemented as a bitstream generated by the above-described encoding method. Furthermore, this disclosure may be implemented as a recording medium on which the program or the bitstream is recorded. For example, this disclosure may be implemented as a non-temporary, computer-readable recording medium on which the program or the bitstream is recorded.

[0436] Furthermore, the division of functional blocks in the block diagram is just one example; multiple functional blocks can be implemented as a single functional block, a single functional block can be divided into multiple parts, or some functions can be moved to other functional blocks. In addition, the functions of multiple functional blocks with similar functions can be processed in parallel or time-sharing by a single piece of hardware or software.

[0437] Furthermore, the order in which each step in the flowchart is performed is illustrative for the purpose of specifically illustrating this disclosure, and may be in a different order. Also, some of the above steps may be performed simultaneously (in parallel) with other steps.

[0438] Although one or more embodiments of encoding and decoding devices, etc., have been described above based on embodiments, this disclosure is not limited to these embodiments. Without departing from the spirit of this disclosure, various modifications that a person skilled in the art could conceive of are applied to these embodiments, and forms constructed by combining components from different embodiments may also be included within the scope of one or more embodiments.

[0439] This disclosure is applicable to encoding and decoding devices.

[0440] 10 Encoding device 11, 21 Processor 12, 22 Memory 20 Decoding device 101 Three-dimensional data encoding system 102 Three-dimensional data decoding system 103 Sensor terminal 104 External connection unit 111 Three-dimensional data generation system 112 Presentation unit 113 Encoding unit 114 Multiplexing unit 115 Input / output unit 116 Control unit 117 Sensor information acquisition unit 118 Three-dimensional data generation unit 121 Sensor information acquisition unit 122 Input / output unit 123 Demultiplexing unit 124 Decoding unit 125 Presentation unit 126 User interface 127 Control unit 130 First encoding unit 131 Location information encoding unit 132 Attribute information encoding unit 133 Additional information encoding unit 134 Multiplexing unit 140 First decoding unit 141 Demultiplexing unit 142 Location information decoding unit 143 Attribute Information Decoding Unit 144 Additional Information Decoding Unit 150 Second Encoding Unit 151 Additional Information Generation Unit 152 Position Image Generation Unit 153 Attribute Image Generation Unit 154 Video Encoding Unit 155 Additional Information Encoding Unit 156 Multiplexing Unit 160 Second Decoding Unit 161 Demultiplexing Unit 162 Video Decoding Unit 163 Additional Information Decoding Unit 164 Position Information Generation Unit 165 Attribute Information Generation Unit 171 Octane Tree Encoding Unit 172 Prediction Tree Encoding Unit 173 Octane Tree Decoding Unit 174 Prediction Tree Decoding Unit 181, 191 Octane Tree Generation Unit 182, 192 Geometric Information Calculation Unit 183, 193 Encoding Table Selection Unit 184 Entropy Encoding Unit 194 Entropy Decoding Unit 201 LoD Attribute Information Encoding Unit 202 Transformation attribute information coding unit 203 LoD attribute information decoding unit 204 Transformation attribute information decoding unit 211 Sorting unit 212 Haar transformation unit 213 Quantization unit 214, 222 Inverse quantization unit 215, 223 Inverse Haar transformation unit 216, 224 Memory 217 Arithmetic coding unit 221 Arithmetic decoding unit 231 Data partitioning unit 232 Coding unit 233 Decoding unit 234 Data merging unit 241 Coding unit 242 TLV storage unit 301 Sensor information input unit 302 Three-dimensional data generation unit 303, 305 Rendering unit 304 Spherical harmonics310 Gaussian data generation unit 311 Encoding unit 312 Multiplexing unit 321 Demultiplexing unit 322 Decoding unit 323 Application unit 324 Input interface unit 325 Rendering unit 326 Presentation unit 330 Encoding device 331 First preprocessing unit 332 Second preprocessing unit 333 G-PCC encoding unit 341 Position information encoding unit 342 Rotation encoding unit 343 Scale encoding unit 344 SH coefficient encoding unit 345 Transparency encoding unit 346 Metadata encoding unit 347 Multiplexing unit 350 Decoding device 351 G-PCC decoding unit 352 Second postprocessing unit 353 First postprocessing unit 361 Demultiplexing unit 362 Position information decoding unit 363 Rotation decoding unit 364 Scale decoding unit 365 SH coefficient decoding unit 366 Transparency decoding unit 367 Metadata decoding unit 401 SFM unit 402 Initialization unit 403 GS update unit 404 Rendering unit 405 Correct value selection unit 406 Error calculation unit 407 GS parameter optimization unit 501 Hierarchical structure determination unit 502 Encoding unit 503, 503A Decoding unit 504 Rendering unit 511 Decoding processing unit 512, 523 GS data restoration unit 513 Hierarchical extraction unit 521 Metadata decoding unit 522 Partial decoding unit 1143 Encoding unit 1144 Formatting unit 1145 Reverse formatting unit 1146 Decoding unit

Claims

1. An encoding method that generates encoded data by hierarchically encoding multiple first components contained in Gaussian data, and generates a bitstream that includes hierarchical information relating to the hierarchical structure in the hierarchical encoding and the encoded data.

2. The encoding method according to claim 1, wherein in the hierarchical arrangement, the plurality of first components are assigned to a base hierarchical which can be used independently and an extended hierarchical which can be used in combination with the base hierarchical, and the hierarchical information indicates whether each of the plurality of first components belongs to the base hierarchical or the extended hierarchical.

3. The encoding method according to claim 2, wherein the base layer includes information that is independent of the line of sight among the plurality of first components, and the extended layer includes information that is dependent on the line of sight among the plurality of first components.

4. The encoding method according to claim 3, wherein the information independent of the line of sight includes a Level 0 SH (Spherical Harmonics) coefficient, and the information dependent on the line of sight includes a Level 1 or higher SH coefficient.

5. The encoding method according to claim 3, wherein the information independent of the line of sight includes a base color feature in the STG (Space-time Splatting) format, and the information dependent on the line of sight includes a view feature in the STG format.

6. The encoding method according to claim 2, wherein the base layer includes time-independent information among the plurality of first components, and the extended layer includes time-dependent information among the plurality of first components.

7. The encoding method according to claim 6, wherein the time-independent information includes position, rotation, scale, and transmittance, and the time-dependent information includes information indicating the time change of at least one of the position, rotation, scale, and transmittance.

8. The encoding method according to claim 6, wherein the time-independent information includes a base color feature in the STG (Space-time Splatting) format, and the time-dependent information includes at least one of Polynomial Trajectory, Polynomial Rotation, Temporal Scale, and Temporal Opacity in the STG format.

9. The encoding method according to claim 1, wherein the encoding involves converting the plurality of first components into a plurality of second components based on an encoding scheme, encoding the converted plurality of second components using the encoding scheme, and the hierarchical information indicates the correspondence between the plurality of first components and the plurality of hierarchies in the hierarchical structure.

10. The encoding method according to claim 1, wherein the encoding involves converting the plurality of first components into a plurality of second components based on an encoding scheme, encoding the converted plurality of second components using the encoding scheme, and the hierarchical information indicates the correspondence between the plurality of second components and the plurality of hierarchies in the hierarchical structure.

11. The encoding method according to claim 1, wherein the encoding involves converting the plurality of first components into a plurality of second components based on an encoding scheme, encoding the converted plurality of second components using the encoding scheme, and the hierarchical information indicates the correspondence between the plurality of first components, the plurality of second components, and the plurality of hierarchies in the hierarchical structure.

12. A decoding method that obtains encoded data generated by hierarchically encoding multiple first components contained in Gaussian data from a bitstream, and hierarchical information relating to the hierarchical structure in the hierarchical encoding, and decodes the encoded data using the hierarchical information.

13. The decoding method according to claim 12, wherein in the hierarchy, the plurality of first components are assigned to a base hierarchy that can be used independently and an extended hierarchy that can be used in combination with the base hierarchy, the hierarchy information indicates whether each of the plurality of first components belongs to the base hierarchy or to the extended hierarchy, and in the decoding, the decoding is performed by decoding only the base hierarchy using the hierarchy information, or by decoding the base hierarchy and the extended hierarchy using the hierarchy information.

14. The decoding method according to claim 13, wherein the base hierarchy includes a component of information that is independent of the viewing direction among the plurality of first components, and the extended hierarchy includes information that is dependent on the viewing direction among the plurality of first components.

15. The decoding method according to claim 14, wherein the information independent of the line of sight includes a Level 0 SH (Spherical Harmonics) coefficient, and the information dependent on the line of sight includes a Level 1 or higher SH coefficient.

16. The decoding method according to claim 14, wherein the information independent of the viewing direction includes a base color feature in the STG (Space-time Splatting) format, and the information dependent on the viewing direction includes a view feature in the STG format.

17. The decoding method according to claim 13, wherein the base layer includes time-independent information among the plurality of first components, and the extended layer includes time-dependent information among the plurality of first components.

18. The decoding method according to claim 17, wherein the time-independent information includes position, rotation, scale, and transmittance, and the time-dependent information includes information indicating the time change of at least one of the position, rotation, scale, and transmittance.

19. The decoding method according to claim 17, wherein the time-independent information includes a base color feature in the STG (Space-time Splatting) format, and the time-dependent information includes at least one of Polynomial Trage, Polynomial Rotation, Temporal Scale, and Temporal Opacity in the STG format.

20. The decoding method according to claim 12, wherein the decoding involves decoding the encoded data using a decoding method to generate a plurality of second components based on the decoding method, converting the plurality of second components into a plurality of first components, and the hierarchical information indicates the correspondence between the plurality of first components and the plurality of hierarchies in the hierarchical structure.

21. The decoding method according to claim 12, wherein the decoding involves decoding the encoded data using a decoding method to generate a plurality of second components based on the decoding method, converting the plurality of second components into a plurality of first components, and the hierarchical information indicates the correspondence between the plurality of second components and the plurality of hierarchies in the hierarchical structure.

22. The decoding method according to claim 12, wherein the decoding process involves decoding the encoded data using a decoding method to generate a plurality of second components based on the decoding method, converting the plurality of second components into a plurality of first components, and the hierarchical information indicates the correspondence between the plurality of first components, the plurality of second components, and the plurality of hierarchies in the hierarchical structure.

23. An encoding device comprising a processor and memory, wherein the processor generates encoded data by encoding a plurality of first components contained in Gaussian data in a hierarchical manner using the memory, and generates a bitstream including hierarchical information relating to the hierarchical structure in the hierarchical arrangement and the encoded data.

24. A decoding device comprising a processor and memory, wherein the processor uses the memory to obtain encoded data generated by hierarchically encoding a plurality of first components contained in Gaussian data from a bitstream, and hierarchical information relating to the hierarchical structure in the hierarchical encoding, and decodes the encoded data using the hierarchical information.

Images