Encoding method, decoding method, encoding device, and decoding device

The encoding method for three-dimensional data reduces processing load by generating a bitstream with control information for partial decoding, addressing the high processing load challenge in three-dimensional data storage and transmission.

WO2026240116A1PCT designated stage Publication Date: 2026-11-19PANASONIC INTELLECTUAL PROPERTY CORP OF AMERICA
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
PCT/JP2026/018021
Authority / Receiving Office
WO · WO
Patent Type
Applications
Current Assignee / Owner
Priority Date
2025-05-13
Filing Date
2026-05-12
Publication Date
2026-11-19

AI Technical Summary

Technical Problem

The processing load on the decoding side is high due to the large volume of data in three-dimensional point clouds, which is a challenge in storage and transmission.

Method used

An encoding method that generates a bitstream with control information containing viewpoint pieces of information, allowing for partial decoding of three-dimensional data by identifying subregions, thereby reducing the processing load on the decoding side.

Benefits of technology

This method enables efficient partial decoding of three-dimensional data by selecting necessary portions, reducing unnecessary data processing and lowering the overall processing load on the decoding side.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure JP2026018021_19112026_PF_FP_ABST
    Figure JP2026018021_19112026_PF_FP_ABST
Patent Text Reader

Abstract

An encoding method executed by an encoding device, said method comprising acquiring one or more pieces of viewpoint information (S1921), and generating a bitstream that includes control information including the one or more pieces of viewpoint information and encoded data obtained by encoding three-dimensional data (S1922), wherein each of the one or more pieces of viewpoint information includes one or more pieces of identification information for identifying, from among one or more partial regions corresponding to at least a part of the three-dimensional data, one or more partial regions included in a region determined by the viewpoint information.
Need to check novelty before this filing date? Find Prior Art

Description

Encoding method, decoding method, encoding device, and decoding device

[0001] The present disclosure relates to an encoding method, a decoding method, an encoding device, and a decoding device.

[0002] In a wide range of fields such as computer vision for autonomous operation of automobiles or robots, map information, surveillance, infrastructure inspection, or video distribution, the spread of devices or services utilizing three-dimensional data is expected in the future. Three-dimensional data is acquired by various methods such as a distance sensor such as a range finder, a stereo camera, or a combination of a plurality of monocular cameras.

[0003] As one of the methods for representing three-dimensional data, there is a representation method called a point cloud that represents the shape of a three-dimensional structure using a point group in a three-dimensional space. In the point cloud, the positions and colors of the point group are stored. The point cloud is expected to become the mainstream representation method for three-dimensional data, but the data volume of the point group is extremely large. Therefore, in the storage or transmission of three-dimensional data, similar to two-dimensional moving images (examples include MPEG-4 AVC (Advanced Video Coding) or HEVC (High Efficiency Video Coding) standardized by MPEG), compression of the data volume by encoding is essential.

[0004] In addition, point cloud compression is partially supported by public libraries (Point Cloud Library) that perform processing related to point clouds.

[0005] In addition, a technology is known that searches for and displays facilities located around a vehicle using three-dimensional map data (see, for example, Patent Document 1).

[0006] International Publication No. 2014 / 020663

[0007] It is desired that the processing load on the decoding side can be reduced.

[0008] The present disclosure aims to provide an encoding method, a decoding method, an encoding device, and a decoding device that can reduce the processing load on the decoding side.

[0009] An encoding method according to one aspect of the present disclosure is an encoding method performed by an encoding device, which acquires one or more viewpoint pieces of information, generates a bitstream including control information containing the one or more viewpoint pieces of information, and encoded data obtained by encoding Gaussian three-dimensional data, wherein each of the one or more viewpoint pieces of information includes one or more identification pieces of information that identify one or more subregions included in the region defined by the viewpoint piece of information, among one or more subregions corresponding to at least a part of the three-dimensional Gaussian data.

[0010] A decoding method according to another aspect of the present disclosure is a decoding method performed by a decoding device, which includes acquiring a bitstream, acquiring control information including one or more viewpoint pieces of information and at least a portion of encoded data corresponding to at least a portion of three-dimensional data from the bitstream, decoding at least a portion of the encoded data to acquire at least a portion of the three-dimensional data, and each of the one or more viewpoint pieces of information includes one or more identification pieces of information that identify one or more sub-regions included in the region defined by the viewpoint piece of one or more sub-regions corresponding to at least a portion of the three-dimensional data.

[0011] This disclosure provides an encoding method, a decoding method, an encoding device, and a decoding device that can reduce the processing load on the decoding side.

[0012] Figure 1 is a diagram showing an example configuration of a three-dimensional data encoding and decoding system according to an embodiment. Figure 2 is a diagram showing an example configuration of point cloud data according to an embodiment. Figure 3 is a diagram showing an example configuration of a data file describing point cloud data information according to an embodiment. Figure 4 is a diagram showing an example configuration of mesh data according to an embodiment. Figure 5 is a diagram showing an example configuration of a data file describing mesh data information according to an embodiment. Figure 6 is a diagram showing the types of three-dimensional data according to an embodiment. Figure 7 is a diagram showing the configuration of the first encoding unit according to an embodiment. Figure 8 is a block diagram of the first encoding unit according to an embodiment. Figure 9 is a diagram showing the configuration of the first decoding unit according to an embodiment. Figure 10 is a block diagram of the first decoding unit according to an embodiment. Figure 11 is a diagram showing the configuration of the second encoding unit according to an embodiment. Figure 12 is a block diagram of the second encoding unit according to an embodiment. Figure 13 is a diagram showing the configuration of the second decoding unit according to an embodiment. Figure 14 is a block diagram of the second decoding unit according to an embodiment. Figure 15 is a block diagram of the position information encoding unit according to an embodiment. Figure 16 is a block diagram of the position information decoding unit according to an embodiment. Figure 17 is a block diagram of the octree coding unit according to the embodiment. Figure 18 is a diagram showing an example of location information according to the embodiment. Figure 19 is a diagram showing an example of the octree representation of location information according to the embodiment. Figure 20 is a block diagram of the octree decoding unit according to the embodiment. Figure 21 is a block diagram of the attribute information coding unit according to the embodiment. Figure 22 is a block diagram of the attribute information decoding unit according to the embodiment. Figure 23 is a block diagram of the attribute information coding unit according to the embodiment. Figure 24 is a block diagram of the attribute information decoding unit according to the embodiment. Figure 25 is a block diagram of the three-dimensional data coding device according to the embodiment. Figure 26 is a block diagram of the three-dimensional data decoding device according to the embodiment. Figure 27 is a diagram showing the relationship between tiles and slices according to the embodiment. Figure 28 is a diagram showing an example of the configuration of a bitstream according to the embodiment. Figure 29 is a block diagram showing an example of the configuration of a three-dimensional data coding device according to the embodiment. Figure 30 is a diagram showing an example of the configuration of coded data and a NAL unit according to the embodiment.Figure 31 is a diagram showing an example of the semantics of pcc_nal_unit_type according to an embodiment. Figure 32 is a diagram showing a part of the processing in a three-dimensional data generation system according to an embodiment. Figure 33 is a diagram showing the components of Gaussian data described in PLY format. Figure 34 is a block diagram showing an example of a device that generates format data from data. Figure 35 is a block diagram showing an example of a device that restores the original data from format data. Figure 36 is a conceptual diagram illustrating an example of how an encoded data bitstream is stored in a system format. Figure 37 is a diagram showing an example of the box structure of ISOBMFF. Figure 38 is a diagram showing the configuration of 2DGS data according to an embodiment. Figure 39 is a diagram showing the configuration of 3DVS data according to an embodiment. Figure 40 is a diagram showing the configuration of STG data according to an embodiment. Figure 41 is a diagram showing the input and output of the spherical harmonic part of color according to an embodiment. Figure 42 is a diagram showing the number of coefficients for each level according to an embodiment. Figure 43 is a diagram showing an example of the coefficients of a spherical harmonic function according to an embodiment. Figure 44 is a diagram showing the rendering process of Gaussian data according to an embodiment. Figure 45 is a diagram showing the rendering process according to an embodiment. Figure 46 is a block diagram of an encoding device according to an embodiment. Figure 47 is a block diagram of a decoding device according to an embodiment. Figure 48 is a diagram showing the detailed processing of a three-dimensional data generation device according to an embodiment. Figure 49 is a block diagram of an encoding device according to an embodiment. Figure 50 is a block diagram of a decoding device according to an embodiment. Figure 51 is a diagram showing an example of Gaussian data mapping according to an embodiment. Figure 52 is a diagram showing an example of mapping information according to an embodiment. Figure 53 is a diagram showing an example of the configuration of conversion information according to an embodiment. Figure 54 is a diagram showing an example of the configuration of encoded data according to an embodiment. Figure 55 is a diagram showing an example of the G-PCC attribute type of Gaussian data according to an embodiment. Figure 56 is a flowchart of the processing by the decoding device according to an embodiment. Figure 57 is a diagram showing an example of the configuration of partial point cloud decoding according to an embodiment. Figure 58 is a diagram showing the relationship between tiles and slices according to an embodiment. Figure 59 is a diagram showing a tile inventory according to an embodiment.Figure 60 is a diagram showing the relationship between the decoding target area and tiles according to the embodiment. Figure 61 is a flowchart showing the partial decoding process according to the embodiment. Figure 62 is a diagram showing point cloud and GS data according to the embodiment. Figure 63 is a diagram showing conventional tile information according to the embodiment. Figure 64 is a diagram showing tiles that are not acquired according to the embodiment. Figure 65 is a diagram showing the configuration for partial decoding of GS data according to the embodiment. Figure 66 is a flowchart showing the processing of the encoding unit according to the embodiment. Figure 67 is a diagram showing the influence range of GS data according to the embodiment. Figure 68 is a diagram showing the influence range of tiles and GS data according to the embodiment. Figure 69 is a diagram showing the tile inventory according to the embodiment. Figure 70 is a diagram showing the relationship between the rendering target area and tiles according to the embodiment. Figure 71 is a flowchart showing the processing of the decoding unit according to the embodiment. Figure 72 is a diagram showing offset information and difference information according to the embodiment. Figure 73 is a diagram showing another syntax example according to the embodiment. Figure 74 is a diagram showing an example of using scale values ​​according to the embodiment. Figure 75 is a diagram showing an example of the configuration of the encoding device according to the embodiment. Figure 76 is a flowchart showing an example of the encoding method executed by the encoding device according to the embodiment. Figure 77 is a diagram showing an example of the configuration of the decoding device according to the embodiment. Figure 78 is a flowchart showing an example of a decoding method executed by the decoding device according to the embodiment. Figure 79 is a diagram showing metadata processing for each viewpoint according to the embodiment. Figure 80 is a flowchart showing the metadata generation process according to the embodiment. Figure 81 is a diagram showing the relationship between viewpoints and tiles according to the embodiment. Figure 82 is a diagram showing an example of the syntax of the tile list for each viewpoint according to the embodiment. Figure 83 is a flowchart showing the processing of the decoding unit according to the embodiment. Figure 84 is a diagram showing an example of the syntax of the slice list for each viewpoint according to the embodiment. Figure 85 is a diagram showing the relationship between tiles corresponding to a plurality of recommended viewpoints according to the embodiment. Figure 86 is a flowchart showing the processing of the decoding unit according to the embodiment. Figure 87 is a diagram showing the configuration of the encoding device according to the embodiment. Figure 88 is a flowchart showing the encoding method according to the embodiment.Figure 89 is a diagram showing the configuration of a decoding device according to an embodiment. Figure 90 is a flowchart showing the decoding method according to an embodiment.

[0013] An encoding method according to a first aspect of the present disclosure is an encoding method performed by an encoding device, which acquires one or more viewpoint pieces of information, generates a bitstream including control information containing the one or more viewpoint pieces of information, and encoded data obtained by encoding three-dimensional data, wherein each of the one or more viewpoint pieces of information includes one or more identification pieces of information that identify one or more subregions included in the region defined by the viewpoint piece of information, among one or more subregions corresponding to at least a part of the three-dimensional data.

[0014] This allows for the correspondence between at least a portion of the three-dimensional data contained in the bitstream and information identifying the sub-regions included in the area defined by each viewpoint. As a result, it is possible to select the necessary portion during decoding and use it for partial decoding, decoding at least a portion of the three-dimensional data, thereby reducing the processing load on the decoding side.

[0015] The encoding method according to the second aspect of this disclosure is the encoding method according to the first aspect, wherein the three-dimensional data is Gaussian data.

[0016] An encoding method according to a third aspect of this disclosure is an encoding method according to a first aspect, wherein each of the one or more viewpoint pieces of information includes identification information for identifying the viewpoint corresponding to the viewpoint piece of information.

[0017] Therefore, each viewpoint information can be handled separately for each corresponding viewpoint. As a result, even when multiple viewpoint information is included, the viewpoint information corresponding to the viewpoint to be decoded can be selected without error, and the selection of unnecessary data can be suppressed.

[0018] The encoding method according to the fourth aspect of this disclosure is the encoding method according to the first aspect, wherein each of the one or more viewpoint pieces of information includes camera parameter information of the viewpoint corresponding to the viewpoint piece of information.

[0019] This allows us to define the region corresponding to each viewpoint using the camera parameter information of that viewpoint. As a result, we can determine the target subregion according to the conditions of each viewpoint, and reduce the selection of subregions that do not correspond to a viewpoint.

[0020] The encoding method according to the fifth aspect of this disclosure is the encoding method according to the first aspect, wherein each of the one or more sub-regions corresponds to sub-three-dimensional data obtained by dividing the three-dimensional data.

[0021] This allows for the association of sub-3D data obtained by dividing the 3D data into sub-regions. As a result, at least a portion of the 3D data can be treated as a target for acquisition or decoding on a sub-region basis.

[0022] The encoding method according to the sixth aspect of this disclosure is the encoding method according to the first aspect, wherein the one or more identification pieces of information include sub-region identification pieces of information that identify each of the one or more sub-regions.

[0023] Therefore, each sub-region can be distinguished using identification information. As a result, the target sub-region can be selected based on viewpoint information, and the acquisition or decoding of data corresponding to unnecessary sub-regions can be reduced.

[0024] The encoding method according to the seventh aspect of this disclosure is the encoding method according to the sixth aspect, wherein each of the one or more subregions is a tile, the subregion identification information is a tile ID that identifies the tile, and the tile ID is set in the slice tag of the header of the encoded data corresponding to the tile.

[0025] This allows a tile ID to be set as a slice tag to identify a tile. As a result, encoded data can be handled in association on a tile-by-tile basis, and the encoded data corresponding to the target tile can be selected.

[0026] The encoding method according to the eighth aspect of this disclosure is the encoding method according to the sixth aspect, wherein the encoded data corresponding to at least a portion of the three-dimensional data includes a slice, the partial region identification information is a slice ID that identifies the slice, and the slice ID is set in the slice ID of the slice header.

[0027] This allows a slice ID to be set in the slice header to identify the slice. As a result, encoded data can be handled separately on a slice-by-slice basis, and the encoded data corresponding to the target slice can be selected.

[0028] An encoding method according to a ninth aspect of the present disclosure is an encoding method according to a first aspect, wherein the control information includes region information indicating at least one of the position and size of each of the one or more subregions.

[0029] This allows the control information to include information indicating at least one of the location and size of each subregion. As a result, the decoding side can determine the target data based on the extent of the subregion.

[0030] An encoding method according to a tenth aspect of the present disclosure is an encoding method according to a first aspect, wherein the control information includes viewpoint type information indicating the type of viewpoint represented by each of the one or more viewpoint pieces of information.

[0031] This allows for the distinction and handling of different viewpoint types. As a result, the viewpoint information used can be narrowed down according to the type of viewpoint, and the use of unnecessary viewpoint information can be reduced.

[0032] A decoding method according to an eleventh aspect of the present disclosure is a decoding method performed by a decoding device, which includes acquiring a bitstream, acquiring control information including one or more viewpoint pieces of information and at least a portion of encoded data corresponding to at least a portion of three-dimensional data from the bitstream, decoding at least a portion of the encoded data to acquire at least a portion of the three-dimensional data, and each of the one or more viewpoint pieces of information includes one or more identification pieces of information that identify one or more sub-regions included in the region defined by the viewpoint piece of information, among one or more sub-regions corresponding to at least a portion of the three-dimensional data.

[0033] This allows for the acquisition of control information and at least a portion of the encoded data from the bitstream, and the decoding of at least a portion of the three-dimensional data. As a result, decoding can be performed while selecting the necessary parts using the identification information contained in the viewpoint information, thereby suppressing the decoding of unnecessary parts.

[0034] The decoding method according to the twelfth aspect of this disclosure is the decoding method according to the eleventh aspect, wherein the three-dimensional data is Gaussian data.

[0035] A decoding method according to a thirteenth aspect of this disclosure is a decoding method according to an eleventh aspect, wherein each of the one or more viewpoint pieces of information includes identification information for identifying the viewpoint corresponding to the viewpoint piece of information.

[0036] Therefore, each viewpoint information can be handled separately for each corresponding viewpoint. As a result, even when multiple viewpoint information is acquired, the viewpoint information corresponding to the viewpoint to be decoded can be selected without error, and the selection of unnecessary data can be suppressed.

[0037] A decoding method according to a fourteenth aspect of this disclosure is a decoding method according to an eleventh aspect, wherein each of the one or more viewpoint pieces of information includes camera parameter information of the viewpoint corresponding to the viewpoint piece of information.

[0038] This allows us to obtain camera parameter information for each viewpoint. As a result, we can determine the target subregion according to the conditions for each viewpoint, and reduce the selection of subregions that do not correspond to a viewpoint.

[0039] The decoding method according to the fifteenth aspect of the present disclosure is the decoding method according to the eleventh aspect, wherein each of the one or more partial regions corresponds to sub-three-dimensional data obtained by dividing the three-dimensional data.

[0040] Thereby, sub-three-dimensional data obtained by dividing the three-dimensional data can be handled in association with each partial region. As a result, at least part of the three-dimensional data to be acquired can be handled in units of partial regions.

[0041] The decoding method according to the sixteenth aspect of the present disclosure is the decoding method according to the eleventh aspect, wherein the one or more pieces of identification information include partial region identification information for identifying each of the one or more partial regions.

[0042] Therefore, each partial region can be distinguished using the identification information. As a result, the partial region to be used for decoding can be selected based on the identification information, and acquisition or decoding of data corresponding to unnecessary partial regions can be suppressed.

[0043] The decoding method according to the seventeenth aspect of the present disclosure is the decoding method according to the sixteenth aspect, wherein each of the one or more partial regions is a tile, the partial region identification information is a tile ID for identifying the tile, and the tile ID is set in a slice tag of a header of the encoded data corresponding to the tile.

[0044] Thereby, the tile ID for identifying the tile can be set in the slice tag of the header of the encoded data corresponding to the tile. As a result, the encoded data can be handled in association with each other in units of tiles, and the encoded data corresponding to the target tile can be selected.

[0045] The decoding method according to the eighteenth aspect of the present disclosure is the decoding method according to the sixteenth aspect, wherein the encoded data corresponding to at least part of the three-dimensional data includes a slice, the partial region identification information is a slice ID for identifying the slice, and the slice ID is set to the slice ID in the header of the slice.

[0046] Thereby, the slice ID for identifying a slice can be set to the slice ID in the header of the slice. As a result, encoded data can be handled distinctively on a slice-by-slice basis, and the encoded data corresponding to the target slice can be selected.

[0047] The decoding method according to the nineteenth aspect of the present disclosure is the decoding method according to the eleventh aspect, wherein the control information includes region information indicating at least one of the position and the size of each of the one or more partial regions.

[0048] Thereby, region information indicating at least one of the position and the size for each partial region can be included in the control information. As a result, the partial region corresponding to the decoding target can be determined based on the control information.

[0049] The decoding method according to the twentieth aspect of the present disclosure is the decoding method according to the eleventh aspect, wherein the control information includes viewpoint type information indicating the type of the viewpoint indicated by each of the one or more pieces of viewpoint information.

[0050] Thereby, the type of the viewpoint can be handled distinctively for each viewpoint. As a result, the viewpoint information to be used can be narrowed down according to the type of the viewpoint, and the use of unnecessary viewpoint information can be suppressed.

[0051] The decoding method according to the twenty-first aspect of the present disclosure is the decoding method according to the nineteenth aspect, and further acquires a decoding target viewpoint, when viewpoint information corresponding to the decoding target viewpoint is included in the one or more pieces of viewpoint information, determines at least one of the one or more pieces of identification information included in the viewpoint information as the identification information of the encoded data to be decoded, and when viewpoint information corresponding to the decoding target viewpoint is not included in the one or more pieces of viewpoint information, determines at least one of the one or more pieces of identification information based on the region information included in the control information as the identification information of the encoded data to be decoded.

[0052] Thereby, the identification information of the encoded data to be decoded can be determined according to the decoding target viewpoint. As a result, the data corresponding to the decoding target viewpoint can be selected and used for decoding, and decoding of unnecessary portions can be suppressed.

[0053] A decoding method according to a 22nd aspect of this disclosure is a decoding method according to an 11th aspect, further rendering an image or model based on at least a portion of the acquired three-dimensional data and at least one viewpoint information from the one or more viewpoint information.

[0054] This allows at least a portion of the acquired three-dimensional data to be used to generate images or models corresponding to viewpoint information. As a result, output can be generated using only the necessary parts of the acquired three-dimensional data, thereby reducing unnecessary data processing.

[0055] The decoding method according to the 23rd aspect of this disclosure is a decoding method according to the 11th aspect, further comprising acquiring at least a portion of the three-dimensional data corresponding to the difference between multiple identification information corresponding to multiple viewpoint information.

[0056] This makes it possible to acquire at least a portion of the three-dimensional data corresponding to the difference between multiple identification information that corresponds to multiple viewpoint information. As a result, only the additional data required when switching viewpoints can be acquired, reducing the processing load.

[0057] An encoding device according to a 24th aspect of the present disclosure comprises a circuit and a memory connected to the circuit, wherein the circuit, in operation, acquires one or more viewpoint pieces of information, generates a bitstream including control information containing the one or more viewpoint pieces of information, and encoded data obtained by encoding three-dimensional data, and each of the one or more viewpoint pieces of information includes one or more identification pieces of information that identify one or more subregions included in the region defined by the viewpoint piece of information, from among one or more subregions corresponding to at least a part of the three-dimensional data.

[0058] This allows a device equipped with circuitry and memory to generate a bitstream that associates at least a portion of the three-dimensional data with information identifying a sub-region corresponding to the viewpoint. As a result, the necessary portion can be selected and used for partial decoding, thereby reducing the processing load on the decoding side.

[0059] A decoding device according to a 25th aspect of the present disclosure comprises a circuit and a memory connected to the circuit, wherein the circuit, in operation, acquires a bitstream, acquires control information including one or more viewpoint pieces of information and at least a portion of encoded data corresponding to at least a portion of three-dimensional data from the bitstream, decodes at least a portion of the encoded data to acquire at least a portion of the three-dimensional data, and each of the one or more viewpoint pieces of information includes one or more identification pieces of information that identify one or more sub-regions included in the region defined by the viewpoint piece of one or more sub-regions corresponding to at least a portion of the three-dimensional data.

[0060] As a result, the device, equipped with circuitry and memory, can acquire control information and at least a portion of encoded data from the bitstream and decode at least a portion of the three-dimensional data. Consequently, the device can perform decoding while selecting the necessary parts using identification information contained in the viewpoint information, thereby suppressing the decoding of unnecessary parts.

[0061] These comprehensive or specific embodiments may be implemented as a system, method, integrated circuit, computer program, or recording medium such as a computer-readable CD-ROM, or as any combination of a system, method, integrated circuit, computer program, and recording medium.

[0062] The embodiments will be described in detail below with reference to the drawings. Note that the embodiments described below are all specific examples of this disclosure. The numerical values, shapes, materials, components, arrangement and connection configurations of components, steps, and the order of steps shown in the following embodiments are examples only and are not intended to limit this disclosure. Furthermore, components in the following embodiments that are not described in an independent claim will be described as optional components.

[0063] (Embodiment) [Three-Dimensional Data Encoding and Decoding System] First, the configuration of the three-dimensional data encoding and decoding system according to this embodiment will be described. Figure 1 is a diagram showing an example of the configuration of the three-dimensional data encoding and decoding system according to this embodiment. As shown in Figure 1, the three-dimensional data encoding and decoding system includes a three-dimensional data encoding system 101, a three-dimensional data decoding system 102, a sensor terminal 103, and an external connection unit 104.

[0064] The three-dimensional data encoding system 101 generates encoded data or multiplexed data by encoding three-dimensional data such as three-dimensional point cloud data or three-dimensional mesh data. The three-dimensional data encoding system 101 may be a three-dimensional data encoding device implemented by a single device, or it may be a system implemented by multiple devices. Furthermore, the three-dimensional data encoding device may include some of the multiple processing units included in the three-dimensional data encoding system 101.

[0065] The three-dimensional data encoding system 101 includes a three-dimensional data generation system 111, a presentation unit 112, an encoding unit 113, a multiplexing unit 114, an input / output unit 115, and a control unit 116. The three-dimensional data generation system 111 also includes a sensor information acquisition unit 117 and a three-dimensional data generation unit 118.

[0066] The sensor information acquisition unit 117 acquires sensor information (sensor signals) from the sensor terminal 103 and outputs the sensor information to the three-dimensional data generation unit 118. The three-dimensional data generation unit 118 generates three-dimensional data from the sensor information and outputs the three-dimensional data to the encoding unit 113.

[0067] The display unit 112 presents sensor information or three-dimensional data to the user. For example, the display unit 112 displays information or images based on sensor information or three-dimensional data.

[0068] The encoding unit 113 encodes (compresses) the three-dimensional data and outputs the resulting encoded data, control information obtained during the encoding process, and other additional information to the multiplexing unit 114. The additional information includes, for example, sensor information.

[0069] The multiplexing unit 114 generates multiplexed data by multiplexing the encoded data input from the encoding unit 113, control information, and additional information. The format of the multiplexed data is, for example, a file format for storage or a packet format for transmission.

[0070] The input / output unit 115 (for example, the communication unit or interface) outputs the multiplexed data to the outside. Alternatively, the multiplexed data is stored in a storage unit such as internal memory. The control unit 116 (or application execution unit) controls each processing unit. In other words, the control unit 116 performs control such as encoding and multiplexing.

[0071] The sensor information may also be input to the encoding unit 113 or the multiplexing unit 114. Furthermore, the input / output unit 115 may output the three-dimensional data or encoded data directly to the outside.

[0072] The transmission signal (multiplexed data) output from the three-dimensional data encoding system 101 is input to the three-dimensional data decoding system 102 via the external connection unit 104.

[0073] The three-dimensional data decoding system 102 generates three-dimensional data, such as three-dimensional point cloud data or three-dimensional mesh data, by decoding encoded data or multiplexed data. The three-dimensional data decoding system 102 may be a three-dimensional data decoding device implemented by a single device, or it may be a system implemented by multiple devices. Furthermore, the three-dimensional data decoding device may include some of the multiple processing units included in the three-dimensional data decoding system 102.

[0074] The three-dimensional data decoding system 102 includes a sensor information acquisition unit 121, an input / output unit 122, a demultiplexing unit 123, a decoding unit 124, a presentation unit 125, a user interface 126, and a control unit 127.

[0075] The sensor information acquisition unit 121 acquires sensor information (sensor signals) from the sensor terminal 103.

[0076] The input / output unit 122 acquires the transmission signal, decodes the multiplexed data (file format or packet) from the transmission signal, and outputs the multiplexed data to the demultiplexing unit 123.

[0077] The demultiplexing unit 123 acquires encoded data, control information, and additional information from the multiplexed data, and outputs the encoded data, control information, and additional information to the decoding unit 124.

[0078] The decoding unit 124 reconstructs the three-dimensional data by decoding the encoded data.

[0079] The presentation unit 125 presents three-dimensional data to the user. For example, the presentation unit 125 displays information or images based on the three-dimensional data. The user interface 126 acquires instructions based on user operations. The control unit 127 (or application execution unit) controls each processing unit. In other words, the control unit 127 performs control such as demultiplexing, decoding, and presentation.

[0080] The input / output unit 122 may acquire three-dimensional data or encoded data directly from an external source. The presentation unit 125 may acquire additional information such as sensor information and present information based on that additional information. The presentation unit 125 may also make presentations based on user instructions acquired through the user interface 126.

[0081] The sensor terminal 103 generates sensor information, which is information obtained from the sensor. The sensor terminal 103 is a terminal equipped with a sensor or camera, and may be, for example, a mobile object such as an automobile, an aerial object such as an airplane, a mobile terminal, or a camera.

[0082] The sensor information that can be acquired by the sensor terminal 103 includes, for example, (1) the distance between the sensor terminal 103 and the object, or the reflectivity of the object, obtained from a LiDAR, millimeter-wave radar, or infrared sensor, and (2) the distance between the camera and the object, or the reflectivity of the object, obtained from multiple monocular camera images or stereo camera images. The sensor information may also include the sensor's attitude, orientation, gyroscope (angular velocity), position (GPS information or altitude), speed, or acceleration. The sensor information may also include temperature, atmospheric pressure, humidity, or magnetism.

[0083] The external connection unit 104 is implemented by an integrated circuit (LSI or IC), an external storage unit, communication with a cloud server via the internet, or broadcasting, etc.

[0084] Next, we will explain three-dimensional point cloud data (hereinafter also referred to as point cloud data). Figure 2 is a diagram showing the structure of point cloud data. Figure 3 is a diagram showing an example of the structure of a data file containing information about point cloud data.

[0085] Point cloud data contains data for multiple points. Each point's data includes location information (three-dimensional coordinates) and attribute information related to that location. A collection of these points is called a point cloud. For example, a point cloud represents the three-dimensional shape of an object.

[0086] Position information, such as three-dimensional coordinates, is sometimes called geometry. Furthermore, the data for each point may include attribute information of multiple attribute types. Attribute types include, for example, color or reflectance.

[0087] One location information may be associated with one attribute information, or multiple attribute information of different attribute types may be associated with one location information. Furthermore, multiple attribute information of the same attribute type may be associated with one location information.

[0088] The example data file structure shown in Figure 3 represents a case where location information and attribute information correspond one-to-one, and it shows the location information and attribute information of the N points that make up the point cloud data.

[0089] Location information includes, for example, information for the three axes: x, y, and z. Attribute information includes, for example, RGB color information. A typical data file is a ply file.

[0090] Next, we will explain three-dimensional mesh data (hereinafter also referred to as mesh data). Figure 4 shows an example of the structure of mesh data. Figure 5 shows an example of the structure of a data file containing mesh data information.

[0091] Mesh data is a data format used in computer graphics (CG). Mesh data represents the three-dimensional shape of an object through a collection of surface information. Surface information consists of polygons such as triangles or quadrilaterals, and is also referred to as a polygon or polygon mesh.

[0092] The components of mesh data include a three-dimensional point cloud (a set of points that have three-dimensional positional information and attribute information corresponding to that positional information), as well as a set of three-dimensional points as vertices, edges connecting two vertices, and faces enclosed by edges.

[0093] Vertices (also expressed as vertex or position) may have attribute information such as color information, reflectivity, or normal vectors. Information indicating the relationships between vertices that constitute an edge or face is also called connectivity. The direction of the normal vector to a point can represent the front and back of a face. Mesh data may also contain attribute information for faces.

[0094] One example of a mesh data file format is an object file. The data file contains the position information G(1) to G(N) and vertex attribute information A(1) to A(N) for the N vertices that make up the mesh. Note that the data file does not have to include attribute information. Also, the attribute information does not have to correspond one-to-one with the vertices. Figure 5 shows an example where the data file has 1 to M attribute information A2.

[0095] Face information is represented by a combination of vertex indices. n[1, 3, 4] indicates that the face is a triangular face composed of three vertices n=1, n=3, and n=4. m[2, 4, 6] indicates that attribute information m=1, m=4, and m=6 correspond to the three vertices, respectively.

[0096] Furthermore, attribute information may be recorded in a separate file from the data file, with the data file indicating its pointer information. For example, attribute information may be stored in a two-dimensional attribute map file, and the file name of the attribute map and its two-dimensional coordinates may be recorded in attribute information A2 of the data file. In any of these methods, it is possible to specify attribute information for a point.

[0097] Next, we will explain the types of three-dimensional data (point cloud data or mesh data). Figure 6 is a diagram showing the types of three-dimensional data. As shown in Figure 6, three-dimensional data includes static objects and dynamic objects.

[0098] A static object is three-dimensional data for any given time (a specific moment). A dynamic object is three-dimensional data that changes over time. Hereafter, three-dimensional point cloud data for a given time will be referred to as a PCC (Point Cloud Compression) frame, or simply a frame. Similarly, three-dimensional mesh data for a given time will be referred to as a mesh frame, or simply a frame.

[0099] The multiple points that make up an object may have limitations in terms of area, pixels, or the number of points, similar to regular video data. Alternatively, the multiple points that make up an object may not have any area limitations, similar to map information.

[0100] Furthermore, there may be point cloud data or mesh data of various densities, including both sparse and dense point cloud data or mesh data.

[0101] The details of each processing unit are described below. Sensor information is acquired by various methods, such as distance sensors like LIDAR or rangefinders, stereo cameras, or combinations of multiple monocular cameras. The three-dimensional data generation unit 118 generates three-dimensional data based on the sensor information obtained by the sensor information acquisition unit 117. The three-dimensional data generation unit 118 generates position information as three-dimensional data and adds attribute information to the position information.

[0102] The three-dimensional data generation unit 118 may process the three-dimensional data when generating position information or adding attribute information. For example, the three-dimensional data generation unit 118 may reduce the amount of data by deleting point clouds with overlapping positions. The three-dimensional data generation unit 118 may also transform the position information (e.g., position shift, rotation, or normalization). Furthermore, the three-dimensional data generation unit 118 may generate mesh data from the point cloud data. Finally, the three-dimensional data generation unit 118 may render attribute information.

[0103] In Figure 1, the three-dimensional data generation system 111 is included in the three-dimensional data encoding system 101, but it may also be provided independently outside of the three-dimensional data encoding system 101.

[0104] The encoding unit 113 generates encoded data by encoding three-dimensional data. There are the following encoding methods. The first is an encoding method using positional information, which will be referred to as the first encoding method hereafter. The second is an encoding method using a video codec, which will be referred to as the second encoding method hereafter.

[0105] The decoding unit 124 decodes the point cloud data by decoding the encoded data. The multiplexing unit 114 generates multiplexed data by multiplexing the encoded data using an existing multiplexing method. The generated multiplexed data is transmitted or stored. In addition to the PCC encoded data, the multiplexing unit 114 multiplexes other media such as video, audio, subtitles, applications, files, or reference time information. Furthermore, the multiplexing unit 114 may also multiplex sensor information or attribute information related to the three-dimensional data.

[0106] Multiplexing methods or file formats include ISOBMFF, ISOBMFF-based transmission methods such as MPEG-DASH, MMT, MPEG-2 TS Systems, and RTP.

[0107] The demultiplexing unit 123 extracts encoded data, other media, and time information from the multiplexed data.

[0108] The input / output unit 115 transmits the multiplexed data using a method appropriate to the transmission medium or storage medium, such as broadcasting or communication. The input / output unit 115 may communicate with other devices via the Internet, or with storage units such as cloud servers.

[0109] Communication protocols such as HTTP, FTP, TCP, or UDP can be used. A pull-type communication method or a push-type communication method may be used.

[0110] Either wired or wireless transmission may be used. Wired transmission methods include Ethernet®, USB, RS-232C, HDMI®, or coaxial cable. Wireless transmission methods include wireless LAN, Wi-Fi®, Bluetooth®, or millimeter wave.

[0111] Furthermore, broadcasting formats such as DVB-T2, DVB-S2, DVB-C2, ATSC3.0, or ISDB-S3 may be used.

[0112] [First Encoding Method] Hereafter, encoding and decoding methods for three-dimensional point clouds or three-dimensional meshes will be described. When the apparatus, processing, or syntax in this disclosure relates to the encoding or decoding of point cloud data, it can also be applied to the encoding or decoding of vertices in a three-dimensional mesh. Furthermore, disclosures relating to the encoding or decoding of vertices in a three-dimensional mesh can also be applied to the encoding or decoding of point cloud data. In addition, disclosures relating to the encoding or decoding of attribute information of point cloud data may be applied to the encoding or decoding of face information, or attribute information for faces or vertices in a three-dimensional mesh. Moreover, the processing may be unified between point cloud encoding and mesh encoding. By unifying the processing, it may be possible to reduce the size of the circuit or software.

[0113] Figure 7 shows the configuration of a first encoding unit 130, which is an example of an encoding unit 113 that performs encoding using the first encoding method. Figure 8 is a block diagram of the first encoding unit 130. The first encoding unit 130 generates encoded data (encoded stream) by encoding point cloud data using the first encoding method. This first encoding unit 130 includes a location information encoding unit 131, an attribute information encoding unit 132, an additional information encoding unit 133, and a multiplexing unit 134.

[0114] The first encoding unit 130 is characterized by performing encoding while being aware of the three-dimensional structure. Furthermore, the first encoding unit 130 is characterized in that the attribute information encoding unit 132 performs encoding using information obtained from the position information encoding unit 131. The first encoding method is also called G-PCC (Geometry-based PCC).

[0115] The point cloud data is PCC point cloud data such as a PLY file, or PCC point cloud data generated from sensor information, and includes position information, attribute information, and other additional information (MetaData). The position information is input to the position information encoding unit 131, the attribute information is input to the attribute information encoding unit 132, and the additional information is input to the additional information encoding unit 133.

[0116] When mesh data is encoded, the position information of the vertex information is input to the position information encoding unit 131, and the face information, or attribute information for faces and vertices, is input to the attribute information encoding unit 132.

[0117] The location information encoding unit 131 generates encoded location information (Compressed Geometry), which is encoded data, by encoding the location information. For example, the location information encoding unit 131 encodes the location information using an N-tree structure such as an octree. Specifically, in an octree, the target space is divided into eight nodes (subspaces), and eight bits of information (occupancy code) are generated to indicate whether or not a point cloud is contained in each node. Furthermore, nodes containing point clouds are further divided into eight nodes, and eight bits of information are generated to indicate whether or not a point cloud is contained in each of these eight nodes. This process is repeated until the number of point clouds contained in a predetermined hierarchy or node falls below a threshold.

[0118] The attribute information encoding unit 132 generates encoded attribute information (Compressed Attribute), which is encoded data, by encoding it using the configuration information generated by the location information encoding unit 131. For example, the attribute information encoding unit 132 determines the reference point (reference node) to be referenced in encoding the target point (target node) to be processed, based on the octave tree structure generated by the location information encoding unit 131. For example, the attribute information encoding unit 132 references a surrounding node or adjacent node whose parent node in the octave tree is the same as the target node. However, the method for determining the reference relationship is not limited to this.

[0119] Furthermore, the attribute information encoding process may include at least one of the following: quantization, prediction, and arithmetic encoding. In this case, a reference means using a reference node to calculate the predicted value of the attribute information, or using the state of a reference node (for example, occupancy information indicating whether or not the reference node contains a point cloud) to determine the encoding parameters. For example, encoding parameters may be quantization parameters in the quantization process, or context in arithmetic encoding.

[0120] The additional information encoding unit 133 generates encoded additional information (Compressed MetaData), which is encoded data, by encoding the compressible data among the additional information.

[0121] The multiplexing unit 134 generates a compressed stream, which is encoded data, by multiplexing encoded position information, encoded attribute information, encoded additional information, and other additional information. The generated compressed stream is output to a processing unit of the system layer (not shown).

[0122] Next, we will describe a first decoding unit 140, which is an example of a decoding unit 124 that performs decoding of the first encoding method. Figure 9 is a diagram showing the configuration of the first decoding unit 140. Figure 10 is a block diagram of the first decoding unit 140. The first decoding unit 140 generates point cloud data by decoding the encoded data (encoded stream) encoded by the first encoding method using the first encoding method. This first decoding unit 140 includes a demultiplexing unit 141, a location information decoding unit 142, an attribute information decoding unit 143, and an additional information decoding unit 144.

[0123] A encoded stream, which is encoded data, is input to the first decoding unit 140 from a processing unit of the system layer (not shown).

[0124] The demultiplexing unit 141 separates encoded location information (Compressed Geometry), encoded attribute information (Compressed Attribute), encoded additional information (Compressed MetaData), and other additional information from the encoded data.

[0125] The location information decoding unit 142 generates location information by decoding the encoded location information. For example, the location information decoding unit 142 reconstructs the location information of a point cloud represented by three-dimensional coordinates from encoded location information represented by an N-tree structure such as an octree.

[0126] The attribute information decoding unit 143 decodes the encoded attribute information based on the configuration information generated by the location information decoding unit 142. For example, the attribute information decoding unit 143 determines the reference point (reference node) to be referenced in the decoding of the target point (target node) to be processed, based on the octave tree structure obtained by the location information decoding unit 142. For example, the attribute information decoding unit 143 references a surrounding node or adjacent node whose parent node in the octave tree is the same as the target node. Note that the method for determining the reference relationship is not limited to this.

[0127] Furthermore, the attribute information decoding process may include at least one of the following: inverse quantization, prediction, and arithmetic decoding. In this case, "reference" means using a reference node to calculate the predicted value of the attribute information, or using the state of the reference node (for example, occupancy information indicating whether or not the reference node contains a point cloud) to determine the decoding parameters. For example, decoding parameters may be quantization parameters in the inverse quantization process, or context in arithmetic decoding.

[0128] The additional information decoding unit 144 generates additional information by decoding the encoded additional information. The first decoding unit 140 uses the additional information necessary for decoding location information and attribute information during decoding and outputs the additional information necessary for the application to the outside.

[0129] [Second Encoding Method] Next, a second encoding unit 150, which is an example of an encoding unit 113 that performs encoding using the second encoding method, will be described. Figure 11 is a diagram showing the configuration of the second encoding unit 150. Figure 12 is a block diagram of the second encoding unit 150.

[0130] The second encoding unit 150 generates encoded data (encoded stream) by encoding the point cloud data using a second encoding method. This second encoding unit 150 includes an additional information generation unit 151, a position image generation unit 152, an attribute image generation unit 153, a video encoding unit 154, an additional information encoding unit 155, and a multiplexing unit 156.

[0131] The second encoding unit 150 is characterized by generating a position image and an attribute image by projecting a three-dimensional structure onto a two-dimensional image, and encoding the generated position image and attribute image using an existing video encoding scheme. The second encoding method is also called VPCC (Video-based PCC).

[0132] The point cloud data is PCC point cloud data such as a PLY file, or PCC point cloud data generated from sensor information, and includes position information, attribute information, and other additional information (MetaData).

[0133] The additional information generation unit 151 generates map information for multiple two-dimensional images by projecting a three-dimensional structure onto a two-dimensional image.

[0134] The location image generation unit 152 generates a location image (Geometry Image) based on location information and map information generated by the additional information generation unit 151. This location image is, for example, a depth image in which the distance is indicated as a pixel value. This depth image may be an image of multiple point clouds viewed from one viewpoint (an image of multiple point clouds projected onto a single two-dimensional plane), or multiple images of multiple point clouds viewed from multiple viewpoints, or a single image formed by integrating these multiple images.

[0135] The attribute image generation unit 153 generates an attribute image based on attribute information and map information generated by the additional information generation unit 151. This attribute image is, for example, an image in which attribute information (e.g., color (RGB)) is shown as pixel values. This image may be an image of multiple point clouds viewed from one viewpoint (an image of multiple point clouds projected onto a single two-dimensional plane), or multiple images of multiple point clouds viewed from multiple viewpoints, or a single image formed by integrating these multiple images.

[0136] When mesh data is encoded, the position information of the vertex information is input to the position image generation unit 152, and the face information, or attribute information for faces and vertices, is input to the attribute image generation unit 153.

[0137] The video encoding unit 154 generates encoded data, namely a encoded geometry image and an encoded attribute image, by encoding the position image and attribute image using a video encoding scheme. Any known encoding method may be used as the video encoding scheme. For example, the video encoding scheme may be AVC or HEVC.

[0138] The additional information encoding unit 155 generates encoded additional information (Compressed MetaData) by encoding additional information and map information included in the point cloud data.

[0139] The multiplexing unit 156 generates a compressed stream, which is encoded data, by multiplexing the encoded position image, encoded attribute image, encoded additional information, and other additional information. The generated compressed stream is output to a processing unit of a system layer (not shown).

[0140] Next, we will describe a second decoding unit 160, which is an example of a decoding unit 124 that performs decoding of the second encoding method. Figure 13 is a diagram showing the configuration of the second decoding unit 160. Figure 14 is a block diagram of the second decoding unit 160. The second decoding unit 160 generates point cloud data by decoding the encoded data (encoded stream) encoded by the second encoding method using the second encoding method. This second decoding unit 160 includes a demultiplexing unit 161, a video decoding unit 162, an additional information decoding unit 163, a location information generation unit 164, and an attribute information generation unit 165.

[0141] A encoded stream (Compressed Stream), which is encoded data, is input to the second decoding unit 160 from a processing unit of the system layer (not shown).

[0142] The demultiplexing unit 161 separates the encoded location image (Compressed Geometry Image), encoded attribute image (Compressed Attribute Image), encoded additional information (Compressed MetaData), and other additional information from the encoded data.

[0143] The video decoding unit 162 generates a position image and an attribute image by decoding the encoded position image and the encoded attribute image using a video encoding scheme. Any known encoding scheme may be used as the video encoding scheme. For example, the video encoding scheme may be AVC or HEVC.

[0144] The additional information decoding unit 163 generates additional information, including map information, by decoding the encoded additional information.

[0145] The location information generation unit 164 generates location information using the location image and map information. The attribute information generation unit 165 generates attribute information using the attribute image and map information.

[0146] The second decoding unit 160 uses the additional information necessary for decoding during decoding and outputs the additional information necessary for the application to the outside.

[0147] [Location Information Coding in the First Coding Method] Figure 15 is a block diagram showing an example configuration of the location information coding unit 131. The location information coding unit 131 comprises an octree coding unit 171 and a prediction tree coding unit 172. The octree coding unit 171 generates coded location information and metadata by coding location information using an octree coding method (octree coding). The prediction tree coding unit 172 generates coded location information and metadata by coding location information using a prediction tree coding method (prediction tree coding).

[0148] The location information encoding unit 131 encodes location information using either octree encoding or predictive tree encoding, or both. The location information encoding unit 131 may switch between these two methods, use an encoding method other than these two, or include the location information in the bitstream as raw data without encoding it. Information indicating the encoding method of the encoded data is stored in metadata and notified to the decoding device.

[0149] Figure 16 is a block diagram showing an example configuration of the location information decoding unit 142. The location information decoding unit 142 comprises an octree decoding unit 173 and a predictive tree decoding unit 174. The octree decoding unit 173 generates location information by decoding the encoded location information using an octree decoding method (octree decoding). The predictive tree decoding unit 174 generates location information by decoding the encoded location information using a predictive tree decoding method (predictive tree decoding). The location information decoding unit 142 also performs decoding using the encoding method notified in the metadata.

[0150] Next, an example of the configuration of the location information coding unit will be described. Figure 17 is a block diagram of the octree coding unit 171 according to this embodiment. The octree coding unit 171 comprises an octree generation unit 181, a geometric information calculation unit 182, a coding table selection unit 183, and an entropy coding unit 184.

[0151] The octree generation unit 181 generates an octree from the input position information and generates occupancy codes for each node in the octree. The geometric information calculation unit 182 obtains information indicating whether the adjacent nodes of the target node are occupied nodes or not. For example, the geometric information calculation unit 182 calculates the occupancy information of adjacent nodes (information indicating whether the adjacent node is an occupied node or not) from the occupancy code of the parent node to which the target node belongs. Alternatively, the geometric information calculation unit 182 may store the encoded nodes in a list and search for adjacent nodes from that list. The geometric information calculation unit 182 may also switch adjacent nodes depending on the position of the target node within the parent node.

[0152] The coding table selection unit 183 selects a coding table to be used for entropy coding of the target node using the occupancy information of adjacent nodes calculated by the geometric information calculation unit 182. For example, the coding table selection unit 183 may generate a bit sequence using the occupancy information of adjacent nodes and select a coding table with an index number generated from that bit sequence.

[0153] The entropy coding unit 184 generates coded location information and metadata by performing entropy coding on the occupancy code of the target node using the coding table of the selected index number. The entropy coding unit 184 may also add information indicating the selected coding table to the coded location information.

[0154] The following describes the octree representation and the scanning order of location information. Location information (location data) is converted into an octree structure (octreeization) and then encoded. An octree structure consists of nodes and leaves. Each node has eight nodes or leaves, and each leaf has voxel (VXL) information. Figure 18 shows an example of the structure of location information containing multiple voxels. Figure 19 shows an example of the location information shown in Figure 18 converted into an octree structure. Here, among the leaves shown in Figure 19, leaves 1, 2, and 3 represent the voxels VXL1, VXL2, and VXL3 shown in Figure 18, respectively, and represent a VXL containing a point cloud (hereinafter referred to as effective VXL).

[0155] Specifically, node 1 corresponds to the overall space encompassing the positional information in Figure 18. The overall space corresponding to node 1 is divided into eight nodes, and of these eight nodes, the node containing the valid VXL is further divided into eight nodes or leaves, and this process is repeated for each level of the tree structure. Here, each node corresponds to a subspace and holds information (occupancy code) indicating the position of the next node or leaf after the division as node information. In addition, the lowest-level block is set as a leaf, and leaf information such as the number of points contained within the leaf is held.

[0156] Next, an example of the configuration of the location information decoding unit will be described. Figure 20 is a block diagram of the octree decoding unit 173 according to this embodiment. The octree decoding unit 173 comprises an octree generation unit 191, a geometric information calculation unit 192, an encoding table selection unit 193, and an entropy decoding unit 194.

[0157] The octane tree generation unit 191 generates an octane tree of a certain space (node) using the header information or metadata of the bitstream. For example, the octane tree generation unit 191 generates a large space (root node) using the size of the x, y, and z axis directions of a certain space attached to the header information, and generates an octane tree by dividing that space into two in each direction of the x, y, and z axes to generate eight small spaces A (nodes A0 to A7). Also, nodes A0 to A7 are set in order as the target nodes.

[0158] The geometric information calculation unit 192 obtains occupancy information indicating whether an adjacent node to the target node is an occupied node. For example, the geometric information calculation unit 192 calculates the occupancy information of an adjacent node from the occupancy code of the parent node to which the target node belongs. Alternatively, the geometric information calculation unit 192 may store the decoded nodes in a list and search for adjacent nodes from that list. The geometric information calculation unit 192 may also switch adjacent nodes depending on the position of the target node within its parent node.

[0159] The coding table selection unit 193 selects a coding table (decoding table) to be used for entropy decoding of the target node using the occupancy information of adjacent nodes calculated by the geometric information calculation unit 192. For example, the coding table selection unit 193 may generate a bit sequence using the occupancy information of adjacent nodes and select a coding table with an index number generated from that bit sequence.

[0160] The entropy decoding unit 194 generates location information by entropy decoding the occupancy code of the target node using the selected coding table. Alternatively, the entropy decoding unit 194 may decode and obtain information from the bitstream regarding the selected coding table, and then entropy decode the occupancy code of the target node using the coding table indicated by this information.

[0161] [Attribute Information Encoding in the First Encoding Scheme] The configuration of the attribute information encoding unit and the attribute information decoding unit will be described below. Figure 21 is a block diagram showing an example configuration of the attribute information encoding unit 132. The attribute information encoding unit may include multiple encoding units that perform different encoding methods. For example, the attribute information encoding unit may switch between the following two methods depending on the use case.

[0162] The attribute information encoding unit 132 includes an LoD attribute information encoding unit 201 and a conversion attribute information encoding unit 202. The LoD attribute information encoding unit 201 uses the positional information of the three-dimensional points to classify each three-dimensional point into multiple layers, predicts the attribute information of the three-dimensional points belonging to each layer, and encodes the predicted residual. Here, each classified layer is called LoD (Level of Detail).

[0163] The attribute information encoding unit 202 encodes attribute information using RAHT (Region Adaptive Hierarchical Transform). Specifically, the attribute information encoding unit 202 generates high-frequency and low-frequency components of each layer by applying RAHT or Haar transform to each attribute information based on the position information of three-dimensional points, and encodes these values ​​using quantization and entropy coding, etc.

[0164] Figure 22 is a block diagram showing an example configuration of the attribute information decoding unit 143. The attribute information decoding unit may include multiple decoding units that perform different decoding methods. For example, the attribute information decoding unit may decode by switching between the following two methods based on the information contained in the header and metadata.

[0165] The attribute information decoding unit 143 includes an LoD attribute information decoding unit 203 and a converted attribute information decoding unit 204. The LoD attribute information decoding unit 203 classifies each three-dimensional point into multiple layers using the position information of the three-dimensional points, and decodes the attribute values ​​while predicting the attribute information of the three-dimensional points belonging to each layer.

[0166] The attribute information decoding unit 204 decodes attribute information using RAHT (Region Adaptive Hierarchical Transform). Specifically, the attribute information decoding unit 204 decodes attribute values ​​by applying inverse RAHT or inverse Haar transform to the high-frequency and low-frequency components of each attribute value based on the position information of the three-dimensional points.

[0167] Figure 23 is a block diagram of a transformation attribute information encoding unit 202, which is an example of the transformation attribute information encoding unit 202. The transformation attribute information encoding unit 202 comprises a sorting unit 211, a Haar transformation unit 212, a quantization unit 213, an inverse quantization unit 214, an inverse Haar transformation unit 215, a memory 216, and an arithmetic encoding unit 217.

[0168] The sorting unit 211 generates Morton codes using the position information of three-dimensional points and sorts the multiple three-dimensional points in Morton code order. The Haar transform unit 212 generates coding coefficients by applying the Haar transform to the attribute information. The quantization unit 213 quantizes the coding coefficients of the attribute information.

[0169] The inverse quantization unit 214 inversely quantizes the encoded coefficients after quantization. The inverse Haar transform unit 215 applies the inverse Haar transform to the encoded coefficients. The memory 216 stores the attribute information values ​​of the decoded three-dimensional points. For example, the attribute information of the decoded three-dimensional points stored in the memory 216 may be used for predicting unencoded three-dimensional points.

[0170] The arithmetic coding unit 217 calculates ZeroCnt from the quantized coding coefficients and arithmetically codes ZeroCnt. The arithmetic coding unit 217 also arithmetically codes the non-zero coding coefficients after quantization. The arithmetic coding unit 217 may also binarize the coding coefficients before arithmetic coding. The arithmetic coding unit 217 may also generate and code various header information.

[0171] Figure 24 is a block diagram of a conversion attribute information decoding unit 204, which is an example of a conversion attribute information decoding unit 204. The conversion attribute information decoding unit 204 comprises an arithmetic decoding unit 221, an inverse quantization unit 222, an inverse Haar conversion unit 223, and a memory 224.

[0172] The arithmetic decoding unit 221 arithmetically decodes the ZeroCnt and coding coefficients contained in the bitstream. The arithmetic decoding unit 221 may also decode various header information.

[0173] The inverse quantization unit 222 inversely quantizes the arithmetic-decoded coding coefficients. The inverse Haar transform unit 223 applies the inverse Haar transform to the coding coefficients after inverse quantization. The memory 224 stores the attribute information values ​​of the multiple decoded three-dimensional points. For example, the attribute information of the decoded three-dimensional points stored in the memory 224 may be used to predict the undecoded three-dimensional points.

[0174] [Slicing] The encoding device may divide the three-dimensional data into one or more three-dimensional data points and encode them. A three-dimensional point cloud divided into multiple parts is called a slice. A slice is a set of points that have geometry information and attribute information.

[0175] Figure 25 is a block diagram showing an example configuration of a three-dimensional data encoding device in this case. The three-dimensional data encoding device comprises a data division unit 231 and an encoding unit 232.

[0176] The data division unit 231 generates multiple divided three-dimensional data by dividing the three-dimensional data. Each divided three-dimensional data corresponds to a slice. The encoding unit 232 generates encoded data by encoding the positional information and attribute information of each of the multiple divided three-dimensional data (slices).

[0177] Figure 26 is a block diagram showing an example configuration of a three-dimensional data decoding device in this case. The three-dimensional data decoding device comprises a decoding unit 233 and a data merging unit 234. The decoding unit 233 generates multiple divided three-dimensional data by decoding encoded data. Each divided three-dimensional data includes slice position information and attribute information. The data merging unit 234 restores the three-dimensional data by merging the multiple divided three-dimensional data.

[0178] In encoding, there may or may not be dependencies between slices. If there are no dependencies, slices can be encoded or decoded independently. Therefore, processing time can be reduced by processing multiple slices in parallel. Also, the amount of processing can be reduced by partial decoding, which decodes only some of the slices.

[0179] If dependencies exist, an identifier indicating the dependency is stored in the bitstream, and encoding or decoding is performed sequentially, starting with the dependent data (the referenced data).

[0180] The number of divisions and the division method can be any method. The 3D data encoding device may determine the shape of an object and divide the point cloud for each object, or it may divide it based on the number of points included in a slice. Alternatively, the 3D data encoding device may use map information or location information to divide the point cloud based on whether or not it is included in 3D space (tile information).

[0181] Figure 27 shows the relationship between tiles and slices. As shown in Figure 27, tiles correspond to three-dimensional space, and slices correspond to divided three-dimensional point clouds. Multiple tiles may overlap. This enables parallel processing in adaptive encoding or decoding depending on the content or object, improving the flexibility of the point cloud encoding or decoding system.

[0182] [Configuration of Encoded Data] Positional and attribute information for each slice are encoded. At least a portion of the resulting encoded data is stored in the data unit's payload. A header is also added to the payload.

[0183] If multiple attribute pieces of information exist for each point, each of the multiple attribute pieces of information is stored in a data unit. Figure 28 shows an example of the structure of a bitstream (encoded data). Figure 28 shows encoded data of a point cloud of two frames with two types of attribute pieces of information. The bitstream has a geometry (position information) data unit (Geometry Data Unit: Geom) and two attribute (attribute information) data units (Attribute Data Unit: Attr(0), Attr(1)) for each slice. For example, if each point has two attribute pieces of information, color and reflectance, the encoded color data is stored in Attr(0) and the encoded reflectance data is stored in Attr(1). The header of the geometry data unit stores the slice identifier (slice_id). The header of the attribute data unit shows the slice_id of the corresponding (referenced) geometry data unit. Note that data units are sometimes called slices, and data unit headers are sometimes called slice headers.

[0184] Furthermore, metadata related to location information coding is stored in the Geometry Parameter Set (GPS). Metadata related to attribute information coding is stored in the Attribute Parameter Set (APS). Metadata common to multiple PCC frames (PCC sequences) is stored in the Sequence Parameter Set (SPS).

[0185] Each data unit and parameter set is converted to either a NAL (Network Access Layer) unit or a TLV (Type Length Value) unit, and the data stream of the unit is output.

[0186] Figure 29 is a block diagram showing an example configuration of a three-dimensional data encoding device that outputs a stream of TLV units. The three-dimensional data encoding device comprises an encoding unit 241 and a TLV storage unit 242. The encoding unit 241 generates encoded data by encoding point cloud data. The TLV storage unit 242 generates a TLV stream (a single stream of TLV units) by storing (encapsulating) the encoded data in multiple TLV units.

[0187] A unit includes a Type field indicating the data type, a Length field indicating length information, and a Value field that stores the data indicated by the Type field. Note that a unit may also be in a different format, such as one that does not include length information.

[0188] Figure 28 shows a sequence of units that store data units and parameter sets. The arrows in Figure 28 indicate dependencies related to the decoding of encoded data. The source of an arrow depends on the data at the end of the arrow, and the header of the source data shows the identifier of the data at the end of the arrow (referenced). The three-dimensional data decoding device decodes the data at the end of the arrow and uses that decoded data to decode the source data of the arrow. For example, the geometry data unit (Geom) shows the GPS ID and SPS ID corresponding to that geometry data unit. The attribute data unit (Attr) shows the APS ID corresponding to that attribute data unit.

[0189] GPS and APS may be provided one per frame. Alternatively, if the encoding scheme changes for each slice, GPS and APS may be provided for each slice. Furthermore, GPS and APS may be shared across multiple frames (sequences).

[0190] Furthermore, APS may be shared across multiple attribute information sets. The parameter set of the referenced object is sent before the referenced object.

[0191] When interframe prediction is used, or when there are dependencies between data units between frames, a Group of Frames (GOF) containing multiple frames may be constructed. A GOF is a random access unit, and the first slice of a GOF is independent of dependencies and is the slice where decoding begins. A stream may have delimiters that indicate frame boundaries or GOF boundaries, or TLV units that indicate boundaries.

[0192] Next, we will explain the structure of the encoded data and how to store the encoded data in the NAL unit.

[0193] For example, a data format is defined for each type of encoded data. Figure 30 shows an example of encoded data and a NAL unit.

[0194] For example, as shown in Figure 30, the encoded data includes a header and a payload. The encoded data may also include length information indicating the length (data volume) of the encoded data, header, or payload. The encoded data may also be in TLV format including length information. Furthermore, the encoded data does not necessarily have to include a header.

[0195] The header includes, for example, identification information to identify the data. This identification information may indicate, for example, the data type or frame number.

[0196] The header contains, for example, identification information indicating a reference relationship. This identification information is stored in the header when there is a dependency between data, and it is information used to reference the referenced data from the source. For example, the header of the referenced data contains identification information to identify that data. The header of the referenced data contains identification information indicating the referenced data.

[0197] Furthermore, if the referenced or source can be identified or derived from other information, the identifying information for identifying the data or identifying information indicating the reference relationship may be omitted.

[0198] The three-dimensional data encoding device stores the encoded data in the payload of the NAL unit. The NAL unit header contains pcc_nal_unit_type, which is the identification information for the encoded data. Figure 31 shows an example of the semantics of pcc_nal_unit_type.

[0199] As shown in Figure 31, when pcc_codec_type is codec 1 (Codec1: first encoding method), the values ​​of pcc_nal_unit_type from 0 to 10 are the encoded position data (Geometry), encoded attribute X data (AttributeX), encoded attribute Y data (AttributeY), position PS (Geom.PS), attribute XPS (AttriX.PS), attribute YPS (AttriY.PS), position SPS (Geometry Sequence PS), attribute XSPS (AttributeX Sequence PS), attribute YSPS (AttributeY Sequence PS), AU header (AU Header), GOF header (GOF It is assigned to the Header. Values ​​11 and above are also assigned to the backup of Codec 1.

[0200] The following describes modifications of this embodiment. PS has levels, such as frame-level PS, sequence-level PS, and PCC sequence-level PS. If the PCC sequence level is considered a higher level and the frame level a lower level, the following method may be used to store the parameters.

[0201] The default PS value is shown in the higher-level PS. If the value of the lower-level PS differs from the value of the higher-level PS, the PS value is shown in the lower-level PS. Alternatively, the PS value is not listed in the higher-level PS, but is listed in the lower-level PS. Alternatively, information on whether the PS value is shown in the lower-level PS, the higher-level PS, or both is shown in either the lower-level PS or the higher-level PS, or both. Alternatively, the lower-level PS may be merged with the higher-level PS. Alternatively, if the lower-level PS and the higher-level PS overlap, the 3D data encoding device may omit the transmission of one of them.

[0202] The three-dimensional data encoding device may divide the data into slices or tiles and send out the divided data. The divided data contains information for identifying the divided data, and the parameters used for decoding the divided data are included in the parameter set. In this case, pcc_nal_unit_type is defined as an identifier indicating that it is data that stores data or parameters related to tiles or slices.

[0203] [Gaussian Data] The following describes Gaussian data (Gaussian Splatting 3D data). Gaussian data is an example of three-dimensional data. First, we will describe the configuration of a system that encodes or decodes three-dimensional data generated using the Gaussian Splatting method.

[0204] Figure 32 shows a part of the processing in the three-dimensional data generation system shown in Figure 1. The sensor information input unit 301 and the three-dimensional data generation unit 302 shown in Figure 32 correspond, for example, to the sensor information acquisition unit 117 and the three-dimensional data generation unit 118 shown in Figure 1.

[0205] The sensor information input unit 301 acquires point cloud data (three-dimensional point cloud data) and outputs the acquired point cloud data to the three-dimensional data generation unit 302. The three-dimensional data generation unit 302 generates Gaussian data (Gaussian Splatting data) from the point cloud data and outputs the generated Gaussian data.

[0206] Specifically, the three-dimensional data generation unit 302 first generates a grid according to the density of the input point cloud data. Each point in the point cloud data is projected onto the nearest grid. At this time, attribute information such as color, reflectance, or normal associated with the point is also linked to the grid. The three-dimensional data generation unit 302 derives parameters such as the mean and variance of the Gaussian function from the information projected onto the grid.

[0207] The method for generating Gaussian data is not limited to the above. For example, Gaussian data may be generated by machine learning using point cloud data generated from multiple two-dimensional images using Structure from Motion (SfM), etc.

[0208] Gaussian data is a collection of multiple 3D Gaussian (ellipsoid) data. Each 3D Gaussian contains data such as three-dimensional point coordinates, a 3x3 covariance matrix, color, and transparency. It is also possible to represent the above data of a 3D Gaussian by converting it into components described in the PLY format.

[0209] Figure 33 shows the components of Gaussian data described in PLY format. The Gaussian data shown in Figure 33 includes Position, Orientation, Scale, SH Coefficient, and Transparency. Transparency is also called transparency, clarity, or opacity.

[0210] Position is the three-dimensional coordinate of a three-dimensional point. Scale and Orientation are parameters of the three-dimensional covariance matrix. Furthermore, the SH coefficient is the coefficient of the spherical harmonics when color information is represented using spherical harmonics.

[0211] In other words, Gaussian data includes a reference three-dimensional coordinate (position) and information associated with that three-dimensional coordinate (rotation, scale, SH coefficient, transparency) for each 3D Gaussian.

[0212] Note that the structure of Gaussian data is not limited to the above. For example, Gaussian data does not have to include a covariance matrix. For example, Gaussian data may include a position (Position) indicating the three-dimensional coordinates of a three-dimensional point, a scale (Scale) indicating the scale along each axis, a color (Color) indicating RGB, and a transmittance (Transparency).

[0213] Furthermore, Gaussian data may include multiple cubes (voxels) that divide 3D space, instead of multiple ellipsoids. Each voxel has attribute information. For example, each voxel may include a position indicating the three-dimensional coordinates of the voxel's center point, attribute information such as color, reflectivity, or material, and a scale indicating the size of the voxel.

[0214] Furthermore, the position may be expressed not in three-dimensional coordinates, but in two-dimensional point coordinates (x, y). In this case, the scale may indicate the size or shape of the ellipsoid on a two-dimensional plane. Specifically, the Gaussian data may include a position indicating the location on a two-dimensional plane, a scale vector indicating the scale (Su, Sv) along each axis, and rotation information indicating the orientation of the ellipsoid.

[0215] Note that the format for describing Gaussian data is not limited to the PLY format. For example, the SPZ format or other formats can also be used to describe Gaussian data.

[0216] Figure 34 is a block diagram showing an example of a device that generates format data from data. This device comprises an encoding unit 1143 and a formatting unit 1144.

[0217] The encoding unit 1143 encodes data consisting of moving images, audio, three-dimensional data (point cloud, mesh, 3D Gaussian splatting, NeRF), neural network models, metadata, SEI, etc., according to a predetermined encoding scheme and generates encoded data. The encoding unit 1143 outputs the generated encoded data as an encoded data bitstream to the formatting unit 1144. In addition to encoded video and audio data, the encoded data may also include metadata such as parameter sets, control information, and SEI. The encoded data may be stored in an encoding unit, which is treated as a processing unit within the encoded data bitstream.

[0218] The formatting unit 1144 formats the input encoded data according to a predetermined system format and outputs it as format data. The formatting unit 1144 multiplexes the encoded data according to, for example, a file format for storage compliant with ISOBMFF or a packet format for transmission compliant with RTP, and can store multiple types of media and related metadata, such as video, audio, three-dimensional data (point cloud, mesh, 3D Gaussian splatting, NeRF), subtitles, application data, file information, reference time information, sensor information, and camera information, in the same format data as needed. The format data may be transmitted externally via input / output means equipped with a communication interface or user interface, or it may be stored in internal memory or storage means.

[0219] Figure 35 is a block diagram showing an example of a device that restores the original data from formatted data.

[0220] The reverse formatting unit 1145 analyzes the input format data according to a predetermined system format, extracts the encoded data stored in the format data, and outputs it as encoded data. The reverse formatting unit 1145 may also extract encoded data according to formats such as ISOBMFF, MPEG-DASH, MMT, AVI, MPEG-2 TS Systems, RTP, glTF, and USD.

[0221] The decoding unit 1146 decodes the encoded data input from the inverse formatting unit 1145 according to a predetermined encoding scheme and outputs data consisting of video, audio, three-dimensional data (point cloud, mesh, 3D Gaussian splatting, NeRF), neural network models, metadata, etc. The output data may be used by applications.

[0222] Figure 36 is a conceptual diagram illustrating an example of how an encoded data bitstream is stored in a system format.

[0223] As shown in this figure, the encoded data bitstream includes not only encoded video and audio data, but also supplementary information such as parameter sets and SEI. The encoded data, parameter sets, and SEI are grouped together as encoding units and treated as NAL units or data units. Examples of these encoding units include the Video NAL unit in video encoding, the TLV unit in G-PCC, and the V3C unit for storing video-based volumetric media, but other types of units and data units may also be used. These encoding units are stored in a system format compliant with ISOBMFF or a transmission format compliant with RTP. Furthermore, the metadata (parameter sets, control information, SEI) of the encoded data described in this embodiment, a part of the metadata, the syntax structure, and the header of the encoded data may be defined as a box structure. These boxes may be contained in the Movie box "moov" or in the Media Data box "mdat". If the metadata is per frame, a new metadata track may be defined and the metadata may be stored in sample entries within the metadata track. Furthermore, the encoded video and three-dimensional data described in this embodiment may also be defined as a box structure and may be stored in an mdat box as a sample or subsample. The header portion may be stored in a moov box and the payload portion in an mdat box. In addition, the SEI described in this embodiment may also be defined as a box structure and may be contained within a moov box or an mdat box. The encoded data may be stored in units, or parts of units may be extracted and stored, or the syntax structure may be modified as needed. Note that the above is an example of storing encoded data in ISOBMFF, and the same approach can be applied when storing encoded data in other formats.

[0224] Figure 37 shows an example of the box structure of ISOBMFF.

[0225] ISOBMFF is an ISO-based media file format specified in ISO / IEC 14496-12, and is a media-independent file format standard for multiplexing and storing various media such as video, audio, and text. In ISOBMFF, a file is composed of multiple boxes, each box consisting of type, length, and data. This embodiment shows an example in which a File type box "ftyp", a Movie box "moov" that stores metadata such as control information, and a Media Data box "mdat" that stores media data such as encoded data are used. The ftyp box indicates the file brand using 4CC and shows compatibility, while the moov box contains a track box that shows media-specific information, including an hdlr box indicating the media type, an stsd box indicating decoding parameters, a tref box indicating reference relationships, and an stco box indicating the data storage location. The mdat box stores samples and subsamples of encoded data corresponding to each media track in the moov box. A sample is a unit of encoded data corresponding to an access unit or a frame at the same time, and a subsample is segmented data corresponding to a part of an access unit or frame. Note that the method of storing each media in ISOBMFF is specified separately; for example, the storage method for AVC video and HEVC video is specified in ISO / IEC 14496-15, and the storage method for three-dimensional data is specified in ISO / IEC 23090-10 and ISO / IEC 23090-18.

[0226] Gaussian data, which represents three-dimensional data using Gaussian splatting, is not limited to the format described as second-order Gaussian data, but can take on multiple formats. Examples include three-dimensional Gaussian format, two-dimensional Gaussian format, voxel splatting format, and spatiotemporal Gaussian format. The rendering unit performs rendering using a rendering method corresponding to the format of the input Gaussian data.

[0227] The format of Gaussian data is sometimes called the data structure of Gaussian data, and sometimes called the representation of Gaussian data. Furthermore, each element included in the format is sometimes called a component.

[0228] Conventionally, Gaussian data is encoded and decoded by an encoding and decoding unit using a point cloud coding scheme. In the preprocessing stage before the encoding unit, the components of the Gaussian data are converted to components of the point cloud coding scheme, and in the postprocessing stage after the decoding unit, the components of the point cloud coding scheme are converted back to components of the Gaussian data. The rendering unit takes Gaussian data with multiple representation methods as input and performs rendering using multiple rendering methods, so it is necessary to understand the representation methods of the Gaussian data.

[0229] However, if the decoding unit does not understand how the Gaussian data is represented, it cannot re-transform the components of the point cloud coding scheme to obtain the components of the original Gaussian data. Similarly, if the rendering unit does not understand how the Gaussian data is represented, it cannot process the Gaussian data properly. For example, if the rendering unit is unsure of the representation method, it cannot select a rendering method, or it may need to analyze the representation method itself.

[0230] Therefore, in this embodiment, the encoding unit generates type information indicating the representation method of Gaussian data. In addition to the type information, the encoding unit generates mapping information indicating the correspondence with the components of the point cloud coding scheme corresponding to the representation method, and conversion information used when converting from the components of the point cloud coding scheme to the components of Gaussian data. The encoding unit generates the type information, the mapping information, and the conversion information as metadata and notifies the decoding unit.

[0231] The conversion information may include, for example, format conversion information used to convert the format of Gaussian data from a first Gaussian format to a second Gaussian format. Furthermore, the conversion information may include conversion parameter information used to format the Gaussian data into data of a data type suitable for encoding or decoding.

[0232] The decoding unit decodes the received encoded data and obtains type information, mapping information, and transformation information from the metadata contained in the decoded data. Based on the mapping information and transformation information, the decoding unit re-converts the components of the decoded point cloud coding scheme to components of Gaussian data and reconstructs the Gaussian data.

[0233] The decoding unit transmits the reconstructed Gaussian data and its type information to the rendering unit. The rendering unit selects a rendering method based on the type information and renders the Gaussian data using the selected rendering method. Furthermore, based on the format conversion information, the rendering unit can switch between outputting Gaussian data expressed in a first Gaussian format, or converting the Gaussian data expressed in the first Gaussian format to second Gaussian data expressed in a second Gaussian format and outputting it, depending on the requirements of the output destination.

[0234] In another embodiment, the three-dimensional Gaussian data generation device may generate Gaussian data, generate type information indicating the representation method of the Gaussian data, and output data including the Gaussian data and type information to the rendering unit. In this case, the rendering unit determines the rendering method based on the type information contained in the input data and performs rendering according to the determined rendering method.

[0235] With the above configuration, Gaussian data in multiple formats can be encoded and decoded using a point cloud coding scheme. Furthermore, the rendering unit can support rendering in multiple formats, eliminating the need for format analysis in the rendering unit.

[0236] Although G-PCC was used as an example of a point cloud coding scheme, this method can be applied not only to G-PCC but also to other coding methods with specific format interfaces, such as volume point cloud coding schemes, volume data compression schemes, and multi-view coding schemes, and similar effects can be obtained.

[0237] Gaussian data (GS data) is data used to represent three-dimensional data using Gaussian splatting, and it contains multiple Gaussian parameters.

[0238] The Gaussian data in question can take on multiple formats depending on the arrangement and structure of the Gaussian parameters.

[0239] Figure 38 shows the structure of the 2DGS data.

[0240] The GS data in the two-dimensional Gaussian format shown in Figure 38 will be referred to as 2DGS data below. Similarly, GS data in the three-dimensional Gaussian format will be referred to as 3DGS data below. 2DGS data is GS data with two-dimensional Scale values. The elliptical shape of 2DGS data is a two-dimensional ellipse, and 2DGS data is particularly suitable for representing planes. As shown in Figure 38, 2DGS data includes Position, Orientation, Scale, SH Coefficient, and Opacity. Here, Position corresponds to positional information, Orientation and Scale may be included in the shape value, and Opacity corresponds to transparency.

[0241] Figure 39 is a diagram showing the structure of 3DVS data.

[0242] The GS data in the 3DVS (Voxel Splating) format shown in Figure 39 will be referred to as 3DVS data below. While 3DGS data contains multiple ellipsoids, 3DVS data contains multiple cubes, i.e., multiple voxels, which are divisions of three-dimensional space. While 3DGS data uses Scale and Orientation to represent ellipsoids centered on Position, 3DVS data contains multiple cubes that divide three-dimensional space centered on Position. As shown in Figure 39, the data structure of 3DVS data consists of Position, SH Coefficient, Opacity, and octree level. Here, octree level is used to represent the size of the voxel. For example, if we represent Position as an octree and let max_octree_level be the maximum number of levels, and octree level be the level from the top level, then the voxel size can be calculated as 2^(max_octree_level - octree level).

[0243] Figure 40 shows the structure of the STG data.

[0244] The GS data in STG (Space-Time Splatting) format shown in Figure 40 will be referred to as STG data below. The STG format is a format that represents the GS data of multiple frames as the GS data of one frame. The STG data shown in Figure 40 includes Position, Orientation, Scale, and Opacity, similar to 3DGS data, as well as Polynomial Trage, Polynomial Rotation, Temporal Scale, Temporal Opacity, Base color feature, View feature, and Time feature. Polynomial Trajectory shows the movement trajectory over time, Polynomial Rotation shows the rotation trajectory over time, Temporal Scale shows the time variation of the visible width of the spatiotemporal Gaussian, and Temporal Opacity shows the opacity of the feature represented by the spatiotemporal Gaussian. Additionally, Base color feature shows the time-independent base color, View feature shows the effect of line of sight direction on the radiance of the spatiotemporal Gaussian, and Time feature shows the time-dependent change in the radiance of the spatiotemporal Gaussian.

[0245] Polynomial Trajectory, Polynomial Rotation, Temporal Scale, and Temporal Opacity are the motion components of position, rotation, scale, and transparency, respectively. By using this motion information as attribute information, GS data from multiple frames can be represented in a single frame. Furthermore, Base color feature, View feature, and Time feature are parameters for indicating color information from multiple frames and multiple viewpoints, and are components similar to the SH coefficients of the spherical harmonics of multiple frames. In addition, Base color feature, View feature, and Time feature are parameters of MLP, and by using them in combination with MLP, color information for any time and any viewpoint can be output. Therefore, when using STG data, MLP configuration information may be used separately.

[0246] Figure 41 shows the input and output of the spherical harmonic region of color.

[0247] The spherical harmonic color unit 1038 expresses three-dimensional color information by representing color using spherical harmonic functions. As shown in Figure 41, the spherical harmonic color unit 1038 accepts information from an arbitrary viewpoint as input and outputs R, G, and B, which are color information as seen from that viewpoint. Therefore, the spherical harmonic color unit 1038 expresses color changes according to differences in viewpoint. Furthermore, the spherical harmonic color unit 1038 expresses color changes when the direction is changed in more detail by increasing the level of the spherical harmonic function.

[0248] Figure 42 shows the number of coefficients for each level.

[0249] A spherical harmonic function has a number of elements corresponding to the level of resolution it represents. For example, if a spherical harmonic function represents Level = 2, it has a total of 9 elements from Level 0 to Level 2, i.e., 9 SH coefficients. For example, if a spherical harmonic function represents Level = 3, it has a total of 16 elements from Level 0 to Level 3, i.e., 16 SH coefficients. As shown in Figure 42, a spherical harmonic function has 1 element at Level 0, 3 elements at Level 1, 5 elements at Level 2, and 7 elements at Level 3. Note that a spherical harmonic function may have Level 4 or higher. Furthermore, the element at Level 0 may be called the DC component, and the elements at Level 1 and above may be called the AC component. The symbols in parentheses in Figure 42 correspond to the variable names of the R component in Figure 43.

[0250] Figure 43 shows an example of the coefficients of spherical harmonics.

[0251] The GSC Spherical Harmonics Coefficient shown in Figure 43 illustrates an example of spherical harmonic coefficients. For example, the example in Figure 38 shows the spherical harmonic function of color at Level = 3, and since the three elements of color, R, G, and B, are each represented by 16 SH coefficients, there are a total of 48 elements. In the coefficient example shown in Figure 43, at Level 0, f_dc0, f_dc1, and f_dc2 are assigned to R, G, and B, respectively. In the coefficient example shown in Figure 43, at Level 1 and above, multiple f_rest values ​​are assigned to each component of R, G, and B, depending on the level.

[0252] Furthermore, spherical harmonics may be used not only to represent color, but also to represent other attribute information such as reflectance or infrared information. In this case, the attribute information will have the SH coefficient of the spherical harmonics as an element. Also, the Level may be changed according to the attribute information or resolution.

[0253] [Rendering of Gaussian Data] Next, the rendering process of Gaussian data will be explained. Figure 44 shows the rendering process of Gaussian data. As shown in Figure 44, the rendering unit 303 generates a 3D model (three-dimensional model) by rendering the Gaussian data. This makes it possible to present the generated 3D model in applications such as 3D display devices. A 3D display device is, for example, a device that displays a stereoscopic image in space like a hologram, or a glasses-free 3D display device.

[0254] For example, the example shown in Figure 33 represents the spherical harmonics of a level 3 color, where the three elements of color (R, G, B) are each represented by 16 SH coefficients. Therefore, the spherical harmonics have 16 × 3 = 48 elements. It is also possible to define levels 4 and above.

[0255] Figure 45 shows the rendering process. As shown in Figure 45, the rendering unit 305 generates a 3D model or 2D image from a specified viewpoint based on Gaussian data and the specified viewpoint. The generated 3D model can be displayed in applications such as VR display. The generated 2D image can also be displayed in applications such as 2D image display devices.

[0256] Furthermore, other attribute information (such as reflectance or infrared information) may be represented using spherical harmonics, not just color. In that case, the Gaussian data will include the SH coefficients of the spherical harmonics for each attribute information. Also, the level may be changed depending on the attribute information or resolution.

[0257] [Gaussian Data Encoding and Decoding] Next, the encoding and decoding processes for Gaussian data will be described. Figure 46 is a block diagram of the encoding device (encoding system) according to this embodiment. This encoding device encodes and multiplexes Gaussian data. Figure 47 is a block diagram of the decoding device (decoding system) according to this embodiment. This decoding device decodes the Gaussian data and presents a 3D model or 2D image in the application. Note that the encoding and decoding devices shown in Figures 46 and 47 are included, for example, in the three-dimensional data generation system shown in Figure 1.

[0258] The encoding device shown in Figure 46 includes an encoding unit 311 and a multiplexing unit 312. The encoding unit 311 generates encoded data (bitstream) by encoding the input Gaussian data using a predetermined encoding scheme, and outputs the generated encoded data to the multiplexing unit 312.

[0259] The multiplexing unit 312 generates multiplexed data by multiplexing the input encoded data using a predetermined multiplexing method, and outputs the generated multiplexed data. This multiplexed data is stored or transmitted.

[0260] The decoding device shown in Figure 47 includes a demultiplexing unit 321, a decoding unit 322, and an application unit 323. The demultiplexing unit 321 generates encoded data by demultiplexing the input multiplexed data using a predetermined multiplexing method, and outputs the generated encoded data to the decoding unit 322.

[0261] The decoding unit 322 generates Gaussian data by decoding the input encoded data using a predetermined encoding method (decoding method), and outputs the generated Gaussian data to the application unit 323.

[0262] The application unit 323 includes an input interface unit 324, a rendering unit 325, and a presentation unit 326. The input interface unit 324 acquires user operations. For example, based on user operations, the input interface unit 324 generates viewpoint information indicating the viewpoint entered by the user.

[0263] The rendering unit 325 generates a three-dimensional model or a two-dimensional image by rendering the input Gaussian data. For example, the rendering unit 325 generates a 3D model or a 2D image viewed from a viewpoint entered by the user, indicated by the viewpoint information. The rendering unit 325 may also generate a 3D model or a 2D image viewed from a predetermined viewpoint.

[0264] The presentation unit 326 presents (displays) the three-dimensional model or two-dimensional image generated by the rendering unit 325. Whether to present a 3D model or a 2D image may be determined by, for example, the following method. For example, the rendering unit 325 first generates a three-dimensional model and generates point coordinate information, as shown in Figure 44. After viewpoint information is input, the rendering unit 325 generates a 2D image viewed from a specified viewpoint. At that time, the rendering unit 325 may also accept information specifying a region along with the specified viewpoint. In this way, by generating a 3D model first and then generating a 2D image of the specified region, the amount of processing can be reduced compared to generating a 2D image of all regions. Therefore, the rendering speed can be increased. Alternatively, whether to present a 3D display (3D model) or a 2D image may be determined according to the processing power of the device performing the rendering. For example, in the case of a device with low processing power, the device may not generate a 2D image and may present a 3D model.

[0265] Next, we will describe the process of learning and generating Gaussian data based on multiple two-dimensional images. Figure 48 shows a detailed view of the three-dimensional data generation device 380.

[0266] The three-dimensional data generation device 380 includes an SFM processing unit 381, an initialization unit 382, ​​a Gaussian update unit 383, a two-dimensional ground truth data selection unit 384, a rendering unit 385, a loss calculation unit 386, and a Gaussian parameter optimization unit 387.

[0267] The SFM processing unit 381 executes Structure from Motion (SfM) based on the input two-dimensional images and generates point cloud data. The initialization unit 382 may use the generated point cloud data to determine the initial values ​​of the position coordinates of the Gaussian data.

[0268] The initialization unit 382 determines initial values ​​for other data elements of the Gaussian data in addition to initial values ​​for the position coordinates, using a predetermined method. These other data elements include, for example, scale, orientation, SH coefficient, and opacity. This generates the initial Gaussian data.

[0269] The rendering unit 385 renders the generated Gaussian data based on multiple viewpoint information to produce multiple rendered two-dimensional images. The viewpoint information is, for example, information indicating the viewpoint corresponding to the multiple input two-dimensional images, and may be information estimated by SfM or information input from an external source. The two-dimensional ground truth data selection unit 384 selects a two-dimensional image to be used as ground truth data from among the multiple input two-dimensional images and outputs the selected two-dimensional image as ground truth data for two-dimensional images.

[0270] The loss calculation unit 386 generates error information by comparing the rendered two-dimensional images with the ground truth data of the two-dimensional images, and calculates the loss based on the generated error information. The Gaussian parameter optimization unit 387 optimizes the Gaussian data by changing the parameter values ​​of the Gaussian data based on the loss calculated by the loss calculation unit 386. In addition to optimizing the parameters of the Gaussian data, the Gaussian parameter optimization unit 387 may, for example, add, delete, merge, and split the Gaussian data.

[0271] The Gaussian update unit 383 updates the Gaussian data based on the optimization results from the Gaussian parameter optimization unit 387. For example, the Gaussian update unit 383 may reflect the parameter update results of the Gaussian data and generate updated Gaussian data.

[0272] The three-dimensional data generation device 380 generates Gaussian data by repeatedly optimizing parameters using the Gaussian parameter optimization unit 387 and adding or deleting Gaussian data, and outputs the generated Gaussian data.

[0273] (First Embodiment) [Gaussian Data Encoding Using G-PCC] Figure 49 is a block diagram showing an example configuration of an encoding device 330 (three-dimensional data encoding device) according to this embodiment. The encoding device 330 generates a bitstream (encoded data) by encoding the first Gaussian data.

[0274] The encoding device 330 includes a first preprocessing unit 331, a second preprocessing unit 332, and a G-PCC encoding unit 333.

[0275] The first Gaussian data includes, for example, a covariance matrix. The first preprocessing unit 331 converts the first Gaussian data into second Gaussian data composed of components of a ply format. For example, the components of a ply file include three-dimensional coordinates, scale, rotation, SH coefficients of spherical harmonics, and transmittance, as shown in Figure 33.

[0276] In this example, first Gaussian data is input to the encoding device 330, and the encoding device 330 converts the first Gaussian data into second Gaussian data. However, second Gaussian data may also be input to the encoding device 330. In this case, the encoding device 330 does not need to include the first preprocessing unit 331.

[0277] The second preprocessing unit 332 converts the multiple components included in the second Gaussian data into a format that can be encoded using the encoding scheme corresponding to each component, and outputs the converted multiple components to the G-PCC encoding unit 333.

[0278] The G-PCC coding unit 333 encodes the three-dimensional coordinates using a geometry (position information) coding method (such as an octave coding method, a predictive tree coding method, or a TriSoup method) within the G-PCC coding method (Geometry-based PCC). The G-PCC coding unit 333 encodes information associated with the three-dimensional coordinates, such as rotation, scale, SH coefficient, and transmittance, using an attribute coding method (such as an LoD-based coding method (LoD attribute information coding method) or a Transform-based coding method (transformation attribute information coding method)) within the G-PCC coding method.

[0279] The G-PCC coding unit 333 includes a position information coding unit 341, a rotation coding unit 342, a scale coding unit 343, an SH coefficient coding unit 344, a transmittance coding unit 345, a metadata coding unit 346, and a multiplexing unit 347.

[0280] The position information encoding unit 341 encodes three-dimensional coordinates. These three-dimensional coordinates are reference or representative three-dimensional coordinates for the Gaussian data. For example, the three-dimensional coordinates may be the center coordinates of an ellipse that constitutes the Gaussian data, or they may be other coordinates. For example, the three-dimensional coordinates may be the origin coordinates of the grid used to generate the Gaussian data.

[0281] Furthermore, the three-dimensional coordinates are not processed by the first preprocessing unit 331, but are directly input to the second preprocessing unit 332. The second preprocessing unit 332 converts the input three-dimensional coordinates into positive integer information (three-dimensional coordinates). The scale value and offset value used during the conversion are stored in metadata such as SPS and notified to the decoding device.

[0282] The position information encoding unit 341 generates encoded data by encoding the three-dimensional coordinates output from the second preprocessing unit 332 using the G-PCC geometry encoding method. This encoded data is stored in a GDU (Geometry Data Unit) and output. The metadata necessary for decoding the GDU is stored in a GPS (Geometry Parameter Set) and output.

[0283] Furthermore, if the multiple three-dimensional coordinates of the multiple Gaussian data are sparse (density is below a predetermined threshold), the location information coding unit 341 may use a predictive tree coding scheme suitable for coding sparse three-dimensional points. Also, if the multiple three-dimensional coordinates of the multiple Gaussian data are dense (density is above a predetermined threshold), the location information coding unit 341 may use an octave tree coding scheme suitable for coding dense three-dimensional points.

[0284] The first preprocessor 331 converts the covariance matrix contained in the first Gaussian data into rotation values ​​and scale values. The first preprocessor 331 converts the color information contained in the first Gaussian data into spherical harmonics and outputs the SH coefficients in the spherical harmonics.

[0285] The second preprocessor 332 converts the data type of each data contained in the second Gaussian data. Specifically, the second preprocessor 332 converts each data into a positive integer type in order to encode it using the G-PCC coding scheme. For example, if the data is of type float, the second preprocessor 332 converts the data into a positive integer type by performing scaling and offset processing on the data. At this time, the scale value and offset value for each attribute information used in the conversion are stored in metadata such as SPS and notified to the decoding device.

[0286] Furthermore, the second preprocessing unit 332 performs format conversion of each data and mapping processing to map each data to attribute components. For example, if the maximum number of elements (dimensions) per attribute component that can be encoded with the G-PCC attribute coding scheme is 3, the second preprocessing unit 332 converts the input components into attribute components that have three dimensions (three elements) per attribute component. For example, since Scale has three elements, it is output as a single attribute component. Orientation has four elements, so it is converted into two two-dimensional attribute components. The number of elements in the SH coefficient changes for each level. For example, SH coefficients with a level exceeding 3 are converted into multiple attribute components.

[0287] In this process, the second preprocessing unit 332 generates mapping information as metadata for each attribute piece of information, indicating which attribute component each element is mapped to.

[0288] The rotation coding unit 342, scale coding unit 343, SH coefficient coding unit 344, and transmittance coding unit 345 each generate multiple coded data by coding rotation, scale, SH coefficient, and transmittance using the attribute coding method in the G-PCC coding scheme. Each of the generated coded data is stored in an ADU (Attribute Data Unit) and output. The metadata necessary for decoding the ADU is stored in an APS (Attribute Parameter Set) and output. In addition, attribute information is coded using three-dimensional coordinates.

[0289] The metadata encoding unit 346 stores conversion information, including conversion parameters used by the first preprocessing unit 331 or the second preprocessing unit 332, in metadata such as SPS or SEI. The metadata encoding unit 346 may also encode the conversion information before storing it in the metadata. The metadata may also include information on the structure of the Gaussian data, or information indicating how the Gaussian data is transmitted using G-PCC encoding.

[0290] The multiplexing unit 347 stores the encoded data (ADU and GDU) and parameter sets (APS, GPS, and SPS) in the TLV unit and transmits them as a bitstream (encoded data). The bitstream may be multiplexed using a predetermined multiplexing scheme. The bitstream may also be formatted.

[0291] [Decoding Gaussian Data Using G-PCC] Figure 50 is a block diagram showing an example configuration of a decoding device 350 (three-dimensional data decoding device) according to this embodiment. The decoding device 350 generates first Gaussian data or second Gaussian data by decoding a bitstream (encoded data). For example, the decoding device 350 decodes a bitstream generated by the encoding device 330 shown in Figure 49.

[0292] The decoding device 350 includes a G-PCC decoding unit 351, a second post-processing unit 352, and a first post-processing unit 353.

[0293] The G-PCC decoding unit 351 generates three-dimensional coordinates, rotation, scale, SH coefficient, and transmittance by decoding the bitstream using the G-PCC coding scheme. The G-PCC decoding unit 351 comprises a demultiplexing unit 361, a position information decoding unit 362, a rotation decoding unit 363, a scale decoding unit 364, an SH coefficient decoding unit 365, a transmittance decoding unit 366, and a metadata decoding unit 367.

[0294] The demultiplexing unit 361 analyzes multiple TLV units contained in the input bitstream and generates data units of encoded data such as GDU, ADU, SPS, GPS, and APS. The position information decoding unit 362, rotation decoding unit 363, scale decoding unit 364, SH coefficient decoding unit 365, transmittance decoding unit 366, and metadata decoding unit 367 decode these data units using the information in the parameter set, with each attribute component using its respective encoding scheme.

[0295] The position information decoding unit 362 decodes three-dimensional coordinates from the GDU. The rotation decoding unit 363, scale decoding unit 364, SH coefficient decoding unit 365, and transmittance decoding unit 366 decode attribute information (rotation, scale, SH coefficient, and transmittance) from the ADU. The attribute information is decoded using three-dimensional coordinates. The metadata decoding unit 367 obtains conversion information, etc., from metadata such as SPS or SEI. The metadata decoding unit 367 may also decode the conversion information from the metadata. The metadata may also include information on the composition of the Gaussian data, or information indicating the correspondence of how the Gaussian data is transmitted using G-PCC coding.

[0296] The second post-processing unit 352 generates second Gaussian data by performing an inverse transformation on the decoded data. Here, the inverse transformation is the reverse process of the transformation process performed in the second pre-processing unit 332, and is performed based on the transformation information (transformation parameters) contained in the metadata.

[0297] The first post-processing unit 353 generates the first Gaussian data by inversely transforming the second Gaussian data. Here, the inverse transformation is the reverse process of the transformation process performed by the first pre-processing unit 331. At least one of the first Gaussian data and the second Gaussian data is output. Note that the decoding device 350 does not necessarily have to include the first post-processing unit 353.

[0298] [Mapping Process] The second preprocessing unit 332 maps the Gaussian data to the attribute components of the G-PCC. The method of this mapping and the mapping information generated at that time will be explained below.

[0299] The following examples illustrate the mapping of rotation, scale, and SH coefficient, omitting the explanation of transmittance. Similar processing can be applied when other attribute information is added.

[0300] Figure 51 shows an example of mapping multiple attribute components contained in Gaussian data to attribute components of the coding scheme G-PCC. Here, we show an example where the maximum number of dimensions (number of subcomponents) that can be coded in the coding of attribute components in G-PCC is 3.

[0301] In the following, attribute components of Gaussian data may be referred to as Gaussian components, and attribute components of G-PCC may be referred to as G-PCC components. Additionally, attribute components may sometimes be simply referred to as components.

[0302] For example, the Gaussian component of the scale has three subcomponents (B1 to B3) with three dimensions, and is mapped to a single G-PCC component that has three-dimensional subcomponents. In this case, the mapping information indicates that the identifier of the Gaussian component of the scale corresponds to the identifier of the G-PCC component.

[0303] Furthermore, the Gaussian component of rotation has four subcomponents (A1 to A4) with a dimension of 4, and cannot be mapped to a single G-PCC component that has three-dimensional subcomponents. Therefore, the four subcomponents of rotation (A1 to A4) are mapped to two G-PCC components that have two-dimensional subcomponents. In this case, the mapping information indicates that the identifier of the Gaussian component of rotation and the identifier indicating the subcomponent number contained in that attribute component correspond to the identifier of the G-PCC component and the identifier indicating the subcomponent number contained in that G-PCC component.

[0304] Since there are 16 SH coefficients for each level, corresponding to three-dimensional (RGB) elements, the SH coefficients have a total of 48 dimensions (C1 to C48). These 48 dimensions are mapped to 16 G-PCC components, each having three-dimensional subcomponents.

[0305] In Figure 51, "ID" is the attribute component ID, which is an identification number used to identify the attribute component of the encoded data. "ND" is the number of dimensions, which indicates the number of elements (number of subcomponents) per attribute component.

[0306] Furthermore, the method for partitioning and mapping Gaussian data is not limited to the examples above; any combination of mapping method and number of dimensions may be used.

[0307] The methods for dividing and mapping each component of Gaussian data are not limited to the examples above. For example, a Gaussian data component having a number of dimensions exceeding the number of dimensions that can be represented by a given attribute component may be divided into multiple attribute components and mapped. The number of dimensions or subcomponents assigned to each attribute component after division is not limited, and each dimension or subcomponent may be assigned to an attribute component in any unit. For example, if the number of dimensions that can be stored in a given attribute component is up to three, a Gaussian data component having a number of dimensions exceeding three may be divided into multiple attribute components that satisfy the dimensional constraint for each attribute component handled by the encoding scheme. In this case, the combination of the order of dimension assignment to each attribute component, the unit of assignment, and the number of dimensions assigned to each attribute component is not limited.

[0308] The second preprocessor unit 332 generates mapping information in the mapping process that indicates the G-PCC component corresponding to the Gaussian component. This mapping information is stored as metadata in the header or SEI.

[0309] The decoding device decodes mapping information from the bitstream. The second post-processing unit 352 reconstructs the Gaussian data by remapping the decoded data for each G-PCC component to Gaussian data based on the mapping information.

[0310] Figure 52 shows an example of mapping information. In Figure 52, the mapping information is stored in SEI (Gaussian Data Attribute Mapping Information SEI). The mapping information includes the number of Gaussian data components. The number of Gaussian data components indicates the number of attribute information included in the Gaussian data. For example, in the example shown in Figure 51, rotation, scale, and SH coefficient are present, and the number of Gaussian data components is 3.

[0311] Furthermore, the mapping information includes, for each Gaussian component, the Gaussian data component ID, the Gaussian data type, and the number of Gaussian data dimensions.

[0312] The Gaussian component ID is an identifier used to uniquely identify a Gaussian component. The Gaussian data type indicates the type of Gaussian component. For example, a value of 0 indicates "rotation," a value of 1 indicates "scale," a value of 2 indicates "SH coefficient," and a value of 3 indicates "transparency." Note that the combinations of values ​​and types shown here are just examples, and the combinations are not limited to these.

[0313] The Gaussian data dimension indicates the number of dimensions (subcomponents) of the Gaussian components. For example, in the example shown in Figure 51, the Gaussian data dimension for rotation is 4, the Gaussian data dimension for scale is 3, and the Gaussian data dimension for SH coefficients is 48.

[0314] Furthermore, the mapping information includes G-PCC component IDs, G-PCC dimension IDs, and transformation information as information for each dimension of the Gaussian component. The G-PCC subcomponent ID corresponds to the "ID" shown in Figure 51. The G-PCC dimension ID is an identifier indicating the subcomponent number. For example, C1 in the SH coefficient shown in Figure 51 has a G-PCC dimension ID of 0, C2 has a G-PCC dimension ID of 1, and C3 has a G-PCC dimension ID of 2.

[0315] Thus, the mapping information shows the ID and dimension ID of the G-PCC component corresponding to each dimension of each component of the Gaussian data. For example, for subcomponent C46 shown in Figure 51, the G-PCC component ID = 18 and the G-PCC dimension ID = 2 are shown.

[0316] In Figure 52, an example of mapping information showing the subcomponents (dimensions) of the G-PCC corresponding to each subcomponent (dimension) of the Gaussian data is shown. However, mapping information showing the subcomponents of the Gaussian data corresponding to each subcomponent of the G-PCC may also be used.

[0317] Furthermore, while Figure 52 shows an example of mapping information indicating the correspondence between subcomponents of Gaussian data and subcomponents of G-PCC, if there are no constraints on the number of subcomponents in the G-PCC component, mapping information indicating the correspondence between the Gaussian component and the G-PCC component may be used. In other words, it is not necessary to show the correspondence between dimensions.

[0318] The conversion information shows the parameters (conversion parameters) used when converting the component data. Figure 53 shows an example of the configuration of the conversion information. For example, if a scale and an offset were used in the conversion, the conversion information would include the scale value and the offset value.

[0319] For example, the transformation is performed using the scale value and offset value according to the following formula.

[0320] Converted data = Original data × Scale value + Offset

[0321] The formula used for the conversion may be a predefined formula, or multiple conversion formulas may be defined and the formula used may be switched according to predetermined conditions. Furthermore, information indicating the used conversion formula may be stored in the bitstream.

[0322] In Figure 52, transformation information is provided for each subcomponent (dimension) of the Gaussian data. However, transformation information may also be provided for each component of the Gaussian data, and may indicate the components of the corresponding encoding scheme, such as the G-PCC component or the parameters used for the transformation to that dimension. In other words, the transformation method may be changed on a component-by-component basis or on a subcomponent-by-subcomponent basis.

[0323] Note that some of the syntax shown in Figure 52 may be omitted. For example, if the order of listing information for rotation, scale, and SH coefficients, and the number of dimensions are predetermined, the Gaussian data component ID, Gaussian data type, and Gaussian data dimension may be omitted.

[0324] Furthermore, each attribute piece of information is encoded based on the mapping information to the attribute components determined by the second preprocessing unit 332. Figure 54 shows an example of the structure of encoded data when Gaussian data is encoded using G-PCC encoding.

[0325] The SPS contains, for each G-PCC component, a G-PCC attribute component ID (attr_id), a G-PCC attribute type (attribute_type), and the number of dimensions of the G-PCC attribute component (num_dimension). Here, a new G-PCC attribute type may be defined to indicate that each element of the Gaussian data is encoded.

[0326] Figure 55 shows an example of G-PCC attribute types for Gaussian data. As shown in Figure 55, for example, G-PCC attribute types may be defined that represent the rotation, scale, SH coefficient, and transmittance of Gaussian data, respectively.

[0327] The ADU header is assigned a G-PCC attribute component ID (attr_id). The decryption device can use the G-PCC attribute component ID contained in the ADU header and the information contained in the SPS to identify which G-PCC attribute type each ADU belongs to.

[0328] Furthermore, mapping information (Gaussian data attribute mapping information SEI) may be transmitted as SEI or included in SPS. For example, SPS may include mapping information indicating the Gaussian component corresponding to the G-PCC component.

[0329] Figure 56 is a flowchart of the processing performed by the decoding device according to this embodiment. First, the decoding device acquires multiple TLV units from the bitstream, and acquires multiple data units (GDU, ADU) and multiple parameter sets (SPS, GPS, APS) from the multiple TLV units (S101).

[0330] Next, the decoding device decodes the three-dimensional coordinates using SPS, GPS, and GDU (S102). Next, the decoding device decodes the multiple attribute components described in SPS using APS, ADU, and the three-dimensional coordinates (S103).

[0331] Next, the decoding device obtains mapping information by analyzing the Gaussian data attribute mapping information SEI (S104). Next, the decoding device reconstructs the second Gaussian data from the decoded attribute components and mapping information (S105). Next, the decoding device obtains transformation information and generates the first Gaussian data by inversely transforming the second Gaussian data using the transformation information (S106).

[0332] [Other] For the sake of simplicity, the above explanation uses an example where the Gaussian data is in a single frame. However, this method can also be applied when there are multiple frames of Gaussian data. In this case, multi-frame coding in the G-PCC method may be used, or the interpretation method in the G-PCC method may be used.

[0333] The encoding device may also generate multiple partitioned data by dividing the Gaussian data into multiple regions and encode each of the multiple partitioned data. Alternatively, the encoding device may assign the partitioned data to tiles or slices in a G-PCC and encode them.

[0334] In the above explanation, an example was shown of encoding attribute information (scale, rotation, SH coefficient, transmittance, etc.) of the second Gaussian data. However, by using the method of this embodiment, it is also possible to encode other attribute information.

[0335] G-PCC supports the encoding of point cloud data and the partial decoding of encoded data from a specific spatial region. The metadata in G-PCC may store multiple spatial region information and identifiers of corresponding data units. The decoding device uses this metadata to perform partial decoding by identifying the identifier of the data unit corresponding to the specific spatial region to be partially encoded.

[0336] On the other hand, when encoding GS data with G-PCC, conventional point cloud-based metadata may not accurately extract the specific spatial region to be encoded, potentially degrading the decoded data. This potential degradation stems from the fact that GS data has not only reference coordinates but also extent, and GS data affecting the region to be rendered may be mapped to space outside that region.

[0337] Therefore, the encoding device divides the GS data into multiple spaces based on the reference coordinates and encodes them. In addition, the encoding device stores, as metadata for partial decoding, spatial region information corresponding to the reference coordinates, as well as information on multiple spatial regions that take into account the spread of the GS data, and identifiers of data units corresponding to the said spatial region information. The spatial region information that takes into account the spread of the GS data is information that corresponds to the spatial region that affects rendering.

[0338] The decoding device, when determining the Gaussian data to be decoded based on the region to be decoded, refers to spatial region information that takes into account the extent of the GS data. Based on the region to be rendered, the decoding device identifies the encoded data of the GS data corresponding to the spatial region that will affect rendering, and partially decodes the data unit corresponding to that encoded data. Through this partial decoding, the decoding device extracts and partially decodes the GS data that will affect rendering.

[0339] Furthermore, the decoding device can improve rendering quality by comprehensively extracting GS data that affects rendering. In particular, the decoding device can improve rendering quality at the edges of the spatial region corresponding to the edges of the region to be decoded by decoding including GS data that affects rendering in that region.

[0340] Figure 57 shows an example of a point cloud partial decoding configuration.

[0341] The encoding device may divide the point cloud frame and generate multiple sub-point clouds during encoding. The encoding device encodes the geometry and attribute information of the point cloud frame that is not subdivided, or of the subdivided sub-point clouds. The encoding device stores the encoded data in data units for geometry and attribute information, respectively. Here, the subdivided sub-point clouds, or the encoded data of those sub-point clouds, are called slices. The regions of the slices may overlap.

[0342] As shown in Figure 57, the bitstream includes SPS, GPS, APS(0), APS(1), SEI for partial decoding, multiple GDUs, and multiple ADUs. The header of the GDU, which is a geometry data unit, includes slice_id, an identifier that identifies a slice, and slice_tag, an identifier that identifies a group of slices. The decoder identifies the slice to be partially decoded by referring to these identifiers.

[0343] Figure 58 shows the relationship between tiles and slices.

[0344] In G-PCC, tiles may correspond to three-dimensional regions, typically representing rectangular prisms. Information corresponding to a tile includes tile_id (tile identification information) and bounding box information. The bounding box information includes origin coordinates and information indicating the length, width, and height. Figure 58 is schematically shown in two dimensions, but it is actually three-dimensional.

[0345] In the example shown in Figure 58, among the multiple tiles, 3D tile #1 contains slice #1 and slice #2, 3D tile #2 contains slice #3, and 3D tile #3 contains slice #4. The decoding device can identify the slices contained in a tile by identifying the tile corresponding to the spatial region to be partially decoded.

[0346] Figure 59 shows the tile inventory.

[0347] Tile information is stored in tile metadata. Tile metadata may be metadata such as tile inventory, data unit, SEI, or user data. The gpcc_tile_inventory() shown in Figure 59 includes num_tiles and tile_id, bbox_xyz, and bbox_wdh corresponding to each tile. Here, num_tiles indicates the number of tiles, tile_id indicates identification information for identifying the tile, bbox_xyz indicates the origin coordinates of the bounding box, and bbox_wdh indicates the width, depth, and height of the bounding box.

[0348] The decoding device identifies the tile corresponding to the spatial region to be partially decoded by referring to the tile information stored in the tile metadata. The decoding device then performs spatial partial decoding of the point cloud by identifying the slice or data unit corresponding to that tile.

[0349] Figure 60 shows the relationship between the area to be decoded and the tiles.

[0350] When partial decoding is used, the encoding device includes the tile inventory in the bitstream. The slice_tag included in the GDU header indicates the tile_id, which is the identifier of the tile to which the slice corresponding to that GDU belongs. When the decoding device performs partial decoding using the tile inventory, it interprets slice_tag as tile_id. Therefore, the decoding device can associate the tile regions and tile_ids included in the tile inventory with the slice_tags of the slices belonging to those regions.

[0351] In the example shown in Figure 60, the tiles intersecting the region to be decoded are 3D tile #1 and 3D tile #3. Therefore, the decoder analyzes the bitstream and extracts and decodes the GDUs whose slice_tag in the GDU header is 1 or 3. In this case, the decoder extracts and decodes slices #1 and #4. The decoder also extracts attribute information with the same slice_id and decodes that attribute information.

[0352] Figure 61 is a flowchart showing the partial decoding process.

[0353] The decryption device first decrypts the tile metadata (S501). This tile metadata corresponds to the tile inventory described in Figure 60 and includes the tile area and tile ID. The decryption device then determines the area to be decrypted (S502). The decryption device then identifies tiles that intersect or overlap with the area to be decrypted (S503). The decryption device then obtains the tile ID of the identified tiles (S504).

[0354] The decoding device then refers to the GDU header, obtains the GDU indicating the corresponding tile ID, and decodes it, leaving other GDUs undecoded (S505). In this way, the decoding device performs partial decoding by selecting GDUs based on the tile corresponding to the area to be decoded. The decoding device also decodes the ADUs corresponding to the selected GDUs as needed.

[0355] Figure 62 shows the point cloud and GS data.

[0356] The tile information for partially decoding a point cloud contains bounding box information based on the position coordinates of the point cloud. The decoding device performs partial decoding by acquiring tiles that intersect with the decoding target area based on this tile information. On the other hand, GS data includes elliptical information that combines Scale, rotation, and Opacity information. Here, Scale and rotation are included in the shape value, and Opacity corresponds to transparency. Such GS data has an extent relative to the position information and affects the rendering result through this extent. Hereinafter, the spatial region in which the GS data affects the rendering result will be called the influence range of the GS data. Therefore, the influence range of the GS data may exceed the region of the bounding box based on the position information of the GS data. More specifically, GS data outside the bounding box region based on the position information of the GS data may affect the data representation of positions within that bounding box region.

[0357] Figure 63 shows a diagram of conventional tile information.

[0358] As shown in Figure 63, in conventional tile information based on the position coordinates of a point cloud, the decoding device performs partial decoding corresponding to the decoding target area by acquiring tiles that intersect the decoding target area. Conventional tile information is based on bounding box information based on the position coordinates of a point cloud, so the decoding device identifies tiles that intersect the decoding target area based on this bounding box information. The tiles shown with diagonal hatching in Figure 63 indicate tiles that intersect the decoding target area.

[0359] Figure 64 shows the tiles that may be missed during acquisition.

[0360] As shown in Figure 64, when GS data has spread relative to positional information, bounding box information based on point cloud position coordinates may require tiles that do not intersect the target region for decoding. The tiles shown by dot hatching in Figure 64 are necessary for decoding the target region but do not intersect it. Therefore, tile information based on point cloud position coordinates may not correctly represent the target region, and it may not be possible to obtain the tiles necessary for the target region. Consequently, the decoding device may not be able to obtain all the GS data that affects rendering, and may not be able to perform partial decoding of high-quality GS data.

[0361] For example, in the example shown in Figure 64, tiles that do not intersect the region to be decoded occur, and therefore the decoding device cannot acquire these tiles. Consequently, the decoding device may not be able to acquire GS data that affects the edges of the region to be decoded, and may not be able to perform high-quality partial decoding of those edges.

[0362] Figure 65 shows the configuration for partial decoding of GS data.

[0363] The system shown in Figure 65 comprises a GS data partitioning unit 501, an encoding unit 502, a decoding unit 503, and a rendering unit 504. The GS data partitioning unit 501 partitions the input GS data into tiles. The GS data partitioning unit 501 generates metadata for the influence range of the GS data for each tile and outputs the generated metadata to the encoding unit 502. The encoding unit 502 encodes the GS data and metadata and generates encoded data. The decoding unit 503 decodes some of the GS data from the encoded data. The rendering unit 504 performs rendering using some of the GS data decoded by the decoding unit 503.

[0364] Figure 66 is a flowchart showing the processing of the encoding unit.

[0365] The encoding unit 502 first acquires GS data and divides the GS data into tiles (S511). At this time, the encoding unit 502 determines whether the position coordinates of the GS data are included in the tiles based on the tile information and performs tile division. That is, the encoding unit 502 uses the same division method as point clouds to divide the GS data so that the position coordinates of the GS data are included in the tiles. The encoding unit 502 may treat the divided GS data as divided GS data, and may treat the divided GS data or the encoded data of the divided GS data as slices. Next, the encoding unit 502 generates metadata of the influence range of the GS data for each tile (S512). Finally, the encoding unit 502 encodes the GS data and metadata (S513).

[0366] Figure 67 shows the scope of influence of GS data.

[0367] As shown in Figure 67, the solid line indicates the bounding box based on the position coordinates of the point cloud. On the other hand, the dashed line indicates the bounding box showing the influence range of the Gaussian data. Here, the influence range of the Gaussian data is the spatial region in which the Gaussian data affects the rendering result. If the position information of the GS data is included in the tile, the encoding unit 502 derives the influence range of the GS data using the position information, scale value, and rotation value of the GS data. For example, the encoding unit 502 may define the influence range as a range that includes the shape of an ellipse. The encoding unit 502 may also define the influence range as an elliptical region with a predetermined transparency range using Opacity.

[0368] Figure 68 shows the scope of influence of tiles and GS data.

[0369] The encoding unit 502 derives the influence range for all GS data contained in the tile. The encoding unit 502 generates bounding box information that shows the influence range of the GS data contained in each tile. Note that the encoding unit 502 may generate region information other than a rectangular bounding box, not limited to rectangular bounding box information. The thick lines in Figure 68 show the influence range of the Gaussian data contained in the tile. The diagonal hatching in Figure 68 shows tiles that intersect the decoding target region. The dot hatching in Figure 68 shows tiles that are necessary for decoding the decoding target region but do not intersect the decoding target region. Therefore, the encoding unit 502 generates information that includes the influence range of the Gaussian data, rather than tile information based on the position coordinates of the GS data. The decoding device refers to this information to obtain tiles that are necessary for decoding the decoding target region but do not intersect the decoding target region.

[0370] Figure 69 shows the tile inventory.

[0371] The encoding unit 502 generates metadata indicating the influence range of the GS data for each tile. The gs_tile_inventory() shown in Figure 69 includes num_tiles and tile_id, 3dgs_bbox_xyz, and 3dgs_bbox_wdh corresponding to each tile. Here, num_tiles indicates the number of tiles, tile_id indicates the tile identification information, 3dgs_bbox_xyz indicates the origin coordinates of the bounding box indicating the influence range of the GS data, and 3dgs_bbox_wdh indicates the width, depth, and height of the bounding box indicating the influence range of the GS data. Note that the syntax description may be omitted. In this case, the decoding device determines that the order in which the data is described is the tile ID.

[0372] Figure 70 shows the relationship between the rendering area and the tiles.

[0373] When decoding GS data, the decoding device decodes metadata indicating the scope of influence of the GS data contained in each tile. Next, the decoding device determines the rendering area. The decoding device identifies the tiles to be decoded by identifying the tiles where the scope of influence of the GS data intersects with the rendering area.

[0374] As shown in Figure 70, diagonal hatching indicates tiles that intersect the rendering area. Dot hatching in Figure 70 indicates tiles that are necessary for decoding the rendering area but do not intersect the rendering area. The thick lines in Figure 70 indicate the influence range of the Gaussian data contained in the tile. The decoding device determines the tiles where the rendering area and the influence range of the GS data intersect and obtains the tile ID of those tiles.

[0375] The decryption device then refers to the GDU header. If 3dgs_slice_tag matches the corresponding tile ID, the decryption device retrieves and decrypts the GDU. The decryption device then retrieves and decrypts the ADU that has the same slice_id as the GDU. In this way, the decryption device performs partial decryption of the GS data.

[0376] Figure 71 is a flowchart showing the processing of the decoding unit.

[0377] The decoding device first decodes the metadata of the tiles within the GS influence range (S521). Next, the decoding device determines the rendering target area (S522). Next, the decoding device identifies the tiles where the rendering target area and the GS data influence range intersect (S523). Next, the decoding device obtains the tile ID of the identified tile (S524).

[0378] The decoding device then refers to the GDU header, obtains and decodes the GDU of the corresponding tile ID, and does not decode other GDUs (S525). The decoding device may, if necessary, decode ADUs that have the same slice_id as the GDU. In this way, the decoding device performs partial decoding of the GS data by obtaining not only tiles that intersect the rendering target area, but also tiles necessary for decoding the rendering target area based on the influence range of the GS data.

[0379] The decoding device may calculate the amount of overlap between the area indicated by the region information corresponding to each of the multiple tiles included in the candidate tile group and the rendering target area. Based on the calculated amount of overlap, the decoding device may select tiles with higher priority from the candidate tile group. For example, the decoding device may determine that tiles with a larger amount of overlap have a higher priority. This allows the decoding device to prioritize the use of tiles that contribute significantly to the rendering target area within the candidate tile group for partial decoding.

[0380] Figure 72 shows the offset information and difference information.

[0381] In the example shown in Figure 72, the metadata includes information indicating the region for each tile, and region information indicating the scope of influence of the GS data. The region information for each tile includes bbox_xyz and bbox_wdh. Here, bbox_xyz indicates the three-dimensional coordinates of the origin of the tile's region, and bbox_wdh indicates the width, depth, and height of the tile's region.

[0382] The region information indicating the influence range of the GS data may include origin_offset_xyz and wdh_offset_xyz. Here, origin_offset_xyz is information that indicates the origin of the region indicating the influence range of the GS data as an offset position relative to the origin of the tile region. wdh_offset_xyz is information that indicates the range of the region indicating the influence range of the GS data as a difference relative to the width, depth, and height of the tile region.

[0383] The decoding device derives region information indicating the influence range of the GS data based on the region information for each tile and the difference information of the region information indicating the influence range of the GS data. Note that the method for deriving the offset position and difference, as well as the reference value of the difference, are not limited to these methods and other methods may be used. Furthermore, the encoding device may store absolute coordinates instead of difference information.

[0384] Figure 73 shows another example of syntax.

[0385] As shown in Figure 73, gs_tile_inventory() includes num_tiles and the corresponding tiles, tile_id, bbox_xyz, bbox_wdh, origin_offset_flag, and wdh_offset_flag for each tile. origin_offset_flag indicates whether or not origin_offset_xyz is included. wdh_offset_flag indicates whether or not wdh_offset_xyz is included. As shown in Figure 73, if origin_offset_flag is true, gs_tile_inventory() further includes origin_offset_xyz. If wdh_offset_flag is true, then gs_tile_inventory() further includes wdh_offset_xyz.

[0386] Furthermore, the encoding device is not limited to storing area information indicating the scope of influence of the GS data in each tile. For example, the encoding device may place a value common to all tiles outside the num_tiles loop. In this case, for example, the encoding device may set the maximum value for all tiles. This increases the scope of influence of the tile, but reduces the amount of syntax information.

[0387] Figure 74 shows an example of using scale values.

[0388] In the example shown in Figure 74, the metadata may include the maximum scale value of the GS data contained in the tile as region information indicating the range of influence of the GS data. The decoding device derives the range of influence of the GS data based on the region information for each tile and the maximum scale value of the GS data contained in the tile. Note that the metadata may indicate the maximum number of bits of the scale value instead of the maximum scale value.

[0389] The metadata of the encoded data described in this embodiment may be included in the parameter set, such as SPS, control information, or SEI. Furthermore, a portion of the metadata, the syntax structure, or the header of the encoded data may be stored in a box structure within the ISOBMFF file. The boxes may be contained within either moov or mdat.

[0390] The representation_type, which is information regarding the format of the GS data, may be shown in the track box in moov, or in the media type in the track box, such as hdlr, stsd, or tref. The GS data may also be stored in mdat and associated with information regarding the format of the GS data stored in mdat.

[0391] Although G-PCC was used as an example, the encoding method is not limited to G-PCC. The encoding method may be an encoding method with an interface for a specific format, in particular, V-PCC, V-DMC, MIV, or V3C schemes that use video encoding such as HEVC or VVC.

[0392] In this embodiment, the region information indicating the range of Gaussian data means information that represents the spatial range that indicates the influence each Gaussian data, i.e., GS data, has on rendering. For example, the region information indicating the range of Gaussian data may be calculated by scaling or deforming the shape of the basic area of ​​the tile based on the basic bounding box information of each tile, bbox_xyz and bbox_wdh, and based on parameters such as position information, scale value, rotation value, and transparency of the GS data contained within the tile, but is not limited to this.

[0393] The region information indicating the range of Gaussian data may be represented as expanded region information or as a bounding box for the Gaussian data, but is not limited to these. In this embodiment, the region information indicating the range of Gaussian data is typically the shape of a box, i.e., a cube or rectangular prism, but is not limited to this. For example, the region information indicating the range of Gaussian data may be defined as the region of an ellipsoid. In this case, the encoding device calculates the shape of an ellipse or ellipsoid based on the scale value and rotation value of each Gaussian data, with the center coordinates as the starting point.

[0394] Furthermore, the region information indicating the range of Gaussian data may be defined as a polygonal or polyhedronal region. In this case, the encoding device may determine the convex or non-convex shape of the data point cloud according to the distribution of GS data within the tile, and define the region information as a polygon in two dimensions and a polyhedron in three dimensions. In addition, the region information indicating the range of Gaussian data may be defined as an contour line region or a contour line region. In this case, the encoding device extracts the portion of the GS data where the value exceeds a threshold, and defines the region information as the region surrounding that portion. This makes it possible to clearly indicate the region where the influence of Gaussian exceeds a certain value.

[0395] Furthermore, region information indicating the range of Gaussian data may be defined by a spline curve or a freeform surface. In this case, the encoding device may approximate the region for more complex GS data distributions using a spline curve, a freeform surface, or a similar curve or surface. Therefore, the encoding device may represent nonlinear and flexible shapes.

[0396] [Encoding Device] Figure 75 is a diagram showing an example configuration of the encoding device 1900. Figure 76 is a flowchart showing an example of an encoding method performed by the encoding device 1900.

[0397] The encoding device 1900 includes a circuit 1901 and a memory 1902. The circuit 1901 executes the processes shown in Figure 76 by running a program stored in the memory 1902. The memory 1902 stores Gaussian group data, region information, bitstream, and various work information that the circuit 1901 references.

[0398] Circuit 1901 first encodes Gaussian group data (S1901). Here, Gaussian group data includes one or more Gaussian data points. Next, circuit 1901 generates region information indicating the range of one or more Gaussian data points (S1902). This region information is generated based on the position information and shape values ​​of one or more Gaussian data points. Finally, circuit 1901 generates a bitstream containing the encoded Gaussian group data and region information (S1903). As a result, circuit 1901 can handle the encoded Gaussian group data in association with the range information generated based on the position information and shape values ​​of the Gaussian data points. Consequently, the decoding device can select the necessary parts during decoding and perform partial decoding, decoding only a portion of the Gaussian group data, thereby reducing the processing load on the decoding side.

[0399] The region information may include bounding box information based on the position information and shape values ​​of one or more Gaussian data. In this case, the circuit 1901 can represent the range based on the position information and shape values ​​of the Gaussian data as bounding box information. The shape values ​​may include at least one of the scale value and rotation value of one or more Gaussian data. This allows the circuit 1901 to reflect at least one of the size and orientation of the Gaussian data in the region information. Furthermore, the bounding box information may be information based on the position information, scale value, rotation value, and transparency of one or more Gaussian data. This allows the circuit 1901 to represent the range of the Gaussian data using information based on the position information, scale value, rotation value, and transparency.

[0400] Circuit 1901 may include in the bitstream, for each of a plurality of tiles, tile identification information indicating the tile, and region information indicating the range of one or more Gaussian data from among the one or more Gaussian data included in the tile. In this way, circuit 1901 can include in the bitstream information indicating the tile and information indicating the range of the Gaussian data included in that tile in association with each other. In this case, the bounding box information included in the region information corresponding to each tile may include information indicating the origin coordinates, as well as information indicating the width, depth, and height. As a result, circuit 1901 can represent the range of the Gaussian data corresponding to each tile as origin coordinates and dimensions in each direction. Therefore, circuit 1901 can treat the position and extent of the range of the Gaussian data corresponding to each tile as coordinate information.

[0401] Furthermore, the bitstream may include offset information relative to the origin of bounding box information containing region information indicating the range of one or more Gaussian data contained in each tile, and at least one of the difference information for at least one of the width, depth, and height of the bounding box information. Therefore, the circuit 1901 can represent the difference in range for each tile as at least one of the offset relative to the origin and the difference for each directional dimension. As a result, the decoder can determine the range to be used for decoding while treating the range information for each tile as a difference.

[0402] Circuit 1901 may include a geometry data unit in the bitstream that corresponds to each of a plurality of tiles. In this case, the header of the geometry data unit may include slice tag information indicating the tile to which the geometry data unit corresponds. This allows the decoder to determine the correspondence between the tile and the geometry data unit from the header information of the geometry data unit. Furthermore, circuit 1901 may include an attribute data unit in the bitstream that corresponds to the geometry data unit. This attribute data unit may have the same slice identification information as the geometry data unit. In this way, circuit 1901 can associate the geometry data unit and the attribute data unit based on the same slice identification information. As a result, the decoder can use the attribute data unit that corresponds to the acquired geometry data unit.

[0403] [Decryption Device] Figure 77 is a diagram showing an example of the configuration of the decoding device 1910. Figure 78 is a flowchart showing an example of a decoding method performed by the decoding device 1910.

[0404] The decoding device 1910 comprises a circuit 1911 and a memory 1912. The circuit 1911 executes the processes shown in Figure 78 by running a program stored in the memory 1912. The memory 1912 stores the bitstream referenced by the circuit 1911, the rendering target area, area information, candidate tile group, and the Gaussian data to be decoded.

[0405] Circuit 1911 first acquires a bitstream (S1911). This bitstream includes encoded data of a Gaussian group of data containing one or more Gaussian data, and region information indicating the range of one or more Gaussian data. This region information is generated based on the position information and shape values ​​of one or more Gaussian data. Next, circuit 1911 decodes the encoded data contained in the bitstream (S1913). As a result, circuit 1911 can acquire both the encoded Gaussian group data and the region information generated based on the position information and shape values ​​of the Gaussian data from the same bitstream. Therefore, when decoding the encoded data, circuit 1911 can acquire the region information indicating the range of the Gaussian data as reference information.

[0406] The region information may include bounding box information based on the positional information and shape values ​​of one or more Gaussian data. Therefore, circuit 1911 can acquire the range based on the positional information and shape values ​​of the Gaussian data as bounding box information. Circuit 1911 may further determine the Gaussian data to be decoded based on the region information and the rendering target region (S1912), and decode the Gaussian data to be decoded (S1913). In this way, circuit 1911 can determine the Gaussian data to be decoded based on the relationship between the rendering target region and the region information. As a result, circuit 1911 can perform partial decoding on the Gaussian data related to the necessary parts, thereby reducing the processing load on the decoding side.

[0407] The shape value may include at least one of the scale value and rotation value of one or more Gaussian data. Therefore, the circuit 1911 can use region information that reflects at least one of the size and orientation of the Gaussian data when determining the data to be decoded. Furthermore, the bounding box information may be information based on the position information, scale value, rotation value, and transparency of one or more Gaussian data. As a result, the circuit 1911 can determine the data to be decoded using bounding box information based on the position information, scale value, rotation value, and transparency.

[0408] Circuit 1911 may further determine the Gaussian data to be decoded based on the region information and the rendering target region, and may refer to tile-level region information when decoding the Gaussian data to be decoded. Specifically, the bitstream may include, for each of a plurality of tiles, tile identification information indicating the tile, and region information indicating the range of one or more Gaussian data included in the tile from one or more Gaussian data. In this way, circuit 1911 can obtain tile identification information and region information indicating the range of Gaussian data included in that tile from the bitstream. When determining the Gaussian data to be decoded, circuit 1911 may refer to the region information associated with the tile identification information. As a result, circuit 1911 can determine the Gaussian data to be decoded by referring to the region information associated with the tile identification information.

[0409] The bounding box information contained in the region information corresponding to each tile may include information indicating the origin coordinates, as well as information indicating the width, depth, and height. According to this, circuit 1911 can use information indicating the range of the Gaussian data corresponding to each tile, expressed as the origin coordinates and dimensions in each direction. Therefore, circuit 1911 can treat the position and extent of the Gaussian data range corresponding to each tile as coordinate information and select the tile or Gaussian data to be used for decoding. Furthermore, the bitstream may include offset information relative to the origin of the bounding box information contained in the region information indicating the range of one or more Gaussian data contained in each tile, and at least one of the difference information for at least one of the width, depth, and height of the said bounding box information. In this case, circuit 1911 may determine the Gaussian data to be decoded based on the bounding box information based on at least one of the offset information and the difference information, and the rendering target area. Therefore, circuit 1911 can determine the target to be used for partial decoding depending on the relationship between the bounding box information and the rendering target area.

[0410] The bitstream may include geometry data units corresponding to each of multiple tiles. Circuit 1911 may obtain a geometry data unit corresponding to the tile containing the Gaussian data to be decoded by referring to the slice tag information contained in the header of the geometry data unit. This allows circuit 1911 to perform partial decoding using the geometry data corresponding to the required tile. The bitstream may further include attribute data units corresponding to the geometry data units. Circuit 1911 may obtain and decode an attribute data unit having the same slice identification information as the geometry data unit. In this way, circuit 1911 can obtain and decode an attribute data unit corresponding to the obtained geometry data unit based on the same slice identification information.

[0411] When determining the Gaussian data to be decoded, circuit 1911 may identify tiles that affect the rendering target area based on region information. This allows circuit 1911 to determine the Gaussian data to be decoded by targeting tiles that affect the rendering target area. Specifically, circuit 1911 may refer to the region information corresponding to each of the multiple tiles included in the bitstream and extract a group of tiles that overlap with the rendering target area as a group of candidate tiles. Therefore, circuit 1911 can determine the target for decoding from the group of candidate tiles and narrow down the target to be handled on the decoding side. Furthermore, circuit 1911 may select tiles with higher priority from the group of candidate tiles based on the amount of overlap between the range indicated by the region information and the rendering target area. As a result, circuit 1911 can determine which tiles from the group of candidate tiles are preferred for partial decoding.

[0412] When determining the Gaussian data to be decoded, circuit 1911 may calculate bounding box information indicating the range of one or more Gaussian data by referring to at least one of the offset information and difference information contained in the bitstream. Based on the calculated bounding box information, circuit 1911 may determine the target to be used for partial decoding. In this way, circuit 1911 can calculate bounding box information indicating the range of Gaussian data by referring to at least one of the offset information and difference information.

[0413] The method described above, which uses region information indicating the influence range of GS data for each tile as metadata for partial decoding of GS data, is useful for obtaining relevant GS data for the area to be decoded (the area to be decoded). In particular, since it can handle any size and shape of the area to be decoded, the decoding unit or application can freely set the size and shape of the area to be decoded. For this reason, this method can be used as a general-purpose partial decoding mechanism applicable to various three-dimensional data such as map data, AR content, or VR content.

[0414] On the other hand, in systems that rely on rendering using GS data, only a portion of the GS data is needed for rendering, and it is not necessary to decode all of the GS data. Also, the rendering target area (the area to be rendered) differs for each viewpoint. Furthermore, since the rendering process of 3D GS is computationally intensive and time-consuming, it is necessary to reduce the amount of GS data input to the rendering unit.

[0415] Therefore, in this embodiment, a mechanism is provided that can identify the data necessary for rendering a viewpoint when the viewpoint is known in advance. In this embodiment, metadata for each viewpoint can be represented as one or more viewpoint information included in the control information. Each viewpoint information may include one or more identification pieces of information that identify the encoded data necessary for rendering the viewpoint corresponding to that viewpoint information. Note that this identification piece of information may be information that identifies one or more sub-regions included in the region defined by the viewpoint information.

[0416] On the encoding side, the encoding data necessary for rendering a specific viewpoint is identified. Next, the encoding side creates a table of information identifying viewpoint information for each viewpoint and the corresponding encoding data for that viewpoint information, and this table can be encoded together with the encoding data as control information. As a result, the bitstream can include not only the encoding data corresponding to the GS data, but also information to identify the encoding data required for each viewpoint.

[0417] Furthermore, the encoding side may set a tile ID that identifies the tile as a slice tag in the header of the encoded data corresponding to each tile. The decoding side may identify and obtain the encoded data corresponding to the tile to be decoded by associating the tile ID included in the tile list for each viewpoint with the slice tag set in the header.

[0418] On the decoding side, identification information for the encoded data corresponding to the viewpoint to be decoded is obtained based on the control information. Next, the decoding side obtains the encoded data containing this identification information from the bitstream and decodes that encoded data. This allows for the selection and decoding of the encoded data necessary for the viewpoint to be decoded.

[0419] Furthermore, if the viewpoint is changed on the decoding side, the decoding side can identify the encoded data corresponding to the changed viewpoint based on the new control information. The decoding side can then acquire and decode the encoded data that has not yet been decoded. In this way, any additional encoded data required in response to the change in viewpoint can be added to the decoded data.

[0420] Thus, in this embodiment, only the necessary encoded data can be acquired and decoded based on the viewpoint. Therefore, decoding time and processing load can be reduced. Furthermore, since only the data necessary for rendering can be input to the rendering unit, the processing load and time related to rendering can be reduced. Moreover, compared to partial decoding using region information for each tile, it is not necessary to detect tiles that intersect with the decoding target region, and it can be used for high-speed partial decoding.

[0421] Figure 79 shows the metadata processing for each viewpoint.

[0422] As shown in Figure 79, the system according to this embodiment comprises a GS data division unit 551, an encoding unit 552, a decoding unit 553, and a rendering unit 554. The GS data division unit 551 divides the GS data into a plurality of tiles. Here, the tiles may be partial regions corresponding to at least a part of the GS data. The encoding unit 552 encodes the GS data divided by the GS data division unit 551 to generate encoded data. The encoding unit 552 may generate a bitstream that includes metadata of the influence range of the GS data for each tile, metadata of identification information of the encoded data for each viewpoint, and encoded data associated with these metadata. The decoding unit 553 may decode the necessary encoded data based on the metadata and output some of the GS data. The rendering unit 554 may perform rendering using some of the GS data input from the decoding unit 553.

[0423] Figure 80 is a flowchart showing the metadata generation process.

[0424] First, the division unit 551 acquires GS data and divides the GS data into tiles (S551). At this time, the division unit 551 may divide the GS data into multiple slices based on the tile information and output each slice to the encoding unit 552.

[0425] Next, the encoding device determines the viewpoint and region to be rendered (S552). For example, the rendering region corresponding to viewpoint A may be the View Frustum shown in Figure 81. The View Frustum is the range of the three-dimensional scene visible from the camera, which is the viewpoint, and may be derived from camera parameter information at the viewpoint. The Frustum may also be called the visibility cone. In Figure 81, the View Frustum is the region enclosed by the field of view from the camera position, the near plane, and the far plane. This View Frustum may be treated as the range to be drawn relative to the viewpoint, similar to the Cul 3DGS and project 3DGS in the rasterization process.

[0426] Next, the encoding device determines for each tile whether or not the tile affects the area to be rendered (S553). For example, the encoding device may use metadata of the GS data's influence range for each tile to determine whether or not the bounding box indicating the influence range of the GS data contained in the tile intersects with the View Frustum area. If the bounding box intersects with the View Frustum area, the encoding device may determine that the tile affects the area to be rendered; if it does not intersect, the encoding device may determine that the tile does not affect the area to be rendered.

[0427] If the encoding device determines that a tile affects the area to be rendered, it treats the tile as the tile corresponding to the viewpoint and stores the tile ID of the tile (S554). For example, if there are three tiles that affect viewpoint A, and their respective tile IDs are 1, 3, and 5, the encoding device may store a list of tile IDs 1, 3, and 5 corresponding to viewpoint A in the metadata. Thus, the metadata may include a list of tiles corresponding to viewpoints. In other words, the metadata may be treated as control information that includes viewpoint information corresponding to each of multiple viewpoints, and a list of tile IDs as identification information for the encoded data corresponding to each viewpoint.

[0428] The encoding device similarly identifies the tiles corresponding to each of the other viewpoints (S555). For example, the encoding device may also determine the tiles that affect the View Frustum of viewpoints B and C, derive a list of tile IDs corresponding to each viewpoint, and store it in the metadata.

[0429] Next, the encoding device generates metadata for tile information for each viewpoint (S556). This metadata may be treated as control information that includes viewpoint information corresponding to each of the multiple viewpoints, and identification information for the encoded data corresponding to each viewpoint.

[0430] Subsequently, the encoding device encodes the segmented GS data and metadata (S557). This generates a bitstream containing encoded data corresponding to the segmented GS data, and metadata for identifying the encoded data required for each viewpoint.

[0431] Figure 81 shows the relationship between the viewpoint and the tiles.

[0432] The tiles affecting the View Frustum from viewpoint A may be different from the tiles affecting the View Frustum from viewpoint B. Therefore, even if the rendering area differs for each viewpoint, the necessary encoded data for each viewpoint can be identified by pre-calculating a list of tile IDs corresponding to each viewpoint.

[0433] Figure 82 shows an example of the syntax for a tile list for each viewpoint.

[0434] As shown in Figure 82, in this embodiment, gs_view_partial_information() can be used as metadata for the tile list for each viewpoint. gs_view_partial_information() may be metadata that includes multiple viewpoint information corresponding to multiple viewpoints. This metadata may be treated as control information that includes viewpoint information corresponding to each viewpoint and identification information for the encoded data corresponding to that viewpoint information.

[0435] The `gs_view_partial_information()` function includes `num_view`, which indicates the number of viewpoints. `num_view` is a value indicating the number of viewpoint information entries included in the metadata. Furthermore, `gs_view_partial_information()` includes a `view_id` for each viewpoint entry. `view_id` is the identifier of the viewpoint. Each viewpoint entry also includes a list of tiles corresponding to that viewpoint. This list of tiles includes `num_tiles[i]`, which indicates the number of tiles corresponding to the viewpoint, and multiple `tile_id`s. `tile_id` is the identifier of the tile. In this way, a list of tile identifiers for multiple viewpoints can be transmitted.

[0436] As shown in Figure 82, in this embodiment, gs_tile_inventory() may be transmitted separately as metadata for area information for each tile. gs_tile_inventory() may include num_tiles indicating the number of tiles, tile_id indicating the identifier of each tile, and 3dgs_bbox_xyz and 3dgs_bbox_wdh, which are three-dimensional bounding box information corresponding to each tile. Here, 3dgs_bbox_xyz may be position information of the bounding box indicating the influence range of the GS data, and 3dgs_bbox_wdh may be size information of the said bounding box. The decoder may obtain area information of the tile corresponding to the tile_id by matching the tile_id included in gs_view_partial_information() with the tile_id included in gs_tile_inventory().

[0437] In this embodiment, metadata including camera parameter information for each viewpoint and information about the tile corresponding to that viewpoint may also be transmitted separately, for example, viewport_information(). viewport_information() may include view_id, view_type, internal camera parameters, external camera parameters, and information about the tile corresponding to that viewpoint. The decoder may obtain information about the viewpoint corresponding to a view_id by matching the view_id included in gs_view_partial_information() with the view_id included in viewport_information(). Here, the viewpoint information may include camera parameter information, view_type, and information about the tile corresponding to that viewpoint.

[0438] Furthermore, the information from gs_tile_inventory() and viewport_information() may or may not be included in gs_view_partial_information(). In other words, the metadata for the tile list for each viewpoint may also include metadata that further includes area information for each tile and camera parameter information for each viewpoint.

[0439] `view_type` may be, for example, the initial viewpoint, the user-recommended viewpoint, or the viewpoint used for training. For example, the viewpoint used for training may be treated as `ground_truth`. In this way, `view_type` may be used as an attribute of viewpoint information that includes a list of tiles related to a specific viewpoint, i.e., identification information for encoded data. Therefore, it can be used as metadata to partially extract encoded data related to the initial viewpoint, the user-recommended viewpoint, and the viewpoint used for training.

[0440] Furthermore, the order in which the tile_ids are listed may be ascending or descending. In addition, the order in which the tile_ids are listed may be determined based on the distance from the viewpoint. For example, the tile_ids may be listed in order of proximity to the viewpoint. In this way, multiple tiles corresponding to each viewpoint can be represented in an ordered manner.

[0441] Figure 83 is a flowchart showing the processing in the decoding unit.

[0442] As shown in Figure 83, the decoding device first determines the viewpoint to be rendered (S561). Here, the viewpoint to be rendered (the viewpoint to be rendered) is the viewpoint that will be rendered.

[0443] Next, the decoding device decodes the metadata of the tile list for each viewpoint (S562). This metadata may be gs_view_partial_information() as described in Figure 82, or it may be treated as control information that includes viewpoint information corresponding to each of the multiple viewpoints, and a list of tile IDs as identification information for the encoded data corresponding to each viewpoint.

[0444] Next, the decoding device obtains a tile list for the viewpoint to be rendered from the metadata (S563). Specifically, the decoding device uses the view_id corresponding to the viewpoint to be rendered to identify the viewpoint information corresponding to that viewpoint and obtains the tile list included in that viewpoint information.

[0445] Next, the decryption device obtains the tile ID of the identified tile (S564). The decryption device may also obtain multiple tile_ids included in the tile list in sequence.

[0446] Subsequently, the decoding device refers to the GDU header, obtains and decodes the GDU corresponding to the tile ID, and does not decode other GDUs (S565). Specifically, the decoding device obtains and decodes the GDU whose slice_tag included in the GDU header matches the tile_id obtained in step S564. In this way, partial decoding of the encoded data corresponding to the rendering target viewpoint is possible. Furthermore, the amount of rendering processing and processing time may be reduced by inputting the partially decoded data into the rendering.

[0447] For example, when accelerating rendering of an initial viewpoint, the decoding apparatus may acquire a tile ID corresponding to a viewpoint whose viewpoint type is the initial viewpoint from the tile list for each viewpoint, and acquire and decode the GDU in which the tile ID is set in slice_tag. When accelerating rendering of a recommended viewpoint, the decoding apparatus may acquire a tile ID corresponding to a viewpoint whose viewpoint type is the recommended viewpoint from the tile list for each viewpoint, and acquire and decode the GDU in which the tile ID is set in slice_tag. Here, the recommended viewpoint may be, for example, a viewpoint that provides high-quality rendering, or a viewpoint that a content creator wants to render and present to the user.

[0448] As described above, the decoding apparatus may determine whether or not the rendering target viewpoint is included in the metadata, and if the rendering target viewpoint is included in the metadata, acquire the tile list and perform partial decoding. Alternatively, the decoding apparatus may determine whether or not the rendering target viewpoint type is included in the metadata, and if the rendering target viewpoint type is included in the metadata, acquire the tile list and perform partial decoding.

[0449] Note that when tile IDs are described in the metadata in the order of tiles closer to the viewpoint, the decoding apparatus may decode data in the order closer to the viewpoint and output the decoded data to the rendering unit. With this configuration, the processing of rearranging data in the order closer to the viewpoint in the rendering unit can be reduced.

[0450] FIG. 84 is a diagram illustrating a syntax example of a slice list for each viewpoint.

[0451] As illustrated in FIG. 84, in the present embodiment, a list of slice IDs for each viewpoint may be presented instead of the tile list for each viewpoint. For example, gs_view_partial_information() can be used as metadata of the slice list for each viewpoint.

[0452] gs_view_partial_information() includes num_view indicating the number of viewpoints. num_view is a value indicating the number of pieces of viewpoint information included in the metadata. In addition, gs_view_partial_information() includes view_id for each piece of viewpoint information. view_id is an identifier of a viewpoint. Furthermore, each piece of viewpoint information may include a list of slices corresponding to the viewpoint. This list of slices may include num_slices[i] indicating the number of slices corresponding to the viewpoint, and a plurality of slice_ids. slice_id is an identifier of a slice.

[0453] Furthermore, the encoding side may set a slice ID for identifying the slice in the header of each slice. The decoding side may acquire the slice ID corresponding to the viewpoint to be decoded from the control information, and acquire and decode the slice whose header is set with the slice ID. This slice ID may be used as partial region identification information for identifying a partial region.

[0454] By doing this, for each of a plurality of viewpoints, a list of a plurality of slice IDs corresponding to the viewpoint can be transmitted. In other words, the metadata may be handled as control information including viewpoint information corresponding to each of the plurality of viewpoints, and a list of slice IDs as identification information of encoded data corresponding to each piece of viewpoint information.

[0455] The decoding device may acquire the list of slice IDs corresponding to the partial decoding target viewpoint (the viewpoint desired to be partially decoded). For example, the decoding device may identify the viewpoint information based on the view_id corresponding to the viewpoint, and acquire a plurality of slice_ids included in the viewpoint information.

[0456] After that, the decoding device may determine whether the slice ID included in the header of the GDU is an ID included in the acquired list. When the slice ID included in the header of the GDU matches the slice_id included in the list, the decoding device may acquire and decode the GDU. On the other hand, when the slice ID included in the header of the GDU is not included in the list, the GDU does not have to be decoded.

[0457] Thus, in this embodiment, partial decoding may be performed based on a list of slice IDs for each viewpoint, instead of a tile list for each viewpoint. Therefore, it is possible to select and partially decode encoded data to be decoded, using slices corresponding to viewpoints as the unit.

[0458] Figure 85 shows the relationship between tiles corresponding to multiple recommended viewpoints.

[0459] As shown in Figure 85, in this embodiment, a list of tile IDs corresponding to each of multiple viewpoints may be used. For example, the decoding device may first partially decode and render the encoded data corresponding to recommended viewpoint A, and then partially decode and render the encoded data corresponding to recommended viewpoint B. In this case, the decoding device may additionally decode only the encoded data corresponding to the difference tiles that have not yet been decoded from the list of tile IDs corresponding to recommended viewpoint B.

[0460] For example, if the list of tile IDs corresponding to recommended viewpoint A is 1, 2, 3, 4, 5, 6, 7, 8, and 9, and the list of tile IDs corresponding to recommended viewpoint B is 7, 8, 9, 10, 11, 12, 13, 14, 15, and 16, then tile IDs 7, 8, and 9, which are included in the list of tile IDs corresponding to recommended viewpoint B, have already been decoded in the partial decoding corresponding to recommended viewpoint A. Therefore, the decoding device only needs to additionally decode the encoded data corresponding to the difference tile IDs, 10 through 16.

[0461] Thus, in this embodiment, based on the difference in the list of tile IDs corresponding to each of the multiple viewpoints, only the encoded data corresponding to the undecoded tiles can be additionally decoded. Therefore, it is not necessary to re-decode the entire encoded data each time the viewpoint is switched, and the amount of processing can be reduced.

[0462] Furthermore, as shown in Figure 85, the decoding target area for recommended viewpoint A and the decoding target area for recommended viewpoint B may partially overlap. In this case, the encoded data corresponding to tiles common to multiple viewpoints may be reused in the partial decoding corresponding to the later viewpoint. This makes it possible to decode only the parts that are additionally required when switching viewpoints.

[0463] For example, the decryption side may calculate the difference between a first set of identification information corresponding to a first viewpoint and a second set of identification information corresponding to a second viewpoint. The decryption side may also acquire additional encoded data corresponding to the identification information included in the difference and decrypt it. For example, if the tile ID list corresponding to recommended viewpoint A and the tile ID list corresponding to recommended viewpoint B are different, the decryption side may exclude the encoded data corresponding to the tile IDs that have already been decrypted, acquire additional encoded data corresponding to the tile IDs included in the difference, and decrypt it.

[0464] Figure 86 is a flowchart showing the processing of the decoding unit.

[0465] As shown in Figure 86, the decoding device first determines the viewpoint to be rendered (S571). Here, the viewpoint to be rendered is the viewpoint that will be rendered. The method for determining the viewpoint to be rendered can be any method as long as the viewpoint is identified. For example, the viewpoint may be determined by the user specifying the angle, position, or zoom level on the user interface. Alternatively, the decoding device may automatically select the viewpoint to be rendered based on predefined initial conditions or a default viewpoint called view_id. Furthermore, the decoding device may determine the viewpoint to be rendered based on the user's behavior history, content characteristics, external sensor information, environmental information, or the results of an analysis of the data distribution, object position, or density within the scene.

[0466] Next, the decoding device decodes the tile list metadata for each viewpoint (S572). This tile list metadata for each viewpoint may be the first control information. The tile list metadata for each viewpoint may be treated as control information that includes viewpoint information corresponding to each of the multiple viewpoints, and a list of tile IDs as identification information for the encoded data corresponding to each viewpoint.

[0467] Next, the decoding device determines whether the viewpoint to be rendered is included in the tile list metadata for each viewpoint (S573). If the viewpoint to be rendered is included in the tile list metadata for each viewpoint, the decoding device obtains the tile list for the viewpoint to be rendered from the metadata (S574). Then, the decoding device obtains the tile ID of the identified tile (S575). In this case, the process of setting the rendering area and detecting tiles that intersect with that area is unnecessary. Therefore, faster processing can be achieved by using the metadata of the tile list for each viewpoint for frequently used viewpoints.

[0468] On the other hand, if the viewpoint to be rendered is not included in the tile list metadata for each viewpoint, the decoding device decodes the tile metadata of the influence range of the GS data (S576). This tile metadata of the influence range of the GS data may be second control information. Next, the decoding device determines the area to be rendered (S577). After that, the decoding device identifies the tile where the area to be rendered and the area influenced by the GS data intersect (S578). Next, the decoding device obtains the tile ID of the identified tile (S579). Thus, even if the viewpoint to be rendered is not included in the tile list metadata for each viewpoint, the tile ID corresponding to the area may be detected and obtained by using the tile metadata of the influence range of the GS data.

[0469] Subsequently, the decryption device refers to the GDU header, obtains the GDU for the corresponding tile ID and decrypts it, and does not decrypt other GDUs (S580). Specifically, the decryption device may obtain and decrypt a GDU in which the slice_tag included in the GDU header matches the tile_id obtained in step S575 or step S579.

[0470] Thus, in this embodiment, if the rendering target viewpoint is included in the first control information, the tile ID may be obtained from the first control information, and if the rendering target viewpoint is not included in the first control information, the tile ID corresponding to the region may be detected and obtained from the second control information. Therefore, if the rendering target viewpoint is frequently used, high-speed processing can be achieved by using the first control information, and for other viewpoints, the decoding target may be determined using the second control information.

[0471] The process shown in Figure 86 may be used to identify the data to be decoded based on the determined rendering target viewpoint, but it may also be used for purposes other than determining the data to be decoded. For example, the decoding device may use this process to identify the data to be rendered after decoding the tile lists for all viewpoints, or for some of multiple viewpoints. In other words, this process may be used as a partial rendering technique.

[0472] This method is suitable, for example, for applications where a large amount of viewpoint information is decoded at once, and then the rendering target is flexibly switched in response to changes in the user's viewpoint. Furthermore, partial decoding and partial rendering may be combined. By combining these, the decoding target can be narrowed down through partial decoding, and the rendering target can be optimized through partial rendering. Therefore, overall processing efficiency and resource conservation can be improved.

[0473] The metadata relating to the encoded data described in this embodiment may be included in the parameter set, such as SPS, control information, or SEI. For example, metadata for the tile list per viewpoint, metadata for the slice list per viewpoint, or metadata for the influence range tiles of the GS data may be stored in any of these.

[0474] Furthermore, some metadata, syntax structures, or headers of encoded data may be stored in a box structure within the ISOBMFF file. For example, this information may be included in the moov file or in the mdat file. In other words, the metadata described in this embodiment can take on multiple storage configurations depending on the system format.

[0475] The representation_type, which is information regarding the format of the GS data, may be shown in the track box of the movie box which is moov, or in hdlr, stsd, or tref, which indicates the type of media in the track box. The GS data may also be stored in mdat. In this case, information regarding the format of the GS data may be associated with the GS data stored in mdat.

[0476] In this way, the metadata, syntax structure, header information, and information regarding the GS data format described in this embodiment may be stored in an appropriate location according to the system format.

[0477] Although this embodiment uses G-PCC as an example, this embodiment is not limited to G-PCC. For example, this embodiment may be applied to encoding methods having an interface for a specific format. Specifically, it may be applied to v-PCC, V-DMC, MIV, or V3C methods that use video encoding such as HEVC or VVC.

[0478] [Encoding device] Figure 87 shows the configuration of the encoding device.

[0479] As shown in Figure 87, the encoding device 1920 includes a circuit 1921 and a memory 1922. The circuit 1921 may include, for example, a processor, CPU, or logic circuit. The memory 1922 is connected to the circuit 1921. The memory 1922 may store, for example, a program, configuration information, working data, viewpoint information, control information, or encoded data. The circuit 1921 may execute the encoding method described in Figure 88 by executing a program stored in the memory 1922.

[0480] FIG. 88 is a flowchart illustrating an encoding method.

[0481] As shown in FIG. 88, the circuit 1921 of the encoding device 1920 acquires one or more pieces of viewpoint information (S1921). Here, the one or more pieces of viewpoint information may be viewpoint information respectively corresponding to a plurality of viewpoints. Each piece of viewpoint information may include one or more pieces of identification information for identifying one or more partial regions included in a region defined by the corresponding viewpoint information.

[0482] Next, the circuit 1921 generates a bitstream (S1922). For example, the circuit 1921 may generate a bitstream including control information containing the one or more pieces of viewpoint information and encoded data obtained by encoding Gaussian data.

[0483] Accordingly, at least part of the Gaussian data included in the bitstream can be handled in association with the information for identifying the partial region included in the region defined for each viewpoint. As a result, a necessary part can be selected in decoding and used for partial decoding that decodes at least part of the Gaussian data, so that the processing load on the decoding side can be reduced.

[0484] For example, the plurality of pieces of viewpoint information may include viewpoint identification information that is a view_id for identifying the viewpoint corresponding to each of the plurality of pieces of viewpoint information (identification information for identifying the viewpoint corresponding to the corresponding viewpoint information).

[0485] Therefore, each piece of viewpoint information can be handled separately for the corresponding viewpoint. As a result, even when a plurality of pieces of viewpoint information are included, the viewpoint information corresponding to the decoding target viewpoint can be selected without mistake, and unnecessary data selection can be suppressed.

[0486] For example, each of the plurality of pieces of viewpoint information may include camera parameter information corresponding to the corresponding viewpoint (camera parameter information of the viewpoint corresponding to the corresponding viewpoint information).

[0487] This allows us to define the region corresponding to each viewpoint using the camera parameter information of that viewpoint. As a result, we can determine the target subregion according to the conditions of each viewpoint, and reduce the selection of subregions that do not correspond to a viewpoint.

[0488] For example, each subregion may correspond to segmented GS data obtained by dividing GS data (sub-Gaussian data obtained by dividing Gaussian data).

[0489] This allows the GS data to be divided into sub-regions, and the resulting segmented GS data to be associated with each sub-region. As a result, at least a portion of the GS data can be treated as a target for acquisition or decoding on a sub-region basis.

[0490] For example, the identification information included in each viewpoint information may include sub-region identification information that identifies each sub-region (one or more identification information includes sub-region identification information that identifies each of the one or more sub-regions).

[0491] Therefore, each sub-region can be distinguished using identification information. As a result, the target sub-region can be selected based on viewpoint information, and the acquisition or decoding of data corresponding to unnecessary sub-regions can be reduced.

[0492] For example, each subregion may be a tile. Subregion identification information may be a tile ID that identifies the tile. The tile ID may be set in a slice tag included in the header of the encoded data corresponding to the tile (the slice tag in the header of the encoded data corresponding to the tile).

[0493] This allows a tile ID to be set as a slice tag to identify a tile. As a result, encoded data can be handled in association on a tile-by-tile basis, and the encoded data corresponding to the target tile can be selected.

[0494] For example, encoded data corresponding to at least a portion of the GS data may include slices. The subregion identification information may be a slice ID that identifies the slice. The slice ID may be set as the slice ID included in the slice header (the slice ID in the slice header).

[0495] This allows a slice ID to be set in the slice header to identify the slice. As a result, encoded data can be handled separately on a slice-by-slice basis, and the encoded data corresponding to the target slice can be selected.

[0496] For example, the control information may include region information indicating at least one of the position and size of each subregion (region information indicating at least one of the position and size of one or more subregions).

[0497] This allows the control information to include information indicating at least one of the location and size of each subregion. As a result, the decoding side can determine the target data based on the extent of the subregion.

[0498] For example, the control information may include viewpoint type information indicating the type of viewpoint shown by each viewpoint information (viewpoint type information indicating the type of viewpoint shown by one or more viewpoint information).

[0499] This allows for the distinction and handling of different viewpoint types. As a result, the viewpoint information used can be narrowed down according to the type of viewpoint, and the use of unnecessary viewpoint information can be reduced.

[0500] [Decoding device] Figure 89 shows the configuration of the decoding device.

[0501] As shown in Figure 89, the decoding device 1930 comprises a circuit 1931 and a memory 1932. The circuit 1931 may include, for example, a processor, CPU, or logic circuit. The memory 1932 is connected to the circuit 1931. The memory 1932 may store, for example, a program, configuration information, working data, control information, encoded data, or decoded Gaussian data. The circuit 1931 may execute the decoding method described in Figure 90 by executing a program stored in the memory 1932.

[0502] Figure 90 is a flowchart showing the decoding method.

[0503] As shown in Figure 90, the circuit 1931 of the decoding device 1930 acquires a bitstream (S1931). This bitstream may include control information containing one or more viewpoint information, and encoded data corresponding to Gaussian data.

[0504] Next, circuit 1931 acquires control information and at least a portion of the encoded data (S1932). For example, circuit 1931 may acquire from the bitstream control information including one or more viewpoint pieces of information and at least a portion of the encoded data corresponding to at least a portion of the Gaussian data.

[0505] Subsequently, circuit 1931 decodes at least a portion of the encoded data (S1933). For example, circuit 1931 may decode at least a portion of the acquired encoded data to obtain at least a portion of the Gaussian data. Circuit 1931 may determine the encoded data to be decoded based on the identification information possessed by each of the one or more viewpoint pieces of information included in the control information.

[0506] This allows us to obtain control information and at least a portion of the encoded data from the bitstream and decode at least a portion of the Gaussian data. As a result, we can decode while selecting the necessary parts using the identification information contained in the viewpoint information, thereby suppressing the decoding of unnecessary parts.

[0507] For example, each of the multiple viewpoint pieces of information may include viewpoint identification information (identification information for identifying the viewpoint corresponding to the viewpoint piece of information), which is the view_id corresponding to that viewpoint.

[0508] Therefore, each viewpoint information can be handled separately for each corresponding viewpoint. As a result, even when multiple viewpoint information is acquired, the viewpoint information corresponding to the viewpoint to be decoded can be selected without error, and the selection of unnecessary data can be suppressed.

[0509] For example, each of the multiple viewpoint pieces of information may include camera parameter information corresponding to that viewpoint (camera parameter information for the viewpoint corresponding to that viewpoint piece of information).

[0510] This allows us to obtain camera parameter information for each viewpoint. As a result, we can determine the target subregion according to the conditions for each viewpoint, and reduce the selection of subregions that do not correspond to a viewpoint.

[0511] For example, each subregion may correspond to segmented GS data obtained by dividing GS data (sub-Gaussian data obtained by dividing Gaussian data).

[0512] This allows for the segmented GS data, obtained by dividing the GS data for each sub-region, to be handled in an associated manner. As a result, at least a portion of the GS data to be acquired can be handled on a sub-region basis.

[0513] For example, the identification information included in each viewpoint information may include sub-region identification information that identifies each sub-region (one or more identification information includes sub-region identification information that identifies each of the one or more sub-regions).

[0514] Therefore, each sub-region can be distinguished using identification information. As a result, the sub-regions used for decoding can be selected based on the identification information, and the acquisition or decoding of data corresponding to unnecessary sub-regions can be suppressed.

[0515] For example, each subregion may be a tile. Subregion identification information may be a tile ID that identifies the tile. The tile ID may be set in a slice tag included in the header of the encoded data corresponding to the tile (the slice tag in the header of the encoded data corresponding to the tile).

[0516] This allows the tile ID, which identifies a tile, to be set as a slice tag in the header of the encoded data corresponding to that tile. As a result, encoded data can be handled in association on a tile-by-tile basis, and the encoded data corresponding to the target tile can be selected.

[0517] For example, encoded data corresponding to at least a portion of the GS data may include slices. The subregion identification information may be a slice ID that identifies the slice. The slice ID may be set as the slice ID included in the slice header (the slice ID in the slice header).

[0518] This allows the slice ID, which identifies a slice, to be set as the slice ID in the slice header. As a result, encoded data can be handled separately on a slice-by-slice basis, and the encoded data corresponding to the target slice can be selected.

[0519] For example, the control information may include region information indicating at least one of the position and size of each subregion (region information indicating at least one of the position and size of one or more subregions).

[0520] This allows region information indicating at least one of the position and size of each subregion to be included in the control information. As a result, the subregion corresponding to the decoding target can be determined based on the control information.

[0521] For example, the control information may include viewpoint type information indicating the type of viewpoint shown by each viewpoint information (viewpoint type information indicating the type of viewpoint shown by one or more viewpoint information).

[0522] This allows for the distinction and handling of different viewpoint types. As a result, the viewpoint information used can be narrowed down according to the type of viewpoint, and the use of unnecessary viewpoint information can be reduced.

[0523] For example, circuit 1931 may further acquire the viewpoint to be decoded. If the multiple viewpoint information (one or more viewpoint information) includes viewpoint information corresponding to the viewpoint to be decoded, circuit 1931 may further determine at least one of the multiple identification information (one or more identification information) included in the viewpoint information as the identification information for the encoded data to be decoded. If the multiple viewpoint information does not include viewpoint information corresponding to the viewpoint to be decoded, circuit 1931 may further determine at least one of the multiple identification information based on the region information included in the control information as the identification information for the encoded data to be decoded.

[0524] This allows the identification information of the encoded data to be decoded to be determined according to the viewpoint to be decoded. As a result, data corresponding to the viewpoint to be decoded can be selected and used for decoding, and the decoding of unnecessary parts can be suppressed.

[0525] For example, the circuit 1931 may further render an image or model based on at least a portion of the acquired GS data (at least a portion of the acquired Gaussian data) and at least one viewpoint information from among multiple viewpoint information (at least one viewpoint information from one or more viewpoint information).

[0526] This allows at least a portion of the acquired GS data to be used to generate images or models corresponding to viewpoint information. As a result, output can be generated using only the necessary parts of the acquired GS data, thereby reducing unnecessary data processing.

[0527] For example, the circuit 1931 may further acquire at least a portion of the GS data corresponding to the difference between multiple identification information corresponding to multiple viewpoint information (at least a portion of the Gaussian data corresponding to the difference between multiple identification information corresponding to multiple viewpoint information).

[0528] This makes it possible to acquire at least a portion of the GS data corresponding to the difference between multiple identification information that corresponds to multiple viewpoint information. As a result, only the additional data required when switching viewpoints can be acquired, reducing the processing load.

[0529] The above describes the encoding device, decoding device, etc., according to embodiments and modifications of the present disclosure, but the present disclosure is not limited to these embodiments.

[0530] Furthermore, each processing unit included in the encoding device and decoding device, etc., according to the above embodiment is typically implemented as an LSI, which is an integrated circuit. These may be individually integrated into a single chip, or some or all of them may be integrated into a single chip.

[0531] Furthermore, integrated circuit implementation is not limited to LSIs; it may also be achieved using dedicated circuits or general-purpose processors. Alternatively, an FPGA (Field Programmable Gate Array), which can be programmed after LSI manufacturing, or a reconfigurable processor capable of reconfiguring the connections and settings of circuit cells within the LSI, may be used.

[0532] Furthermore, in each of the above embodiments, each component may be implemented by being composed of dedicated hardware or by executing a software program suitable for each component. Each component may also be implemented by a program execution unit such as a CPU or processor reading and executing a software program recorded on a recording medium such as a hard disk or semiconductor memory.

[0533] Furthermore, this disclosure may be implemented as an encoding method (three-dimensional data encoding method) or a decoding method (three-dimensional data decoding method), etc., performed by an encoding device (three-dimensional data encoding device) and a decoding device (three-dimensional data decoding device), etc.

[0534] Furthermore, this disclosure may be implemented as a program that causes a computer, processor, or device to execute the above-described encoding or decoding method. Alternatively, this disclosure may be implemented as a bitstream generated by the above-described encoding method. Furthermore, this disclosure may be implemented as a recording medium on which the program or the bitstream is recorded. For example, this disclosure may be implemented as a non-temporary, computer-readable recording medium on which the program or the bitstream is recorded.

[0535] Furthermore, the division of functional blocks in the block diagram is just one example; multiple functional blocks can be implemented as a single functional block, a single functional block can be divided into multiple parts, or some functions can be moved to other functional blocks. In addition, the functions of multiple functional blocks with similar functions can be processed in parallel or time-sharing by a single piece of hardware or software.

[0536] Furthermore, the order in which each step in the flowchart is performed is illustrative for the purpose of specifically illustrating this disclosure, and may be in a different order. Also, some of the above steps may be performed simultaneously (in parallel) with other steps.

[0537] Although one or more embodiments of encoding and decoding devices, etc., have been described above based on embodiments, this disclosure is not limited to these embodiments. Without departing from the spirit of this disclosure, various modifications that a person skilled in the art could conceive of are applied to these embodiments, and forms constructed by combining components from different embodiments may also be included within the scope of one or more embodiments.

[0538] This disclosure is applicable to encoding and decoding devices.

[0539] 101 Three-dimensional data encoding system 102 Three-dimensional data decoding system 103 Sensor terminal 104 External connection unit 111 Three-dimensional data generation system 112 Presentation unit 113 Encoding unit 114 Multiplexing unit 115 Input / output unit 116 Control unit 117 Sensor information acquisition unit 118 Three-dimensional data generation unit 121 Sensor information acquisition unit 122 Input / output unit 123 Demultiplexing unit 124 Decoding unit 125 Presentation unit 126 User interface 127 Control unit 130 First encoding unit 131 Location information encoding unit 132 Attribute information encoding unit 133 Additional information encoding unit 134 Multiplexing unit 140 First decoding unit 141 Demultiplexing unit 142 Location information decoding unit 143 Attribute information decoding unit 144 Additional information decoding unit 150 Second encoding unit 151 Additional Information Generation Unit 152 Position Image Generation Unit 153 Attribute Image Generation Unit 154 Video Coding Unit 155 Additional Information Coding Unit 156 Multiplexing Unit 160 Second Decoding Unit 161 Demultiplexing Unit 162 Video Decoding Unit 163 Additional Information Decoding Unit 164 Position Information Generation Unit 165 Attribute Information Generation Unit 171 Octane Tree Coding Unit 172 Prediction Tree Coding Unit 173 Octane Tree Decoding Unit 174 Prediction Tree Decoding Unit 181 Octane Tree Generation Unit 182 Geometric Information Calculation Unit 183 Coding Table Selection Unit 184 Entropy Coding Unit 191 Octane Tree Generation Unit 192 Geometric Information Calculation Unit 193 Coding Table Selection Unit 194 Entropy Decoding Unit 201 LoD Attribute Information Coding Unit 202 Transformation Attribute Information Coding Unit 203 LoD attribute information decoding unit 204 Transform attribute information decoding unit 211 Sorting unit 212 Haar transformation unit 213 Quantization unit 214 Inverse quantization unit 215 Inverse Haar transformation unit 216 Memory 217 Arithmetic coding unit 221 Arithmetic decoding unit 222 Inverse quantization unit 223 Inverse Haar transformation unit 224 Memory 231 Data splitting unit 232 Coding unit 233 Decoding unit 234 Data merging unit 241 Coding unit 242 TLV storage unit 301 Sensor information input unit 302 Three-dimensional data generation unit 303 Rendering unit 305 Rendering unit 311 Coding unit 312 Multiplexing unit 321 Inverse multiplexing unit 322 Decoding unit 323 Application unit 324 Input interface unit 325 Rendering unit 326 Presentation section330 Encoding device 331 First preprocessing unit 332 Second preprocessing unit 333 G-PCC encoding unit 341 Position information encoding unit 342 Rotation encoding unit 343 Scale encoding unit 344 SH coefficient encoding unit 345 Transparency encoding unit 346 Metadata encoding unit 347 Multiplexing unit 350 Decoding device 351 G-PCC decoding unit 352 Second postprocessing unit 353 First postprocessing unit 361 Demultiplexing unit 362 Position information decoding unit 363 Rotation decoding unit 364 Scale decoding unit 365 SH coefficient decoding unit 366 Transparency decoding unit 367 Metadata decoding unit 380 Three-dimensional data generation device 381 SFM processing unit 382 Initialization unit 383 Gaussian update unit 384 Two-dimensional ground truth data selection unit 385 Rendering unit 386 Loss calculation unit 387 Gaussian parameter optimization unit 501 GS data partitioning unit 502 Encoding unit 503 Decoding unit 504 Rendering unit 551 GS data partitioning unit 552 Encoding unit 553 Decoding unit 554 Rendering unit 1038 Spherical harmony unit for colors 1143 Encoding unit 1144 Formatting unit 1145 Inverse formatting unit 1146 Decoding unit 1900, 1920 Encoding device 1901, 1921 Circuit 1902, 1922 Memory 1910, 1930 Decoding device 1911, 1931 Circuit 1912, 1932 Memory

Claims

1. An encoding method performed by an encoding device, comprising: acquiring one or more viewpoint pieces of information; generating a bitstream including control information containing the one or more viewpoint pieces of information and encoded data obtained by encoding three-dimensional data; and each of the one or more viewpoint pieces of information includes one or more identification pieces of information that identify one or more subregions included in the region defined by the viewpoint piece of information, among one or more subregions corresponding to at least a part of the three-dimensional data.

2. The encoding method according to claim 1, wherein the three-dimensional data is Gaussian data.

3. The encoding method according to claim 1, wherein each of the one or more viewpoint pieces of information includes identification information for identifying the viewpoint corresponding to the viewpoint piece of information.

4. The encoding method according to claim 1, wherein each of the one or more viewpoint pieces of information includes camera parameter information of the viewpoint corresponding to the viewpoint piece of information.

5. The encoding method according to claim 1, wherein each of the one or more subregions corresponds to sub-three-dimensional data obtained by dividing the three-dimensional data.

6. The encoding method according to claim 1, wherein the one or more identification pieces of information include sub-region identification pieces of information that identify each of the one or more sub-regions.

7. The encoding method according to claim 6, wherein each of the one or more subregions is a tile, the subregion identification information is a tile ID that identifies the tile, and the tile ID is set in the slice tag of the header of the encoded data corresponding to the tile.

8. The encoding method according to claim 6, wherein the encoded data corresponding to at least a portion of the three-dimensional data includes a slice, the partial region identification information is a slice ID that identifies the slice, and the slice ID is set to the slice ID of the header of the slice.

9. The encoding method according to claim 1, wherein the control information includes region information indicating at least one of the position and size of each of the one or more subregions.

10. The encoding method according to claim 1, wherein the control information includes viewpoint type information indicating the type of viewpoint represented by each of the one or more viewpoint pieces of information.

11. A decoding method performed by a decoding device, comprising: acquiring a bitstream; acquiring control information including one or more viewpoint pieces of information and at least a portion of encoded data corresponding to at least a portion of three-dimensional data from the bitstream; decoding at least a portion of the encoded data to acquire at least a portion of the three-dimensional data; and each of the one or more viewpoint pieces of information includes one or more identification pieces of information that identify one or more subregions included in the region defined by the viewpoint piece of information, among one or more subregions corresponding to at least a portion of the three-dimensional data.

12. The decoding method according to claim 11, wherein the three-dimensional data is Gaussian data.

13. The decoding method according to claim 11, wherein each of the one or more viewpoint pieces of information includes identification information for identifying the viewpoint corresponding to the viewpoint piece of information.

14. The decoding method according to claim 11, wherein each of the one or more viewpoint pieces of information includes camera parameter information of the viewpoint corresponding to the viewpoint piece of information.

15. The decoding method according to claim 11, wherein each of the one or more subregions corresponds to sub-three-dimensional data obtained by dividing the three-dimensional data.

16. The decoding method according to claim 11, wherein the one or more identification pieces of information include sub-region identification pieces of information that identify each of the one or more sub-regions.

17. The decoding method according to claim 16, wherein each of the one or more subregions is a tile, the subregion identification information is a tile ID that identifies the tile, and the tile ID is set in the slice tag of the header of the encoded data corresponding to the tile.

18. The decoding method according to claim 16, wherein the encoded data corresponding to at least a portion of the three-dimensional data includes a slice, the partial region identification information is a slice ID that identifies the slice, and the slice ID is set to the slice ID of the header of the slice.

19. The decoding method according to claim 11, wherein the control information includes region information indicating at least one of the position and size of each of the one or more subregions.

20. The decoding method according to claim 11, wherein the control information includes viewpoint type information indicating the type of viewpoint represented by each of the one or more viewpoint pieces of information.

21. The decoding method according to claim 19, further comprising: obtaining a viewpoint to be decoded; if the one or more viewpoint pieces of viewpoint information include viewpoint pieces of viewpoint information corresponding to the viewpoint to be decoded, determining at least one of the one or more identification pieces of viewpoint information included in that viewpoint piece of viewpoint information as identification information for the encoded data to be decoded; and if the one or more viewpoint pieces of viewpoint information do not include viewpoint pieces of viewpoint information corresponding to the viewpoint to be decoded, determining at least one of the one or more identification pieces of viewpoint information based on the region information included in the control information as identification information for the encoded data to be decoded.

22. The decoding method according to claim 11, further comprising rendering an image or model based on at least a portion of the acquired three-dimensional data and at least one viewpoint information from the one or more viewpoint information.

23. The decoding method according to claim 11, further comprising obtaining at least a portion of the three-dimensional data corresponding to the difference between multiple identification information corresponding to multiple viewpoint information.

24. An encoding device comprising a circuit and a memory connected to the circuit, wherein the circuit, in operation, acquires one or more viewpoint pieces of information, generates a bitstream including control information containing the one or more viewpoint pieces of information, and encoded data obtained by encoding three-dimensional data, and each of the one or more viewpoint pieces of information includes one or more identification pieces of information that identify one or more subregions included in the region defined by the viewpoint piece of information, among one or more subregions corresponding to at least a part of the three-dimensional data.

25. A decoding device comprising a circuit and a memory connected to the circuit, wherein the circuit, in operation, acquires a bitstream, acquires control information including one or more viewpoint pieces of information and at least a portion of encoded data corresponding to at least a portion of three-dimensional data from the bitstream, decodes at least a portion of the encoded data to acquire at least a portion of the three-dimensional data, and each of the one or more viewpoint pieces of information includes one or more identification pieces of information that identify one or more sub-regions included in the region defined by the viewpoint piece of one or more sub-regions corresponding to at least a portion of the three-dimensional data.