Encoding method, decoding method, encoding device, and decoding device

The encoding method for three-dimensional data improves efficiency and image quality by incorporating rendering information, addressing large data volume challenges and ensuring compatibility with existing standards.

WO2026234955A1PCT designated stage Publication Date: 2026-11-12PANASONIC INTELLECTUAL PROPERTY CORP OF AMERICA
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
PCT/JP2026/015337
Authority / Receiving Office
WO · WO
Patent Type
Applications
Current Assignee / Owner
Priority Date
2025-05-09
Filing Date
2026-04-15
Publication Date
2026-11-12

AI Technical Summary

Technical Problem

Existing methods for encoding and decoding three-dimensional data, such as point clouds, face challenges in managing large data volumes and require improvements for efficient storage and transmission.

Method used

An encoding method that generates encoded three-dimensional data with additional information for rendering, including viewpoint restrictions and quality control, to reduce processing load and improve image quality.

Benefits of technology

The method reduces processing load and enhances the quality of rendered images by controlling rendering based on additional information, ensuring compatibility with existing standards.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure JP2026015337_12112026_PF_FP_ABST
    Figure JP2026015337_12112026_PF_FP_ABST
Patent Text Reader

Abstract

This encoding method comprises: generating encoded three-dimensional data by encoding three-dimensional data (S321); and generating a bit stream including the encoded three-dimensional data and additional information including first information relating to rendering of the three-dimensional data (S322). For example, the first information may include second information indicating a range of viewpoints for which rendering is prohibited. For example, the second information includes viewpoint positions and information indicating distances from the viewpoint positions, and rendering from viewpoints in a range within a distance from the viewpoint positions may be prohibited.
Need to check novelty before this filing date? Find Prior Art

Description

Encoding method, decoding method, encoding apparatus and decoding apparatus

[0001] The present disclosure relates to an encoding method, a decoding method, an encoding apparatus, and a decoding apparatus.

[0002] In a wide range of fields such as computer vision for autonomous operation of automobiles or robots, map information, surveillance, infrastructure inspection, or video distribution, the spread of devices or services utilizing three-dimensional data is expected in the future. Three-dimensional data is acquired by various methods such as a distance sensor like a range finder, a stereo camera, or a combination of a plurality of monocular cameras.

[0003] As one of the methods for representing three-dimensional data, there is a representation method called a point cloud that represents the shape of a three-dimensional structure using a point group in a three-dimensional space. In the point cloud, the positions and colors of the point group are stored. The point cloud is expected to become the mainstream as a representation method for three-dimensional data, but the data volume of the point group is extremely large. Therefore, in the storage or transmission of three-dimensional data, similar to two-dimensional moving images (examples include MPEG-4 AVC (Advanced Video Coding) or HEVC (High Efficiency Video Coding) standardized by MPEG), compression of the data volume by encoding is essential.

[0004] In addition, point cloud compression is partially supported by public libraries (Point Cloud Library) that perform point cloud-related processing and the like.

[0005] In addition, a technology is known that searches for and displays facilities located around a vehicle using three-dimensional map data (see, for example, Patent Document 1).

[0006] International Publication No. 2014 / 020663

[0007] Further improvements are desired regarding the encoding and decoding of three-dimensional data as described above. The present disclosure aims to provide an encoding method, a decoding method, an encoding apparatus, or a decoding apparatus capable of realizing further improvements.

[0008] An encoding method according to one aspect of the present disclosure generates encoded three-dimensional data by encoding three-dimensional data, and generates a bitstream including additional information including first information relating to the rendering of the three-dimensional data and the encoded three-dimensional data.

[0009] A decoding method according to one aspect of this disclosure generates three-dimensional data by decoding encoded three-dimensional data contained in a bitstream, and obtains first information relating to the rendering of the three-dimensional data from additional information contained in the bitstream.

[0010] This disclosure can provide an encoding method, a decoding method, an encoding device, or a decoding device that can achieve further improvements.

[0011] Figure 1 is a diagram showing an example configuration of a three-dimensional data encoding and decoding system according to an embodiment. Figure 2 is a diagram showing the configuration of point cloud data in the embodiment. Figure 3 is a diagram showing an example configuration of a data file describing information about point cloud data in the embodiment. Figure 4 is a diagram showing the configuration of three-dimensional mesh data in the embodiment. Figure 5 is a diagram showing an example configuration of a data file describing information about three-dimensional mesh data in the embodiment. Figure 6 is a diagram for explaining a three-dimensional model in the embodiment. Figure 7 is a diagram showing the types of three-dimensional data in the embodiment. Figure 8 is a diagram for explaining the encoding process of three-dimensional data in the embodiment. Figure 9 is a diagram for explaining the decoding process of three-dimensional data in the embodiment. Figure 10 is a diagram schematically showing tiles and slices of three-dimensional data in two dimensions in the embodiment. Figure 11 is a diagram showing an example of a screen for a user to select three-dimensional data according to the embodiment. Figure 12 is a diagram showing an example of automatic switching of three-dimensional data according to the embodiment. Figure 13 is a block diagram showing an example of the functional configuration of a server and a terminal in the embodiment. Figure 14 is a block diagram showing another example of the data generation unit of the server in the embodiment. Figure 15 is a diagram showing an example of mesh data according to the embodiment. Figure 16 is a diagram showing an example of point cloud data according to the embodiment. Figure 17 is a diagram showing an example of mesh data according to the embodiment. Figure 18 is a diagram showing an example of point cloud data according to the embodiment. Figure 19 is a diagram illustrating the relationship between three-dimensional space and encoded data in the embodiment. Figure 20 is a diagram showing an example of the syntax of an encoding scheme unit in the embodiment. Figure 21 is a diagram showing an example of the syntax of an encoded point cloud in the embodiment. Figure 22 is a diagram showing an example of the syntax of an encoded mesh in the embodiment. Figure 23 is a diagram showing an example of the syntax of an encoded three-dimensional model in the embodiment. Figure 24 is a diagram showing an example of the syntax of three-dimensional data information in the embodiment. Figure 25 is a diagram illustrating the data structure of an encoded point cloud in the embodiment.Figure 26 is a diagram illustrating the data structure of an encoded mesh in an embodiment. Figure 27 is a diagram illustrating the data structure of an encoded three-dimensional model in an embodiment. Figure 28 is a two-dimensional diagram showing an example of multiple three-dimensional spaces in an embodiment. Figure 29 is a diagram showing an example of a bounding box in an embodiment. Figure 30 is a diagram showing an example of the syntax of three-dimensional spatial information in an embodiment. Figure 31 is a flowchart showing an example of partial decoding in an embodiment. Figure 32 is a diagram showing an example of a three-dimensional spatial region targeted for partial decoding in an embodiment. Figure 33 is a diagram showing an example of the data structure of an encoded point cloud to be partially decoded in an embodiment. Figure 34 is a diagram showing an example of the data structure of an encoded mesh to be partially decoded in an embodiment. Figure 35 is a diagram showing an example of the data structure of an encoded three-dimensional model to be partially decoded in an embodiment. Figure 36 is a diagram showing an example of the coordinate system of mesh data according to an embodiment. Figure 37 is a diagram showing an example of the coordinate system of point cloud data according to an embodiment. Figure 38 is a diagram showing an example of 3D data information (3Ddata_info) according to an embodiment. Figure 39 is a diagram showing an example of 3D spatial information (3Dspace_info) according to the embodiment. Figure 40 is a block diagram of a terminal that receives 3D data according to the embodiment. Figure 41 is a flowchart of the synchronization process by the terminal according to the embodiment. Figure 42 is a block diagram showing an example of a device that generates format data from data according to the embodiment. Figure 43 is a block diagram showing an example of a device that restores the original data from the format data according to the embodiment. Figure 44 is a conceptual diagram illustrating an example of how an encoded data bitstream is stored in a system format according to the embodiment. Figure 45 is a diagram showing an example of the box structure of ISOBMFF according to the embodiment. Figure 46 is a diagram showing a part of the processing in the three-dimensional data generation system according to the embodiment. Figure 47 is a diagram showing the components of Gaussian data according to the embodiment. Figure 48 is a diagram showing the rendering process of Gaussian data according to the embodiment.Figure 49 is a diagram showing an example of the number of SH coefficients of a spherical harmonic function according to the embodiment. Figure 50 is a diagram showing an example of input and output of a spherical harmonic function according to the embodiment. Figure 51 is a diagram showing the rendering process according to the embodiment. Figure 52 is a block diagram of the encoding device according to the embodiment. Figure 53 is a block diagram of the decoding device according to the embodiment. Figure 54 is a block diagram of the encoding device according to the embodiment. Figure 55 is a block diagram of the decoding device according to the embodiment. Figure 56 is a block diagram of the data processing system according to Embodiment 1. Figure 57 is a block diagram of the GS generation unit according to Embodiment 1. Figure 58 is a diagram showing an example of learning method information according to Embodiment 1. Figure 59 is a diagram showing an example of quality information according to Embodiment 1. Figure 60 is a flowchart showing an example of processing using GS data according to Embodiment 1. Figure 61 is a block diagram of the GS encoding unit according to Embodiment 2. Figure 62 is a diagram showing an example of encoding method information indicating the encoding method of GS data according to Embodiment 2. Figure 63 is a diagram showing an example of encoding method information when using a profile according to Embodiment 2. Figure 64 is a diagram showing an example of a profile according to Embodiment 2. Figure 65 is a diagram showing an example of tools supported by the profile according to Embodiment 2. Figure 66 is a diagram showing an example of quality information according to Embodiment 2. Figure 67 is a flowchart showing an example of processing using GS data according to Embodiment 2. Figure 68 is a block diagram of the data processing system according to Embodiment 3. Figure 69 is a diagram showing an example of a system that performs switching operations according to Embodiment 3. Figure 70 is a flowchart of the switching operation according to Embodiment 3. Figure 71 is a diagram showing an example of hierarchical structure information according to Embodiment 3. Figure 72 is a diagram showing an example of the configuration of basic data and extended data according to Embodiment 3. Figure 73 is a diagram showing an example of the configuration of basic data and extended data according to Embodiment 3. Figure 74 is a diagram showing an example of the configuration of basic data and extended data according to Embodiment 3. Figure 75 is a flowchart of the 3D data switching process according to Embodiment 3. Figure 76 is a diagram showing an example of a setting simulation according to Embodiment 3. Figure 77 is a diagram showing an example of an appearance simulation according to Embodiment 3.Figure 78 is a diagram showing an example of quality information according to Embodiment 4. Figure 79 is a diagram showing an example of quality information according to Embodiment 4. Figure 80 is a diagram showing an example of the syntax of rendering assistance information according to Embodiment 5. Figure 81 is a diagram showing an example of a method for setting rendering prohibited areas according to Embodiment 5. Figure 82 is a flowchart showing an example of processing in a playback device according to Embodiment 5. Figure 83 is a flowchart of processing by an encoding device according to an embodiment. Figure 84 is a block diagram of an encoding device according to an embodiment. Figure 85 is a flowchart of processing by a decoding device according to an embodiment. Figure 86 is a block diagram of a decoding device according to an embodiment.

[0012] [Introduction] An encoding method according to one aspect of the present disclosure generates encoded three-dimensional data by encoding three-dimensional data, and generates a bitstream including additional information including first information relating to the rendering of the three-dimensional data and the encoded three-dimensional data.

[0013] According to this, a data processing device that renders decoded three-dimensional data can perform rendering appropriately using the first information. Therefore, further improvements can be achieved, such as reducing the processing load in the data processing device or improving the quality of the rendered image.

[0014] For example, the first information may include second information indicating the range of viewpoints for which rendering is prohibited. This can, for example, prevent the generation of low-quality rendered images.

[0015] For example, the second information may include a viewpoint position and information indicating the distance from that viewpoint position, and rendering from viewpoints within the range of the distance from the viewpoint position may be prohibited. This allows for appropriate control over the range in which rendering is prohibited.

[0016] For example, the second information may include information indicating the range and the line of sight direction, and rendering from a viewpoint within the range to the line of sight direction may be prohibited. This allows for fine-grained control over the prohibition of rendering.

[0017] For example, the first information may include information indicating the quality of the two-dimensional image generated from the three-dimensional data for each viewpoint, and rendering from viewpoints where the quality is set to or below a predetermined value may be prohibited. This allows for the control of the region where rendering is prohibited based on the information indicating the quality of the two-dimensional image. Therefore, since there is no need to add new information to the bitstream, the amount of data in the bitstream can be reduced.

[0018] For example, the first information may include a third piece of information relating to the lighting used for rendering. In this case, the data processing device can improve the quality of the rendered image by using the third piece of information.

[0019] For example, the third information may be provided for each viewpoint. In this case, the data processing device can improve the quality of the rendered image by using the third information for each viewpoint.

[0020] For example, the first information may include information indicating a viewpoint used to generate an overhead image of the three-dimensional data. This allows the data processing device to easily generate an overhead image, thereby reducing the processing load on the data processing device.

[0021] For example, the first information may include information indicating the distance between viewpoints that is permissible during viewpoint movement. This makes it possible to suppress large changes in the viewpoint of the rendered image. Therefore, the processing load on the data processing device can be reduced and the processing speed can be improved.

[0022] For example, the first information may include information about the engine used for rendering. This allows the data processing device to improve the quality or reproducibility of the rendered image by using the information.

[0023] For example, the first information may include parameters for projecting the three-dimensional data onto a two-dimensional image. This allows the data processing device to improve the quality or reproducibility of the rendered image by using the information.

[0024] For example, the first information may include parameters for inversely transforming a two-dimensional image into three-dimensional data. This allows the data processing device to improve the quality or reproducibility of the rendered image by using the information.

[0025] For example, the additional information may be an SEI (Supplemental Enhancement Information) message. This ensures, for example, compatibility between the generated bitstream and existing encoding standards.

[0026] For example, the three-dimensional data may include data for multiple objects, and the first information may include information indicating the quality of each of the data for the multiple objects. In this case, the data processing device can acquire the quality for each object, and thus enable detailed control based on that quality.

[0027] For example, the first information may include a fourth piece of information indicating the quality of the three-dimensional position information of the three-dimensional data for each viewpoint. This allows the data processing device to acquire the quality of the three-dimensional position information for each viewpoint, thereby enabling detailed control based on that quality.

[0028] For example, each of the multiple pieces of information described above may include information indicating a viewpoint position, information indicating a range of three-dimensional space based on the viewpoint position, and information indicating the quality of the three-dimensional position information of the three-dimensional data within the range. In this case, the data processing device can acquire the quality of the three-dimensional position information for each spatial range corresponding to the viewpoint, and thus realize detailed control based on that quality.

[0029] For example, the first information may include a fifth piece of information indicating the quality of the attribute information of the three-dimensional data for each viewpoint. This allows the data processing device to acquire the quality of the attribute information for each viewpoint, thereby enabling detailed control based on that quality.

[0030] For example, each of the multiple pieces of information described above may include information indicating a viewpoint position, information indicating a range of three-dimensional space based on the viewpoint position, and information indicating the quality of attribute information of the three-dimensional data within the range. In this case, the data processing device can acquire the quality of attribute information for each spatial range corresponding to the viewpoint, and thus realize detailed control based on that quality.

[0031] For example, the first information may include information common to the three-dimensional position information and attribute information of the three-dimensional data for each viewpoint, and may include information indicating the quality of the three-dimensional position information and attribute information of the three-dimensional data. This reduces the amount of data compared to the case where quality information is generated separately for the position information and attribute information.

[0032] A decoding method according to one aspect of this disclosure generates three-dimensional data by decoding encoded three-dimensional data contained in a bitstream, and obtains first information relating to the rendering of the three-dimensional data from additional information contained in the bitstream.

[0033] According to this, a data processing device that renders decoded three-dimensional data can perform rendering appropriately using the first information. Therefore, further improvements can be achieved, such as reducing the processing load in the data processing device or improving the quality of the rendered image. The data processing device may also include a decoding device that decodes the encoded three-dimensional data.

[0034] For example, the first information may include second information indicating the range of viewpoints for which rendering is prohibited. This can, for example, prevent the generation of low-quality rendered images.

[0035] For example, the second information may include a viewpoint position and information indicating the distance from that viewpoint position, and rendering from viewpoints within the range of the distance from the viewpoint position may be prohibited. This allows for appropriate control over the range in which rendering is prohibited.

[0036] For example, the second information may include information indicating the range and the line of sight direction, and rendering from a viewpoint within the range to the line of sight direction may be prohibited. This allows for fine-grained control over the prohibition of rendering.

[0037] For example, the first information may include information indicating the quality of the two-dimensional image generated from the three-dimensional data for each viewpoint, and rendering from viewpoints where the quality is set to or below a predetermined value may be prohibited. This allows for the control of the region where rendering is prohibited based on the information indicating the quality of the two-dimensional image. Therefore, since there is no need to add new information to the bitstream, the amount of data in the bitstream can be reduced.

[0038] For example, the first information may include a third piece of information relating to the lighting used for rendering. In this case, the data processing device can improve the quality of the rendered image by using the third piece of information.

[0039] For example, the third information may be provided for each viewpoint. In this case, the data processing device can improve the quality of the rendered image by using the third information for each viewpoint.

[0040] For example, the first information may include information indicating a viewpoint used to generate an overhead image of the three-dimensional data. This allows the data processing device to easily generate an overhead image, thereby reducing the processing load on the data processing device.

[0041] For example, the first information may include information indicating the distance between viewpoints that is permissible during viewpoint movement. This makes it possible to suppress large changes in the viewpoint of the rendered image. Therefore, the processing load on the data processing device can be reduced and the processing speed can be improved.

[0042] For example, the first information may include information about the engine used for rendering. This allows the data processing device to improve the quality or reproducibility of the rendered image by using the information.

[0043] For example, the first information may include parameters for projecting the three-dimensional data onto a two-dimensional image. According to this, the data processing apparatus can improve the quality or reproducibility of the rendered image by using the information.

[0044] For example, the first information may include parameters for inversely transforming a two-dimensional image into the three-dimensional data. According to this, the data processing apparatus can improve the quality or reproducibility of the rendered image by using the information.

[0045] For example, the additional information may be a SEI (Supplemental Enhancement Information) message. According to this, compatibility between the generated bitstream and existing coding standards can be ensured, for example.

[0046] For example, the three-dimensional data includes data of a plurality of objects, and the first information may include information indicating the quality of each of the data of the plurality of objects. According to this, the data processing apparatus can acquire the quality for each object, so that detailed control based on the quality can be realized.

[0047] For example, the first information may include, for each viewpoint, fourth information indicating the quality of the three-dimensional position information of the three-dimensional data. According to this, the data processing apparatus can acquire the quality of the three-dimensional position information for each viewpoint, so that detailed control based on the quality can be realized.

[0048] For example, each of the plurality of pieces of fourth information may include information indicating a viewpoint position, information indicating a range of a three-dimensional space based on the viewpoint position, and information indicating the quality of the three-dimensional position information of the three-dimensional data in the range. According to this, the data processing apparatus can acquire the quality of the three-dimensional position information for each spatial range corresponding to the viewpoint, so that detailed control based on the quality can be realized.

[0049] For example, the first information may include, for each viewpoint, fifth information indicating the quality of the attribute information of the three-dimensional data. According to this, the data processing apparatus can acquire the quality of the attribute information for each viewpoint, so that detailed control based on the quality can be realized.

[0050] For example, each of the multiple pieces of information described above may include information indicating a viewpoint position, information indicating a range of three-dimensional space based on the viewpoint position, and information indicating the quality of attribute information of the three-dimensional data within the range. In this case, the data processing device can acquire the quality of attribute information for each spatial range corresponding to the viewpoint, and thus realize detailed control based on that quality.

[0051] For example, the first information may include information common to the three-dimensional position information and attribute information of the three-dimensional data for each viewpoint, and may include information indicating the quality of the three-dimensional position information and attribute information of the three-dimensional data. This reduces the amount of data compared to the case where quality information is generated separately for the position information and attribute information.

[0052] An encoding method according to one aspect of the present disclosure comprises a processor and a memory, wherein the processor generates encoded three-dimensional data by encoding three-dimensional data using the memory, and generates a bitstream including additional information including first information relating to the rendering of the three-dimensional data and the encoded three-dimensional data.

[0053] According to this, a data processing device that renders decoded three-dimensional data can perform rendering appropriately using the first information. Therefore, further improvements can be achieved, such as reducing the processing load in the data processing device or improving the quality of the rendered image.

[0054] A decoding method according to one aspect of the present disclosure comprises a processor and a memory, wherein the processor generates three-dimensional data by decoding encoded three-dimensional data contained in a bitstream using the memory, and obtains first information relating to the rendering of the three-dimensional data from additional information contained in the bitstream.

[0055] According to this, a data processing device that renders decoded three-dimensional data can perform rendering appropriately using the first information. Therefore, further improvements can be achieved, such as reducing the processing load in the data processing device or improving the quality of the rendered image.

[0056] These comprehensive or specific embodiments may be implemented as a system, integrated circuit, computer program, or recording medium such as a computer-readable CD-ROM, or as any combination of system, method, integrated circuit, computer program, and recording medium.

[0057] The embodiments will be described in detail below with reference to the drawings. Note that the embodiments described below are all specific examples of this disclosure. The numerical values, shapes, materials, components, arrangement and connection configurations of components, steps, and the order of steps shown in the following embodiments are examples only and are not intended to limit this disclosure. Furthermore, among the components in the following embodiments, those not described in the independent claim representing the highest-level concept will be described as optional components.

[0058] (Embodiment) The configuration of the three-dimensional data encoding and decoding system according to this embodiment will be described. Figure 1 is a diagram showing an example of the configuration of the three-dimensional data encoding and decoding system according to this embodiment. As shown in Figure 1, the three-dimensional data encoding and decoding system includes a three-dimensional data encoding system 1001, a three-dimensional data decoding system 1002, a sensor terminal 1003, and an external connection unit 1004.

[0059] The three-dimensional data encoding system 1001 generates encoded data or multiplexed data by encoding three-dimensional data. The three-dimensional data encoding system 1001 may be a three-dimensional data encoding device implemented by a single device, or it may be a system implemented by multiple devices. Furthermore, the three-dimensional data encoding device may include some of the multiple processing units included in the three-dimensional data encoding system 1001.

[0060] The three-dimensional data encoding system 1001 includes a three-dimensional data generation system 1011, a presentation unit 1012, an encoding unit 1013, a multiplexing unit 1014, an input / output unit 1015, and a control unit 1016. The three-dimensional data generation system 1011 also includes a sensor information acquisition unit 1017 and a three-dimensional data generation unit 1018.

[0061] The sensor information acquisition unit 1017 acquires a sensor signal from the sensor terminal 1003 and outputs the sensor signal to the three-dimensional data generation unit 1018. The three-dimensional data generation unit 1018 generates three-dimensional data from the sensor signal and outputs the three-dimensional data to the encoding unit 1013.

[0062] The display unit 1012 presents sensor signals or three-dimensional data to the user. For example, the display unit 1012 displays information or images based on sensor signals or three-dimensional data.

[0063] The encoding unit 1013 encodes (compresses) the three-dimensional data and outputs the resulting encoded data, control information obtained during the encoding process, and other additional information to the multiplexing unit 1014. The additional information includes, for example, sensor signals.

[0064] The multiplexing unit 1014 generates multiplexed data by multiplexing the encoded data input from the encoding unit 1013, control information, and additional information. The format of the multiplexed data is, for example, a file format for storage or a packet format for transmission.

[0065] The input / output unit 1015 (for example, the communication unit or interface) outputs the multiplexed data to the outside. Alternatively, the multiplexed data is stored in a storage unit such as internal memory. The control unit 1016 (or the application execution unit) controls each processing unit. In other words, the control unit 1016 performs control such as encoding and multiplexing. The control unit 1016 may also perform demultiplexing, decoding, or presentation control.

[0066] The sensor signal may also be input to the encoding unit 1013 or the multiplexing unit 1014. Furthermore, the input / output unit 1015 may output the three-dimensional data or encoded data directly to the outside.

[0067] The transmission signal (multiplexed data) output from the three-dimensional data encoding system 1001 is input to the three-dimensional data decoding system 1002 via the external connection unit 1004.

[0068] The three-dimensional data decoding system 1002 generates three-dimensional data by decoding encoded data or multiplexed data. The three-dimensional data decoding system 1002 may be a three-dimensional data decoding device implemented by a single device, or it may be a system implemented by multiple devices. Furthermore, the three-dimensional data decoding device may include some of the multiple processing units included in the three-dimensional data decoding system 1002.

[0069] The three-dimensional data decoding system 1002 includes a sensor information acquisition unit 1021, an input / output unit 1022, a demultiplexing unit 1023, a decoding unit 1024, a presentation unit 1025, a user interface 1026, and a control unit 1027.

[0070] The sensor information acquisition unit 1021 acquires sensor signals from the sensor terminal 1003.

[0071] The input / output unit 1022 acquires the transmission signal, decodes the multiplexed data (file format or packet) from the transmission signal, and outputs the multiplexed data to the demultiplexing unit 1023.

[0072] The demultiplexing unit 1023 acquires encoded data, control information, and additional information from the multiplexed data, and outputs the encoded data, control information, and additional information to the decoding unit 1024.

[0073] The decoding unit 1024 reconstructs the point cloud data by decoding the encoded data.

[0074] The presentation unit 1025 presents point cloud data to the user. For example, the presentation unit 1025 displays information or images based on the point cloud data. The user interface 1026 acquires instructions based on user operations. The control unit 1027 (or application execution unit) controls each processing unit. In other words, the control unit 1027 performs control such as demultiplexing, decoding, and presentation.

[0075] The input / output unit 1022 may acquire point cloud data or encoded data directly from an external source. The presentation unit 1025 may acquire additional information such as sensor signals and present information based on that additional information. The presentation unit 1025 may also make presentations based on user instructions acquired through the user interface 1026.

[0076] The sensor terminal 1003 generates a sensor signal, which is information obtained from the sensor. The sensor terminal 1003 is a terminal equipped with a sensor or camera, and may be, for example, a mobile object such as an automobile, an aerial object such as an airplane, a mobile terminal, or a camera.

[0077] The sensor signals obtainable by the sensor terminal 1003 include, for example, (1) signals indicating the distance between the sensor terminal 1003 and the object, or the reflectivity of the object, obtained from a LiDAR, millimeter-wave radar, or infrared sensor, and (2) signals indicating the distance between the camera and the object, or the reflectivity of the object, obtained from multiple monocular camera images or stereo camera images. The sensor signals may also include the sensor's attitude, orientation, gyroscope (angular velocity), position (GPS information or altitude), speed, or acceleration. Furthermore, the sensor signals may include temperature, atmospheric pressure, humidity, or magnetism.

[0078] The external connection unit 1004 is implemented by an integrated circuit (LSI or IC), an external storage unit, communication with a cloud server via the internet, or broadcasting, etc.

[0079] Next, we will explain point cloud data. Figure 2 shows the structure of point cloud data. Figure 3 shows an example of the structure of a data file containing information about point cloud data.

[0080] Point cloud data contains data for multiple points. Each point's data includes location information (three-dimensional coordinates) and attribute information related to that location. A collection of these points is called a point cloud. For example, a point cloud represents the three-dimensional shape of an object.

[0081] Position information, such as three-dimensional coordinates, is sometimes called geometry. Furthermore, the data for each point may include attribute information of multiple attribute types. Attribute types include, for example, color or reflectance.

[0082] One location information may be associated with one attribute information, or multiple attribute information of different attribute types may be associated with one location information. Furthermore, multiple attribute information of the same attribute type may be associated with one location information.

[0083] The example data file structure shown in Figure 3 represents a case where location information and attribute information correspond one-to-one, and it shows the location information and attribute information of the N points that make up the point cloud data.

[0084] Location information includes, for example, information for the three axes: x, y, and z. Attribute information includes, for example, RGB color information. A typical data file is a ply file.

[0085] Next, we will explain three-dimensional mesh data. Figure 4 is a diagram showing the structure of three-dimensional mesh data. Figure 5 is a diagram showing an example of the structure of a data file that describes the information of three-dimensional mesh data.

[0086] Three-dimensional mesh data is a data format used in computer graphics (CG), representing the three-dimensional shape of an object through a collection of multiple surface information. Each of these surface information points represents a polygon, such as a triangle or quadrilateral. Three-dimensional mesh data is also referred to as polygon or polygon mesh.

[0087] The constituent elements are a three-dimensional point cloud, vertices (multiple three-dimensional points in the three-dimensional point cloud), edges (connecting two vertices in multiple three-dimensional points), and faces (enclosed by multiple edges). A three-dimensional point cloud is a set of points that contain positional information in three-dimensional space and attribute information corresponding to that positional information. Note that a three-dimensional point may simply be referred to as a point.

[0088] A vertex may have attribute information such as color information, reflectivity, and normal vectors for a three-dimensional point. The relationships between vertices constituting an edge or face may be represented by information called connectivity. A vertex may also be expressed as a position. The front and back of a face may be represented by the direction of the normal vector for a three-dimensional point. Furthermore, a vertex may have attribute information for the face.

[0089] One example of a mesh data file format is an object file. In a mesh data file like the one shown in Figure 5, the position information G(1) to G(N) and the attribute information A(1) to A(N) of the N vertices that make up the mesh are shown as vertex information. In a mesh data file, the vertex information does not necessarily have to include attribute information.

[0090] Furthermore, attribute information does not need to have a one-to-one correspondence with vertices. In the mesh data file shown in Figure 5, an example is shown where the three-dimensional mesh data has M attribute information A2s.

[0091] Face information is represented by a combination of vertex indices. n[1,3,4] indicates a triangular face composed of three vertices: n=1, n=3, and n=4.

[0092] Furthermore, m[2, 4, 6] indicates that the attribute information for m=2, m=4, and m=6 from attribute information A2 correspond to the three vertices, respectively. Note that although an example of a face being formed with three vertices is shown here, a face can have any number of vertices greater than or equal to three, and is not limited to three. For example, if the face is a quadrilateral, the number of vertices is four, and if the face is a polygon, the number of vertices is equal to the number of vertices that make up the polygon.

[0093] Furthermore, attribute information A2 may be represented in a separate file from the mesh data file, and may include pointer information to that file. For example, attribute information may be stored in a two-dimensional attribute map file, and the attribute map file name and the two-dimensional coordinates in the attribute map may be represented in attribute information A2 of the mesh data file. Thus, attribute information A2 may be included in the mesh data file or represented in a separate file from the mesh data file, and either method makes it possible to specify attribute information for three-dimensional points.

[0094] Next, we will explain the three-dimensional model. Figure 6 is a diagram illustrating the three-dimensional model.

[0095] A three-dimensional model is a model generated based on two-dimensional or three-dimensional data.

[0096] The three-dimensional model learning unit 1031 generates a three-dimensional model, which is a network model in which the three-dimensional shape and attribute information corresponding to the three-dimensional shape have been learned using a Neural Network or the like, by learning two-dimensional data (two-dimensional images) or three-dimensional data (point clouds or meshes).

[0097] The three-dimensional model learning unit 1031 may generate a three-dimensional model by learning using NeRF (Neural Radiance Fields) based on two-dimensional images. Alternatively, the three-dimensional model learning unit 1031 may generate a three-dimensional model after converting two-dimensional images into three-dimensional data by performing photogrammetry using the two-dimensional images. The three-dimensional model may also be generated using three-dimensional data acquired by a sensor (distance sensor).

[0098] Three-dimensional model data consists of elements that make up a three-dimensional model and includes information that describes the structure of a network model, such as features. Three-dimensional model data includes, for example, information about the components of a neural network. This information about components includes, for example, multiple layers such as input layers, hidden layers, and output layers, nodes in each layer, weight coefficients for the nodes, and transformation functions for the nodes.

[0099] Furthermore, the three-dimensional model may be data generated based on learning methods such as 3DGaussian Platting (3DGS). The three-dimensional model may also be data that does not include the structure of a network model.

[0100] The three-dimensional model encoding unit 1032 may encode the three-dimensional model data and transmit the encoded three-dimensional model data.

[0101] The three-dimensional model decoding unit 1033 receives the transmitted, encoded three-dimensional model data and decodes the three-dimensional model based on the encoded three-dimensional model data.

[0102] The rendering reconstruction unit 1034 reconstructs (generates) two-dimensional data (two-dimensional image) or three-dimensional data (point cloud or mesh) based on the decoded three-dimensional model. For example, when using a three-dimensional model modeled with 3DGS or NeRF, the rendering reconstruction unit 1034 acquires viewpoint position or line-of-sight vector information, generates rendered two-dimensional data (two-dimensional image) based on the three-dimensional model and the viewpoint position or line-of-sight vector, and outputs the two-dimensional data. The generated two-dimensional data shows a two-dimensional image of a three-dimensional object as seen from the viewpoint position, or a three-dimensional image of a three-dimensional object as seen from the line of sight indicated by the line-of-sight vector. The three-dimensional object is the three-dimensional object of the subject that was the basis for the two-dimensional data or three-dimensional data input to the three-dimensional model learning unit 1031. Figure 7 is a diagram showing the types of three-dimensional data. As shown in Figure 7, three-dimensional data includes static objects and dynamic objects.

[0103] A static object is three-dimensional data for any given time (a specific moment). A dynamic object is three-dimensional data that changes over time. Hereafter, point cloud data for a given time will be called a PCC frame, or frame. Similarly, mesh data for any given time will be called a mesh frame, or frame.

[0104] The object may be three-dimensional data with a somewhat limited area, like regular video data, or it may be three-dimensional data with no area limitations, like map information.

[0105] Furthermore, there may be points of varying densities, and both sparse point cloud data (sparse mesh data) and dense point cloud data (dense mesh data) may exist.

[0106] The details of each processing unit are described below. Sensor information is acquired by various methods, such as distance sensors like LIDAR or rangefinders, stereo cameras, or combinations of multiple monocular cameras. The three-dimensional data generation unit 1018 generates point cloud data based on the sensor information obtained by the sensor information acquisition unit 1017. The three-dimensional data generation unit 1018 generates position information (geometry information) as point cloud data and adds attribute information to the position information.

[0107] The three-dimensional data generation unit 1018 may process point cloud data when generating position information or adding attribute information. For example, the three-dimensional data generation unit 1018 may reduce the amount of data by deleting point clouds with overlapping positions. The three-dimensional data generation unit 1018 may also transform the position information (such as shifting, rotating, or normalizing it), or process the point cloud data to generate mesh data. Furthermore, the three-dimensional data generation unit 1018 may render attribute information.

[0108] In Figure 1, the three-dimensional data generation system 1011 is included in the three-dimensional data encoding system 1001, but it may also be provided independently outside of the three-dimensional data encoding system 1001.

[0109] The encoding unit 1013 generates encoded data by encoding three-dimensional data based on a predetermined encoding method. The encoding methods include G-PCC (a method using positional information), V-PCC (a method using a video codec), Draco (a mesh encoding method), and V-DMC (a mesh encoding method). The encoding method is not limited to these methods; for example, it may be a method for encoding a dynamic mesh, or another method combining these methods.

[0110] The decoding unit 1024 decodes the three-dimensional data by decoding the encoded data based on a predetermined encoding method.

[0111] The multiplexing unit 1014 generates multiplexed data by multiplexing the encoded data using an existing multiplexing method. The generated multiplexed data is transmitted or stored. In addition to the encoded data of the three-dimensional data, the multiplexing unit 1014 multiplexes other media such as video, audio, subtitles, applications, files, or reference time information. Furthermore, the multiplexing unit 1014 may also multiplex attribute information related to sensor information or point cloud data.

[0112] Multiplexing methods or file formats include ISOBMFF, ISOBMFF-based transmission methods such as MPEG-DASH, MMT, MPEG-2 TS Systems, and RTP.

[0113] The demultiplexing unit 1023 extracts encoded data of the three-dimensional data, other media, and time information from the multiplexed data.

[0114] The input / output unit 1015 transmits the multiplexed data using a method appropriate to the transmission medium or storage medium, such as broadcasting or communication. The input / output unit 1015 may communicate with other devices via the Internet, or with storage units such as cloud servers.

[0115] Communication protocols such as HTTP, FTP, TCP, or UDP can be used. A pull-type communication method or a push-type communication method may be used.

[0116] Either wired or wireless transmission may be used. Wired transmission methods include Ethernet®, USB, RS-232C, HDMI®, or coaxial cable. Wireless transmission methods include wireless LAN, Wi-Fi®, Bluetooth®, or millimeter wave.

[0117] Furthermore, broadcasting formats such as DVB-T2, DVB-S2, DVB-C2, ATSC3.0, or ISDB-S3 may be used.

[0118] Next, we will explain the process of dividing three-dimensional data into one or more three-dimensional data points. Figure 8 is a diagram illustrating the encoding process of three-dimensional data. Figure 9 is a diagram illustrating the decoding process of three-dimensional data.

[0119] As shown in Figure 8, the data division unit 1041 divides the three-dimensional data into one or more three-dimensional spaces and generates one or more divided three-dimensional data (i.e., one or more divided three-dimensional data). The encoding unit 1042 may encode one or more divided three-dimensional data to generate encoded data. The data division unit 1041 and the encoding unit 1042 may be included in a single encoding device as components of that single encoding device, or they may be included in separate devices.

[0120] Each of the one or more three-dimensional spaces may be referred to as a tile or space. A three-dimensional space is, for example, a bounding box. The three-dimensional data contained in each of the divided three-dimensional spaces may be referred to as a slice. A slice is divided three-dimensional data and includes either a point cloud, a mesh, or a three-dimensional model having geometry or attribute information. Each of the multiple slices is encoded by the encoding unit 1042 for each component and output as encoded data. The encoded data includes the multiple encoded slices.

[0121] As shown in Figure 9, in the decoding process, the decoding unit 1051 decodes one or more divided three-dimensional data (one or more slices) based on the encoded data. The data merging unit 1052 merges one or more divided three-dimensional data to restore (generate) three-dimensional data. The decoding unit 1051 and the data merging unit 1052 may be included in a single decoding device as components of the same device, or they may be included in separate devices. The one or more divided three-dimensional data decoded by the decoding unit 1051 do not have to be merged. The decoding unit 1051 may decode a portion of the divided three-dimensional data from one or more divided three-dimensional data based on a portion of the encoded data and output the decoded portion of divided three-dimensional data. In this case, the decoding device does not have to have a data merging unit 1052.

[0122] Figure 10 is a schematic two-dimensional representation of tiles and slices of three-dimensional data.

[0123] When encoding multiple slices, the encoding device may use dependencies between the slices or may not use dependencies. When encoding without dependencies, the encoding device can encode each slice independently, and processing time can be reduced by encoding multiple slices in parallel. Similarly, when multiple slices are encoded without dependencies, the decoding device can decode each slice independently, and processing time can be reduced by decoding multiple slices in parallel. Furthermore, the decoding device can reduce the amount of processing by performing partial decoding, which decodes only some of the multiple slices.

[0124] When encoding using dependencies, the encoding device signals identifiers indicating dependencies and encodes the dependent data in order. When multiple slices are encoded using dependencies, the decoding device decodes them in order, starting with the dependent data, based on the identifiers.

[0125] In the division of three-dimensional data, any number of divisions and any division method may be used. The division of three-dimensional data may be based on the shape of an object, with multiple three-dimensional points being assigned to each object. Alternatively, the division may be based on the number of three-dimensional points contained in a slice; that is, an upper limit may be set for the number of three-dimensional points in a single slice. Furthermore, three-dimensional data may be divided based on whether or not it is included in three-dimensional space (tile information) using map information or location information. Multiple tile shapes may overlap.

[0126] By dividing three-dimensional data into multiple segmented three-dimensional data in this way, adaptive encoding according to the content or object, and parallel processing in decoding become possible.

[0127] [Use Cases] The following describes use cases for selecting and switching between multiple three-dimensional data sets. The server stores multiple three-dimensional data (3D data) for the same space. For example, the server stores point cloud data and mesh data for the same space.

[0128] The terminal switches the 3D data acquired from the server based on the terminal's intended use and displays (presents) the acquired 3D data. For example, a terminal that analyzes 3D data switches the 3D data acquired from the server based on its intended use, such as analysis or display, and user operation.

[0129] The following describes how to switch between displaying point cloud data and mesh data, and how to transmit data between them. The terminal selects the data and sends the selected result to the server. The terminal downloads the selected data from the server and displays the downloaded data.

[0130] The point cloud data may or may not be encoded by the server. If the point cloud data is encoded, the terminal decodes the received data and displays the decoded data.

[0131] First, let's explain an example of switching data according to user requests. The user can choose whether to display point cloud data or mesh data. In other words, the terminal obtains the data selection result based on the user's operation and notifies the server of the obtained selection result. The server sends the 3D data based on the selection result to the terminal.

[0132] Figure 11 shows an example of a screen for a user to select three-dimensional data. The terminal displays a user interface that allows the user to select point cloud data or mesh data, for example, as shown in Figure 11.

[0133] Here, mesh data has the advantage of being suitable for display because it requires less processing for display. On the other hand, point cloud data has the advantage of being suitable for measurement because it provides high accuracy in the positional information of 3D models. Therefore, by selecting mesh data when a user wants to observe a 3D model and point cloud data when they want to perform measurements, it is possible to achieve both reduced display processing and improved measurement accuracy.

[0134] Next, we will explain an example of automatic switching based on user operation. Figure 12 shows an example of this automatic switching of three-dimensional data. For example, as shown in Figure 12, with mesh data displayed, the user operates the terminal to zoom in on the mesh data. As a result of this operation, for example, when the display exceeds a predetermined zoom level, the terminal downloads point cloud data corresponding to a portion of the 3D data that is of interest due to the zoom. The terminal decodes the downloaded point cloud data and displays the decoded point cloud data.

[0135] The above operation is just one example, and automatic switching may occur when other operations are performed. For example, if a user selects a portion of the mesh data by tapping or other actions, the displayed data may switch to the point cloud data corresponding to the selected area.

[0136] Furthermore, the point cloud data may be displayed superimposed on the mesh data, or it may be displayed independently of the mesh data. Also, the terminal may anticipate reaching a predetermined zoom level and begin downloading before that level is reached. The terminal may also begin decrypting the downloaded data.

[0137] Furthermore, if the point cloud data and mesh data are spatially synchronized (coordinate alignment is achieved), the terminal will display the point cloud data as is. On the other hand, if the point cloud data and mesh data are not spatially synchronized, the terminal will perform alignment based on the synchronization information and display the data after alignment. Note that even if the point cloud data and mesh data are not spatially synchronized, the terminal may display the point cloud data without performing alignment.

[0138] In this way, the terminal downloads and decodes only the necessary point cloud data and displays the decoded data. This reduces communication capacity, reduces processing load on the terminal, and lowers display latency.

[0139] Generally, mesh data has a small data size, while point cloud data has a large data size. Therefore, by downloading the mesh data first and displaying it as the initial state, the delay until the initial display can be reduced.

[0140] Furthermore, displaying mesh data reduces the processing load for observing 3D models. Additionally, displaying mesh data and point cloud data simultaneously improves visibility and facilitates measurement.

[0141] The terminal may switch based on the network bandwidth of the communication. For example, the terminal may select mesh data when the bandwidth is narrow and the network speed is slow, and select point cloud data when the network speed is fast.

[0142] Furthermore, the terminal may switch based on its processing performance or CPU usage. For example, if the terminal is displaying point cloud data and it becomes difficult to process the point cloud data (i.e., the terminal's processing performance is insufficient for displaying point cloud data), it may switch the data to display to mesh data. Alternatively, if the terminal is displaying mesh data and it determines that there is sufficient processing power available (i.e., sufficient processing power), it may switch the data to display to point cloud data.

[0143] Next, we will describe an example of a use case for switching between multiple 3D data sets with different resolutions.

[0144] For example, the above process may be used to display 3D data at a construction site. First, the terminal uses mesh data for the appearance of a building or map. The terminal also uses point cloud data for areas where measurement is necessary, such as strain, cracks, or distortions.

[0145] For example, in a space such as a stadium, hall, or factory, if a user wants to check the appearance, such as the brightness, color, or atmosphere of the lighting, the terminal will display mesh data. Similarly, if a user wants to check the studio shape, equipment layout, or aisle layout, the terminal will display point cloud data.

[0146] For example, when rendering and displaying a 3D model, the terminal selects which 3D model to display from multiple 3D models with different resolutions, such as multiple point cloud data or multiple mesh data with different resolutions.

[0147] For example, when a device displays 3D data of an object far from the viewpoint, it uses low-resolution mesh data. When the viewpoint moves closer to the object, the device switches to displaying high-resolution mesh data for the 3D data. This improves the accuracy of the 3D data display.

[0148] For example, when displaying a 3D model related to sports, the device displays mesh data until the user identifies a specific movement or location when analyzing an athlete's movements. After the specific movement or location is identified, the device switches the displayed 3D data to point cloud data. For example, this point cloud data can be used to measure three-dimensional shapes.

[0149] For example, in games, a device can select which 3D model to display from multiple 3D models with different resolutions. For instance, the device might use high-resolution 3D data for important parts and low-resolution 3D data for less important parts. This can improve overall processing performance. Whether a model is important or not is determined based on factors such as its relevance to the game's operation. For example, in a shooting game, the player, target, and gun would be considered important. Alternatively, whether a model is important or not is determined based on its relevance to the player's choices. For example, in a purchasing scenario, product information would be considered important. The importance of each model may also be pre-defined for each game.

[0150] For example, when a device displays terrain data, it selects the terrain data to display from multiple terrain data sets with different resolutions. This allows the device to display more detailed terrain information as needed.

[0151] For example, when a device displays thumbnails on the web, it uses low-resolution 3D data. When the device uses the 3D data selected on the thumbnail for measurement by an application, or when displaying the selected 3D data, it uses high-resolution 3D data.

[0152] For example, if display speed is a priority, the terminal will display mesh data that is easier to process. When measurement is performed, the terminal will replace the 3D data used with point cloud data suitable for measurement. This makes it possible to create a lightweight application.

[0153] Next, we will explain how to select the three-dimensional data to present or transmit from among multiple three-dimensional data sets.

[0154] The server stores multiple three-dimensional data sets for the same space. For example, the server stores point cloud data and mesh data for the same space. The server is an example of an encoding device. The terminal switches the three-dimensional data it acquires from the server based on its intended use and displays the switched three-dimensional data. The terminal may be, for example, a terminal for analyzing three-dimensional data. In this case, the terminal may switch the three-dimensional data it displays based on its intended use, such as analysis or display, and user operation. The terminal is an example of a decoding device.

[0155] When switching between 3D data, the system may switch between presenting a point cloud or a mesh as 3D data. Furthermore, when switching between 3D data, the system may switch between transmitting a point cloud or a mesh as 3D data. For example, a terminal may send the user's selection result to a server, receive (download) 3D data based on that selection result from the server, and then present the received 3D data. The 3D data (point cloud or mesh) may or may not be encoded on the server. If the 3D data is encoded, the terminal may receive the encoded 3D data from the server, decode the 3D data based on the received encoded 3D data, and then present the decoded 3D data.

[0156] Next, the configuration of the server 1070 and the terminal 1090 will be described. Figure 13 is a block diagram showing an example of the functional configuration of the server and the terminal.

[0157] The server 1070 comprises a data generation unit 1071, a synchronization unit 1075, a point cloud coding unit 1076, a mesh coding unit 1077, a model coding unit 1078, a multiplexing unit 1079, and a data extraction unit 1080.

[0158] The data generation unit 1071 generates three-dimensional data based on at least one of two-dimensional data and three-dimensional data. The generated three-dimensional data includes point cloud data, mesh data, and at least two of three-dimensional model data. The data generation unit 1071 has a point cloud generation unit 1072, a mesh generation unit 1073, and a model generation unit 1074. The data generation unit 1071 only needs to have at least two of the point cloud generation unit 1072, the mesh generation unit 1073, and the model generation unit 1074. The point cloud generation unit 1072 generates point cloud data based on at least one of two-dimensional data and three-dimensional data. The mesh generation unit 1073 generates mesh data based on at least one of two-dimensional data and three-dimensional data. The model generation unit 1074 generates three-dimensional model data by machine learning based on at least one of two-dimensional data and three-dimensional data.

[0159] The two-dimensional data input to the data generation unit 1071 may be a two-dimensional image acquired by a camera. The three-dimensional data input to the data generation unit 1071 may be, for example, point cloud data acquired by a sensor such as LiDAR in a space such as a construction site, factory, or office. The data generation unit 1071 may generate color information corresponding to each point in the point cloud data of the three-dimensional data as attribute information, using the two-dimensional image of the two-dimensional data. The three-dimensional data generated by the data generation unit 1071 may be divided into arbitrary spaces. The point cloud data, mesh data, and three-dimensional model data may each be divided into arbitrary spaces.

[0160] The synchronization unit 1075 synchronizes the spatial position or time (playback time, decoding time, acquisition time, etc.) of the point cloud data, mesh data, and three-dimensional model data generated by the data generation unit 1071. The time of each data is the playback time, decoding time, acquisition time, etc. The synchronization unit 1075 may generate synchronization information for synchronization without synchronizing the point cloud data, mesh data, and three-dimensional model data. The synchronization unit 1075 only needs to synchronize at least two types of three-dimensional data from the point cloud data, mesh data, and three-dimensional model data generated by the data generation unit 1071, or generate synchronization information (synchronization signal) for synchronization, and does not need to perform a synchronization process for all three types of three-dimensional data.

[0161] The point cloud encoding unit 1076 encodes the point cloud data after synchronization processing has been performed by the synchronization unit 1075. The point cloud encoding unit 1076 does not necessarily have to encode the point cloud data. The point cloud data may be pre-encoded or encoded in response to a request from the terminal 1090.

[0162] The mesh encoding unit 1077 encodes the mesh data after synchronization processing has been performed by the synchronization unit 1075.

[0163] The model encoding unit 1078 encodes the three-dimensional model data after synchronization processing has been performed by the synchronization unit 1075.

[0164] The multiplexing unit 1079 multiplexes encoded point cloud data (encoded point cloud), encoded mesh data (encoded mesh data), encoded three-dimensional model data, and synchronization information using a predetermined format or predetermined multiplexing method. Note that multiplexing by the multiplexing unit 1079 is not required. In this case, the server 1070 does not need to be equipped with the multiplexing unit 1079.

[0165] The data extraction unit 1080 extracts some of the three-dimensional data from the multiplexed three-dimensional data in response to a request from the terminal 1090, and transmits the extracted portion of the three-dimensional data to the terminal 1090. Note that data extraction by the data extraction unit 1080 is not required. In this case, the server 1070 does not need to be equipped with the data extraction unit 1080. If data extraction by the data extraction unit 1080 is not performed, the server 1070 may transmit the three-dimensional data multiplexed by the multiplexing unit 1079 to the terminal 1090. Furthermore, if multiplexing by the multiplexing unit 1079 is not performed, the server 1070 may transmit encoded point cloud data, encoded mesh data, encoded three-dimensional model data, and synchronization information to the terminal 1090, or it may transmit a bitstream containing encoded point cloud data, encoded mesh data, encoded three-dimensional model data, and synchronization information to the terminal 1090.

[0166] The terminal 1090 comprises a control unit 1091, a decoding unit 1092, and a display unit 1093.

[0167] The control unit 1091 sends a request to the server 1070 for some of the three-dimensional data to be presented. The control unit 1091 may also accept user input to identify some of the three-dimensional data.

[0168] The decoding unit 1092 decodes some of the three-dimensional data based on the bitstream (encoded data) obtained from the server 1070.

[0169] The display unit 1093 renders and displays a portion of the decoded three-dimensional data.

[0170] The data generation unit 1071 in Figure 13 may be implemented by the data generation unit 1110 shown in Figure 14. Figure 14 is a block diagram showing another example of the server's data generation unit.

[0171] The data generation unit 1110 comprises a point cloud generation unit 1111, a mesh generation unit 1112, and a model generation unit 1113.

[0172] The point cloud generation unit 1111 has the same function as the point cloud generation unit 1072. The point cloud generation unit 1111 acquires point cloud data obtained from the point cloud sensor 1101 and a two-dimensional image obtained from the camera 1102, and generates point cloud data based on the point cloud data and the two-dimensional image. The point cloud data generated by the point cloud generation unit 1111 includes position information for each point and attribute information corresponding to each point indicated by the position information, and includes attribute information (such as color information) extracted from the two-dimensional image.

[0173] The mesh generation unit 1112 generates mesh data based on the point cloud data generated by the point cloud generation unit 1111.

[0174] The model generation unit 1113 has the same functions as the model generation unit 1074. The model generation unit 1113 acquires point cloud data obtained from the point cloud sensor 1101 and two-dimensional images obtained from the camera 1102, and generates three-dimensional model data by performing machine learning based on the point cloud data and two-dimensional images.

[0175] Point cloud data, mesh data, and 3D model data may be generated independently, as explained in Figure 13. Mesh data may be generated from point cloud data, as explained in Figure 14. Note that point cloud data may also be generated from mesh data.

[0176] The mesh may be generated from the point cloud, or the point cloud may be generated from the mesh.

[0177] The point cloud data, mesh data, and three-dimensional model data may be generated by the server 1070, or by the sensor or the terminal 1090 equipped with the sensor. The sensor is, for example, a point cloud sensor 1101 and a camera 1102.

[0178] [Spatial Synchronization of Multiple Three-Dimensional Formats] This section explains spatial synchronization of data in multiple three-dimensional formats. First, we will explain coordinate system synchronization. Even for data relating to the same space, the origin position of the local coordinate system handled by a system may differ from the actual coordinates due to differences in the systems or coordinate systems used. For example, if mesh data is generated from point cloud data, the origin positions of the two sets of data are the same. On the other hand, if the mesh data and point cloud data are generated by different systems, the origin positions of the two sets of data may differ.

[0179] Figure 15 shows an example of mesh data. Figure 16 shows an example of point cloud data. Figure 17 shows an example of mesh data. Figure 18 shows an example of point cloud data. In the examples shown in Figures 15 to 18, the origin coordinates of the coordinate system of the mesh data and the origin coordinates of the coordinate system of the point cloud data are both, for example, (x1, y1, z1) in world coordinates. In other words, the origins of the mesh data and the point cloud data are the same. Also, the coordinate systems of the mesh data and the point cloud data are the same. If the origins of the mesh data and the point cloud data are not the same, the device may correct the position of the origin of at least one of the mesh data and the point cloud data to align the origins of the two data sets.

[0180] Next, we will explain the synchronization of the divided three-dimensional spatial information. For example, the device sets the three-dimensional spatial region for dividing the mesh data and the three-dimensional spatial region for dividing the point cloud data to be the same.

[0181] In the examples shown in Figures 15 and 16, the number, size, and position of multiple three-dimensional spatial regions (bounding boxes, tiles, or slices) in the mesh data coordinate system and the multiple three-dimensional spatial regions (bounding boxes, tiles, or slices) in the point cloud coordinate system are the same.

[0182] Focusing on the three-dimensional space represented by the thick dotted line, the origin and maximum value of the bounding box representing the three-dimensional spatial region are the same in both the mesh data and the point cloud data. Furthermore, the vertical, horizontal, and depth dimensions of the bounding box are also the same in both the mesh data and the point cloud data.

[0183] As shown in the examples in Figures 17 and 18, the number, size, and position of multiple three-dimensional spatial regions in the mesh data and the point cloud data do not have to be exactly the same. In the examples in Figures 17 and 18, the mesh data is divided into large regions, and the regions in the point cloud data correspond to regions obtained by further dividing the regions in the mesh data. In other words, the combined unit of multiple regions in the point cloud data is the same as the region in the mesh data.

[0184] Next, we will explain the relationship between three-dimensional space and encoded data. Figure 19 is a diagram illustrating the relationship between three-dimensional space and encoded data.

[0185] As mentioned above, three-dimensional data includes, for example, point cloud data, mesh data, and three-dimensional models.

[0186] As shown in Figure 19, when three-dimensional data is divided into three three-dimensional data in three three-dimensional spaces (tiles or spaces), the encoding device encodes each of the three divided three-dimensional data and adds a header to create a data unit. The header signals (assigns) the identifier of the space to which the encoded data of the data unit belongs (Space_ID) and the identifier of the data unit (DataUnit_ID).

[0187] A data unit is further given a header containing information such as the data unit identifier or the data unit length, and when these are unitized, an encoding scheme unit is generated.

[0188] Next, we will explain the syntax of the coding scheme unit. Figure 20 shows an example of the syntax of the coding scheme unit. Figure 21 shows an example of the syntax of the coded point cloud. Figure 22 shows an example of the syntax of the coded mesh. Figure 23 shows an example of the syntax of the coded three-dimensional model.

[0189] The `unit_type` parameter indicates the type of data unit stored in the encoding scheme unit. This specifies the type of data unit to be stored in the encoding scheme unit.

[0190] The length indicates the length of the data unit.

[0191] `data()` represents the body of the data unit.

[0192] In Figure 21, if unit_type is 0, it indicates that the data unit is the location information (geometry) of the encoded point cloud. If unit_type is 1, it indicates that the data unit is the attribute information of the encoded point cloud. If unit_type is 2, it indicates that the data unit is the metadata of the encoded point cloud.

[0193] In Figure 22, if unit_type is 0, it indicates that the data unit is the location information (geometry) of the encoded mesh. If unit_type is 1, it indicates that the data unit is the attribute information of the encoded mesh. If unit_type is 2, it indicates that the data unit is the metadata of the encoded mesh.

[0194] In Figure 23, if unit_type is 0, it indicates that the data unit is element 1 of the encoded 3D model. If unit_type is 1, it indicates that the data unit is element 2 of the encoded 3D model. If unit_type is 2, it indicates that the data unit is metadata of the encoded 3D model.

[0195] Note that the syntax shown in Figures 21 to 23 is just an example and is not limited to the above configuration. Some of the syntax components may be used, types (categories) not described above may be used, and the order of the syntax components may be changed. For example, in the syntax for an encoding scheme unit, a common encoding scheme unit configuration may be used for multiple encoding schemes, as shown in Figure 20, and the unit_type, length, and data() shown in Figures 21 to 23 may be shown.

[0196] Furthermore, a header may be added to the encoding scheme unit to indicate the type of encoding scheme unit. Examples of encoding scheme types include point_cloud_codec_unit for point cloud data, mesh_codec_unit for mesh data, and model_codec_unit for three-dimensional model data. This makes it possible to handle multiple encoding schemes in an integrated manner.

[0197] Figure 24 shows an example of the syntax for three-dimensional data information.

[0198] In the syntax, when multiple encoding schemes are stored in a single format, the number of 3D data points (number_of_3Dformat) and the type of 3D data (format_type) contained in that format are indicated, and data for each format may be stored. This makes it possible to handle multiple encoding schemes or 3D data points in an integrated manner, and also to identify multiple encoding schemes or 3D data points.

[0199] 3Ddata_info indicates the format structure information for storing multiple three-dimensional data.

[0200] `number_of_3Dformat` indicates the number of 3D formats used.

[0201] The `format_type` parameter indicates the format of the stored 3D data. For example, the `format_type` number and the corresponding format may be defined as follows: If `format_type` is 0, it indicates that the stored 3D data is in the format of point cloud data. If `format_type` is 1, it indicates that the stored 3D data is in the format of mesh data. If `format_type` is 2, it indicates that the stored 3D data is in the format of G-PCC data (g-pcc). If `format_type` is 3, it indicates that the stored 3D data is in the format of V-DMC data (v-dmc). If `format_type` is 4, it indicates that the stored 3D data is in the format of 3D model data (3Dmodel).

[0202] Next, we will explain the data structure of encoded data for multiple three-dimensional data sets, categorized by the type of three-dimensional data. Figure 25 is a diagram illustrating the data structure of encoded point clouds. Figure 26 is a diagram illustrating the data structure of encoded meshes. Figure 27 is a diagram illustrating the data structure of encoded three-dimensional models.

[0203] The encoding device divides each of the multiple types of three-dimensional data into multiple three-dimensional data for each of the multiple spatial regions, encodes each of the divided three-dimensional data (i.e., multiple divided three-dimensional data), and generates encoded data.

[0204] Each encoded data entry is assigned a header containing at least one of the data_unit_id and space_id.

[0205] Here, `data_unit_id` is an identifier that identifies a data unit within the encoded data, and is unique within the encoded data. `space_id` indicates the identification information of a spatial region. If `data_unit_id` or `space_id` is common across multiple 3D data sets, the same value will be shown across those sets.

[0206] In the examples in Figures 25 to 27, the data unit with data_unit_id=0 in the encoded point cloud, the data unit with data_unit_id=3 in the encoded mesh, and the data unit with data_unit_id=0 in the encoded three-dimensional model are all assigned space_id=1. This means that they are three-dimensional data contained in a common three-dimensional space indicated by Space_ID #1.

[0207] Data such as data and headers may be contained in a bitstream structure such as a data unit or encoding scheme, or they may be stored in a predetermined file format such as each BOX of ISOBMFF.

[0208] Next, we will explain three-dimensional spatial information. Figure 28 is a two-dimensional diagram showing an example of multiple three-dimensional spaces. Figure 29 is a diagram showing an example of a bounding box. Figure 30 is a diagram showing an example of the syntax of three-dimensional spatial information.

[0209] In the syntax for three-dimensional spatial information, 3Dspace_info represents information indicating a divided three-dimensional space. 3Dspace_info can be used for partial decoding.

[0210] `number_of_space` indicates the number of partitions in the three-dimensional space.

[0211] `space_id` indicates the identifier of the partitioned three-dimensional space.

[0212] The three-dimensional spatial information includes bounding box information as information for defining the bounding box shown in Figure 29.

[0213] Bounding box information includes bounding_box_xyz and bounding_box_whd.

[0214] `bounding_box_xyz` indicates the coordinates of the reference point of the bounding box. In the example in Figure 29, it is represented, for example, by the coordinate values ​​of x, y, and z (x0, y0, z0).

[0215] `bounding_box_whd` indicates the size of the bounding box. In the example in Figure 29, it is represented, for example, by width w, height h, and depth d (w0, h0, d0).

[0216] Furthermore, the three-dimensional spatial information may include an identifier for each data unit of the encoded data. However, the three-dimensional spatial information does not necessarily have to include such an identifier. In other words, the identifier does not need to be signaled.

[0217] The `pointcloud_id` indicates the identifier of the data unit of the encoded point cloud in the space corresponding to the `space_id`.

[0218] The mesh_id indicates the identifier of the data unit of the encoded mesh in the space corresponding to the space_id.

[0219] The `model_id` indicates the identifier of the data unit of the encoded 3D model for the space corresponding to `space_id`.

[0220] Furthermore, if a data unit does not have a space_id but does have a data_unit_id, the identifier of the data unit for each encoded data may be stored in the information indicating each space of the three-dimensional spatial information. This allows the three-dimensional spatial information to be associated with the divided three-dimensional encoded data.

[0221] Furthermore, if a space_id is indicated in the data unit, the space_id may be used to associate the three-dimensional spatial information with the data unit identifier for each encoded data. In this case, the data unit identifier for each encoded data does not need to be stored.

[0222] The three-dimensional spatial information of the point cloud data and the three-dimensional spatial information of the mesh data may be made common by making the division method, the origin of each divided space, and the size of the bounding box the same for both the mesh data and the point cloud data. Alternatively, the same three-dimensional spatial information may be used for both the point cloud data and the mesh data. In this way, three-dimensional spatial information may be made common among multiple different types of three-dimensional data, or the same three-dimensional spatial information may be used. By making the three-dimensional spatial information common, it becomes easier to switch between different types of three-dimensional data (for example, switching presentation or transmission). Furthermore, in a format that handles multiple three-dimensional data integrally, it is not necessary to provide three-dimensional spatial information for each three-dimensional data, and one set of three-dimensional spatial information can be used for each three-dimensional data, thus reducing the amount of three-dimensional spatial information data.

[0223] In addition to point cloud data and mesh data, the three-dimensional spatial information of the three-dimensional model may be synchronized with other types of three-dimensional data, or the three-dimensional spatial information may be shared with other types of three-dimensional data.

[0224] Next, we will explain the relationship between the data structure of three-dimensional data and partial decoding. Figure 31 is a flowchart showing an example of partial decoding. Figure 32 shows an example of a three-dimensional spatial region targeted for partial decoding. Figure 33 shows an example of the data structure of an encoded point cloud to be partially decoded. Figure 34 shows an example of the data structure of an encoded mesh to be partially decoded. Figure 35 shows an example of the data structure of an encoded three-dimensional model to be partially decoded.

[0225] In partial decoding, the decoding device first determines the three-dimensional spatial region to be partially decoded (S1001).

[0226] Next, the decoding device uses three-dimensional spatial information (3Dspace_info) to identify regions that overlap with the target three-dimensional spatial region from the bounding box information of multiple three-dimensional spatial regions, and obtains the space_id corresponding to the identified region (S1002).

[0227] Next, the decoding device obtains a data unit having the acquired space_id from the encoded data and decodes it (S1003). In this way, the decoding device performs partial decoding, which decodes only a portion of the three-dimensional data. In partial decoding, the decoding device decodes only a portion of the three-dimensional data, not the entire three-dimensional data.

[0228] For example, as shown in Figure 32, if the three-dimensional spatial region to be partially decoded is the region indicated by the thick line, the space_id of the three-dimensional space to be obtained is determined to be #2 from the three-dimensional spatial information.

[0229] Then, as shown in Figures 33 to 35, the data unit associated with Space_id=#2 among the encoded data of multiple types of three-dimensional data is acquired and decoded.

[0230] The decoding device may also obtain a data unit ID instead of a space_id from the three-dimensional spatial information, and then obtain a data unit having the obtained data unit ID to perform partial decoding.

[0231] In the above embodiment, point cloud data, mesh data, and three-dimensional model data were given as examples of three-dimensional data representing a three-dimensional object, but the embodiment is not limited to these. For example, a three-dimensional object may be represented by multiple sets, each containing line-of-sight information indicating a line of sight and a two-dimensional image of the three-dimensional object as seen from that line of sight. In other words, data containing such multiple sets may be treated as a type of three-dimensional data. Furthermore, the three-dimensional data may be in other formats, such as Gaussian splatting data.

[0232] [Spatial Synchronization of 3D Data] This section describes how to handle cases where multiple 3D data (mesh data and point cloud data) are not spatially synchronized. Figure 36 shows an example of the coordinate system of mesh data. Figure 37 shows an example of the coordinate system of point cloud data. The origin of the coordinate system of the mesh data shown in Figure 36 is (x1, y1, z1), and the origin of the coordinate system of the point cloud data shown in Figure 37 is (x2, y2, z2). In this way, when the origin coordinates differ due to the difference in coordinate systems for data in the same space, the device may align the coordinate systems of the two data by correcting at least one of the coordinates using the relative coordinate values ​​(x1-x2, y1-y2, z1-z2).

[0233] Alternatively, the device may notify the terminal of this relative value as synchronization information. The synchronization information may be the relative position of the origin of the point cloud data to the origin of the mesh data, or the relative position of the origin of the mesh data to the origin of the point cloud data. The same applies to 3D models. If there are three or more types of 3D data to be synchronized, one of the data will be used as the reference.

[0234] Figure 38 shows an example of 3D data information (3Ddata_info). As shown in Figure 38, 3D data information includes space_sync_information. space_sync_information is spatial synchronization information for the 3D data, and for example, it indicates the amount of spatial displacement (the difference between the reference coordinates and the current coordinates).

[0235] 3D data information (3Ddata_info) is included in the bitstream and contains information about the data formats of multiple 3D data. Synchronization information is provided for each data format. For example, the synchronization information for the data format that serves as the basis for synchronization is placed at the beginning of the loop. The synchronization information for data formats in the second and subsequent loops indicates their relative position from the first data format.

[0236] Figure 39 shows an example of 3D spatial information (3Dspace_info). 3D spatial information is contained in a bitstream and includes information about multiple 3D spaces. As shown in Figure 39, 3D spatial information includes synchronization information (space_sync_information). Note that if the spatial positions are shifted for each 3D space, synchronization information may be included in the loop for each space. In other words, 3D spatial information may include synchronization information for each space.

[0237] Figure 40 is a block diagram of a terminal that receives 3D data. The terminal 200 shown in Figure 40 comprises a decoding unit 201 and a synchronization presentation unit 202. The decoding unit 201 decodes synchronization information and multiple 3D data from a bitstream (encoded data). The synchronization presentation unit 202 aligns the multiple 3D data based on the synchronization information and presents (displays) the aligned multiple 3D data.

[0238] Figure 41 is a flowchart of the synchronization process performed by the terminal. First, the terminal determines whether spatial synchronization processing is necessary for the display process (S201). For example, the terminal determines whether spatial synchronization processing is necessary depending on the application or use of the 3D data. For example, if the 3D data is used for measurement purposes, accurate positioning is required, so spatial synchronization processing is determined to be necessary. Also, if the 3D data is to be displayed to the user, accurate positioning is not required, so spatial synchronization processing is determined to be unnecessary. Note that a level of positioning may be defined, and the synchronization accuracy may be switched based on the level.

[0239] If spatial synchronization processing is required (Yes in S201), the terminal determines whether or not spatial synchronization is achieved between the point cloud data and the mesh data (S202). If spatial synchronization is achieved between the point cloud data and the mesh data (Yes in S202), the terminal displays the 3D data (point cloud data and mesh data) as is (S203). On the other hand, if spatial synchronization is not achieved between the point cloud data and the mesh data (No in S202), the terminal performs spatial synchronization processing on the point cloud data and the mesh data based on the synchronization information and displays the point cloud data and mesh data after spatial synchronization processing (S204). Also, if spatial synchronization processing is not required (No in S201), the terminal does not perform spatial synchronization processing and displays the 3D data (point cloud data and mesh data) as is (S205).

[0240] [Other] In the above explanation, spatial synchronization was used as an example of 3D data synchronization, but other methods such as time synchronization, such as matching the display time (presentation time), decoding time, or acquisition time, may also be used. Furthermore, attribute synchronization may be performed to set parameters such as the color matrix, number of color bits, or HDR to be the same for multiple 3D data in order to seamlessly switch the displayed color representation when switching 3D data.

[0241] The above describes the process of switching between multiple 3D data of different types, such as point cloud data and mesh data. However, multiple point cloud data with different resolutions or numbers of points may be switched, or multiple mesh data with different numbers of points or numbers of faces may be switched. Furthermore, three or more 3D data may be switched.

[0242] Furthermore, it may be possible to switch between multiple point cloud data sets acquired at different times. For example, at a construction site, point cloud data from before construction, point cloud data from after construction, point cloud data from 10 years later, and modeled mesh data may be prepared, and these data may be switched between.

[0243] In addition to 3D data, 3D model data such as NeRF may also be used. 3D model data is a model for displaying 3D data. 3D model data may or may not be encoded.

[0244] Multiple 3D models may be prepared for the same space, and these multiple 3D models may be switched between. Alternatively, 3D data and 3D models may be prepared and these may be switched between. In this case, the spatial synchronization method described above can be used. Furthermore, a 3D model identifier may be added to the data format that handles point cloud data and mesh data in an integrated manner. This makes it possible to handle 3D data and 3D models in an integrated manner.

[0245] Furthermore, through synchronization, the spatial information (origin, bounding box, and division method, etc.) of the 3D data and the 3D model are set to be identical, thereby enabling the sharing of 3D region information. This makes partial decoding easier.

[0246] Alternatively, the displayed data may be switched in the following order: low-resolution mesh data, high-resolution mesh data, low-resolution point cloud data, and high-resolution point cloud data.

[0247] Regarding the timing of data transmission and encoding, the server may pre-encode multiple 3D models and store the data of the encoded 3D models. Based on a request from the terminal, the server may extract the requested data from the stored data and send it to the terminal. Alternatively, the server may encode the requested 3D model at the time of the request from the terminal. Furthermore, the terminal may request data from the server in advance and download the data from the server. This reduces the time required to display the 3D data.

[0248] Furthermore, the terminal may choose to display point cloud data, mesh data, a 3D model, or a combination of two or more of these for each segmented data. For example, the terminal could display the segmented data corresponding to the avatar's torso as mesh data and the segmented data corresponding to the face as point cloud data. This would enable a more accurate display.

[0249] Figure 42 is a block diagram showing an example of a device that generates format data from data. This device comprises an encoding unit 1143 and a formatting unit 1144.

[0250] The encoding unit 1143 encodes data consisting of moving images, audio, three-dimensional data (point cloud, mesh, 3D Gaussian splatting, NeRF), neural network models, metadata, SEI, etc., according to a predetermined encoding scheme and generates encoded data. The encoding unit 1143 outputs the generated encoded data as an encoded data bitstream to the formatting unit 1144. In addition to encoded video and audio data, the encoded data may also include metadata such as parameter sets, control information, and SEI. The encoded data may be stored in an encoding unit, which is treated as a processing unit within the encoded data bitstream.

[0251] The formatting unit 1144 formats the input encoded data according to a predetermined system format and outputs it as format data. The formatting unit 1144 multiplexes the encoded data according to, for example, a file format for storage compliant with ISOBMFF or a packet format for transmission compliant with RTP, and can store multiple types of media and related metadata, such as video, audio, three-dimensional data (point cloud, mesh, 3D Gaussian splatting, NeRF), subtitles, application data, file information, reference time information, sensor information, and camera information, in the same format data as needed. The format data may be transmitted externally via input / output means equipped with a communication interface or user interface, or it may be stored in internal memory or storage means.

[0252] Figure 43 is a block diagram showing an example of a device that restores the original data from formatted data.

[0253] The reverse formatting unit 1145 analyzes the input format data according to a predetermined system format, extracts the encoded data stored in the format data, and outputs it as encoded data. The reverse formatting unit 1145 may also extract encoded data according to formats such as ISOBMFF, MPEG-DASH, MMT, AVI, MPEG-2 TS Systems, RTP, glTF, and USD.

[0254] The decoding unit 1146 decodes the encoded data input from the inverse formatting unit 1145 according to a predetermined encoding scheme and outputs data consisting of video, audio, three-dimensional data (point cloud, mesh, 3D Gaussian splatting, NeRF), neural network models, metadata, etc. The output data may be used by applications.

[0255] Figure 44 is a conceptual diagram illustrating an example of how an encoded data bitstream is stored in a system format.

[0256] As shown in this figure, the encoded data bitstream includes not only encoded video and audio data, but also supplementary information such as parameter sets and SEI. The encoded data, parameter sets, and SEI are grouped together as encoding units and treated as NAL units or data units. Examples of these encoding units include the Video NAL unit in video encoding, the TLV unit in G-PCC, and the V3C unit for storing video-based volumetric media, but other types of units and data units may also be used. These encoding units are stored in a system format compliant with ISOBMFF or a transmission format compliant with RTP. Furthermore, the metadata (parameter sets, control information, SEI) of the encoded data described in this embodiment, a part of the metadata, the syntax structure, and the header of the encoded data may be defined as a box structure. These boxes may be contained in the Movie box "moov" or in the Media Data box "mdat". If the metadata is per frame, a new metadata track may be defined and the metadata may be stored in sample entries within the metadata track. Furthermore, the encoded video and three-dimensional data described in this embodiment may also be defined as a box structure and may be stored in an mdat box as a sample or subsample. The header portion may be stored in a moov box and the payload portion in an mdat box. In addition, the SEI described in this embodiment may also be defined as a box structure and may be contained within a moov box or an mdat box. The encoded data may be stored in units, or parts of units may be extracted and stored, or the syntax structure may be modified as needed. Note that the above is an example of storing encoded data in ISOBMFF, and the same approach can be applied when storing encoded data in other formats.

[0257] Figure 45 shows an example of the box structure of ISOBMFF.

[0258] ISOBMFF is an ISO-based media file format specified in ISO / IEC 14496-12, and is a media-independent file format standard for multiplexing and storing various media such as video, audio, and text. In ISOBMFF, a file is composed of multiple boxes, each box consisting of type, length, and data. This embodiment shows an example in which a File type box "ftyp", a Movie box "moov" that stores metadata such as control information, and a Media Data box "mdat" that stores media data such as encoded data are used. The ftyp box indicates the file brand using 4CC and shows compatibility, while the moov box contains a track box that shows media-specific information, including an hdlr box indicating the media type, an stsd box indicating decoding parameters, a tref box indicating reference relationships, and an stco box indicating the data storage location. The mdat box stores samples and subsamples of encoded data corresponding to each media track in the moov box. A sample is a unit of encoded data corresponding to an access unit or a frame at the same time, and a subsample is segmented data corresponding to a part of an access unit or frame. Note that the method of storing each media in ISOBMFF is specified separately; for example, the storage method for AVC video and HEVC video is specified in ISO / IEC 14496-15, and the storage method for three-dimensional data is specified in ISO / IEC 23090-10 and ISO / IEC 23090-18.

[0259] [Gaussian Data] The following describes Gaussian data (Gaussian Splatting 3D data). First, we will explain the configuration of a system that encodes or decodes three-dimensional data generated using the Gaussian Splatting method.

[0260] Figure 46 shows a part of the processing in the three-dimensional data generation system shown in Figure 1. The sensor information input unit 301 and the three-dimensional data generation unit 302 shown in Figure 46 correspond, for example, to the sensor information acquisition unit 1017 and the three-dimensional data generation unit 1018 shown in Figure 1.

[0261] The sensor information input unit 301 acquires point cloud data (three-dimensional point cloud data) and outputs the acquired point cloud data to the three-dimensional data generation unit 302. The three-dimensional data generation unit 302 generates Gaussian data (Gaussian Splatting data) from the point cloud data and outputs the generated Gaussian data.

[0262] Specifically, the three-dimensional data generation unit 302 first generates a grid according to the density of the input point cloud data. Each point in the point cloud data is projected onto the nearest grid. At this time, attribute information such as color, reflectance, or normal associated with the point is also linked to the grid. The three-dimensional data generation unit 302 derives parameters such as the mean and variance of the Gaussian function from the information projected onto the grid.

[0263] The method for generating Gaussian data is not limited to the above. For example, Gaussian data may be generated by machine learning using point cloud data generated from multiple two-dimensional images using Structure from Motion (SfM), etc.

[0264] Gaussian data is a collection of multiple 3D Gaussian (ellipsoid) data. Each 3D Gaussian contains data such as three-dimensional point coordinates, a 3x3 covariance matrix, color, and transparency. It is also possible to represent the above data of a 3D Gaussian by converting it into components described in the PLY format.

[0265] Figure 47 shows the components of Gaussian data described in PLY format. The Gaussian data shown in Figure 47 includes Position, Orientation, Scale, SH Coefficient, and Transparency. Transparency is also called transparency, clarity, or opacity.

[0266] Position is the three-dimensional coordinate of a three-dimensional point. Scale and Orientation are parameters of the three-dimensional covariance matrix. Furthermore, the SH coefficient is the coefficient of the spherical harmonics when color information is represented using spherical harmonics.

[0267] In other words, Gaussian data includes a reference three-dimensional coordinate (position) and information associated with that three-dimensional coordinate (rotation, scale, SH coefficient, transparency) for each 3D Gaussian.

[0268] Note that the structure of Gaussian data is not limited to the above. For example, Gaussian data does not have to include a covariance matrix. For example, Gaussian data may include a position (Position) indicating the three-dimensional coordinates of a three-dimensional point, a scale (Scale) indicating the scale along each axis, a color (Color) indicating RGB, and a transmittance (Transparency).

[0269] Furthermore, Gaussian data may include multiple cubes (voxels) that divide 3D space, instead of multiple ellipsoids. Each voxel has attribute information. For example, each voxel may include a position indicating the three-dimensional coordinates of the voxel's center point, attribute information such as color, reflectivity, or material, and a scale indicating the size of the voxel.

[0270] Furthermore, the position may be expressed not in three-dimensional coordinates, but in two-dimensional point coordinates (x, y). In this case, the scale may indicate the size or shape of the ellipsoid on a two-dimensional plane. Specifically, the Gaussian data may include a position indicating the location on a two-dimensional plane, a scale vector indicating the scale (Su, Sv) along each axis, and rotation information indicating the orientation of the ellipsoid.

[0271] Note that the format for describing Gaussian data is not limited to the PLY format. For example, the SPZ format or other formats can also be used to describe Gaussian data.

[0272] [Rendering Gaussian Data] Next, we will explain the rendering process for Gaussian data. By rendering Gaussian data, a 3D model or a 2D image is generated. The rendered 2D image can be displayed in applications such as viewers. In addition, rendering is sometimes used to evaluate Gaussian data when generating it. Rendering is the process of converting an entire 3D scene into a 2D image, and rendering three-dimensional data is called volume rendering.

[0273] Rendering techniques include, for example, ray tracing and rasterizing. Hereafter, rendering may also be referred to as rasterizing or rasterization.

[0274] Figure 48 shows the rendering process of Gaussian data. As shown in Figure 48, the rendering unit 303 generates a 3D model by rendering the Gaussian data. This makes it possible to display the generated 3D model in applications such as 3D display devices. A 3D display device is, for example, a device that displays a three-dimensional image in space, such as a hologram, or a glasses-free 3D display device.

[0275] Figure 49 shows an example of the number of SH coefficients in a spherical harmonic function. The number of elements (number of SH coefficients) in a spherical harmonic function is determined according to the level of resolution it represents. For example, a higher level (a larger numerical value of the level) indicates higher resolution. For example, as the level increases, more information about high-frequency components is added.

[0276] For example, to represent level 2, the spherical harmonics have a total of nine elements (SH coefficients) from levels 0 to 2. For example, to represent level 3, the spherical harmonics have a total of sixteen elements (SH coefficients) from levels 0 to 3.

[0277] For example, the example shown in Figure 47 illustrates the spherical harmonics of a level 3 color, where the three elements of color (R, G, B) are each represented by 16 SH coefficients. Therefore, the spherical harmonics have 16 × 3 = 48 elements. It is also possible to define levels 4 and above.

[0278] By representing color using spherical harmonics, it is possible to represent three-dimensional color information. Figure 50 shows an example of input and output of spherical harmonics. For example, as shown in Figure 50, by inputting viewpoint information into the spherical harmonics 304 of the color, the color information (R, G, B) as seen from that viewpoint is output.

[0279] Figure 51 shows the rendering process. As shown in Figure 51, the rendering unit 305 generates a 3D model or 2D image based on Gaussian data and a specified viewpoint, viewed from that viewpoint. The generated 3D model can be displayed in applications such as VR display. The generated 2D image can also be displayed in applications such as 2D image display devices.

[0280] Furthermore, other attribute information (such as reflectance or infrared information) may be represented using spherical harmonics, not just color. In that case, the Gaussian data will include the SH coefficients of the spherical harmonics for each attribute information. Also, the level may be changed depending on the attribute information or resolution.

[0281] [Gaussian Data Encoding and Decoding] Next, the encoding and decoding processes for Gaussian data will be described. Figure 52 is a block diagram of the encoding device (encoding system) according to this embodiment. This encoding device encodes and multiplexes Gaussian data. Figure 53 is a block diagram of the decoding device (decoding system) according to this embodiment. This decoding device decodes the Gaussian data and presents a 3D model or 2D image in the application. Note that the encoding and decoding devices shown in Figures 52 and 53 are included, for example, in the three-dimensional data generation system shown in Figure 1.

[0282] The encoding device shown in Figure 52 includes a Gaussian data generation unit 310, an encoding unit 311, and a multiplexing unit 312. The Gaussian data generation unit 310 generates Gaussian data. The encoding unit 311 generates encoded data (bitstream) by encoding the input Gaussian data using a predetermined encoding method, and outputs the generated encoded data to the multiplexing unit 312.

[0283] The multiplexing unit 312 generates multiplexed data by multiplexing the input encoded data using a predetermined multiplexing method, and outputs the generated multiplexed data. This multiplexed data is stored or transmitted.

[0284] The decoding device shown in Figure 53 includes a demultiplexing unit 321, a decoding unit 322, and an application unit 323. The demultiplexing unit 321 generates encoded data by demultiplexing the input multiplexed data using a predetermined multiplexing method, and outputs the generated encoded data to the decoding unit 322.

[0285] The decoding unit 322 generates Gaussian data by decoding the input encoded data using a predetermined encoding method (decoding method), and outputs the generated Gaussian data to the application unit 323.

[0286] The application unit 323 includes an input interface unit 324, a rendering unit 325, and a presentation unit 326. The input interface unit 324 acquires user operations. For example, based on user operations, the input interface unit 324 generates viewpoint information indicating the viewpoint entered by the user.

[0287] The rendering unit 325 generates a three-dimensional model or a two-dimensional image by rendering the input Gaussian data. For example, the rendering unit 325 generates a 3D model or a 2D image viewed from a viewpoint entered by the user, indicated by the viewpoint information. The rendering unit 325 may also generate a 3D model or a 2D image viewed from a predetermined viewpoint.

[0288] The presentation unit 326 presents (displays) the three-dimensional model or two-dimensional image generated by the rendering unit 325. Whether to present a 3D model or a 2D image may be determined by, for example, the following method. For example, the rendering unit 325 first generates a three-dimensional model and generates point coordinate information, as shown in Figure 48. After viewpoint information is input, the rendering unit 325 generates a 2D image viewed from a specified viewpoint. At that time, the rendering unit 325 may also accept information specifying a region along with the specified viewpoint. In this way, by generating a 3D model first and then generating a 2D image of the specified region, the amount of processing can be reduced compared to generating a 2D image of the entire region. Therefore, the rendering speed can be increased. Alternatively, whether to present a 3D display (3D model) or a 2D image may be determined according to the processing power of the device performing the rendering. For example, in the case of a device with low processing power, the device may not generate a 2D image and may present a 3D model.

[0289] (First Embodiment) [Gaussian Data Encoding Using G-PCC] Figure 54 is a block diagram showing an example configuration of an encoding device 330 (three-dimensional data encoding device) according to this embodiment. The encoding device 330 generates a bitstream (encoded data) by encoding the first Gaussian data.

[0290] The encoding device 330 includes a first preprocessing unit 331, a second preprocessing unit 332, and a G-PCC encoding unit 333.

[0291] The first Gaussian data includes, for example, a covariance matrix. The first preprocessing unit 331 converts the first Gaussian data into second Gaussian data composed of components of the ply format. For example, the components of the ply file include three-dimensional coordinates, scale, rotation, SH coefficients of spherical harmonics, and transmittance, as shown in Figure 47.

[0292] In this example, first Gaussian data is input to the encoding device 330, and the encoding device 330 converts the first Gaussian data into second Gaussian data. However, second Gaussian data may also be input to the encoding device 330. In this case, the encoding device 330 does not need to include the first preprocessing unit 331.

[0293] The second preprocessing unit 332 converts the multiple components included in the second Gaussian data into a format that can be encoded using the encoding scheme corresponding to each component, and outputs the converted multiple components to the G-PCC encoding unit 333.

[0294] The G-PCC coding unit 333 encodes the three-dimensional coordinates using a geometry (position information) coding method (such as an octave coding method, a predictive tree coding method, or a TriSoup method) within the G-PCC coding method (Geometry-based PCC). The G-PCC coding unit 333 encodes information associated with the three-dimensional coordinates, such as rotation, scale, SH coefficient, and transmittance, using an attribute coding method (such as an LoD-based coding method (LoD attribute information coding method) or a Transform-based coding method (transformation attribute information coding method)) within the G-PCC coding method.

[0295] The G-PCC coding unit 333 includes a position information coding unit 341, a rotation coding unit 342, a scale coding unit 343, an SH coefficient coding unit 344, a transmittance coding unit 345, a metadata coding unit 346, and a multiplexing unit 347.

[0296] The position information encoding unit 341 encodes three-dimensional coordinates. These three-dimensional coordinates are reference or representative three-dimensional coordinates for the Gaussian data. For example, the three-dimensional coordinates may be the center coordinates of an ellipse that constitutes the Gaussian data, or they may be other coordinates. For example, the three-dimensional coordinates may be the origin coordinates of the grid used to generate the Gaussian data.

[0297] Furthermore, the three-dimensional coordinates are not processed by the first preprocessing unit 331, but are directly input to the second preprocessing unit 332. The second preprocessing unit 332 converts the input three-dimensional coordinates into positive integer information (three-dimensional coordinates). The scale value and offset value used during the conversion are stored in metadata such as SPS and notified to the decoding device.

[0298] The position information encoding unit 341 generates encoded data by encoding the three-dimensional coordinates output from the second preprocessing unit 332 using the G-PCC geometry encoding method. This encoded data is stored in a GDU (Geometry Data Unit) and output. The metadata necessary for decoding the GDU is stored in a GPS (Geometry Parameter Set) and output.

[0299] Furthermore, if the multiple three-dimensional coordinates of the multiple Gaussian data are sparse (density is below a predetermined threshold), the location information coding unit 341 may use a predictive tree coding scheme suitable for coding sparse three-dimensional points. Also, if the multiple three-dimensional coordinates of the multiple Gaussian data are dense (density is above a predetermined threshold), the location information coding unit 341 may use an octave tree coding scheme suitable for coding dense three-dimensional points.

[0300] The first preprocessor 331 converts the covariance matrix contained in the first Gaussian data into rotation values ​​and scale values. The first preprocessor 331 converts the color information contained in the first Gaussian data into spherical harmonics and outputs the SH coefficients in the spherical harmonics.

[0301] The second preprocessor 332 converts the data type of each data contained in the second Gaussian data. Specifically, the second preprocessor 332 converts each data into a positive integer type in order to encode it using the G-PCC coding scheme. For example, if the data is of type float, the second preprocessor 332 converts the data into a positive integer type by performing scaling and offset processing on the data. At this time, the scale value and offset value for each attribute information used in the conversion are stored in metadata such as SPS and notified to the decoding device.

[0302] Furthermore, the second preprocessing unit 332 performs format conversion of each data and mapping processing to map each data to attribute components. For example, if the maximum number of elements (dimensions) per attribute component that can be encoded with the G-PCC attribute coding scheme is 3, the second preprocessing unit 332 converts the input components into attribute components that have three dimensions (three elements) per attribute component. For example, since Scale has three elements, it is output as a single attribute component. Orientation has four elements, so it is converted into two two-dimensional attribute components. The number of elements in the SH coefficient changes for each level. For example, SH coefficients with a level exceeding 3 are converted into multiple attribute components.

[0303] In this process, the second preprocessing unit 332 generates mapping information as metadata for each attribute piece of information, indicating which attribute component each element is mapped to.

[0304] The rotation coding unit 342, scale coding unit 343, SH coefficient coding unit 344, and transmittance coding unit 345 each generate multiple coded data by coding rotation, scale, SH coefficient, and transmittance using the attribute coding method in the G-PCC coding scheme. Each of the generated coded data is stored in an ADU (Attribute Data Unit) and output. The metadata necessary for decoding the ADU is stored in an APS (Attribute Parameter Set) and output. In addition, attribute information is coded using three-dimensional coordinates.

[0305] The metadata encoding unit 346 stores conversion information, including conversion parameters used by the first preprocessing unit 331 or the second preprocessing unit 332, in metadata such as SPS or SEI. The metadata encoding unit 346 may also encode the conversion information before storing it in the metadata. The metadata may also include information on the structure of the Gaussian data, or information indicating how the Gaussian data is transmitted using G-PCC encoding.

[0306] The multiplexing unit 347 stores the encoded data (ADU and GDU) and parameter sets (APS, GPS, and SPS) in the TLV unit and transmits them as a bitstream (encoded data). The bitstream may be multiplexed using a predetermined multiplexing scheme. The bitstream may also be formatted.

[0307] [Decoding Gaussian Data Using G-PCC] Figure 55 is a block diagram showing an example configuration of a decoding device 350 (three-dimensional data decoding device) according to this embodiment. The decoding device 350 generates first Gaussian data or second Gaussian data by decoding a bitstream (encoded data). For example, the decoding device 350 decodes a bitstream generated by the encoding device 330 shown in Figure 54.

[0308] The decoding device 350 includes a G-PCC decoding unit 351, a second post-processing unit 352, and a first post-processing unit 353.

[0309] The G-PCC decoding unit 351 generates three-dimensional coordinates, rotation, scale, SH coefficient, and transmittance by decoding the bitstream using the G-PCC coding scheme. The G-PCC decoding unit 351 comprises a demultiplexing unit 361, a position information decoding unit 362, a rotation decoding unit 363, a scale decoding unit 364, an SH coefficient decoding unit 365, a transmittance decoding unit 366, and a metadata decoding unit 367.

[0310] The demultiplexing unit 361 analyzes multiple TLV units contained in the input bitstream and generates data units of encoded data such as GDU, ADU, SPS, GPS, and APS. The position information decoding unit 362, rotation decoding unit 363, scale decoding unit 364, SH coefficient decoding unit 365, transmittance decoding unit 366, and metadata decoding unit 367 decode these data units using the information in the parameter set, with each attribute component using its respective encoding scheme.

[0311] The position information decoding unit 362 decodes three-dimensional coordinates from the GDU. The rotation decoding unit 363, scale decoding unit 364, SH coefficient decoding unit 365, and transmittance decoding unit 366 decode attribute information (rotation, scale, SH coefficient, and transmittance) from the ADU. The attribute information is decoded using three-dimensional coordinates. The metadata decoding unit 367 obtains conversion information, etc., from metadata such as SPS or SEI. The metadata decoding unit 367 may also decode the conversion information from the metadata. The metadata may also include information on the composition of the Gaussian data, or information indicating the correspondence of how the Gaussian data is transmitted using G-PCC coding.

[0312] The second post-processing unit 352 generates second Gaussian data by performing an inverse transformation on the decoded data. Here, the inverse transformation is the reverse process of the transformation process performed in the second pre-processing unit 332, and is performed based on the transformation information (transformation parameters) contained in the metadata.

[0313] The first post-processing unit 353 generates the first Gaussian data by inversely transforming the second Gaussian data. Here, the inverse transformation is the reverse process of the transformation process performed by the first pre-processing unit 331. At least one of the first Gaussian data and the second Gaussian data is output. Note that the decoding device 350 does not necessarily have to include the first post-processing unit 353.

[0314] (Embodiment 1) Figure 56 is a block diagram of a data processing system according to this embodiment. The data processing system shown in Figure 56 includes a GS generation unit 501 (Gaussian data generation unit), a GS encoding unit 502 (Gaussian data encoding unit), a GS decoding unit 503 (Gaussian data decoding unit), and an application 504. For example, the GS generation unit 501 and the GS encoding unit 502 are included in a data generation device (e.g., a server) that generates and transmits Gaussian data. The GS decoding unit 503 and the application 504 are included in a data processing device (e.g., a terminal) that receives and decodes Gaussian data and performs processing (rendering or measurement) using the decoded Gaussian data.

[0315] The GS generation unit 501 generates 3D GS (3D Gaussian Splating) (GS data) from multiple images through learning. The GS encoding unit 502 (GS compression unit) generates encoded GS data by encoding (compressing) the GS data using a predetermined encoding method. The GS encoding unit 502 transmits the generated encoded GS data to the GS decoding unit 503.

[0316] The GS decoding unit 503 generates decoded GS data by decoding the encoded GS data using a predetermined decoding method. The application 504 includes a rendering unit 505 and a measurement unit 506. The rendering unit 505 generates a two-dimensional image by rendering the decoded GS data and displays the two-dimensional image. The measurement unit 506 performs measurements using the position information contained in the decoded GS data. For example, the measurement is a distance measurement that determines the distance between two points in three-dimensional space. The distance determined by the distance measurement is not limited to the distance between points, but may also be the distance between combinations of points, lines, and planes. Furthermore, the measurement is not limited to distance measurement, but may be any process that uses position information in three-dimensional space.

[0317] In the generation or encoding (compression) of GS data, the quality of the GS data during the generation process is measured. Based on this quality of GS data, the generation or compression of the GS data is performed in a way that ensures the quality of the GS data is high or not low.

[0318] The quality of GS data can be categorized into, for example, 3D geometry quality, 3D attribute quality, or 2D quality. 3D geometry quality is an indicator that evaluates how well the positional information of the GS data matches the positional information of the original 3D data. 3D attribute quality is an indicator that evaluates how well the attribute information (such as color) of the GS data matches the attribute information of the original 3D data. 2D quality (2D metrics) is an indicator that evaluates how well the multiple 2D images generated by rendering the GS data match the attribute information of the original multiple images. For example, the source images used for training are called Ground Truth. Note that 3D geometry quality and / or 3D attribute quality are also sometimes referred to as 3D quality (3D metrics).

[0319] The evaluation methods include, for example, SSIM (Structural Similarity Index Measure), PSNR (Peak Signal-to-Noise Ratio), MSE (Mean Squared Error), VIF (Visual Information Fidelity), or NIQE (Natural Image Quality Evaluator).

[0320] Furthermore, GS data includes, as its components, three-dimensional coordinates (Position) indicating the position of the ellipse, and scale and rotation (Orientation) indicating the shape of the ellipse. GS data also includes SH coefficients (SH Coefficient) and transparency (Opacity) indicating color information. For example, the position information of the GS data includes three-dimensional coordinates, and the attribute information of the GS data includes scale, rotation, SH coefficient, and transparency. Note that the inclusion of each of the above pieces of information in either the position information or the attribute information is not limited to the above example. For example, in addition to three-dimensional coordinates, scale and rotation may be included in the position information. Or, at least one of scale, rotation, SH coefficient, and transparency may be included in the position information.

[0321] In the generation of GS data, if the GS data is rendered during the processing and generated based on 2D quality, the generated GS data will be optimized for 2D image quality. Therefore, the 3D quality may be poor, and the 3D geometry quality and 3D attribute quality may be compromised. In that case, while high-quality 2D images can be obtained in applications where 2D quality is important, such as rendering, high-quality measurement results may not be obtained in applications where 3D position coordinates are important, such as distance measurement. In other words, whether the decoded GS data is suitable for use in an application depends on the method used in the learning or compression process.

[0322] Therefore, in this embodiment, the data generation device generates information indicating the quality of the GS data during learning or compression. The following describes how to notify this information and the data format of this information. Furthermore, use cases for switching data in an application are described. Finally, methods for hierarchizing GS data of different qualities, hierarchical coding methods, and hierarchical decoding methods are described.

[0323] Figure 57 is a block diagram of the GS generation unit 501. The GS generation unit 501 includes a learning unit 511 and a quality measurement unit 512.

[0324] The learning unit 511 generates GS data (3DGS) using the learned data. The quality measurement unit 512 measures the quality of the generated GS data. The learning unit 511 also updates the GS data based on this quality. This process is repeated to generate final GS data with improved quality, and this GS data is output. Here, this iterative process using quality is called learning. Note that the above process does not necessarily have to be repeated in learning. For example, the quality of multiple GS data generated by multiple methods may be measured, and the GS data to be output from the multiple GS data may be selected based on the measured quality. For example, the GS data with the highest quality among the multiple GS data may be selected.

[0325] The GS generation unit 501 generates quality information indicating the quality derived during the learning process. For example, the quality information indicates the quality of the final output GS data. Alternatively, the GS generation unit 501 generates learning method information indicating whether the method used during learning was a learning method that emphasized 2D quality or a learning method that emphasized 3D quality.

[0326] The GS generation unit 501 outputs both the generated quality information or learning method information and the GS data. The GS generation unit 501 may generate and output either the quality information or the learning method information, or it may generate and output both.

[0327] Here, a learning method that prioritizes 2D quality includes, for example, the following processes. In the process of generating GS data, the GS generation unit 501 adds, deletes, combines, separates, or changes the color of the GS data in order to improve the quality of the initially created GS data. At that time, the GS generation unit 501 generates a 2D image by rendering the GS data and determines how to modify the GS data based on the quality of the 2D image. The GS generation unit 501 also outputs learning method information indicating a learning method that prioritizes 2D quality along with the GS data.

[0328] Furthermore, a learning method that prioritizes 3D quality includes, for example, the following processes: The GS generation unit 501 measures the 3D geometry quality and attribute quality based on the positional information or attribute information of the GS data without rendering the GS data. Based on the geometry quality and attribute quality, the GS generation unit 501 determines how to modify the GS data. The GS generation unit 501 also outputs learning method information indicating a learning method that prioritizes 3D quality along with the GS data.

[0329] Furthermore, both 3D geometry quality and 3D attribute quality may be used, or the method for modifying GS data may be determined based on either 3D geometry quality or 3D attribute quality. For example, if location information is important for distance measurement, the method for modifying GS data may be determined based on 3D geometry quality.

[0330] Furthermore, a learning method that prioritizes 2D quality means that the generated GS data is of a quality suitable for rendering. Conversely, a learning method that prioritizes 3D quality means that the generated GS data is of a quality suitable for applications that prioritize location information, such as distance measurement.

[0331] Next, an example of training method information, which indicates the training method for GS data, will be explained. Figure 58 shows an example of training method information. Training method information is stored in metadata such as parameter sets, control information, or SEI. Training method information (GS_training_information) includes training method (training_metrics_method) and quality type (training_metrics_type). training_metrics_method indicates whether 3D quality (3D metrics) or 2D quality (2D metrics) was used in training. Note that training_metrics_method may also indicate whether a training method emphasizing 2D quality or a training method emphasizing 3D quality was used. The training_metrics_method may indicate whether 3D geometry quality was used or 3D attribute quality was used, or whether a training method that emphasizes 3D geometry quality was used or whether a training method that emphasizes 3D attribute quality was used. Furthermore, the training_metrics_method may indicate whether 3D geometry quality was used, 3D attribute quality was used, or 2D quality was used. Alternatively, the training_metrics_method may indicate whether a training method that emphasizes 3D geometry quality was used, whether a training method that emphasizes 3D attribute quality was used, or whether a training method that emphasizes 2D quality was used. Additionally, the training_metrics_method may indicate whether the GS data is of rendering quality or quality suitable for applications that emphasize location information, such as distance measurement.

[0332] The quality type (training_metrics_type) indicates the quality evaluation method. For example, the evaluation method may be SSIM, PSNR, MSE, VIF, or NIQE.

[0333] Furthermore, a flag (training_metrics_info_flag) may be provided to indicate the presence or absence of this data (training method and quality type).

[0334] Furthermore, the learning method information indicating the learning method for GS data is not limited to the examples shown here. For example, the two syntaxes may be combined into one, or only a part of the above syntax or information may be used. Also, quality other than 2D quality and 3D quality may be used.

[0335] Next, an example of quality information will be explained. Figure 59 shows an example of quality information. Quality information is stored in metadata such as parameter sets, control information, or SEI. Quality information (GS_quality_information) includes GS_3D_geom_metrics, GS_3D_attr_metrics, and GS_2D_metrics.

[0336] GS_3D_geom_metrics indicates the 3D geometry quality obtained during the learning process. GS_3D_attr_metrics indicates the 3D attribute quality obtained during the learning process. GS_2D_metrics indicates the 2D quality obtained during the learning process. numView indicates the number of rendering viewpoints. For each viewpoint where rendering is performed, viewpoint information (view_information) and GS_2D_metrics are stored. The viewpoint information is, for example, camera parameters and includes at least one of external and internal parameters.

[0337] Furthermore, the quality information may include at least one of GS_3D_geom_metrics, GS_3D_attr_metrics, and GS_2D_metrics.

[0338] Furthermore, a flag (3D_metrics_flag) may be provided to indicate the presence or absence of information indicating 3D quality (GS_3D_geom_metrics and GS_3D_attr_metrics). Additionally, a flag (2D_metrics_flag) may be provided to indicate the presence or absence of information indicating 2D quality (GS_2D_metrics).

[0339] Furthermore, the quality information may include at least one of training_metrics_method and training_metrics_type. Also, the quality information may be combined with the training method information described above.

[0340] Furthermore, at least one learning method information (GS_training_information) is provided for each GS data. For example, if the GS data consists of multiple frames and the learning method information is the same for all frames, a single learning method information common to all frames is assigned to the GS data. On the other hand, if the learning method of the GS data changes from frame to frame, learning method information is assigned to each frame. It is also possible to provide learning method information only for the frame at which the learning method information changes.

[0341] Furthermore, at least one quality information field (GS_quality_information) is assigned to each GS data. For example, if the GS data consists of multiple frames and the quality information is the same across all frames, a single quality information field common to all frames is assigned to the GS data. On the other hand, if the quality information of the GS data changes from frame to frame, quality information is assigned to each frame. It is also possible to assign quality information only to the frame at which the quality information changes.

[0342] For example, among the GS data learning methods, those that emphasize 3D quality or improve geometric accuracy include FeatureGS, GausSurf, Dynamic 3D Gaussians, MeshSplats, 3D Gaussian Splatting for Real-Time Radiance Field Rendering, SG-Splatting, 4D Gaussian Splatting for Real-Time Dynamic Scene Rendering, and GSEDiTPro.

[0343] FeatureGS is a learning method that optimizes the alignment of Gaussian centers and surfaces in 3D space to improve geometric accuracy. GausSurf is a technique that provides geometric guidance to achieve high-quality surface reconstruction. Dynamic 3D Gaussians is a technique that focuses on tracking and reconstructing dynamic scenes and emphasizes the accuracy of positional information in 3D space. MeshSplats is a technique that converts Gaussian elements into mesh faces and achieves advanced lighting effects using ray tracing. 3D Gaussian Splatting for Real-Time Radiance Field Rendering is a technique that emphasizes the accuracy of positional information in 3D space to achieve real-time viewpoint synthesis. SG-Splatting is a technique that improves computational efficiency using spherical Gaussian and enables real-time processing. 4D Gaussian Platting for Real-Time Dynamic Scene Rendering is a technology that emphasizes the accuracy of positional information in 3D space in order to achieve dynamic scene rendering. GSE Edit Pro is a technology that emphasizes accurate positioning and control during editing.

[0344] Furthermore, metadata may also contain information indicating not only the learning method but also techniques such as GS data editing methods (adding, deleting, merging, and splitting Gaussian data).

[0345] Furthermore, even the learning methods exemplified above may be classified as learning methods that prioritize 2D quality depending on the quality. Also, the metrics or quality used in the above methods may be included in the quality type (training_metrics_type).

[0346] Next, as an example of using the above technology, we will describe an example of using the above information in an application. Figure 60 is a flowchart showing an example of processing using GS data.

[0347] First, the GS generation unit 501 generates learning method information indicating a method for learning GS data, and quality information indicating the 2D quality or 3D quality of the GS data (S211). The generated information is sent to the application 504 along with the GS data.

[0348] Application 504 selects a distance measurement mode using GS data (S212). Application 504 determines whether or not the GS data can be used for distance measurement based on at least one of the learning method information and the quality information (S213). For example, Application 504 determines that the GS data can be used for distance measurement if the 3D quality is above a predetermined threshold, and determines that the GS data cannot be used for distance measurement if the 3D quality is below the threshold. For example, Application 504 determines the 3D quality based on the learning method information and makes the above determination based on the determined 3D quality. Alternatively, Application 504 makes the above determination based on the 3D quality indicated in the quality information.

[0349] If application 504 determines that the GS data can be used for distance measurement (Yes in S214), application 504 performs distance measurement using the GS data (S215). On the other hand, if application 504 determines that the GS data cannot be used for distance measurement (No in S214), application 504 displays a message indicating that distance measurement using the GS data is not permitted (S216).

[0350] (Embodiment 2) Embodiment 1 described an example using information related to learning GS data. Embodiment 2 describes an example using information related to encoding GS data.

[0351] Various encoding tools can be used for encoding GS data. For example, if the encoding tool is lossless compression, the encoded and decoded data will be lossless and undegraded compared to the input data. Therefore, the 2D and 3D quality of the input data and the decoded data will be identical.

[0352] On the other hand, if the encoding tool is lossy, it is possible to adjust the quality and bitstream size. For example, it is possible to reduce the compression size while sacrificing a certain degree of quality. When using such lossy tools, the degree of degradation of the 2D and 3D quality of the decoded GS data relative to the input data will differ depending on whether 2D quality or 3D quality is prioritized during encoding. For example, there are encoding methods (tools) that prioritize 2D quality and encoding methods (tools) that prioritize 3D quality, and quality can be controlled by using different tools.

[0353] The following describes examples of encoding methods (tools) that prioritize 2D quality and encoding methods (tools) that prioritize 3D quality. GS data is a collection of ellipses, and its components include three-dimensional coordinates indicating the position of the ellipses, and scale and orientation indicating the shape of the ellipses. GS data also includes SH coefficients and opacity, which indicate color information.

[0354] Lossy compression uses tools that, for example, involve adding and deleting ellipses, as well as merging and separating ellipses. Alternatively, the position, scale, and orientation values ​​that indicate the position and shape of ellipses are modified. Furthermore, if quantization is applied, the position and shape of the ellipses will change. In these cases, the 3D geometry quality of 3DGS may degrade.

[0355] Furthermore, changes in the position and shape of the ellipse alter the influence of its color and transparency. In this case, the 3D attribute quality of the 3DGS may deteriorate. If the 3D geometry quality and 3D attribute quality deteriorate in this way, the quality of the rendered image (2D quality) may also deteriorate.

[0356] The following describes encoding methods (tools) that prioritize 3D geometry quality. For example, if the color and transparency values ​​of an ellipse are changed or quantization is applied without altering its position and shape, the 3D geometry quality of the GS data will not deteriorate, but the 3D attribute quality and 2D quality may deteriorate. Tools that perform such processing can be said to prioritize 3D geometry quality.

[0357] For example, not changing the position and shape of an ellipse is called lossless coding of geometry. Near-lossless coding, which is also lossy coding but does not use tools that distort the three-dimensional shape, is another coding method that prioritizes 3D geometry quality.

[0358] Next, we will explain encoding methods (tools) that prioritize 2D quality. For example, when (1) adding or deleting ellipses that do not contribute to rendering, (2) changing the position and shape of ellipses, or (3) shifting the position of ellipses in the rendering line of sight, the 3D quality may deteriorate, while the 2D quality after rendering may not deteriorate significantly. Tools that perform such processing can be said to be encoding tools that prioritize 2D quality.

[0359] Furthermore, even with the same tool, the emphasis on 3D geometry quality, 3D attribute quality, and 2D quality can be switched using parameters or combinations of parameters. In this way, it is possible to switch between encoding methods that emphasize 2D quality and encoding methods that emphasize 3D quality using tools, parameters, or combinations thereof.

[0360] Figure 61 is a block diagram of the GS encoding unit 502 according to this embodiment. The GS encoding unit 502 shown in Figure 61 comprises an encoding unit 521 and a quality measurement unit 522. The encoding unit 521 generates encoded GS data by encoding GS data. The encoding unit 521 also generates encoding method information regarding the encoding method used for encoding.

[0361] The quality measurement unit 522 measures the quality of the generated encoded GS data and generates quality information indicating that quality. For example, the quality measurement unit 522 generates decoded GS data by decoding the encoded GS data and calculates the quality based on the difference between the original GS data and the decoded GS data. For example, the smaller the difference, the higher the quality is determined to be. The encoding unit 521 changes the encoding parameters or method to improve the quality and updates the encoded GS data. This process is repeated to generate final encoded GS data with improved quality, and this encoded GS data is output.

[0362] It should be noted that the above process does not necessarily need to be repeated during encoding. For example, the quality of each of the multiple encoded GS data generated by multiple methods may be measured, and based on the measured quality of the multiple encoded GS data, the encoded GS data to be output from the multiple encoded GS data may be selected. For example, the GS data with the highest quality among the multiple encoded GS data may be selected. Furthermore, the GS encoding unit 502 may generate and output either quality information or encoding method information, or it may generate and output both.

[0363] Next, we will explain the quality information (profile) that indicates the quality in the encoding of GS data. When the GS encoding unit 502 uses an encoding method (tool) that prioritizes 2D quality, it stores encoding method information indicating the encoding method that prioritizes 2D quality in the metadata. Also, when the GS encoding unit 502 uses an encoding method (tool) that prioritizes 3D quality, it stores encoding method information indicating the encoding method that prioritizes 3D quality in the metadata.

[0364] The GS encoding unit 502 encodes the metadata and GS data respectively, and outputs the obtained encoded metadata and encoded GS data as a bitstream.

[0365] Furthermore, when an encoding method prioritizing 2D quality is used, the encoded GS data is of a quality suitable for rendering. Similarly, when an encoding method prioritizing 3D quality is used, the encoded GS data is of a quality suitable for applications such as distance measurement.

[0366] Figure 62 shows an example of encoding method information indicating the encoding method of GS data. Encoding method information is stored in metadata such as parameter sets, control information, or SEI. Encoding method information (GS_encoding_information) includes the encoding method (encoding_tool). encoding_tool indicates whether a tool prioritizing 2D quality or a tool prioritizing 3D quality was used in the encoding.

[0367] Furthermore, a flag (encoding_info_flag) may be provided to indicate the presence or absence of this data (encoding method).

[0368] Furthermore, profiles and / or levels may be defined that contain a set of tools suitable for a specific quality or application, and profile information indicating the profile and / or level may be stored in metadata instead of encoding_tool. The profiles may include, for example, profiles for 2D applications and profiles for 3D applications. Alternatively, the profiles may include a 2D-focused profile and a 3D quality-focused profile, or a rendering-friendly quality profile and a location-focused quality profile such as distance measurement. The profiles may also include lossless compression profiles and lossy compression profiles, or lossless compression profiles, semi-lossy compression profiles, and lossy compression profiles.

[0369] For each profile based on the type of GS data, a tool suitable for that type of GS data may be defined. Types of GS data include, for example, (1) GS data based on images taken with one camera, (2) GS data based on images taken with two or more cameras, (3) object-centric GS data taken around a person or object, and (4) GS data used for scene movement.

[0370] Figure 63 shows an example of encoding method information when using profiles. For example, the SequenceParameterSet, which is metadata for each sequence (multiple frames), stores information indicating the profile (profile) and information indicating the level (level). Note that both the profile and the level may be used, or only one of them may be used.

[0371] Figure 64 shows an example of a profile. Here, we explain an example of adding a new profile to the Gaussian Splatting Coding profile, which is a G-PCC profile. However, a new profile may be added to a profile other than G-PCC, or the above profile may be shown separately from the G-PCC profile.

[0372] The G-PCC sequence parameter set is signaled with a profile indicating a set of tools, and a level to constrain the parameters by the required processing power. The G-PCC profiles include simple, predictive, main, and dense profiles (0-3). Additional profiles for Gaussian Platting coding using G-PCC are the Gaussian Platting for 3D applications (4) and the Gaussian Platting for 2D applications (5).

[0373] Furthermore, each profile specifies whether or not it supports each tool. Figure 65 shows an example of the tools supported by a profile. For example, if tools A and B are tools that irreversibly compress geometry, it is not suitable for 3D applications. Therefore, a Gaussian coding profile for 3D applications specifies that tools A and B are not used.

[0374] Next, we will explain an example of quality information in GS data encoding. Figure 66 shows an example of quality information (GS_quality_information) indicating the 2D quality and 3D quality of GS data. The quality information shows the metrics information obtained during the encoding process. The quality information is stored in metadata such as parameter sets, control information, or SEI.

[0375] Quality information includes 3D_metrics_flag, 3D_metrics_type_present_flag, 3D_metrics_type, GS_3D_metrics, 2D_metrics_flag, 2D_rendering_type_present_flag, 2D_rendering_type, 2D_metrics_type_present_flag, 2D_metrics_type, numView, view_information, and GS_2D_metrics.

[0376] The 3D_metrics_flag flag indicates whether or not 3D quality information (GS_3D_metrics), which indicates the 3D quality of the GS data, is included in the quality information. For example, if 3D_metrics_flag is 1, 3D quality information is included in the quality information. If 3D_metrics_flag is 0, 3D quality information is not included in the quality information.

[0377] 3D_metrics_type_present_flag is a flag that indicates whether or not information indicating the type of 3D quality information (3D_metrics_type) is included in the quality information. For example, if 3D_metrics_type_present_flag is 1, the quality information includes information indicating the type of 3D quality information. If 3D_metrics_type_present_flag is 0, the quality information does not include information indicating the type of 3D quality information.

[0378] 3D_metrics_type indicates the type of 3D quality information. For example, a value of 0 indicates D1 metrics, a value of 1 indicates D2 metrics, a value of 2 indicates Euclidean distance, and a value of 3 indicates Manhattan distance.

[0379] For example, the presence or absence of 3D_metrics_type_present_flag and 3D_metrics_type can be switched depending on 3D_metrics_flag. For example, if 3D_metrics_flag is 1, the quality information will include 3D_metrics_type_present_flag and 3D_metrics_type. If 3D_metrics_flag is 0, the quality information will not include 3D_metrics_type_present_flag and 3D_metrics_type.

[0380] GS_3D_metrics indicates the 3D quality of the GS data and represents the overall quality of the 3D data. If 3D_metrics_type exists, GS_3D_metrics indicates the type of 3D quality indicated by 3D_metrics_type. If 3D_metrics_type does not exist, GS_3D_metrics indicates a predetermined type of 3D quality.

[0381] Here, D1 metrics and D2 metrics are terms commonly used in the fields of 3D data processing and computer vision, for example, as evaluation metrics for standardization bodies (such as MPEG) and point cloud coding. D1 metrics and D2 metrics are also known as Chamfer Distance and Hausdorff Distance.

[0382] Specifically, the D1 metric is an index that evaluates the point-to-point distance between the decoded point cloud and the reference point cloud. The D1 metric is used to directly evaluate the fidelity of the geometry.

[0383] D2 metrics are indicators that evaluate the distance between a decoded point and a reference point, projected in the direction of the reference point's normal. D2 metrics are effective for evaluating surface smoothness and local shape reproducibility.

[0384] The 2D_metrics_flag flag indicates whether or not the quality information includes 2D quality information (GS_2D_metrics), which represents the 2D quality of the GS data. For example, if 2D_metrics_flag is 1, the quality information includes 2D quality information. If 2D_metrics_flag is 0, the quality information does not include 2D quality information.

[0385] The flag 2D_rendering_type_present_flag indicates whether or not information indicating the 2D rendering method for 2D quality evaluation (2D_rendering_type) is included in the quality information. For example, if 2D_rendering_type_present_flag is 1, the quality information includes information indicating the 2D rendering method. If 2D_rendering_type_present_flag is 0, the quality information does not include information indicating the 2D rendering method.

[0386] 2D_rendering_type indicates a 2D rendering method for 2D quality evaluation. For example, a value of 0 indicates renderer 1, and a value of 2 indicates renderer 2. Alternatively, 2D_rendering_type may indicate the URL or version of the renderer software. Or, 2D_rendering_type may indicate the parameters used for rendering, or the command to execute the renderer. This information may also be shown as a string.

[0387] 2D_metrics_type_present_flag is a flag that indicates whether or not information indicating the type of 2D quality information (2D_metrics_type) is included in the quality information. For example, if 2D_metrics_type_present_flag is 1, the quality information includes information indicating the type of 2D quality information. If 2D_metrics_type_present_flag is 0, the quality information does not include information indicating the type of 2D quality information.

[0388] 2D_metrics_type indicates the type of 2D quality information. For example, the type includes at least one of the following: PSNR (Peak Signal-to-Noise Ratio), WS-PSNR (Weighted Spherical PSNR), IV-PSNR, SSIM (Structural Similarity Index Measure), IV-SSIM (Immersive Video SSIM), and LPIPS (Learned Perceptual Image Patch Similarity).

[0389] PSNR represents the peak signal-to-noise ratio of an image. WS-PSNR is a PSNR adapted for omnidirectional video. IV-PSNR is a PSNR for immersive video. SSIM represents the structural similarity of an image. IV-SSIM is an SSIM for immersive video. LPIPS represents the perceptual loss function in an image generation or transformation task. In addition, at least one of the software, software version, and parameters used for quality evaluation may be indicated.

[0390] For example, depending on 2D_metrics_flag, the presence or absence of 2D_rendering_type_present_flag, 2D_rendering_type, 2D_metrics_type_present_flag, and 2D_metrics_type can also be switched. For example, if 2D_metrics_flag is 1, this information is included in the quality information, and if 2D_metrics_flag is 0, this information is not included in the quality information.

[0391] `numView` indicates the number of rendering viewpoints. `view_information` is viewpoint information. `GS_2D_metrics` is 2D quality information indicating the quality of the 2D rendered image. For example, `view_information` and `GS_2D_metrics` are provided for each viewpoint.

[0392] Furthermore, if 2D_rendering_type exists, GS_2D_metrics indicates the 2D quality when the rendering method indicated by 2D_rendering_type is used. If 2D_rendering_type does not exist, GS_2D_metrics indicates the 2D quality when a predetermined rendering method is used.

[0393] Furthermore, if 2D_metrics_type exists, GS_2D_metrics indicates the 2D quality of the type specified by 2D_metrics_type. If 2D_metrics_type does not exist, GS_2D_metrics indicates the 2D quality of a predetermined type.

[0394] Next, as an example of using the above technology, we will describe an example of using the above information in encoding processing. Figure 67 is a flowchart of an example of processing using GS data.

[0395] First, the GS generation unit 501 generates learning method information indicating a method for learning GS data, and quality information indicating the 2D quality and 3D quality of the GS data (S221). The generated information is sent to the GS encoding unit 502 along with the GS data.

[0396] The GS coding unit 502 determines the 2D quality and 3D quality of the GS data based on at least one of the learning method information and the quality information (S222). Next, the GS coding unit 502 determines whether the determined 3D quality is higher than the determined 2D quality (S223).

[0397] If the determined 3D quality is higher than the determined 2D quality (Yes in S223), the GS encoding unit 502 encodes the GS data using an encoding tool that prioritizes 3D quality (S224). On the other hand, if the determined 3D quality is lower than the determined 2D quality (No in S223), the GS encoding unit 502 encodes the GS data using an encoding tool that prioritizes 2D quality (S225).

[0398] In this example, in steps S222 to S225, the GS encoding unit 502 selects an encoding tool based on whether the 3D quality or the 2D quality is higher, but it is not limited to this. For example, the GS encoding unit 502 may use an encoding tool that prioritizes 3D quality if the 3D quality is higher than a predetermined quality, and use an encoding tool that prioritizes 2D quality otherwise. Alternatively, the GS encoding unit 502 may use an encoding tool that prioritizes 2D quality if the 2D quality is higher than a predetermined quality, and use an encoding tool that prioritizes 3D quality otherwise.

[0399] Furthermore, the GS coding unit 502 may select an coding tool based on the learning method indicated in the learning method information. For example, the GS coding unit 502 may use a coding tool that prioritizes 3D quality when the learning method is a learning method that emphasizes 3D quality, and may use a coding tool that prioritizes 2D quality when the learning method is a learning method that emphasizes 2D quality.

[0400] (Embodiment 3) This embodiment describes the hierarchical arrangement of GS data. Figure 68 is a block diagram of a data processing system according to this embodiment. The data processing system shown in Figure 68 includes a GS generation unit 531, a GS encoding unit 532, a GS decoding unit 533, and applications 534 and 535. For example, the GS generation unit 531 and the GS encoding unit 532 are included in a data generation device (e.g., a server) that generates and transmits Gaussian data. The GS decoding unit 533 and applications 534 and 535 are included in a data processing device (e.g., a terminal) that receives and decodes Gaussian data and performs processing (rendering or measurement) using the decoded Gaussian data. The data processing device may include either application 534 or 535. For example, the data processing system may include a data processing device including a GS decoding unit 533 and application 534, and a data processing device including a GS decoding unit 533 and application 535.

[0401] The GS generation unit 531 generates GS data through learning. The GS data includes basic data, which is GS data of the basic hierarchy, and extended data, which is GS data of the extended hierarchy. For example, the basic data is GS data that prioritizes 2D quality after rendering, and includes multiple GS data. The extended data is extended data for improving 3D quality, and also includes multiple GS data. Furthermore, by combining the extended data with the basic data, GS data that prioritizes 3D quality is generated. In other words, the basic data is GS data that can be used on its own. The extended data cannot be used on its own, and GS data that can be used is generated by using both the basic data and the extended data.

[0402] In the following, GS data that prioritizes 3D quality will be referred to as "3D quality GS data," and GS data that prioritizes 2D quality will be referred to as "2D quality data."

[0403] The GS generation unit 531 generates basic data based on 2D quality and also generates extended data based on 3D quality.

[0404] The GS encoding unit 532 generates encoded basic data by encoding basic data and generates encoded extended data by encoding extended data. The GS encoding unit 532 stores the encoded basic data and encoded extended data in different units. Specifically, the GS encoding unit 532 stores the encoded basic data in a unit that stores encoded data of 2D quality GS data. The GS encoding unit 532 also stores the data of the encoded extended hierarchy in a unit that stores encoded data of 3D quality GS data. Each unit stores an identifier that indicates whether the data stored in the unit is basic data or extended data. This allows the GS decoding unit 533 to identify the data within the unit.

[0405] In this example, hierarchical processing is shown in the GS generation unit 531, but hierarchical processing may also be performed in the GS encoding unit 532 instead of the GS generation unit 531. For example, the GS encoding unit 532 may generate basic data and extended data from the GS data and encode each of the data.

[0406] When outputting 3D quality GS data, the GS decoding unit 533 generates basic data and extended data by decoding the encoded basic data and encoded extended data included in the encoded GS data. The GS decoding unit 533 also generates decoded GS data by merging (integrating) the basic data and extended data. This decoded GS data is 3D quality GS data.

[0407] On the other hand, when the GS decoding unit 533 outputs 2D quality GS data, it decodes only the encoded basic data included in the encoded GS data to generate basic data. The GS decoding unit 533 then outputs this basic data as decoded GS data. This decoded GS data is 2D quality GS data. In this case, the GS decoding unit 533 does not need to decode the extended data. Furthermore, whether to output 3D quality GS data or 2D quality GS data can be switched based on the data requested by, for example, application 534 or 535.

[0408] Application 534 includes a rendering unit 536. The rendering unit 536 generates a two-dimensional image by rendering decoded GS data containing only basic data, and displays the two-dimensional image. Application 535 includes a measurement unit 537. The measurement unit 537 performs measurements using position information contained in decoded GS data in which basic data and extended data have been merged.

[0409] Furthermore, hierarchical data may be generated either during GS data generation or encoding, or both. Here, we describe an example where 2D quality GS data is obtained from basic data, and 3D quality GS data is obtained from integrated data formed by combining the basic data and extended data. This means that the 3D quality of the integrated data is higher than that of the basic data. In this case, the 2D quality of the integrated data may be equivalent to that of the basic data, lower than that of the basic data, or higher than that of the basic data.

[0410] Furthermore, while a measurement application was described here as an example of an application that emphasizes 3D quality, applications that emphasize 3D quality are not limited to this. An application that emphasizes 3D quality is typically an application in which the accuracy of location information is important, and its use may be not limited to measurement but also to the reproduction, recording, or analysis of accurate location information. For example, when using 3D data in a digital archive of cultural properties, accurate shape and location information are important, and distance measurement is performed as a measurement process within the application, but the measurement results are not necessarily shown to the user as numerical values. Therefore, "measurement application" in this specification includes not only those in which measurement results are shown to the user, but also those in which measurement processing using location information is performed internally. Moreover, applications that emphasize 3D quality are not limited to applications that perform measurement processing.

[0411] Next, we will describe an example of a switching operation that switches the data to decode or output based on the data requested by the application. Figure 69 shows an example of a system that performs this switching operation. As shown in Figure 69, the system includes an application 541 and a terminal 542.

[0412] Figure 70 is a flowchart of the switching operation. First, terminal 542 notifies application 541 that it supports both first GS data suitable for rendering and second GS data suitable for measurement (S231). Application 541 requests either first GS data suitable for rendering or second GS data suitable for measurement from terminal 542 (S232).

[0413] If application 541 requests first GS data (Yes in S233), the decoder in terminal 542 decodes only the basic data from the hierarchical data and sends the decoded basic data to application 541 (S234). Application 541 processes the sent basic data (S236).

[0414] On the other hand, if application 541 requests second GS data (No in S233), the decoder in terminal 542 decodes the basic data and extended data. Terminal 542 integrates the decoded basic data and decoded extended data and sends the integrated data to application 541 (S235). Application 541 processes the data that has been sent (S236).

[0415] Next, we will explain the hierarchical structure information that shows the structure of the hierarchy. Figure 71 is a diagram showing an example of hierarchical structure information. Hierarchical structure information is stored in metadata such as parameter sets, control information, or SEI. The hierarchical structure information (GS_spatial_scalable_information) shown in Figure 71 includes quality information for each hierarchy or for each cumulative hierarchy. The hierarchical structure information includes num_layer, layer_index, and the quality information (GS_quality_information) mentioned above.

[0416] `num_layer` indicates the number of layers. `layer_index` is an index (identifier) ​​used to identify the layer. For example, as shown in Figure 71, quality information is stored for each layer.

[0417] For example, if two layers are used, a basic layer and an extended layer, num_layer=2 is set. In the first loop, the quality of the basic layer is stored as quality information, and in the second loop, the quality of the extended layer is stored as quality information. The second loop may also store the quality of the data that integrates the basic and extended layers. Also, layer_index=0 may be defined as the basic layer.

[0418] The above explanation describes an example where basic data prioritizing 2D quality is stored in the base layer and extended data that can improve 3D quality is stored in the extended layer, but this is not the only example. For instance, basic data prioritizing 3D quality may be stored in the base layer and extended data that can improve 2D quality may be stored in the extended layer.

[0419] The hierarchical structure may be set commonly for geometry and attributes, or it may be set individually. The extended hierarchy may contain both geometry data and attribute data, or only one of them. If the extended hierarchy contains only either geometry data or attribute data, the metadata of the hierarchical structure may include information indicating that "the extended hierarchy contains only either geometry data or attribute data."

[0420] Figures 72-74 illustrate examples of the structure of basic data and extended data. Figure 72 shows an example where the hierarchical structure is common to both geometry and attributes. Furthermore, the data is arranged in an order that prioritizes geometry. That is, the geometry data is arranged together, followed by the attribute data. Specifically, the data is arranged in the following order: basic geometry data (basic G), extended geometry data (extended G), basic attribute data (basic A), and extended attribute data (extended A).

[0421] Figure 73 shows an example where the hierarchical structure of geometry and attributes is common, and the data is arranged in a hierarchical order. In other words, the basic data is arranged together, and then the extended data is arranged together. Specifically, the data is arranged in the following order: basic data for geometry (basic G), basic data for attributes (basic A), extended data for geometry (extended G), and extended data for attributes (extended A).

[0422] Note that the order shown here is just one example, and other orders are also acceptable. Furthermore, information indicating which of the above multiple orders is being used may be stored in metadata.

[0423] Figure 74 shows an example where the extended hierarchy is applied only to geometry. That is, the geometry data includes basic data (basic G) and extended data (extended G), while the attribute data includes only basic data (basic A). Note that the extended hierarchy may also be applied only to attributes.

[0424] Next, we will describe use cases for switching between 3D-quality GS data and 2D-quality GS data. While the following examples focus on switching between 3D-quality and 2D-quality GS data, these can also be applied to switching between GS data and other 3D data (such as point cloud data or mesh data).

[0425] For example, the above processing may be used to display 3D data at a construction site. The terminal uses 2D quality GS data for the appearance of buildings or maps, etc. The terminal uses 3D quality GS data for areas where measurement is required, such as strain, cracks, and distortions.

[0426] For example, in a space such as a stadium, hall, or factory, if a user wants to check the appearance, such as the brightness, color, or atmosphere of the lighting, the terminal will display 2D quality GS data. Also, if a user wants to check the studio layout, equipment placement, or aisle layout, the terminal will display 3D quality GS data.

[0427] For example, when rendering and displaying a 3D model, the terminal may switch which three-dimensional model to display. For example, the terminal may select data to display from multiple point cloud data with different resolutions, or select data to display from multiple mesh data with different resolutions.

[0428] For example, when a device displays 3D data of an object far from the viewpoint, it uses low-resolution mesh data. When the viewpoint moves closer to the object, the device switches to displaying high-resolution mesh data for the 3D data. This improves the accuracy of the 3D data display.

[0429] For example, when displaying a 3D model related to sports, the device displays mesh data until the user identifies a specific movement or location when analyzing an athlete's movements. After the specific movement or location is identified, the device switches the displayed 3D data to point cloud data. For example, this point cloud data can be used to measure three-dimensional shapes.

[0430] For example, in games, a device can select which 3D model to display from multiple 3D models with different resolutions. For instance, the device might use high-resolution 3D data for important parts and low-resolution 3D data for less important parts. This can improve overall processing performance. Whether a model is important or not is determined based on factors such as its relevance to the game's operation. For example, in a shooting game, the player, target, and gun would be considered important. Alternatively, whether a model is important or not is determined based on its relevance to the player's choices. For example, in a purchasing scenario, product information would be considered important. The importance of each model may also be pre-defined for each game.

[0431] For example, when a device displays terrain data, it selects the terrain data to display from multiple terrain data sets with different resolutions. This allows the device to display more detailed terrain information as needed.

[0432] For example, when a device displays thumbnails on the web, it uses low-resolution 3D data. When the device uses the 3D data selected on the thumbnail for measurement by an application, or when displaying the selected 3D data, it uses high-resolution 3D data.

[0433] For example, in a simulation of furniture, the terminal uses a 3D model of the furniture under consideration and a 3D model of the space where the furniture is planned to be installed. When simulating the appearance, the terminal uses 2D quality GS data suitable for rendering, and when simulating the installation, it uses 3D quality GS data specialized for geometric shapes.

[0434] For example, a device uses the above processing to optimize performance in games. The device uses 2D quality GS data specialized for rendering to represent characters or environments, and 3D quality GS data specialized for geometric shapes to perform calculations related to movement, such as collision detection.

[0435] For example, the terminal uses the above processing for virtual try-on in online shopping. The terminal uses 2D quality GS data specialized for rendering to display the design or color of the clothing for the customer to check, and 3D quality GS data specialized for geometric shapes to accurately simulate the size or fit.

[0436] For example, if display speed is a priority, the terminal will display mesh data that is easier to process. When measurement is performed, the terminal will replace the 3D data used with point cloud data suitable for measurement. This makes it possible to create a lightweight application.

[0437] In this way, by notifying the terminal and the application of quality information or application usage, the terminal or application can determine whether the GS data is suitable for the application.

[0438] Furthermore, the hierarchical structure allows applications to switch between data.

[0439] In addition to information indicating 3D and 2D quality, the following information may also be transmitted: A flag indicating whether or not the data is suitable for the specific use of the application may be transmitted. For example, at least one of the following flags may be transmitted: a flag indicating whether or not the GS data can be used for collision detection, and a flag indicating whether or not the GS data can be used for distance detection.

[0440] The following describes an example of switching 3D data in a furniture placement simulation. A 3D model of the furniture under consideration and a 3D model of the planned space are used. Figure 75 is a flowchart of this 3D data switching process.

[0441] First, the terminal performs a simulation of the setup (installation), including measurements. The terminal replaces the 3D data with GS data of geometric quality specialized for the shape and determines whether the furniture can be placed in the planned space (S241). Figure 76 shows an example of this setup simulation. As shown in Figure 76, 3D data with geometric quality suitable for the shape is used for both the 3D model of the furniture under consideration and the 3D model of the planned space.

[0442] Next, if furniture can be configured, the terminal proceeds to an appearance simulation including rendering (S242). When simulating the appearance, the terminal switches the 3D data to be used to 2D quality GS data suitable for rendering and performs rendering using the 2D quality 3D data. Figure 77 shows an example of this appearance simulation. As shown in Figure 77, 3D data with 2D quality suitable for rendering is used for both the 3D model of the furniture under consideration and the 3D model of the space to be configured.

[0443] Next, we will describe examples of evaluation metrics for determining quality information. The quality information disclosed in this specification may be rephrased as information based on the results of evaluating data based on a certain evaluation metric. In this specification, we have mainly described quality information obtained by evaluating 3D data using 2D quality and 3D quality evaluation metrics as an example, but the evaluation metrics corresponding to quality information are not limited to these. Other examples of evaluation metrics are shown below.

[0444] (1) The evaluation metrics may also relate to computational efficiency or rendering performance. For example, the evaluation metrics may be rendering time, FPS (frame rate), or computational resource usage when various parameters (such as Gaussian number, sample density, and level of detail) are changed in the same scene. Computational resource usage refers to the amount of computational resources used by, for example, a GPU (Graphics Processing Unit) or CUDA (Compute Unified Device Architecture).

[0445] For example, for real-time applications, the frame rate or response time on specific hardware may be prioritized for evaluation. Since Gaussian Splatting essentially leverages parallel computing and hardware acceleration, the effect of its optimization (e.g., Gaussian culling strategies or the use of hierarchical structures) may be used as an evaluation metric.

[0446] (2) The evaluation metrics may relate to scalability, data compression, or data optimization. For example, the evaluation metric may be the change in rendering speed or memory consumption as the complexity of the scene (a scene with a large number of Gaussians or a wide-ranging scene) increases.

[0447] For example, the evaluation metric may be a result comparing the degree of quality degradation and the amount of performance improvement before and after the introduction of a data compression method or optimization method (e.g., integration of similar Gaussians, or a Level of Detail (LOD) strategy). For example, the evaluation metric is whether the compression ratio can be increased while maintaining reproducibility at different resolutions or scales, and the evaluation value that is the evaluation result may be indicated by quality information.

[0448] (3) The evaluation metrics may relate to temporal consistency or dynamic representation. For example, the evaluation metric may be whether the Gaussian parameters have consistency between frames in the generation of Gaussian data. For example, the parameters are the number of initial Gaussians, or whether there is consistency between frames in the deletion, addition, merging, or splitting of Gaussians. For example, it is determined whether there is consistency within a Group of Frames.

[0449] For example, inter-frame metrics may be quantitatively evaluated for consecutive frames accompanying camera movement or changes in objects within the scene. For example, the evaluation value may be determined according to whether variations in Gaussian parameters (position, size, attributes) are smoothly interpolated without causing flicker or abrupt jumps.

[0450] For example, quantitative evaluation using visual indicators between frames (such as temporal SSIM or dynamic error analysis) may be performed on temporal changes in a moving image or animation. For example, discontinuities specific to Gaussian Splatting that occur due to Gaussian redistribution or update processing may be detected, and the evaluation value may be determined based on the result.

[0451] (4) The evaluation indicator may relate to physical accuracy or realism. For example, the evaluation indicator may be how accurately the Gaussians reproduce behaviors such as light scattering, absorption, and reflection when checked against a physically based rendering (PBR) model for in-scene lighting, shadows, or material representation. For example, by comparing GS data with real-shot data or simulation results, it may be analyzed whether the luminance distribution, color temperature, and specular reflection behavior are faithful. For example, the consistency between the rendering result and actual optical phenomena (for example, comparison with photon mapping), or whether the Gaussian distribution causes unnatural blur is detected, and the evaluation value may be determined based on the result.

[0452] (5) The evaluation indicator may relate to usability or interactivity. For example, the quality information may be information based on the evaluation result of whether the user can easily adjust the parameters of Gaussian Splatting (such as the position, variance, color, or transparency of each Gaussian). For example, results of verifying the usability of an interface or tool through user interviews or heuristic evaluation may be used.

[0453] For example, the response during operation (presence or absence of real-time preview, or update speed after operation) may be measured, and how intuitively the rendering result can be confirmed or corrected may be evaluated. Further, the evaluation may be performed based on the richness of editors or debugging tools, the understandability of documents, or the function of visually feeding back the influence of various parameters.

[0454] (6) Evaluation metrics may also relate to flexibility or adaptability. For example, evaluation metrics may be the results of evaluating applicability to different scenes (indoor / outdoor, dynamic scenes, or complex geometry) or ease of integration with other rendering pipelines. For example, results validated in case studies or actual projects may be used.

[0455] For example, the accuracy of the conversion from data input (different sensor data, point clouds, or conventional meshes, etc.) to a Gaussian representation, or the flexibility of exporting GS data to other formats, may be evaluated, and quality information may be based on the results.

[0456] The evaluation of flexibility or adaptability may be based on the degree to which it can handle new effects (e.g., special lighting effects or post-processing filters) or dynamic adjustments. Alternatively, the evaluation may be based on factors such as plug-in functionality, modularity, or API extensibility.

[0457] (Embodiment 4) Figure 66 and others illustrate an example of SEI indicating the quality of encoding GS data, but its application is not necessarily limited to GS data. The encoding device may add quality information regarding the encoding of 3D data, such as 3D point cloud data or mesh data, to the bitstream of the 3D data as metadata such as SEI. This allows the decoding device to know the 3D quality and 2D quality of the 3D data contained in the bitstream by decoding the SEI in the bitstream. For example, the decoding device uses the quality of the 3D data to determine whether the decoded 3D data should be used as 3D data or rendered to be used as 2D data. The decoding device can also use the quality of the 3D data in use cases such as switching between 3D quality GS data and 2D quality GS data as described above.

[0458] Figure 78 shows an example of quality information indicating the 2D quality and 3D quality of 3D data. The quality information represents metric information obtained during the encoding process. The 2D quality of 3D data refers to the quality of the 2D image obtained by projecting the 3D data onto a plane at a certain viewpoint. For example, the 2D quality may include PSNR. Note that the 2D quality can be any information that indicates the quality of the 2D image.

[0459] Furthermore, 3D quality may include at least one of 3D geometry quality, which indicates the quality of the positional information (geometry information) of the 3D data, and 3D attribute quality, which indicates the quality of the attribute information of the 3D data. For example, 3D geometry quality may include D1 metrics, which indicate the positional accuracy of the point cloud, or D2 metrics, which indicate the surface accuracy of the point cloud. Note that 3D geometry quality may include any quality information related to the positional information of the 3D data.

[0460] Furthermore, 3D attribute quality includes, for example, the PSNR (Photons Processing Rating) which indicates the quality of the color information of a 3D point cloud if that point cloud has color information as attribute information. Note that 3D attribute quality may include any information related to the attribute information of 3D data.

[0461] In this way, by enabling the addition of various types of information as information indicating 2D or 3D quality, the range of use cases for utilizing 3D data can be broadened. Note that quality information is not limited to quality information obtained after encoding; it may include any quality information obtained during the process of generating the 3D data. For example, quality information may be information indicating quality obtained by comparing the 3D data before processing with the 3D data after processing. Here, processing refers to any process that modifies the 3D data. This allows quality information indicating any quality of the 3D data, not just encoding, to be added to the bitstream, enabling the decoder to switch to an appropriate method of utilizing the 3D data using that quality information. In this case, information indicating the process by which the added quality information was calculated may also be added to the bitstream. For example, the bitstream may include information indicating that the quality information shows the quality before and after encoding, or the quality before and after data processing. This allows the decoder to know the process by which the added quality information was calculated.

[0462] The quality information shown in Figure 78 includes 3D_geo_metrics_flag, 3D_geo_metrics_type_present_flag, 3D_geo_metrics_type, and 3D_geo_metrics as information regarding 3D geometry quality.

[0463] The 3D_geo_metrics_flag flag indicates whether or not the quality information includes 3D geometry quality information (3D_geo_metrics), which indicates 3D geometry quality. For example, if 3D_geo_metrics_flag is 1, the quality information includes 3D geometry quality information. If 3D_geo_metrics_flag is 0, the quality information does not include 3D geometry quality information.

[0464] The 3D_geo_metrics_type_present_flag is a flag that indicates whether or not information indicating the type of 3D geometry quality (3D_geo_metrics_type) is included in the quality information.

[0465] 3D_geo_metrics_type indicates the type of 3D geometry quality. For example, a value of 0 indicates D1 metrics, a value of 1 indicates D2 metrics, a value of 2 indicates Euclidean distance, and a value of 3 indicates Manhattan distance. If 3D_geo_metrics_type is not attached to the bitstream, the decoder may estimate that the value of 3D_geo_metrics_type is a predetermined default value. For example, if the default value is D1 metrics, the decoder may estimate the value of 3D_geo_metrics_type to be 0. This can reduce the number of bits.

[0466] 3D_geo_metrics is 3D geometry quality information that indicates the quality of the 3D geometry. 3D_geo_metrics indicates the type of 3D geometry quality indicated by 3D_geo_metrics_type. For example, if 3D_geo_metrics_type = 0, it indicates the value of the D1 metric; if 3D_geo_metrics_type = 1, it indicates the value of the D2 metric. The quality information may also include information indicating the unit (e.g., dB) for each type.

[0467] Furthermore, the quality information shown in Figure 78 includes 3D_attr_metrics_flag, numAttr, 3D_attr_metrics_type_present_flag, 3D_attr_metrics_type, and 3D_attr_metrics as information regarding 3D attribute quality.

[0468] 3D_attr_metrics_flag is a flag that indicates whether or not 3D attribute quality information (3D_attr_metrics), which indicates 3D attribute quality, is included in the quality information. For example, if 3D_attr_metrics_flag is 1, the quality information includes 3D attribute quality information, and if 3D_attr_metrics_flag is 0, the quality information does not include 3D attribute quality information.

[0469] `numAttr` indicates the number of attribute pieces of attribute information included in the 3D data to which 3D attribute quality information is added. For example, if only 3D attribute quality information for color is added as attribute information, `numAttr` is set to 1. If 3D attribute quality information for both color and reflectance is added as attribute information, `numAttr` is set to 2. This allows you to specify the number of attribute pieces to which 3D attribute quality information is added to the bitstream when the 3D data has one or more attribute pieces of information. In other words, the number of attribute pieces of attribute information indicated by `numAttr` may be less than the number of attribute pieces of attribute information that the 3D data has. For example, if the 3D data has two types of attribute information, color and reflectance, only one of the 3D attribute quality information pieces may be added. In this case, `numAttr` is set to 1.

[0470] Furthermore, information indicating the type of attribute information (such as color or reflectance) to which 3D attribute quality information has been added may be added to the bitstream. For example, information indicating that the first 3D attribute quality information indicates the 3D attribute quality of color, and the second 3D attribute quality information indicates the 3D attribute quality of reflectance may be added to the bitstream. This allows the decoder to know which attribute information of the 3D data the 3D attribute quality information added to the bitstream belongs to.

[0471] If numAttr is 1 or greater, numAttr_minus1, which represents numAttr minus 1, may be added to the bitstream instead of numAttr. In this case, the decoder calculates numAttr by adding 1 to numAttr_minus1. This reduces the number of bits.

[0472] 3D_attr_metrics_type_present_flag, 3D_attr_metrics_type, and 3D_attr_metrics are provided for each attribute information.

[0473] The flag 3D_attr_metrics_type_present_flag indicates whether or not the bitstream contains information indicating the type of 3D attribute information quality (3D_attr_metrics_type).

[0474] 3D_attr_metrics_type indicates the type of 3D attribute quality. For example, a value of 0 indicates PSNR, and a value of 1 indicates SSIM. If 3D_attr_metrics_type is not attached to the bitstream, the decoder may assume that the value of 3D_attr_metrics_type is a predetermined default value. For example, if the default value is PSNR, the decoder may assume that the value of 3D_attr_metrics_type is 0. This can reduce the number of bits.

[0475] 3D_attr_metrics is 3D attribute quality information that indicates 3D attribute quality. 3D_attr_metrics indicates the type of 3D attribute quality indicated by 3D_attr_metrics_type. For example, if 3D_attr_metrics_type = 0, it indicates the PSNR value, and if 3D_attr_metrics_type = 1, it indicates the SSIM value. The quality information may also include information indicating the unit (e.g., dB) for each type.

[0476] As shown in Figure 78, quality information may include at least one 3D quality information item and at least one 2D quality information item related to the 3D data. This allows the decoding device to obtain the 3D quality information of the 3D data, or the 2D quality information of one or more viewpoints, by decoding the quality information. By obtaining this information before decoding or rendering the 3D data, the decoding device can quickly determine whether to decode or render the 3D data depending on the application. For example, the decoding device can know the 3D or 2D quality of the 3D data bitstream before decoding it. Therefore, if the 3D data is used in an application that requires a certain level of 3D or 2D quality, the decoding device can reduce the processing load by not decoding the bitstream of 3D data that does not meet the quality standard.

[0477] Furthermore, the quality information shown in Figure 78 includes, as information relating to 2D attribute quality, 2D_metrics_flag, 2D_rendering_type_present_flag, 2D_rendering_type, 2D_metrics_type_present_flag, 2D_metrics_type, numView, view_information, and 3D_2D_metrics.

[0478] The 2D_metrics_flag flag indicates whether or not the quality information includes 2D quality information (3D_2D_metrics), which represents the 2D quality of 3D data. For example, if 2D_metrics_flag is 1, the quality information includes 2D quality information. If 2D_metrics_flag is 0, the quality information does not include 2D quality information.

[0479] The flag 2D_rendering_type_present_flag indicates whether or not information (2D_rendering_type) indicating the 2D rendering method for evaluating 2D quality is included in the quality information.

[0480] 2D_rendering_type indicates a 2D rendering method for evaluating 2D quality. For example, a value of 0 indicates renderer 1, and a value of 2 indicates renderer 2. Alternatively, 2D_rendering_type may indicate the URL or version of the renderer software. Or, 2D_rendering_type may indicate the parameters used for rendering, or the command to run the renderer. This information may also be shown as a string.

[0481] If 2D_rendering_type is not attached to the bitstream, the decoder may assume that the value of 2D_rendering_type is a predetermined default value. For example, if the default value is renderer 1, the decoder may assume that the value of 2D_rendering_type is 0. This reduces the number of bits.

[0482] Furthermore, a recommended value may be set as 2D_rendering_type. For example, a value indicating a recommended rendering method may be set as 2D_rendering_type. For example, the recommended rendering method may include ray tracing or the Z-buffer method. Furthermore, the recommended rendering method may include any rendering method. This allows the decoder to know the recommended rendering method for a given viewpoint. Therefore, the decoder can, for example, generate a high-quality rendered image by using the recommended rendering method.

[0483] The flag 2D_metrics_type_present_flag indicates whether or not information indicating the type of 2D quality (2D_metrics_type) is included in the quality information.

[0484] 2D_metrics_type indicates the type of 2D quality. For example, the type of 2D quality includes at least one of PSNR (Peak Signal-to-Noise Ratio), WS-PSNR (Weighted Spherical PSNR), IV-PSNR, SSIM (Structural Similarity Index Measure), IV-SSIM (Immersive Video SSIM), and LPIPS (Learned Perceptual Image Patch Similarity).

[0485] PSNR indicates the peak signal-to-noise ratio of an image. WS-PSNR is a PSNR adapted for omnidirectional video. IV-PSNR is a PSNR for immersive video. SSIM indicates the structural similarity of images. IV-SSIM is an SSIM for immersive video. LPIPS indicates a perceptual loss function in image generation or conversion tasks. Note that at least one of the software, software version, and parameters used for additional quality evaluation may be indicated.

[0486] Note that when 2D_metrics_type is not added to the bitstream, the decoding apparatus may infer that the value of 2D_metrics_type is a predetermined default value. For example, when the default value is PSNR, the decoding apparatus may infer that the value of 2D_metrics_type is 0. This can reduce the bit amount.

[0487] numView indicates the number of rendering viewpoints to which 2D quality information is added. Note that when numView is 1 or more, numView_minus1 indicating a value obtained by subtracting 1 from numView may be added to the bitstream instead of numView. In this case, the decoding apparatus calculates numView by adding 1 to numView_minus1. This can reduce the bit amount.

[0488] view_information and 3D_2D_metrics are provided for each viewpoint. view_information is the i-th viewpoint information. For example, the viewpoint information includes the information necessary to generate the rendered image of the i-th viewpoint. For example, the viewpoint information includes viewport_type, viewport_id, extrinsic_flag, intrinsic_flag, camera pose information, and camera parameter information.

[0489] The `viewport_type` parameter indicates the type of viewpoint information. For example, a value of 0 indicates that the viewpoint information is user-recommended viewpoint information. A value of 1 indicates that the viewpoint information is the viewpoint information of the image used for training.

[0490] `viewport_id` is the identifier for the viewpoint. `extrinsic_flag` is a flag indicating whether the viewpoint information includes camera pose information representing external parameters. `intrinsic_flag` is a flag indicating whether the viewpoint information includes camera parameter information, which is an internal parameter.

[0491] Camera pose information includes camera position (e.g., x, y, z coordinates) and a direction vector indicating the camera direction (e.g., x, y, z vector). Camera parameter information includes, for example, at least one of the following: angle of view, image sensor size, focal length, aperture value, shutter speed, ISO sensitivity, depth of field, and frame rate.

[0492] This allows the decoder to generate a 2D rendering image of the i-th viewpoint by using the i-th view_information appended to the bitstream.

[0493] 3D_2D_metrics is 2D quality information indicating the 2D quality of the i-th viewpoint. 3D_2D_metrics indicates the value of the 2D quality of the type specified by 2D_metrics_type when rendered using the rendering method specified by 2D_rendering_type. For example, if 2D_metrics_type is PSNR, 3D_2D_metrics will show the PSNR value, and if 2D_metrics_type is SSIM, it will show the SSIM value. The quality information may also include information indicating the unit of each type (e.g., dB).

[0494] In the example shown in Figure 78, 2D_rendering_type and 2D_metrics_type are provided in common for multiple rendering viewpoints. This reduces the amount of code in the header. Alternatively, 2D_rendering_type or 2D_metrics_type may be added for each viewpoint. This allows specifying the rendering method or 2D quality type for each viewpoint, enabling more flexible generation of 2D quality information.

[0495] Furthermore, while the example shown in Figure 78 illustrates a case where 2D quality information is provided for each viewpoint, 2D quality information common to multiple viewpoints may also be provided. For example, the common 2D quality information could indicate the average value of the 2D quality for each viewpoint. Additionally, information indicating that the added 2D quality information is the average value of the 2D quality for each viewpoint may be added to the bitstream. By adding 2D quality information common to multiple viewpoints to the bitstream in this way, the amount of code in the header can be reduced.

[0496] Furthermore, if the encoding device is unable to calculate the 3D_2D_metrics for the i-th viewpoint, it may set the value of 3D_2D_metrics for the i-th viewpoint to a value indicating that the 2D quality could not be calculated. For example, if the value of 3D_2D_metrics for the i-th viewpoint is 0, it indicates that the 2D quality for the i-th viewpoint could not be calculated. This allows the decoding device to know that if the value of 3D_2D_metrics for the i-th viewpoint is 0, the encoding device or the transmitting device was unable to calculate the 2D quality for the i-th viewpoint. This allows the decoding device to decide, for example, not to render and display the image for the i-th viewpoint. Alternatively, the decoding device may decide to display a different image obtained or generated by some method when displaying an image for the i-th viewpoint. This prevents the rendering and display of an image for a viewpoint for which the 2D quality could not be calculated.

[0497] Furthermore, a value indicating rendering prohibition may be set in 3D_2D_metrics for the i-th viewpoint. For example, setting the value of 3D_2D_metrics for the i-th viewpoint to 0 may indicate that rendering of the i-th viewpoint is prohibited. This allows the decoder to know that rendering of 3D data from the i-th viewpoint is prohibited if the value of 3D_2D_metrics for the i-th viewpoint is 0. This allows the decoder to decide, for example, not to render and display the image from the i-th viewpoint. Alternatively, the decoder may decide to display a different image obtained or generated by some means when displaying an image from the i-th viewpoint based on a user request. This prevents the display of an image from a viewpoint where rendering is prohibited. Alternatively, the decoder may, in some way, inform the user that rendering of the i-th viewpoint is prohibited if the value of 3D_2D_metrics for the i-th viewpoint is 0. This allows the user to know that rendering of the i-th viewpoint is prohibited.

[0498] Furthermore, quality information may be provided in common for multiple objects included in 3D data. For example, if 3D data includes objects A and B, quality information may be added to the bitstream as common information for objects A and B. This reduces the amount of code in the header. In this case, for example, 3D quality information may be calculated using both objects A and B. Also, 2D quality information may be calculated using a 2D image rendered by projecting both objects A and B onto a certain viewpoint. This allows the decoder to obtain quality information common to both objects A and B. Therefore, the decoder can know the 3D quality or 2D quality of both objects A and B.

[0499] Alternatively, quality information may be provided separately for each of the multiple objects included in the 3D data. For example, if the 3D data includes objects A and B, quality information for object A and quality information for object B may be provided separately. This allows the decoding device to know separate quality information for each object. In this case, the 3D quality of object A may be calculated using the information for object A, and the 3D quality of object B may be calculated using the information for object B. Furthermore, the 2D quality of object A may be calculated using a 2D image rendered in a 3D space without object B, or by treating object B as a transparent object and projecting object A onto a certain viewpoint. Similarly, the 2D quality of object B may be calculated using a 2D image rendered in a 3D space without object A, or by treating object A as a transparent object and projecting object B onto a certain viewpoint. This allows the decoding device to obtain the respective quality information for object A and object B. Therefore, the decoding device can know the 3D quality or 2D quality when either object A or B exists as 3D data.

[0500] In the example shown in Figure 78, one type or rendering method is specified for 3D quality information or 2D quality information, but multiple types or multiple rendering methods may be specified. For example, 3D quality information may include multiple types of information and the 3D quality of each of the multiple types. Similarly, 2D quality information may include multiple types of information and the 2D quality of each of the multiple types. For example, D1 metrics and D2 metrics may be specified as 3D quality information. Or, PSNR and SSIM may be specified as 2D quality information. This allows the decoder to switch how it uses 3D data using multiple types of 3D quality or 2D quality.

[0501] In the example shown in Figure 78, 3D quality information is provided regardless of viewpoint or distance, but 3D quality information may also be provided for each viewpoint or distance. Figure 79 shows an example of quality information in this case. The quality information shown in Figure 79 includes, in addition to the quality information shown in Figure 78, 3D_geo_numView, 3D_geo_view_information, 3D_geo_numDepth, 3D_geo_depth, 3D_attr_numView, 3D_attr_view_information, 3D_attr_numDepth, and 3D_attr_depth. Furthermore, it differs from Figure 78 in that 3D_geo_metrics and 3D_attr_metrics are provided for each viewpoint.

[0502] 3D_geo_numView indicates the number of viewpoints to which 3D geometry quality information (3D_geo_metrics) is added. If 3D_geo_numView is 1 or greater, 3D_geo_numView_minus1, which is the value obtained by subtracting 1 from 3D_geo_numView, may be added to the bitstream instead. In this case, the decoder calculates 3D_geo_numView by adding 1 to 3D_geo_numView_minus1. This reduces the number of bits.

[0503] 3D_geo_view_information and 3D_geo_numDepth are provided for each viewpoint. 3D_geo_view_information is the i-th viewpoint information. The details of the viewpoint information are similar to, for example, the view_information shown in Figure 78 above. As a result, the decoding device can obtain the camera information for the i-th viewpoint by using the i-th 3D_geo_view_information added to the bitstream.

[0504] 3D_geo_numDepth indicates the distance or depth (depth number) added to the 3D geometry quality information. If 3D_geo_numDepth is 1 or greater, 3D_geo_numDepth_minus1, which represents the value obtained by subtracting 1 from 3D_geo_numDepth, may be added to the bitstream instead. In this case, the decoder calculates 3D_geo_numDepth by adding 1 to 3D_geo_numDepth_minus1. This reduces the number of bits.

[0505] 3D_geo_depth and 3D_geo_metrics are provided for each distance (depth). 3D_geo_depth indicates the distance range of the 3D data used to calculate the 3D geometry quality. For example, the 3D geometry quality of the 3D data is calculated using 3D data contained within a radius of 3D_geo_depth from the camera position indicated by 3D_geo_view_information. This allows the quality information to indicate the 3D geometry quality according to the distance from a certain viewpoint. Therefore, even if the 3D geometry quality changes with distance, the encoding device can accurately inform the decoding device of the 3D geometry quality.

[0506] Figure 79 shows an example where 3D geometry quality information for 3D data contained within a radius 3D_geo_depth from the camera position indicated by 3D_geo_view_information is added to the bitstream. However, bounding box information may be used instead of 3D_geo_depth. Bounding box information specifies a bounding box, which is a rectangular space based on the camera position indicated by 3D_geo_view_information. For example, the bounding box includes the camera position indicated by 3D_geo_view_information. The 3D geometry quality information indicates the 3D geometry quality of the 3D data contained within the bounding box. For example, the bounding box information indicates the width, height, and depth of the bounding box. The bounding box may be a cube, and the bounding box information may indicate the length of one side of the cube. This adds 3D geometry quality information for the 3D data within a bounding box containing a specific viewpoint to the bitstream. Therefore, even if the 3D geometry quality changes depending on the size of the bounding box, the decoding device can be accurately informed of the 3D geometry quality.

[0507] 3D_geo_metrics indicates the 3D geometry quality of 3D data contained within the distance 3D_geo_depth from the i-th viewpoint. By adding 3D geometry quality information for each distance in this way, the decoding device can determine, for example, which distances of 3D data or rendered images of that 3D data should be presented to the user. For example, the decoding device checks the 3D geometry quality information for each distance. The decoding device then presents 3D data or rendered images using 3D data where the value of the 3D geometry quality information is above a predetermined threshold. This allows the user to be presented with results with less image quality degradation. Furthermore, if the value of the 3D geometry quality information is below the threshold, the decoding device determines that the 3D data contained within that range is of low accuracy. By avoiding the use of such 3D data, the decoding device can, for example, present the user with highly accurate measurement results.

[0508] 3D_attr_numView indicates the number of viewpoints to which 3D attribute quality information is added. If 3D_attr_numView is 1 or greater, 3D_attr_numView_minus1, which is the value obtained by subtracting 1 from 3D_attr_numView, may be added to the bitstream instead. In this case, the decoder calculates 3D_attr_numView by adding 1 to 3D_attr_numView_minus1. This reduces the number of bits.

[0509] 3D_attr_view_information and 3D_attr_numDepth are provided for each viewpoint. 3D_attr_view_information is the i-th viewpoint information for the k-th attribute. The details of the viewpoint information are similar to, for example, the view_information shown in Figure 78 above. As a result, the decoding device can obtain the camera information for the i-th viewpoint by using the i-th 3D_attr_view_information for the k-th attribute attached to the bitstream.

[0510] 3D_attr_numDepth indicates the distance or depth (depth number) to which 3D attribute quality information is added. If 3D_attr_numDepth is 1 or greater, 3D_attr_numDepth_minus1, which is the value obtained by subtracting 1 from 3D_attr_numDepth, may be added to the bitstream instead. In this case, the decoder calculates 3D_attr_numDepth by adding 1 to 3D_attr_numDepth_minus1. This reduces the number of bits.

[0511] 3D_attr_depth and 3D_attr_metrics are provided for each distance (depth). 3D_attr_depth indicates the distance range of the 3D data used to calculate the 3D attribute quality. For example, the 3D attribute quality of the 3D data is calculated using 3D data contained within a radius of 3D_attr_depth from the camera position indicated by 3D_attr_view_information. This allows the quality information to indicate the 3D attribute quality according to the distance from a certain viewpoint. Therefore, even if the 3D attribute quality changes with distance, the encoding device can accurately inform the decoding device of the 3D attribute quality.

[0512] In Figure 79, an example is shown where 3D attribute quality information for 3D data contained within a radius of 3D_attr_depth from the camera position indicated by 3D_attr_view_information is added to the bitstream. However, bounding box information may be used instead of 3D_attr_depth. For example, the bounding box includes the camera position indicated by 3D_attr_view_information. The 3D geometry quality information indicates the 3D geometry quality of the 3D data contained within the bounding box. For example, the bounding box information indicates the width, height, and depth of the bounding box. The bounding box may be a cube, and the bounding box information may indicate the length of one side of the cube. As a result, 3D attribute quality information for 3D data within a bounding box containing a certain viewpoint position is added to the bitstream. Therefore, even if the 3D attribute quality changes depending on the size of the bounding box, the 3D geometry quality can be accurately communicated to the decoding device.

[0513] 3D_attr_metrics indicates the 3D attribute quality of the 3D data contained within the distance 3D_attr_depth from the i-th viewpoint of the k-th attribute information. By adding 3D attribute quality information for each distance in this way, the decoding device can determine, for example, which distances of 3D data or rendered images of that 3D data should be presented to the user. For example, the decoding device checks the 3D attribute quality information for each distance. The decoding device then presents 3D data or rendered images using 3D data where the value of the 3D attribute quality information is above a predetermined threshold. This allows the user to be presented with results with less image quality degradation. Furthermore, if the value of the 3D attribute quality information is below the threshold, the decoding device determines that the attribute information of the 3D data contained within that range is of low accuracy. By avoiding the use of such 3D data, the decoding device can, for example, present the user with highly accurate measurement results.

[0514] In the example shown in Figure 79, different viewpoint information (3D_geo_numView, 3D_geo_view_information, 3D_attr_numView, and 3D_attr_view_information) and depth information (3D_geo_numDepth, 3D_geo_depth, 3D_attr_numDepth, and 3D_attr_depth) are set for the location information and attribute information of the 3D data. However, common viewpoint information and depth information may be set for both the location information and the attribute information. Furthermore, at least a portion of this information may be common to both the location information and the attribute information. This can reduce the amount of code in the header.

[0515] Furthermore, although the syntax for 2D quality information is omitted in Figure 79, the quality information may include 2D quality information for each viewpoint, similar to Figure 78. This provides the following benefits: Improved flexibility in quality evaluation. Specifically, the decoder can evaluate the quality of the 2D image or texture for each viewpoint (view) to gain a detailed understanding of the quality of the rendered image from a particular viewpoint.

[0516] Furthermore, it becomes easier to optimize the playback device (rendering device) that plays back the decoded 3D data. The playback device may be included in the decoding device or it may be a separate device. Specifically, the playback device can select and play back high-quality viewpoints. Therefore, the user's visual experience can be improved. Also, by indicating the texture quality for each viewpoint, the playback device can select and play back viewpoints with high signal-to-noise ratios. In addition, by extending 2D metrics to 3D metrics and including metrics for each viewpoint, the playback device can select and play back high-quality viewpoints.

[0517] Furthermore, the data generation device that uses learning to generate 3D data selects viewpoints with a quality above a threshold and uses those images to train a new 3D data generation model. This improves the learning accuracy.

[0518] The range of viewpoints for rendering 3D data may be indicated using quality (S / N ratio) as shown below. Alternatively, the range may be indicated using viewpoint information (or viewport_id).

[0519] First, let's explain an example of using quality (signal-to-noise ratio). For example, a playback device may use the signal-to-noise ratio to evaluate the quality of each viewpoint and select only viewpoints with high signal-to-noise ratios for playback. Alternatively, the playback device may skip viewpoints with low signal-to-noise ratios and play only those with high ratios. This provides the user with a high-quality visual experience. By performing playback in a way that avoids areas with low signal-to-noise ratios, the playback device can avoid rendering areas that are not visible or of low quality.

[0520] Next, we will explain an example of how viewpoint information can be used. For example, the playback device may display a range of viewpoints that can be selected as viewpoints to be used for rendering. In other words, the playback device may use viewpoint information to visually indicate the area that the user can move to. For example, the playback device presents the user with multiple viewpoint positions attached to the SEI. The user selects a viewpoint to be used for rendering from the presented multiple viewpoint positions.

[0521] Alternatively, the playback device may linearly connect multiple viewpoint positions attached to the SEI and present the resulting shape to the user. This allows the user to visually understand the area where viewpoint selection is possible, thereby improving user operability.

[0522] Alternatively, the playback device may use viewpoint information to clearly indicate areas that the user cannot move to or cannot see. Furthermore, the playback device may effectively indicate areas that can be moved by combining the above-mentioned quality (S / N ratio) with viewpoint information.

[0523] Furthermore, the playback device may indicate the range in which the viewpoint can be moved using at least one of the quality information and viewpoint information. For example, the encoding device may add information indicating the range in which the viewpoint can be moved to the SEI. The encoding device may also add information indicating viewpoint positions where rendering is prohibited to the SEI. The encoding device may also add information indicating viewpoint positions that it does not want to show, such as areas in the 3D data where degradation is noticeable.

[0524] Furthermore, the encoding device may add information to the SEI indicating the space in which the viewpoint can move, so that the viewpoint does not move beyond a predetermined range. For example, the encoding device may add positional information regarding the renderable area to the SEI for a given 3D space. For example, the renderable area is indicated by a three-dimensional bounding box. The encoding device adds information indicating the reference position of the bounding box and information indicating the size of the bounding box to the SEI. The playback device then generates a rendered image if the location requested by the user for rendering is within the renderable area, but does not need to generate a rendered image if the location is outside the renderable area.

[0525] If the location in question is outside the renderable area, the playback device may indicate to the user in some way that an area outside the renderable area has been specified. For example, the playback device may display on the screen that an area outside the renderable area has been specified. Alternatively, the playback device may simultaneously display the renderable area on the screen. This allows the user to know that an area outside the renderable area has been specified, and also to know the renderable area instead.

[0526] This setting of the renderable area can also be used for security purposes. For example, by setting areas that should not be displayed for security reasons to be outside the renderable area, a secure rendered image can be provided. If the user specifies rendering outside the renderable area, the playback device may instead display an image generated using, for example, AI. This ensures that some image can be presented even when the user requests rendering outside the renderable area. When informing the user that an area is outside the renderable area, this can be done in any way, such as by displaying text or representing it with an AI-generated image.

[0527] (Embodiment 5) Figures 78 and 79 show an example in which information regarding the quality of 3D data is added to the bitstream as SEI. This allows the decoding device to use the quality information to, for example, provide the user with a high-quality rendering image and prevent the user from receiving a low-quality rendering image. This embodiment describes an example of providing the user with a high-quality rendering image by another method.

[0528] Figure 80 shows an example of the syntax for rendering hint information (3D_rendering_hint_information) used when rendering 3D data. For example, this rendering hint information is included in SEI.

[0529] Rendering support information includes quality information (3D_quality_information) as shown in Figure 78 or Figure 79. Furthermore, rendering support information includes rendering-prohibited area information, lighting information, and overhead view information. Note that rendering support information may include all of this information, or only some of it.

[0530] First, let's explain the rendering-prohibited area information. Using 3D data, it is possible to generate rendered images from any viewpoint position. On the other hand, if there are missing areas in the 3D data, or if the accuracy of the positional information in the 3D data is not high, the quality of the rendered image using that 3D data will be poor. Therefore, if you want to provide users with rendered images of a certain quality or higher, you should prohibit rendering from viewpoint positions that would result in poor rendering image quality. In this embodiment, information to prohibit rendering is added to the SEI.

[0531] As shown in Figure 80, the rendering assistance information includes, as information related to rendering prohibited area information, prohibit_area_present_flag, numArea, position(x,y,z), and radius.

[0532] The `prohibit_area_present_flag` flag indicates whether or not the rendering assistance information includes rendering-prohibited area information. For example, if `prohibit_area_present_flag` is 1, the rendering assistance information includes rendering-prohibited area information, and if `prohibit_area_present_flag` is 0, the rendering assistance information does not include rendering-prohibited area information.

[0533] numArea indicates the number of rendering-prohibited areas. If numArea is 1 or greater, numArea_minus1, which represents numArea minus 1, may be added to the bitstream instead of numArea. In this case, the decoder calculates numArea by adding 1 to numArea_minus1. This reduces the number of bits.

[0534] `position(x,y,z)` and `radius` are rendering aids that indicate rendering-prohibited areas. `position(x,y,z)` and `radius` are provided for each of the multiple rendering-prohibited areas if they are set. `position(x,y,z)` indicates the center coordinates (x,y,z) of the rendering-prohibited area, which is represented as a sphere. `radius` indicates the radius of the rendering-prohibited area, which is represented as a sphere.

[0535] Thus, the rendering-prohibited area may be represented by a sphere. The encoding device may add the center coordinates (x, y, z) and radius radius of the sphere to the SEI. In this case, for example, when the playback device receives a rendering request from the user for 3D data of a certain 3D space from a position contained within a sphere indicated by the center coordinates (x, y, z) and radius radius, it performs at least one of the following processes. As a result, the playback device does not provide the user with a rendered image from within the rendering-prohibited area.

[0536] For example, the playback device may inform the user that rendering is prohibited from that position and refrain from rendering. For example, the playback device may display a different image that has been prepared in advance. The playback device may also inform the user, through text or other means, that it is displaying a different image that has been prepared in advance.

[0537] For example, the playback device may present a rendered image from a position outside the rendering prohibited area as a separate image. The playback device may also indicate to the user the viewpoint from which the presented image was rendered.

[0538] Therefore, by appropriately setting rendering-prohibited areas, it is possible to prevent users from being provided with rendered images from locations that you do not want them to see. This improves the quality of the service.

[0539] In the above example, a sphere was used as a method for specifying a rendering-prohibited area, but rendering-prohibited areas may be specified using shapes other than a sphere. For example, a rendering-prohibited area may be specified using a three-dimensional bounding box. Alternatively, an ellipsoid may be used. In this case, the encoding device may add information indicating the shape of the rendering-prohibited area to the bitstream. For example, if a bounding box is used, information indicating the three-dimensional point coordinates of the reference position of the bounding box, as well as its width, height, and depth, may be added to the bitstream. If an ellipsoid is used, the center coordinates of the ellipsoid and information regarding the ellipsoid's surface may be added to the bitstream.

[0540] Furthermore, if multiple rendering-prohibited areas are set, information indicating the shape of each rendering-prohibited area may be set for each area. This allows the encoding device to flexibly configure rendering-prohibited areas. In addition, whether or not rendering is prohibited may be determined based on the rendering-prohibited area (viewpoint position) and the viewpoint direction. For example, rendering from within a rendering-prohibited area towards direction A may be prohibited, while rendering towards direction B may be permitted. This allows for more flexible configuration, as rendering can be prohibited or prohibited for each rendering-prohibited area and direction.

[0541] Next, we will explain the lighting information. The quality of the rendered image of 3D data changes depending on the lighting information used during rendering. In addition, the appropriate lighting information may change depending on the viewpoint position. Therefore, the quality of the rendered image can be improved by providing lighting information for each viewpoint. In this embodiment, the lighting information used during rendering for each viewpoint is added to the SEI.

[0542] As shown in Figure 80, the rendering assistance information includes lighting_info_present_flag, numView, view_information, and lighting_data as information related to lighting information.

[0543] The `lighting_info_present_flag` parameter indicates whether the rendering aid information includes lighting information for rendering. For example, if `lighting_info_present_flag` is 1, the rendering aid information includes lighting information. If `lighting_info_present_flag` is 0, the rendering aid information does not include lighting information.

[0544] `numView` indicates the number of rendering viewpoints. If `numView` is 1 or greater, `numView_minus1`, which represents `numView` minus 1, may be added to the bitstream instead of `numView`. In this case, the decoder calculates `numView` by adding 1 to `numView_minus1`. This reduces the number of bits.

[0545] view_information and lighting_data are provided for each of the multiple viewpoints when multiple viewpoints are set. view_information is the i-th viewpoint information. The details of the viewpoint information are the same as, for example, the view_information shown in Figure 78 above. As a result, the decoding device can generate a rendered image of the i-th viewpoint by using the i-th view_information added to the bitstream.

[0546] `lighting_data` is the lighting information used when rendering the i-th viewpoint. For example, `lighting_data` may be recommended lighting information. Here, lighting information includes at least one of the following pieces of information: lighting position, intensity of light, brightness, color of light, radius of light, type of light source (e.g., point light source or area light source), and the environment in which the light is installed (e.g., indoor or outdoor). Note that lighting information is not limited to these pieces of information and may include any information related to lighting that is necessary during rendering. Furthermore, combinations of the above pieces of information may be defined as multiple presets, and lighting information may refer to one of these multiple presets.

[0547] Furthermore, the lighting information may include information indicating the recommended probe positions. Here, a probe is a sensor placed at a specific location in the scene to collect light information. For example, it may be calculated how an object reflects light or casts shadows at the probe position. Also, by avoiding placing probes in areas with poor 3D data quality, those areas will be rendered darker as a result. This makes distortion less visible. Additionally, many probes may be placed in areas with large changes in light, and fewer probes in areas with small changes in light. This can improve computational efficiency.

[0548] Next, we will explain the overhead view information. As shown in Figure 80, the rendering aid information includes birdey_present_flag and birdey_view_information as information related to the overhead view.

[0549] The `birdey_present_flag` flag indicates whether the rendering aid information includes overhead view information. For example, if `birdey_present_flag` is 1, the rendering aid information includes overhead view information; if `birdey_present_flag` is 0, the rendering aid information does not include overhead view information.

[0550] birdey_view_information is overhead view information that indicates the viewpoint for generating an overhead view image of 3D data as if viewed from above. For example, the information included in the overhead view information is the same as the view_information shown in Figure 78 above. As a result, the decoding device can generate an overhead view image of the 3D data by using birdey_view_information attached to the bitstream.

[0551] Furthermore, depending on the 3D data, rendering images from certain directions may be of poor quality. Therefore, rendering from those areas may be prohibited using the following method. Figure 81 shows an example of how to set a rendering prohibited area.

[0552] For example, rendering-prohibited areas may be set by setting the value of 3D_2D_metrics, as shown in Figure 78, to 0. In other words, rendering of viewpoints with a quality (value of 3D_2D_metrics) of 0 may be prohibited. The following points may also be considered when selecting a rendering viewpoint for 3D data.

[0553] For example, the distance the viewpoint can move may be restricted. For instance, if viewpoint A is selected, the next viewpoint to be selected may be restricted to viewpoints within a certain distance from viewpoint A. This promotes the reuse of rendering processes and improves real-time performance. Also, if the user significantly changes the viewpoint position for rendering, the system becomes heavy because rendering is required each time. On the other hand, the amount of processing load can be reduced by restricting the distance the viewpoint can move.

[0554] For example, information regarding eye movement may be included in the bitstream. This allows the user to select the optimal viewpoint. For instance, the playback device evaluates the quality of each viewpoint and prioritizes displaying viewpoints with a high signal-to-noise ratio. This provides the user with a high-quality viewpoint.

[0555] For example, the distance the viewpoint can move may be limited to reduce loading times associated with viewpoint changes. This promotes the reuse of rendering processes. Furthermore, reduced loading times allow for smoother display even when users frequently switch viewpoints.

[0556] As a result of the above, by limiting the distance the viewpoint moves and including information about eye movement in the bitstream, loading times can be reduced and a high-quality viewpoint can be provided to the user.

[0557] Furthermore, rendering aid information may include the following information: Rendering aid information may include engine information regarding the rendering engine. Engine information is information about the rendering engine recommended for rendering. For example, engine information may include at least one of the type and version of the rendering engine used for rendering. Engine information may also include information indicating which engine the 3D data was created for.

[0558] Rendering assistance information may include information about the projection matrix. For example, this information may show projection parameters used when projecting a 3D object onto a 2D image. It may also show parameters used for transforming camera coordinates. For example, this information may show a transformation matrix for converting camera coordinates to world coordinates.

[0559] Rendering support information may include information indicating rendering parameters. For example, this information may indicate lighting conditions or the position of light sources during rendering. It may also indicate the camera position or viewpoint information during rendering.

[0560] Rendering aid information may include information about the inverse transform. For example, this information may also include information about reprojection. For example, this information may indicate the inverse transform parameters used when converting a 2D image back to a 3D object. This information may also indicate the transform matrix used for the inverse transform.

[0561] Including the above-mentioned information in rendering aids can improve the quality and reproducibility of rendered images from 3D data.

[0562] Furthermore, if you wish to prohibit rendering from within a 3D object, you can do so using rendering prohibition area information included in the rendering assistance information. For example, by setting an area including the inside of an object as a rendering prohibition area, rendering from inside the object to the outside will be prohibited. The rendering assistance information may also include information indicating whether or not to prohibit rendering from inside the object for each object. This allows you to switch whether or not to prohibit rendering from inside to outside for each object, thereby improving the flexibility of the settings.

[0563] For example, as shown in Figure 81, a viewpoint prohibition area may be set using rendering prohibition area information included in the rendering assistance information. Rendering from viewpoints within the viewpoint prohibition area is prohibited. This prevents rendering from prohibited viewpoints, such as the one shown in Figure 81.

[0564] Furthermore, the playback device may prohibit rendering from camera viewpoints corresponding to the 3D_2D_metrics value shown in Figure 78. For example, the playback device may prohibit rendering from camera viewpoints where the 3D_2D_metrics value is 0. The playback device may also indicate to the user in some way that rendering of this viewpoint is prohibited. This allows the user to know that rendering of this viewpoint is prohibited.

[0565] Figure 81 shows examples of usable camera viewpoints, camera viewpoints with a quality of 0, camera positions used for training, prohibited viewpoints, viewpoint restriction areas, and viewpoint direction restriction areas. A viewpoint direction restriction area is a region where rendering in a specific direction (for example, outward) from a viewpoint position within that area is prohibited.

[0566] Figure 82 is a flowchart showing an example of processing in a playback device (decoder). For example, this processing is used in an application performed on the playback device. First, the playback device determines whether the user has instructed rendering from an overhead view (S301). If there is an instruction for rendering from an overhead view (Yes in S301), the playback device determines whether birdey_info_present_flag included in the rendering assistance information is 1 (S302).

[0567] If birdey_info_present_flag is 1 (Yes in S302), the playback device generates an overhead view image by rendering using the overhead view information (birdey_view_information). The playback device presents (displays) the overhead view image to the user (S303). On the other hand, if birdey_info_present_flag is 0 (No in S302), the playback device informs the user that it cannot generate an overhead view image (S304). The playback device may also present a message prompting the user to specify a different viewpoint.

[0568] If overhead viewpoint information is unavailable, the playback device may generate a rendered image using predetermined default viewpoint information and present that rendered image to the user. For example, the playback device may present a rendered image of the 3D data viewed from the viewpoint position (0,0,0).

[0569] On the other hand, if there is no instruction to render from an overhead view (No in S301), the playback device determines whether the user has instructed rendering from viewpoint A (S305). If there is an instruction to render from viewpoint A (Yes in S305), the playback device determines whether viewpoint A is included within the rendering prohibited area set in the rendering prohibited area information (S306).

[0570] If viewpoint A is located within the rendering prohibited area (Yes in S306), the playback device informs the user that a rendered image from viewpoint A cannot be generated (S307). The playback device may also display a message prompting the user to specify a different viewpoint.

[0571] On the other hand, if viewpoint A is not included in the rendering prohibited area (No in S306), the playback device determines whether there is lighting information for rendering viewpoint A (S308). If there is lighting information for rendering viewpoint A (Yes in S308), the playback device generates a rendered image from viewpoint A by performing rendering using the viewpoint information and lighting information of viewpoint A, and presents the generated rendered image to the user (S309).

[0572] On the other hand, if there is no lighting information for rendering viewpoint A (No in S308), the playback device generates a rendered image from viewpoint A by performing rendering using viewpoint information for viewpoint A without using lighting information, and presents the generated rendered image to the user (S310). If there is no lighting information, the playback device may use predetermined default values ​​as lighting information.

[0573] Furthermore, if the result is No after steps S303, S304, S307, S309, S310, or in step S305, steps S301 onwards are executed again.

[0574] The following describes implementation examples of other applications. First, let's describe an example of application functionality. For example, the above technology may be used in an intelligent viewer control function. For example, the playback device first determines whether overhead viewpoint information is available in response to a viewpoint request specified by the user. If overhead viewpoint information is available, the playback device automatically selects the optimal overhead view.

[0575] For example, if a playback device receives a viewpoint request that falls within a rendering-prohibited area, it will automatically select an alternative viewpoint and present the alternative viewpoint, or a rendered image from the alternative viewpoint, to the user.

[0576] For example, when a user requests enhanced rendering, the playback device automatically switches whether or not to perform enhanced rendering depending on the presence or absence of information for enhanced rendering. If enhanced rendering is necessary, the playback device performs the appropriate display mode (e.g., highlighting).

[0577] Furthermore, the above technology may be applied to a quality-assured rendering mode. For example, the playback device, in response to an input viewpoint request, uses quality information before rendering to determine whether the rendered image of the requested viewpoint meets the image quality standards. If the playback device determines that the rendered image of the requested viewpoint is of low quality, it automatically generates and presents alternative content or a corrected rendered image (such as a filtered rendered image) instead of the normal rendered image.

[0578] Furthermore, the above technology may be used in a dynamic rendering path selection function. For example, the playback device dynamically selects the optimal rendering path according to user requests based on multiple decision steps defined in a flowchart (such as confirmation of overhead view information, confirmation of rendering prohibited areas, and confirmation of the presence or absence of information for enhanced rendering). This allows the playback device to make high-speed branch decisions within the processing flow using the information added to the SEI. Thus, rendering efficiency can be improved.

[0579] Furthermore, the above technology can be applied to the following use cases. For example, it can be applied to virtual tours or real estate viewing systems. For instance, when a user clicks on a spot (viewpoint), the system determines whether the clicked viewpoint is of high quality. If the system determines that the clicked viewpoint is of high quality, it proceeds with rendering. The system also automatically suggests the optimal viewpoint based on overhead view information and rendering prohibited area information. This ensures that users always experience clear and sharp images.

[0580] Furthermore, the above technology can be applied to industrial inspection or 3D scan data viewers. For example, when viewing 3D scan data of a factory or plant, it can prevent low-quality or inappropriate rendering from certain viewpoints. This allows users to understand the situation accurately. For instance, a playback device can display only the parts the user needs in high quality. This improves the efficiency and safety of inspection work.

[0581] Furthermore, the above technology can be applied to simulations for education or training. For example, in 3D simulations used for learning purposes, it is possible to enable the selection of dynamic rendering paths that guide the user to the optimal viewpoint. This can improve learning effectiveness by avoiding user confusion caused by low-quality displays.

[0582] Furthermore, the above technology can be applied to the UX (User Experience) portion of an application. For example, the playback device can provide visual feedback on quality. For instance, when a user requests a viewpoint, the playback device can display the quality evaluation results or indicators of rendering-prohibited areas (e.g., a heatmap or warning mark) on the screen in real time. This allows the playback device to clearly present the current rendering range to the user.

[0583] Furthermore, the above technology can be applied to alternative viewpoints and guidance interfaces. For example, if the requested viewpoint is included in a rendering-prohibited area, the playback device will present available alternative viewpoints as candidates on the screen in addition to the requested viewpoint. In this case, a user-friendly UI (e.g., a slider or map display) may be used. Also, if the user requests enhanced rendering, the playback device will display an interface for comparing the current viewpoint with alternative viewpoints that can be enhanced.

[0584] Furthermore, the above technology can be applied to smooth transitions using animation or transitions. For example, when a playback device switches the viewpoint from a viewpoint deemed low-quality to an alternative viewpoint, it uses a fade or slide animation. This allows the user to experience a high-quality progression without any sense of incongruity. In addition, the playback device overlays viewpoint transition guidelines (e.g., arrows or path lines) based on SEI information onto the screen. This allows the user to intuitively understand which direction to move their viewpoint in next.

[0585] [Summary] An encoding method according to one aspect of this disclosure is a method for encoding 3D data, wherein when encoding the 3D data, quality information (e.g., 3D quality information) relating to the 3D data is added to the bitstream as metadata (additional information). According to this, the encoding method adds quality information as metadata such as SEI to various types of 3D data (e.g., 3D point clouds or meshes, etc.) at the time of encoding. As a result, a decoding device (or data processing device) that decodes and processes the 3D data can evaluate the quality of the 3D data before decoding it. Furthermore, the decoding device can use the added quality information to grasp the quality of the 3D data (e.g., 3D quality or 2D quality) in advance from within the bitstream. As a result, the decoding device can quickly select the optimal processing method according to the application of the 3D data (e.g., a method of directly using the 3D data, or a method of rendering and using the 3D data). As a result, the decoding device can avoid unnecessary decoding processing. Therefore, a reduction in the overall system load, saving of communication bandwidth, and an improvement in user experience can be expected.

[0586] For example, the quality information includes quality evaluation parameters based on the location information or attribute information of the 3D data. Here, the quality evaluation parameters include, for each information area of ​​the 3D data, (a) instruction information regarding the presence or absence and classification of quality information, and (b) information that comprehensively shows the corresponding quantitative evaluation value. According to this, details of the quality information based on the location information and attribute information of the 3D data are added to the metadata. As a result, the decoding device can evaluate the quality of the point cloud's location accuracy (e.g., D1 metrics or D2 metrics) or attributes (e.g., color information). Furthermore, the encoding device can flexibly manage the presence or absence of quality information and the type of quality information using various flags (3D_geo_metrics_flag or 3D_attr_metrics_flag) and type information. In addition, the encoding device can reduce the amount of bits. Furthermore, the decoding device can accurately extract the necessary quality information. This enables control of processing according to the quality requirements of each application.

[0587] For example, the quality information includes 2D quality evaluation parameters for a 2D rendered image generated based on the projection process of the 3D data. Here, the 2D quality evaluation parameters include information that comprehensively indicates whether or not evaluation is performed at each rendering viewpoint, the evaluation method, and the quantitative quality value in the evaluation method. As a result, the metadata includes quality information (e.g., 2D quality information) for a 2D rendered image obtained by projecting 3D data to a specific viewpoint position. Therefore, the decoding device can perform evaluation of display quality and evaluation for each viewpoint. The metadata also includes the presence or absence of 2D quality information, the rendering method used, and viewpoint information. As a result, the decoding device can select the optimal rendering viewpoint using the quality of each viewpoint before decoding. This makes it possible to realize an optimal processing flow according to the usage scenario (e.g., an application where 2D display is essential), leading to an improvement in the user experience.

[0588] For example, the quality information is not limited to quality information calculated after encoding, but also includes quality information calculated in the 3D data generation or processing process. The quality information further includes process identification information indicating the process in which the quality information was calculated. As a result, the metadata includes not only quality information calculated after encoding, but also quality information calculated at different stages, such as the 3D data generation or processing process. Therefore, the system can comprehensively manage quality variations in each process. Furthermore, the metadata includes process identification information. This allows the decoding device to understand at which process the quality information was calculated. Therefore, highly reliable quality evaluation and traceability are ensured. This enhances overall system quality control and promotes flexible decision-making and processing for diverse applications.

[0589] For example, the decoding device pre-evaluates the 3D or 2D quality of the 3D data by decoding the quality information, and based on the evaluation, decides whether to use the 3D data as is or to render the 3D data and use it as 2D data. This allows the decoding device to pre-evaluate the quality information and decode or render only data that meets a certain quality standard. As a result, the decoding device can avoid processing data that does not meet the quality standard. Therefore, unnecessary computational load or consumption of communication resources can be reduced. Consequently, the system's processing efficiency can be improved, power consumption reduced, and communication bandwidth saved, leading to an improved user experience.

[0590] For example, the metadata includes individual quality information for each of the multiple objects included in the 3D data, or quality information common to all of the multiple objects. This means that the metadata includes quality information common to each object or to the entire dataset of 3D data containing multiple objects. Therefore, granular quality evaluation is achieved. Furthermore, the decoding device can perform high-quality processing on only certain objects, omitting others, enabling partial decoding or rendering according to the application. This promotes the optimal allocation of system resources. Additionally, the decoding device receives detailed feedback based on the quality information of individual objects, improving the accuracy of data processing or correction.

[0591] For example, the quality information may include multiple quality values ​​as 3D quality information, such as D1 metrics, D2 metrics, Euclidean distance, and Manhattan distance, or multiple quality indicators as 2D quality information, such as PSNR, SSIM, and LPIPS. This allows for a variety of indicators, such as D1 metrics, D2 metrics, Euclidean distance, and Manhattan distance, as 3D quality information. Therefore, a multifaceted quality evaluation tailored to the application or situation is realized. Furthermore, multiple image quality indicators, such as PSNR, SSIM, and LPIPS, are used for 2D quality information. This enables more accurate quality checks. As a result, optimal quality management and utilization strategies can be adopted in various applications (e.g., medical image processing, VR / AR, and remote sensing).

[0592] For example, the quality information is added to the bitstream as an SEI message. This ensures compatibility between the technology and existing encoding standards, and allows for easy integration of the technology into standard technologies. Furthermore, the metadata method minimizes the overhead of the added information while transmitting sufficient quality information, thereby improving communication efficiency. In addition, it improves the ease of implementation of the entire system and reduces operating costs.

[0593] For example, the decoding device dynamically selects whether or not to decode or render the 3D data, or the processing path, based on the quality information. This allows the decoding device to achieve optimal processing based on real-time quality evaluation. It also promotes the efficient use of system resources. For example, by performing high-load processing only when necessary, the decoding device improves the overall processing efficiency of the system. Furthermore, by thinning out poor-quality data, the decoding device ensures that the final output to the user is of guaranteed quality, thereby improving reliability.

[0594] For example, based on the quality information, the decoding device evaluates the quality of the 3D data before decoding it and avoids decoding or rendering 3D data that does not meet the quality standards. As a result, the decoding device can evaluate the quality of the 3D data before decoding it, and therefore can avoid decoding or rendering data that does not meet the quality standards. Thus, unnecessary decoding or rendering processes can be eliminated, significantly reducing processing load, power consumption, and communication bandwidth. Furthermore, in real-time processing systems or large-scale systems, efficient resource management and rapid user interface response can be achieved. Thus, the overall user experience and system stability are improved.

[0595] For example, the metadata includes a 2D_metrics_flag. Here, if the 2D_metrics_flag is 1, the metadata includes the 2D quality information, and if the 2D_metrics_flag is 0, the metadata does not include the 2D quality information. As a result, the decoding device can quickly determine whether or not the metadata contains 2D quality information based on the 2D_metrics_flag. Therefore, the decoding device can avoid reading unnecessary information and can analyze the bitstream efficiently. In addition, since unnecessary 2D quality information in the bitstream can be omitted, the amount of data in the bitstream can be reduced.

[0596] For example, the metadata includes 2D_rendering_type_present_flag and 2D_rendering_type with respect to the 2D quality information. The decoding device interprets 2D_rendering_type as a predetermined default value (e.g., 0 indicating renderer 1) if the metadata does not include 2D_rendering_type. This allows the decoding device to obtain the recommended 2D rendering method (e.g., renderer 1 or renderer 2), related parameters, and command information using 2D_rendering_type_present_flag and 2D_rendering_type. Therefore, the decoding device can generate high-quality 2D images. Furthermore, the decoding device uses the default value when 2D_rendering_type is not included in the metadata. This enables a reduction in the number of bits and simplification of processing. Therefore, the efficiency of communication and data storage is improved.

[0597] For example, the metadata includes 2D_metrics_type_present_flag and 2D_metrics_type to indicate the type of 2D quality information. Here, 2D_metrics_type indicates a type such as PSNR, WS-PSNR, IV-PSNR, SSIM, IV-SSIM, or LPIPS. The metadata also includes software or parameter information used for quality evaluation as needed. This allows the encoding device to select appropriate 2D quality information from among multiple evaluation indices such as PSNR, SSIM, and LPIPS. Thus, flexible quality evaluation according to the application of 3D data becomes possible. Furthermore, by using the version or parameter information of the quality evaluation software, the reproducibility of evaluation results is improved, and quality comparisons in the decoding device become easier.

[0598] For example, the 2D quality information includes numView (or numView_minus1) indicating the number of rendering viewpoints, view_information corresponding to each viewpoint, and the quality value (3D_2D_metrics) of the 2D rendered image for each viewpoint. If the value of 3D_2D_metrics is 0, it indicates that the 2D quality information for that viewpoint could not be calculated or rendering is prohibited. This allows the decoder to use view_information and 3D_2D_metrics for each rendering viewpoint to determine which viewpoint's image should be rendered based on the quality of the 2D rendered images from multiple viewpoints. Furthermore, a value of 3D_2D_metrics of 0 clearly indicates that the 2D quality information for that viewpoint cannot be calculated or rendering is prohibited, thus preventing unnecessary processing or the generation of inappropriate images by the decoder. The decoder can also calculate an integrated evaluation (average value, etc.) of multiple viewpoints using the quality information for each viewpoint. Additionally, the amount of code in the header is reduced.

[0599] For example, the 2D quality information may be (a) provided in common to multiple objects present in the 3D data, or (b) provided separately for each of the multiple objects. The metadata may further include one or more quality evaluation types or rendering methods (e.g., multiple 2D_metrics_type or 2D_rendering_type). This allows for the provision of common or individual 2D quality information for multiple objects in the 3D data. As a result, optimal quality evaluation for each object becomes possible, improving the flexibility and accuracy of the system. Furthermore, since multiple quality evaluation types and rendering methods can be specified together, optimal 2D image generation or quality judgment can be performed according to the application or environment. As a result, the overall user experience and system efficiency are improved.

[0600] For example, the 3D quality information relating to the positional information of the 3D data includes 3D_geo_numView (or represented by 3D_geo_numView_minus1) based on 3D_geo_metrics_flag. The 3D quality information further includes 3D_geo_view_information for each viewpoint indicated by the value of 3D_geo_numView (e.g., viewport_id, extrinsic_flag, intrinsic_flag, camera pose information, and camera parameter information). The 3D quality information further includes 3D_geo_numDepth (or using 3D_geo_numDepth_minus1), which is the depth information of the 3D quality information at each viewpoint. The 3D quality information further includes 3D_geo_depth (distance from the viewpoint or range information such as a bounding box) at each depth j. The 3D quality information further includes 3D_geo_metrics (3D quality values ​​based on position information) at the viewpoint and depth.

[0601] As a result, the metadata includes detailed quality information for each viewpoint and distance (depth) relative to the location information of the 3D data. Therefore, changes in quality according to the distance from the camera position are accurately transmitted. Furthermore, the decoding device can select which viewpoint or distance data to use based on the quality for each viewpoint and depth. Therefore, the decoding device can achieve optimal rendering or display control. In addition, the decoding device can present only high-quality areas to the user. Therefore, the decoding device can provide highly reliable results with minimal image degradation.

[0602] For example, the 3D quality information relating to the attribute information of the 3D data includes the number of attributes numAttr (or numAttr_minus1) based on 3D_attr_metrics_flag. The 3D quality information further includes 3D_attr_metrics_type (when 3D_attr_metrics_type_present_flag is 1) for each attribute k. The 3D quality information further includes 3D_attr_numView (or 3D_attr_numView_minus1) for each attribute k. The 3D quality information further includes 3D_attr_view_information (camera information, etc., corresponding to each attribute) and 3D_attr_numDepth (or 3D_attr_numDepth_minus1), which is the depth of the attribute quality information for each viewpoint. The 3D quality information further includes (α) 3D_attr_depth (distance from the viewpoint or bounding box information) and (β) 3D_attr_metrics (3D quality values ​​based on attribute information) for each depth j.

[0603] As a result, the metadata includes attribute quality information for each attribute within the 3D data, as well as for each viewpoint and depth. Therefore, the decoding device can individually evaluate the quality of multiple attributes, such as color or reflectance. Furthermore, the decoding device can refer to the individual quality information for each attribute and perform processing tailored to the characteristics of the object or scene (for example, highlighting important attributes or presenting highly accurate measurement results). In addition, since attribute information is managed individually, the entire scene is optimized, improving the user experience.

[0604] For example, the number of viewpoints (3D_geo_numView, 3D_attr_numView) and the number of depths (3D_geo_numDepth, 3D_attr_numDepth) are represented by subtracting 1 from the actual value (e.g., 3D_geo_numView_minus1). This reduces the number of bits in the header. Furthermore, by encoding values ​​obtained by subtracting 1 from the actual values ​​of 3D_geo_numView, 3D_geo_numDepth, 3D_attr_numView, or 3D_attr_numDepth, the header information is efficiently compressed, reducing the overall bitrate. This minimizes the encoding load caused by the additional detailed quality information, improving the overall communication and storage efficiency of the system.

[0605] For example, the decoding device presents 3D data or rendered images within a depth range only if the 3D quality information within that depth range, based on the 3D_geo_depth or 3D_attr_depth information, is above a predetermined threshold. Alternatively, the decoding device avoids using the 3D data within that depth range if the 3D quality information within that depth range is below the threshold. This allows the decoding device to present high-quality and accurate results to the user. Furthermore, the decoding device can use the 3D_geo_depth and 3D_attr_depth information to set quality evaluation thresholds for each viewpoint or distance. Therefore, the decoding device can selectively use only high-quality data. This mechanism eliminates low-quality data, providing the user with highly accurate measurement results or high-quality rendered images. In addition, since the user is always presented with high-quality information, overall user confidence and satisfaction are improved.

[0606] For example, 3D quality information related to location and 3D quality information related to attribute information can be set independently or in common. This enables optimization of the header's encoding amount and flexible decoding and rendering process selection based on individual quality evaluations for each object or attribute. This increased flexibility allows for optimal encoding according to the scene configuration or number of objects. In addition, fine-grained quality control at the object or attribute level is achieved simultaneously with the optimization of the header's encoding amount. As a result, the decoder can make flexible processing selections (e.g., partial rendering or selective decoding as needed). Furthermore, the overall system processing load can be reduced. Finally, the user experience is improved.

[0607] For example, individual 2D quality information is provided for each viewpoint. For instance, each viewpoint is provided with its own view_information and 2D quality metrics (including signal-to-noise ratio). This allows the decoder to accurately understand the texture or image quality for each viewpoint, enabling it to select a high-quality viewpoint during playback. The decoder also extracts viewpoint images that meet a certain quality standard and uses these viewpoint images to train the 3D generation model. This improves the training accuracy.

[0608] For example, metadata includes information indicating a movable or renderable area (e.g., position or size information of the 3D bounding box) based on viewpoint information attached to the SEI. This allows the decoder to use the viewpoint information attached to the SEI and the above information to present a high-quality and safe viewpoint (movable area) to the user. Thus, the user can intuitively recognize and select a viewpoint. In this way, the usability of the user interface is improved. Furthermore, the rendering range desired by the user is clearly indicated, improving the overall visual experience.

[0609] For example, metadata includes information for generating alternative images in response to rendering-prohibited areas or rendering requests. This allows the decoder to avoid outputting video from viewpoints with low rendering quality, thus always presenting high-quality video. Furthermore, the decoder can reduce user confusion by presenting alternative images if the user mistakenly specifies a rendering-prohibited area.

[0610] For example, an encoding device generates a bitstream by encoding 3D data. The bitstream includes the encoded 3D data and additional information. The additional information includes 3D rendering hint information (3D rendering support information). In this way, the encoding device can include additional metadata in the bitstream by adding rendering support information (rendering hint information) when encoding 3D data. The decoding device can use the rendering hint information to optimize or flexibly utilize the rendering process.

[0611] For example, the 3D rendering hint information includes quality information for the 3D data. The decoding device controls the generation of high-quality rendered images or the suppression of low-quality rendering based on the quality information. This allows the decoding device to evaluate the quality of the data before rendering based on the quality information attached to the 3D data. As a result, the decoding device can generate high-quality rendered images. Furthermore, the decoding device can suppress inappropriate rendering.

[0612] For example, the 3D rendering hint information includes rendering-prohibited area information or lighting information. Based on the rendering-prohibited area information, the decoding device processes rendering requests from a specified area. For example, this processing may involve stopping the generation of a rendering image from the specified area or presenting an alternative image. This allows the decoding device to control rendering according to user requests. For example, the decoding device can prevent rendering from low-quality or unnecessary areas. As a result, overall video quality and service quality are improved.

[0613] For example, the rendering prohibited area information includes (a) a flag indicating the existence of a prohibited area, (b) information indicating the number of prohibited areas (e.g., numArea or numArea_minus1), and (c) at least the center position (e.g., position(x, y, z)) and shape information (e.g., radius radius in the case of a sphere) for each prohibited area. In this way, the encoding device can precisely specify areas where rendering is inappropriate by adding the presence or absence of prohibited areas, the number of prohibited areas, and the center position and shape information of each area to the metadata. As a result, the decoding device can accurately determine where rendering is prohibited, and can effectively suppress rendering from locations that it does not want to present to the user.

[0614] For example, the lighting information includes (a) a flag (lighting_info_present_flag) indicating the presence of lighting information, and (b) lighting parameters for each viewpoint applied during rendering (e.g., viewpoint information (view_information) and lighting data (lighting_data)). This ensures that appropriate lighting conditions are reflected during rendering, improving the reproducibility of the scene's brightness, color, and shadows. It also enhances the overall quality of the rendered image, resulting in a more realistic visual representation.

[0615] For example, the 3D rendering hint information further includes overhead view information (e.g., birdey_view_information) for generating an overhead view image of the 3D data. This allows the user to get an overview of the entire scene, facilitating navigation assistance and enabling the user to grasp the overall scene. Thus, user operability and interactivity are improved.

[0616] For example, the additional information is added to the bitstream as an SEI (Supplemental Enhancement Information) message. The SEI message conforms to a syntax that allows the insertion of various variations of 3D rendering hint information (3D quality information, rendering prohibited area information, lighting information, and overhead view information, etc.). This allows the system to flexibly apply various variations of 3D rendering hint information (quality information, rendering prohibited information, lighting information, and overhead view information), providing a mechanism that can accommodate different scenes or future expansions. Furthermore, the standard-compliant syntax ensures proper processing by the decoding device.

[0617] For example, the decoding device dynamically selects or controls the rendering process of 3D data based on quality information, rendering-prohibited area information, or lighting information included in the additional information. This improves real-time performance and ensures that users always receive high-quality rendered images. As a result, the overall system performance and user experience are improved.

[0618] For example, the 3D rendering hint information includes rendering viewpoint control information. The rendering viewpoint control information includes at least one of (1) rendering prohibition setting information for prohibiting rendering output from a specific direction or region, and (2) an allowable travel distance parameter for limiting the travel distance between adjacent viewpoints. This prevents low-quality rendering by pre-controlling the rendering output from a specific direction or region. Furthermore, the use of a parameter that limits the travel distance between adjacent viewpoints facilitates the reuse of the rendering process.

[0619] For example, the rendering viewpoint control information includes (a) rendering prohibition information for prohibiting rendering of a viewpoint or direction when the value of 3D_2D_metrics is equal to a predetermined threshold, and (b) gaze movement recommendation information based on a quality evaluation index for each viewpoint. This enables quality control of the rendered image by using the rendering prohibition information based on the 3D_2D_metrics threshold. In addition, the gaze movement recommendation information based on a quality evaluation index for each viewpoint supports the selection of the optimal viewpoint.

[0620] For example, the rendering viewpoint control information includes recommended viewpoint movement information and parameter information related to viewpoint movement. This makes user operation easier when switching viewpoints and suppresses display delays.

[0621] For example, the 3D rendering hint information includes rendering assistance technical parameter information. The rendering assistance technical parameter information includes at least one of (a) rendering engine information, (b) projection matrix information, (c) rendering parameter information, and (d) inverse transformation information. This improves the accuracy and reproducibility of 3D-2D conversion. Furthermore, by flexibly combining multiple technical parameters, it is possible to generate images that are suitable for various scenes or rendering environments.

[0622] For example, based on the 3D rendering hint information, when a user requests rendering from a specific viewpoint, the decoding device performs a first determination step of determining whether the viewpoint is based on overhead view information, and a second determination step of determining whether the viewpoint falls within a rendering prohibited area. This allows the decoding device to respond flexibly and efficiently to the user's rendering request through a hierarchical determination flow. Furthermore, the decoding device can select different rendering paths based on each determination step and provide the optimal rendering method.

[0623] For example, the 3D rendering hint information includes quality evaluation information for evaluating the quality of the rendered image. Based on the quality evaluation information, the decoding device performs a quality evaluation step to determine whether the quality of the rendered image presented to the user meets a predetermined standard. This allows the decoding device to prevent the output of low-quality rendered images by pre-evaluating the quality of the rendered image, thereby guaranteeing a minimum standard of image quality for the image presented to the user. Furthermore, processing based on the quality evaluation information enables efficient use of processing resources.

[0624] For example, the 3D rendering hint information includes alternative display information for providing alternative display means when rendering is prohibited. Based on the alternative display information, the decoding device performs an alternative display step to display pre-prepared alternative content if the requested viewpoint is included in the rendering prohibited area. This allows the decoding device to continue the user experience without interruption by providing alternative display means when rendering is prohibited. Furthermore, the use of pre-prepared alternative content enables smooth display switching, thereby improving user satisfaction.

[0625] Furthermore, the encoding device according to the embodiment performs the processing shown in Figure 83. Figure 83 is a flowchart of the processing performed by the encoding device. The encoding device includes, for example, the GS encoding unit 502 shown in Figure 56. The encoding device may also further include a GS generation unit 501.

[0626] The encoding device generates encoded three-dimensional data by encoding the three-dimensional data (S321), and generates a bitstream including the encoded three-dimensional data and additional information (e.g., metadata or SEI) which includes first information (e.g., rendering assistance information) related to rendering the three-dimensional data.

[0627] According to this, a data processing device that renders decoded three-dimensional data can perform rendering appropriately using the first information. Therefore, further improvements can be achieved, such as reducing the processing load in the data processing device or improving the quality of the rendered image. The data processing device may also include a decoding device that decodes the encoded three-dimensional data.

[0628] For example, the first piece of information includes information indicating the range of viewpoints where rendering is prohibited (e.g., rendering prohibited area information). This can, for example, prevent the generation of low-quality rendered images.

[0629] For example, the second piece of information includes the viewpoint position (e.g., position(x, y, z)) and information indicating the distance from that viewpoint position (e.g., radius), and rendering from viewpoints within the distance from the viewpoint position is prohibited. This allows for appropriate control over the range in which rendering is prohibited.

[0630] For example, the second piece of information includes the viewpoint position and information indicating the distance from that viewpoint position, and rendering from viewpoints within a certain distance range from the viewpoint position is prohibited. This allows for fine-grained control over which rendering is prohibited.

[0631] For example, the first piece of information includes information indicating the quality of the two-dimensional image generated from the three-dimensional data for each viewpoint (e.g., 2D quality information or 3D_2D_metrics), and rendering from viewpoints where the quality is set to or below a predetermined value (e.g., zero) is prohibited. This allows the region where rendering is prohibited to be controlled based on the information indicating the quality of the two-dimensional image. Therefore, since there is no need to add new information to the bitstream, the amount of data in the bitstream can be reduced.

[0632] For example, the first piece of information includes a third piece of information (e.g., lighting information) related to the lighting used for rendering. This allows the data processing device to improve the quality of the rendered image by using the third piece of information.

[0633] For example, third-party information is provided for each viewpoint. This allows the data processing device to improve the quality of the rendered image by using this third-party information for each viewpoint.

[0634] For example, the first piece of information includes information indicating the viewpoint used to generate an overhead view of the three-dimensional data (e.g., overhead viewpoint information). This allows the data processing device to easily generate the overhead view. Therefore, the processing load on the data processing device can be reduced.

[0635] For example, the first piece of information includes information indicating the distance between viewpoints that is permissible during viewpoint movement. This helps to suppress large changes in the viewpoint of the rendered image. Therefore, the processing load on the data processing device can be reduced and the processing speed can be improved.

[0636] For example, the first piece of information includes information about the rendering engine (e.g., engine information). This allows the data processing device to improve the quality or reproducibility of the rendered image by using this information.

[0637] For example, the first piece of information includes parameters for projecting three-dimensional data onto a two-dimensional image. This allows the data processing device to improve the quality or reproducibility of the rendered image by using this information.

[0638] For example, the first piece of information includes parameters for inversely transforming a two-dimensional image into three-dimensional data. This allows the data processing device to improve the quality or reproducibility of the rendered image by using this information.

[0639] For example, the additional information is an SEI (Supplemental Enhancement Information) message. This ensures, for instance, compatibility between the generated bitstream and existing encoding standards.

[0640] For example, three-dimensional data includes data for multiple objects, and the first information includes information indicating the quality of each of the object's data (e.g., quality information). This allows the data processing device to acquire the quality for each object, enabling detailed control based on that quality.

[0641] For example, the first piece of information includes a fourth piece of information (3D geometry quality or 3D_geo_metrics) for each viewpoint that indicates the quality of the three-dimensional position information of the three-dimensional data. This allows the data processing device to acquire the quality of the three-dimensional position information for each viewpoint, enabling detailed control based on that quality.

[0642] For example, each of the multiple fourth pieces of information includes information indicating the viewpoint position (e.g., 3D_geo_view_information), information indicating the range of the three-dimensional space relative to the viewpoint position (e.g., 3D_geo_depth), and information indicating the quality of the three-dimensional position information of the three-dimensional data within the range (e.g., 3D_geo_metrics). This allows the data processing device to acquire the quality of the three-dimensional position information for each spatial range corresponding to the viewpoint, thereby enabling detailed control based on that quality.

[0643] For example, the first piece of information includes a fifth piece of information (e.g., 3D attribute quality or 3D_attr_metrics) for each viewpoint that indicates the quality of the attribute information of the three-dimensional data. This allows the data processing device to acquire the quality of the attribute information for each viewpoint, enabling detailed control based on that quality.

[0644] For example, each of the multiple fifth pieces of information includes information indicating the viewpoint position (e.g., 3D_attr_view_information), information indicating the range of the three-dimensional space relative to the viewpoint position (e.g., 3D_attr_depth), and information indicating the quality of the attribute information of the three-dimensional data within the range (e.g., 3D_attr_metrics). This allows the data processing device to acquire the quality of attribute information for each spatial range corresponding to the viewpoint, thereby enabling detailed control based on that quality.

[0645] For example, the first piece of information is common to the three-dimensional position information and attribute information of the three-dimensional data for each viewpoint, and includes information indicating the quality of the three-dimensional position information and attribute information of the three-dimensional data. This reduces the amount of data compared to the case where quality information is generated separately for the position information and attribute information.

[0646] Figure 84 is a block diagram of the encoding device 10. For example, the encoding device 10 comprises a processor 11 and a memory 12, and the processor 11 performs the above processing using the memory 12.

[0647] Furthermore, the decoding device according to the embodiment performs the processing shown in Figure 85. Figure 85 is a flowchart of the processing performed by the decoding device. The decoding device includes, for example, the GS decoding unit 503 shown in Figure 56. The decoding device may further include at least one of the rendering unit 505 and the measurement unit 506.

[0648] The decoding device generates three-dimensional data by decoding the encoded three-dimensional data contained in the bitstream (S331), and obtains first information related to the rendering of the three-dimensional data (e.g., rendering assistance information) from additional information contained in the bitstream (e.g., metadata or SEI) (S332).

[0649] According to this, a data processing device that renders decoded three-dimensional data can perform rendering appropriately using the first information. Therefore, further improvements can be achieved, such as reducing the processing load in the data processing device or improving the quality of the rendered image. The data processing device may also include a decoding device that decodes the encoded three-dimensional data.

[0650] For example, the first piece of information includes information indicating the range of viewpoints where rendering is prohibited (e.g., rendering prohibited area information). This can, for example, prevent the generation of low-quality rendered images.

[0651] For example, the second piece of information includes the viewpoint position (e.g., position(x, y, z)) and information indicating the distance from that viewpoint position (e.g., radius), and rendering from viewpoints within the distance from the viewpoint position is prohibited. This allows for appropriate control over the range in which rendering is prohibited.

[0652] For example, the second piece of information includes the viewpoint position and information indicating the distance from that viewpoint position, and rendering from viewpoints within a certain distance range from the viewpoint position is prohibited. This allows for fine-grained control over which rendering is prohibited.

[0653] For example, the first piece of information includes information indicating the quality of the two-dimensional image generated from the three-dimensional data for each viewpoint (e.g., 2D quality information or 3D_2D_metrics), and rendering from viewpoints where the quality is set to or below a predetermined value (e.g., zero) is prohibited. This allows the region where rendering is prohibited to be controlled based on the information indicating the quality of the two-dimensional image. Therefore, since there is no need to add new information to the bitstream, the amount of data in the bitstream can be reduced.

[0654] For example, the first piece of information includes a third piece of information (e.g., lighting information) related to the lighting used for rendering. This allows the data processing device to improve the quality of the rendered image by using the third piece of information.

[0655] For example, third-party information is provided for each viewpoint. This allows the data processing device to improve the quality of the rendered image by using this third-party information for each viewpoint.

[0656] For example, the first piece of information includes information indicating the viewpoint used to generate an overhead view of the three-dimensional data (e.g., overhead viewpoint information). This allows the data processing device to easily generate the overhead view. Therefore, the processing load on the data processing device can be reduced.

[0657] For example, the first piece of information includes information indicating the distance between viewpoints that is permissible during viewpoint movement. This helps to suppress large changes in the viewpoint of the rendered image. Therefore, the processing load on the data processing device can be reduced and the processing speed can be improved.

[0658] For example, the first piece of information includes information about the rendering engine (e.g., engine information). This allows the data processing device to improve the quality or reproducibility of the rendered image by using this information.

[0659] For example, the first piece of information includes parameters for projecting three-dimensional data onto a two-dimensional image. This allows the data processing device to improve the quality or reproducibility of the rendered image by using this information.

[0660] For example, the first piece of information includes parameters for inversely transforming a two-dimensional image into three-dimensional data. This allows the data processing device to improve the quality or reproducibility of the rendered image by using this information.

[0661] For example, the additional information is an SEI (Supplemental Enhancement Information) message. This ensures, for instance, compatibility between the generated bitstream and existing encoding standards.

[0662] For example, three-dimensional data includes data for multiple objects, and the first information includes information indicating the quality of each of the object's data (e.g., quality information). This allows the data processing device to acquire the quality for each object, enabling detailed control based on that quality.

[0663] For example, the first piece of information includes a fourth piece of information (3D geometry quality or 3D_geo_metrics) for each viewpoint that indicates the quality of the three-dimensional position information of the three-dimensional data. This allows the data processing device to acquire the quality of the three-dimensional position information for each viewpoint, enabling detailed control based on that quality.

[0664] For example, each of the multiple fourth pieces of information includes information indicating the viewpoint position (e.g., 3D_geo_view_information), information indicating the range of the three-dimensional space relative to the viewpoint position (e.g., 3D_geo_depth), and information indicating the quality of the three-dimensional position information of the three-dimensional data within the range (e.g., 3D_geo_metrics). This allows the data processing device to acquire the quality of the three-dimensional position information for each spatial range corresponding to the viewpoint, thereby enabling detailed control based on that quality.

[0665] For example, the first piece of information includes a fifth piece of information (e.g., 3D attribute quality or 3D_attr_metrics) for each viewpoint that indicates the quality of the attribute information of the three-dimensional data. This allows the data processing device to acquire the quality of the attribute information for each viewpoint, enabling detailed control based on that quality.

[0666] For example, each of the multiple fifth pieces of information includes information indicating the viewpoint position (e.g., 3D_attr_view_information), information indicating the range of the three-dimensional space relative to the viewpoint position (e.g., 3D_attr_depth), and information indicating the quality of the attribute information of the three-dimensional data within the range (e.g., 3D_attr_metrics). This allows the data processing device to acquire the quality of attribute information for each spatial range corresponding to the viewpoint, thereby enabling detailed control based on that quality.

[0667] For example, the first piece of information is common to the three-dimensional position information and attribute information of the three-dimensional data for each viewpoint, and includes information indicating the quality of the three-dimensional position information and attribute information of the three-dimensional data. This reduces the amount of data compared to the case where quality information is generated separately for the position information and attribute information.

[0668] Figure 86 is a block diagram of the decoding device 20. For example, the decoding device 20 comprises a processor 21 and a memory 22, and the processor 21 performs the above processing using the memory 22.

[0669] The above describes the encoding device (three-dimensional data encoding device) and decoding device (three-dimensional data decoding device), etc., according to embodiments and modifications of the present disclosure. However, the present disclosure is not limited to these embodiments.

[0670] Furthermore, each processing unit included in the encoding device and decoding device, etc., according to the above embodiment is typically implemented as an LSI, which is an integrated circuit. These may be individually integrated into a single chip, or some or all of them may be integrated into a single chip.

[0671] Furthermore, integrated circuit implementation is not limited to LSIs; it may also be achieved using dedicated circuits or general-purpose processors. Alternatively, an FPGA (Field Programmable Gate Array), which can be programmed after LSI manufacturing, or a reconfigurable processor capable of reconfiguring the connections and settings of circuit cells within the LSI, may be used.

[0672] Furthermore, in each of the above embodiments, each component may be implemented by being composed of dedicated hardware or by executing a software program suitable for each component. Each component may also be implemented by a program execution unit such as a CPU or processor reading and executing a software program recorded on a recording medium such as a hard disk or semiconductor memory.

[0673] Furthermore, this disclosure may be implemented as an encoding method (three-dimensional data encoding method) or a decoding method (three-dimensional data decoding method), etc., performed by an encoding device (three-dimensional data encoding device) and a decoding device (three-dimensional data decoding device), etc.

[0674] Furthermore, this disclosure may be implemented as a data generation device or data generation method for generating three-dimensional data, or as a data processing device or data processing method for processing three-dimensional data. The data generation device may or may not include an encoding device for encoding the three-dimensional data. The data processing device may or may not include a decoding device for decoding the three-dimensional data.

[0675] Furthermore, this disclosure may be implemented as a program that causes a computer, processor, or device to execute the above-described encoding or decoding method. Alternatively, this disclosure may be implemented as a bitstream generated by the above-described encoding method. Furthermore, this disclosure may be implemented as a recording medium on which the program or the bitstream is recorded. For example, this disclosure may be implemented as a non-temporary, computer-readable recording medium on which the program or the bitstream is recorded.

[0676] Furthermore, the division of functional blocks in the block diagram is just one example; multiple functional blocks can be implemented as a single functional block, a single functional block can be divided into multiple parts, or some functions can be moved to other functional blocks. In addition, the functions of multiple functional blocks with similar functions can be processed in parallel or time-sharing by a single piece of hardware or software.

[0677] Furthermore, the order in which each step in the flowchart is performed is illustrative for the purpose of specifically illustrating this disclosure, and may be in a different order. Also, some of the above steps may be performed simultaneously (in parallel) with other steps.

[0678] Although one or more embodiments of encoding and decoding devices, etc., have been described above based on embodiments, this disclosure is not limited to these embodiments. Without departing from the spirit of this disclosure, various modifications that a person skilled in the art could conceive of are applied to these embodiments, and forms constructed by combining components from different embodiments may also be included within the scope of one or more embodiments.

[0679] This disclosure is applicable to encoding and decoding devices.

[0680] 10 Encoding device 11, 21 Processor 12, 22 Memory 20 Decoding device 200 Terminal 201 Decoding unit 202 Synchronization display unit 301 Sensor information input unit 302 Three-dimensional data generation unit 303, 305 Rendering unit 304 Spherical harmonics 310 Gaussian data generation unit 311 Encoding unit 312 Multiplexing unit 321 Demultiplexing unit 322 Decoding unit 323 Application unit 324 Input interface unit 325 Rendering unit 326 Display unit 330 Encoding device 331 First preprocessing unit 332 Second preprocessing unit 333 G-PCC encoding unit 341 Position information encoding unit 342 Rotation encoding unit 343 Scale encoding unit 344 SH coefficient encoding unit 345 Transmittance encoding unit 346 Metadata encoding unit 347 Multiplexing unit 350 Decoder 351 G-PCC decoding unit 352 Second post-processing unit 353 First post-processing unit 361 Demultiplexing unit 362 Position information decoding unit 363 Rotation decoding unit 364 Scale decoding unit 365 SH coefficient decoding unit 366 Transmittance decoding unit 367 Metadata decoding unit 501, 531 GS generation unit 502, 532 GS coding unit 503, 533 GS decoding unit 504, 534, 535, 541 Application 505, 536 Rendering unit 506, 537 Measurement unit 511 Learning unit 512, 522 Quality measurement unit 521 Coding unit 542 Terminal 1001 Three-dimensional data coding system 1002 Three-dimensional data decoding system 1003 Sensor terminal 1004 External connection unit 1011 Three-dimensional data generation system 1012 Presentation unit 1013 Encoding unit 1014 Multiplexing unit 1015 Input / output unit 1016 Control unit 1017 Sensor information acquisition unit 1018 Three-dimensional data generation unit 1021 Sensor information acquisition unit 1022 Input / output unit 1023 Demultiplexing unit 1024 Decoding unit 1025 Presentation unit 1026 User interface 1027 Control unit 1031 Three-dimensional model learning unit 1032 Three-dimensional model encoding unit 1033 Three-dimensional model decoding unit 1034 Rendering reconstruction unit 1041 Data partitioning unit1042 Encoding Unit 1051 Decoding Unit 1052 Data Merging Unit 1070 Server 1071 Data Generation Unit 1072 Point Cloud Generation Unit 1073 Mesh Generation Unit 1074 Model Generation Unit 1075 Synchronization Unit 1076 Point Cloud Encoding Unit 1077 Mesh Encoding Unit 1078 Model Encoding Unit 1079 Multiplexing Unit 1080 Data Extraction Unit 1090 Terminal 1091 Control Unit 1092 Decoding Unit 1093 Display Unit 1101 Point Cloud Sensor 1102 Camera 1110 Data Generation Unit 1111 Point Cloud Generation Unit 1112 Mesh Generation Unit 1113 Model Generation Unit 1143 Encoding Unit 1144 Formatting Unit 1145 Reverse Formatting Unit 1146 Decoding Unit

Claims

1. An encoding method that generates encoded three-dimensional data by encoding three-dimensional data, and generates a bitstream that includes additional information including first information relating to the rendering of the three-dimensional data, and the encoded three-dimensional data.

2. The encoding method according to claim 1, wherein the first information includes second information indicating a range of viewpoints from which rendering is prohibited.

3. The encoding method according to claim 2, wherein the second information includes a viewpoint position and information indicating the distance from said viewpoint position, and rendering from viewpoints within the range of the distance from said viewpoint position is prohibited.

4. The encoding method according to claim 2, wherein the second information includes information indicating the range and the line of sight direction, and rendering from a viewpoint within the range to the line of sight direction is prohibited.

5. The encoding method according to claim 1, wherein the first information includes information indicating the quality of a two-dimensional image generated from the three-dimensional data for each viewpoint, and rendering from viewpoints where the quality is set to or below a predetermined value is prohibited.

6. The encoding method according to claim 1, wherein the first information includes a third information relating to lighting used for rendering.

7. The encoding method according to claim 6, wherein the third information is provided for each viewpoint.

8. The encoding method according to claim 1, wherein the first information includes information indicating a viewpoint used to generate an overhead image of the three-dimensional data.

9. The encoding method according to claim 1, wherein the first information includes information indicating the distance between viewpoints that is permissible with respect to viewpoint movement.

10. The encoding method according to claim 1, wherein the first information includes information about an engine used for rendering.

11. The encoding method according to claim 1, wherein the first information includes parameters for projecting the three-dimensional data onto a two-dimensional image.

12. The encoding method according to claim 1, wherein the first information includes parameters for inversely transforming a two-dimensional image into three-dimensional data.

13. The encoding method according to claim 1, wherein the additional information is an SEI (Supplemental Enhancement Information) message.

14. The encoding method according to claim 1, wherein the three-dimensional data includes data of a plurality of objects, and the first information includes information indicating the quality of each of the data of the plurality of objects.

15. The encoding method according to claim 1, wherein the first information includes a fourth piece of information indicating the quality of the three-dimensional position information of the three-dimensional data for each viewpoint.

16. The encoding method according to claim 15, wherein each of the plurality of fourth pieces of information includes information indicating a viewpoint position, information indicating a range of three-dimensional space based on the viewpoint position, and information indicating the quality of the three-dimensional position information of the three-dimensional data within the range.

17. The encoding method according to claim 1, wherein the first information includes a fifth piece of information indicating the quality of the attribute information of the three-dimensional data for each viewpoint.

18. The encoding method according to claim 17, wherein each of the plurality of fifth pieces of information includes information indicating a viewpoint position, information indicating a range of three-dimensional space based on the viewpoint position, and information indicating the quality of attribute information of the three-dimensional data within the range.

19. The encoding method according to claim 1, wherein the first information is common to the three-dimensional position information and attribute information of the three-dimensional data for each viewpoint, and includes information indicating the quality of the three-dimensional position information and attribute information of the three-dimensional data.

20. A decoding method for generating three-dimensional data by decoding encoded three-dimensional data contained in a bitstream, and obtaining first information relating to the rendering of the three-dimensional data from additional information contained in the bitstream.

21. The decoding method according to claim 20, wherein the first information includes second information indicating a range of viewpoints for which rendering is prohibited.

22. The decoding method according to claim 21, wherein the second information includes a viewpoint position and information indicating the distance from said viewpoint position, and rendering from viewpoints within the range of the distance from said viewpoint position is prohibited.

23. The decoding method according to claim 21, wherein the second information includes information indicating the range and the line of sight direction, and rendering from a viewpoint within the range to the line of sight direction is prohibited.

24. The decoding method according to claim 20, wherein the first information includes information indicating the quality of a two-dimensional image generated from the three-dimensional data for each viewpoint, and rendering from viewpoints where the quality is set to or below a predetermined value is prohibited.

25. The decoding method according to claim 20, wherein the first information includes third information relating to lighting used for rendering.

26. The decoding method according to claim 25, wherein the third information is provided for each viewpoint.

27. The decoding method according to claim 20, wherein the first information includes information indicating a viewpoint used to generate an overhead image of the three-dimensional data.

28. The decoding method according to claim 20, wherein the first information includes information indicating the distance between viewpoints that is permissible with respect to viewpoint movement.

29. The decoding method according to claim 20, wherein the first information includes information about an engine used for rendering.

30. The decoding method according to claim 20, wherein the first information includes parameters for projecting the three-dimensional data onto a two-dimensional image.

31. The decoding method according to claim 20, wherein the first information includes parameters for inversely transforming the two-dimensional image into the three-dimensional data.

32. The decoding method according to claim 20, wherein the additional information is an SEI (Supplemental Enhancement Information) message.

33. The decoding method according to claim 20, wherein the three-dimensional data includes data of a plurality of objects, and the first information includes information indicating the quality of each of the data of the plurality of objects.

34. The decoding method according to claim 20, wherein the first information includes a fourth piece of information indicating the quality of the three-dimensional position information of the three-dimensional data for each viewpoint.

35. The decoding method according to claim 34, wherein each of the plurality of fourth pieces of information includes information indicating a viewpoint position, information indicating a range of three-dimensional space based on the viewpoint position, and information indicating the quality of the three-dimensional position information of the three-dimensional data within the range.

36. The decoding method according to claim 20, wherein the first information includes a fifth piece of information indicating the quality of the attribute information of the three-dimensional data for each viewpoint.

37. The decoding method according to claim 36, wherein each of the plurality of fifth pieces of information includes information indicating a viewpoint position, information indicating a range of three-dimensional space based on the viewpoint position, and information indicating the quality of attribute information of the three-dimensional data within the range.

38. The decoding method according to claim 20, wherein the first information is common to the three-dimensional position information and attribute information of the three-dimensional data for each viewpoint, and includes information indicating the quality of the three-dimensional position information and attribute information of the three-dimensional data.

39. An encoding device comprising a processor and memory, wherein the processor generates encoded three-dimensional data by encoding three-dimensional data using the memory, and generates a bitstream including additional information including first information relating to the rendering of the three-dimensional data and the encoded three-dimensional data.

40. A decoding device comprising a processor and memory, wherein the processor generates three-dimensional data by decoding encoded three-dimensional data contained in a bitstream using the memory, and obtains first information relating to the rendering of the three-dimensional data from additional information contained in the bitstream.