Information processing device and method

Abstract

The present disclosure relates to an information processing device and method that make it possible to control the quality of a reconstructed 3D model. The device and method generate a scene description including a quality control information buffer format that defines a buffer for storing quality control information pertaining to the quality of the reconstructed 3D model. In addition, on the basis of the scene description, the device and method acquire content data and store the quality control information included in the acquired content data in the buffer. Then, on the basis of the scene description, the device and method read the quality control information from the buffer and reconstruct a 3D model of the quality corresponding to the read quality control information by using the content data. The present disclosure is applicable, for example, to an information processing device, an information processing method, an information processing system, or the like.

Description

Information processing device and method

[0001] This disclosure relates to an information processing device and method, and more particularly to an information processing device and method that can suppress a reduction in the processing efficiency of the decoding process.

[0002] Conventionally, there was glTF (The GL Transmission Format) (registered trademark) 2.0, a scene description format for arranging and rendering 3D objects in three-dimensional space (see, for example, Non-Patent Document 1). Furthermore, in MPEG-I scene descriptions (MPEG (Moving Picture Experts Group)-I Scene Description), an extension of glTF 2.0 has been proposed to support 6DoF scene streaming (see, for example, Non-Patent Document 2 and Patent Document 1).

[0003] Incidentally, a codec using Visual Volumetric Video-Based Coding (also known as the V3C codec), a technology that compresses 3D objects generated by camera capture using a video codec, has been proposed (see, for example, Non-Patent Documents 3 to 5). By applying this V3C codec, it is possible to transmit 3D objects with a higher compression ratio than conventional methods, and support for this V3C codec is being advanced in MPEG-I scene descriptions as one of the 3D object encoding technologies that constitute a scene. In this V3C codec, quality control information contained in the bitstream of atlas information can be used to improve the quality of the 3D model during the 3D model reconstruction process.

[0004] Also, the client processing of MPEG-I scene description consists of two processing blocks: a Presentation Engine (PE) and a Media Access Function (MAF). The V3C codec, which is a compression codec for 3D models, supports a method of reconstructing 3D models in the PE (PE reconstruction).

[0005] Saurabh Bhatia, Patrick Cozzi, Alexey Knyazev, Tony Parisi, "Khronos glTF2.0", https: / / github.com / KhronosGroup / glTF / tree / master / specification / 2.0, June 9, 2017"Information technology - Coded representation of immersive media -Part 14:Scene description Technologies de l'information - Representation codee de media immersifs - Partie 14: Description de scenes", ISO / IEC DIS 23090-14:2024(E), MDS23831_WG03_N01221_23090-14_2nd_edition, MPEG 146th meeting, 2024-04, WG03"Information technology - Coded representation of immersive media - Part 5: Visual volumetric video-based coding (V3C) and video-based point cloud compression (V-PCC)", A ISO / IEC DIS 23090-5:2024(en), ISO / IEC JTC 1 / SC 29 / WG 07, 2024-11-08, Secretariat: JISC, WG07N1038 FDIS 23090-5, MPEG 148th meeting, 2024-11, WG07"Information technology - Coded representation of immersive media - Part 12: MPEG Immersive video", ISO / IEC 23090-12:2021(E) / AMD.1:2022, ISO / IEC JTC 1 / SC 29, Secretariat: JISC, WG04N0270 FDIS 23090-12, MPEG 140th meeting, 2022-10, WG04"Information technology - Coded representation of immersive media - Part 29: Video-based dynamic mesh coding (V-DMC)", ISO 23090-29:2024(E), ISO / IEC JTC 1 / SC 29 / WG 7, Date: 2024-12-13, MDS24469_WG07_N01027, MPEG 148th meeting, 2024-11, WG07.

[0006] International Publication No. 2024 / 190805

[0007] However, in the case of PE reconstruction, the buffer that stores atlas information only has an area for storing patch data. Therefore, it was not possible to transmit quality control information contained in the atlas information from MAF to PE. Consequently, it was not possible to utilize quality control information in the PE reconstruction process, making it difficult to generate 3D models of a quality that matched the creator's intentions. In other words, it was not possible to control the quality of the reconstructed 3D model (the 3D model generated by the PE reconstruction process).

[0008] This disclosure is made in light of these circumstances and aims to enable control over the quality of the reconstructed 3D model.

[0009] One aspect of this technology is an information processing device that includes a scene description generation unit that generates a scene description including a quality control information buffer format that defines a buffer for storing quality control information regarding the quality of the reconstructed 3D model.

[0010] One aspect of this technology is an information processing method that generates a scene description including a quality control information buffer format that defines a buffer for storing quality control information related to the quality of the reconstructed 3D model.

[0011] An information processing device for another aspect of this technology includes a content processing unit that acquires content data based on a scene description including a quality control information buffer format that defines a buffer for storing quality control information regarding the quality of the reconstructed 3D model, and stores the quality control information contained in the acquired content data in the buffer; and a reconstruction unit that reads the quality control information from the buffer based on the scene description, and reconstructs the 3D model with a quality corresponding to the read quality control information using the content data.

[0012] Another aspect of this technology is an information processing method that includes acquiring content data based on a scene description which includes a quality control information buffer format which defines a buffer for storing quality control information relating to the quality of the reconstructed 3D model, storing the quality control information contained in the acquired content data in the buffer, and reading the quality control information from the buffer based on the scene description, and using the content data to reconstruct the 3D model with a quality corresponding to the read quality control information.

[0013] In one aspect of this technology, the information processing device and method generate a scene description that includes a quality control information buffer format that defines a buffer for storing quality control information regarding the quality of the reconstructed 3D model.

[0014] In other aspects of this technology, the information processing device and method include the following processes: acquiring content data based on a scene description which includes a quality control information buffer format that defines a buffer for storing quality control information relating to the quality of the reconstructed 3D model; storing the quality control information contained in the acquired content data in the buffer; reading the quality control information from the buffer based on the scene description; and using the content data to reconstruct a 3D model with a quality corresponding to the read quality control information.

[0015] This figure shows the main configuration example of glTF2.0. This figure shows an example of glTF objects and reference relationships. This figure shows an example of scene description writing. This figure explains how to access binary data. This figure shows an example of scene description writing. This figure explains how to extend objects. This figure explains the configuration of client processing in MPEG-I scene descriptions. This figure shows an example of extension configuration for handling timed metadata. This figure shows an example of scene description writing. This figure explains the configuration of client processing in MPEG-I scene descriptions to which V-PCC is applied. This figure shows an example of scene description structure in the case of MAF reconstruction processing. This figure shows an example of scene description structure in the case of PE reconstruction processing. This figure shows an example of buffer format in MPEG-I scene descriptions to which the V3C codec is applied. This figure shows an example of buffer format data structure that defines a buffer to store baseline atlas information. This figure shows an example of buffer format data structure that defines a buffer to store extended atlas information. This figure shows an example of buffer format data structure that defines a buffer to store atlas information for MIV. This figure explains the configuration of client processing in MPEG-I scene descriptions to which V-DMC is applied. This figure shows an example of the scene description structure in the case of PE reconstruction processing. This figure shows an example of a buffer format that defines a buffer to store atlas information when V-DMC is applied. This figure shows an example of a decode block diagram of V3C data. This figure shows an example of quality control information for V-PCC. This figure shows an example of the syntax of quality control information for V-PCC. This figure shows an example of the syntax of quality control information for V-PCC. This figure shows an example of quality control information for MIV. This figure shows an example of the syntax of quality control information for MIV. This figure shows an example of the syntax of quality control information for MIV. This figure shows an example of the syntax of quality control information for MIV.This figure shows an example of quality control information for V-DMC. This figure shows an example of the syntax for quality control information for V-DMC. This figure shows an example of the syntax for quality control information for V-DMC. This figure shows an example of the syntax for quality control information for V-DMC. This figure shows an example of the MAF reconstruction process. This figure shows an example of the PE reconstruction process. This figure shows an example of how quality control information is transmitted from MAF to PE. This figure shows an example of a quality control information buffer format. This figure shows an example of the data structure of a buffer format that defines a buffer to store baseline atlas information and quality control information. This figure shows an example of the data structure of a buffer format that defines a buffer to store extended atlas information and quality control information. This figure shows an example of the data structure of a buffer format that defines a buffer to store atlas information and quality control information for MIV. This figure shows an example of the data structure of a buffer format that defines a buffer to store atlas information and quality control information for V-DMC. This figure shows an example of quality control information. This figure shows an example of the data structure of a buffer format that defines a buffer to store only quality control information. This figure shows an example of the appearance of a buffer that stores only quality control information. This figure shows an example of the structure of a scene description when applying a buffer that stores only quality control information. This figure shows an example of a property that includes a buffer format that defines a buffer to store quality control information. This figure shows an example of the scene description structure when applying a property that includes a buffer format that defines a buffer for storing quality control information. This is a block diagram showing a main configuration example of a file generation device. This is a flowchart illustrating an example of the file generation process flow. This is a block diagram showing a main configuration example of a playback device. This is a flowchart illustrating an example of the playback process flow. This is a block diagram showing a main configuration example of a content viewing system. This is a block diagram showing a main configuration example of a content viewing system. This is a block diagram showing a main configuration example of a computer.

[0016] The following describes the embodiments for implementing this disclosure. The description will be given in the following order: 1. Supporting literature, etc., for technical content and technical terminology 2. MPEG-I scene description compatible with the V3C codec 3. Use of quality control information in PE reconstruction processing 4. First embodiment (file generation device) 5. Second embodiment (playback device) 6. Third embodiment (content viewing system) 7. Appendix

[0017] <1. Supporting Documents for Technical Content and Terminology> The scope disclosed in this technology includes not only the contents described in the embodiments, but also the contents described in the following non-patent and patent documents that were publicly known at the time of filing, as well as the contents of other documents referenced in the following non-patent and patent documents.

[0018] Non-patent document 1: (described above) Non-patent document 2: (described above) Non-patent document 3: (described above) Non-patent document 4: (described above) Non-patent document 5: (described above) Patent document 1: (described above)

[0019] In other words, the content described in the aforementioned non-patent and patent documents, as well as the content of other documents referenced in those documents, will also serve as a basis for determining the support requirements.

[0020] <2. MPEG-I Scene Description Compatible with V3C Codec> <gltf2.0> Conventionally, as described in Non-Patent Document 1, for example, there was glTF (The GL Transmission Format) (registered trademark) 2.0, a Scene Description format for arranging 3D (three-dimensional) objects within a region (e.g., three-dimensional space) and rendering them. glTF2.0 consists of a JSON format file (.glTF), a binary file (.bin), and an image file (.png, .jpg, etc.), as shown in Figure 1, for example. The binary file stores binary data such as geometry and animation. The image file stores data such as textures.

[0021] A JSON format file is a scene description file written in JSON (JavaScript® Object Notation). A scene description is metadata that describes a scene in 3D content. The description of this scene defines what kind of scene it is. A scene description file is a file that stores such scene descriptions.

[0022] A JSON format file is composed of a list of key-value pairs. An example of this format is shown below: "KEY":"VALUE"

[0023] The key consists of a string. The value consists of a number, string, boolean, array, object, or null, etc.

[0024] Furthermore, multiple key-value pairs ("KEY":"VALUE") can be grouped together using curly braces {}. This grouped structure is also called a JSON object. An example of its format is shown below: "user":{"id":1, "name":"tanaka"}

[0025] In this example, a JSON object is defined that combines the pairs "id":1 and "name":"tanaka" as the values ​​corresponding to the key (user).

[0026] Additionally, zero or more values ​​can be enclosed in square brackets [] to create an array. This array is also called a JSON array. For example, a JSON object can be used as an element of this JSON array. An example of its format is shown below: "test":["hoge", "fuga", "bar"] "users":[{"id":1, "name":"tanaka"},{"id":2,"name":"yamada"},{"id":3, "name":"sato"}]

[0027] Figure 2 shows the glTF objects that can be placed at the top level of a JSON format file and the reference relationships they can have. In the tree structure shown in Figure 2, the elongated ovals represent objects, and the arrows between those objects indicate the reference relationships. As shown in Figure 2, objects such as "scene", "node", "mesh", "camera", "skin", "material", and "texture" are placed at the top level of a JSON format file.

[0028] An example of such a JSON format file (scene description) is shown in Figure 3. The JSON format file 20 in Figure 3 shows a partial example of the top level description. In this JSON format file 20, all top-level objects 21 used are described at the very top. These top-level objects 21 are the glTF objects shown in Figure 2. In addition, the JSON format file 20 shows the reference relationships between objects, as indicated by the arrow 22. More specifically, these reference relationships are shown by specifying the index of the element in the array of referenced objects in the property of the parent object.

[0029] Figure 4 illustrates how to access binary data. As shown in Figure 4, binary data is stored in a buffer object. In other words, the buffer object contains information for accessing the binary data (e.g., a URI (Uniform Resource Identifier)). In a JSON format file, as shown in Figure 4, objects such as meshes, cameras, and skins can access their buffer objects via accessor objects and bufferView objects.

[0030] In other words, for objects such as meshes, cameras, and skins, the accessor object to be referenced is specified. Figure 5 shows an example of how a mesh object is described in a JSON format file. For example, as shown in Figure 5, in a mesh object, vertex attributes such as NORMAL, POSITION, TANGENT, and TEXCORD_0 are defined as keys, and for each attribute, the accessor object to be referenced is specified as the value.

[0031] Next, we will explain how to extend such scene description objects. Each object in glTF2.0 can store newly defined objects within an extension object. Figure 6 shows an example of how to specify a newly defined object (ExtensionExample). As shown in Figure 6, when using a newly defined extension, the extension object name (ExtensionExample in the example in Figure 6) is written in "extensionUsed" and "extensionRequired". This indicates that this extension is an extension that will be used or an extension that is required for loading.

[0032] <Client Processing> MPEG-I Scene Description (MPEG (Moving Picture Experts Group)-I Scene Description) is a standard for controlling the playback of 6DoF (Degree of Freedom) visuals using scene descriptions compliant with glTF2.0. Here, "visual" refers to visual information such as images (information transmitted using vision), and a 6DoF visual represents a visual that corresponds to the viewer's (receiver of visual information) 6 degrees of freedom (6DoF) movement (so-called free viewpoint). In other words, in MPEG-I Scene Description, the playback of 6DoF visuals (i.e., playback of the visual scene) is controlled using a scene description that represents the visual scene. Here, a visual scene refers to a scene related to visuals. A visual scene consists of visual objects placed in a region (e.g., three-dimensional space). Visual objects are objects that exist within the region and are composed of visuals. In other words, the playback device in MPEG-I Scene Description plays the 6DoF visual and reconstructs the visual scene indicated by the scene description.

[0033] A scene description conforming to this MPEG-I scene description standard represents a visual scene composed of high-definition visual objects. In this specification, a scene description conforming to the MPEG-I scene description standard may also be referred to as an MPEG-I scene description.

[0034] Next, we will explain the processing performed by the client device in this MPEG-I scene description. The client device obtains the scene description, and based on that scene description, obtains 3D object data, that is, 3D data representing the three-dimensional structure of the 3D object, and generates a display image using that scene description and 3D data. The 3D object represented by the 3D data is also called a 3D model. In other words, 3D data can be said to be data representing a 3D model. Furthermore, a 3D model can be said to be the three-dimensional structure represented by the 3D data.

[0035] As described in Non-Patent Document 2, the client processing performed by the client device consists of two processing blocks: a Presentation Engine and a Media Access Function, as shown in the example in Figure 7. In this specification, the Presentation Engine is also referred to as PE, and the Media Access Function is also referred to as MAF.

[0036] The presentation engine (PE) acquires and analyzes the scene description. Based on the analysis results (i.e., the scene description), the presentation engine performs processes such as rendering, creating and discarding pipelines, requesting media (content data) acquisition, and specifying buffers. For example, the presentation engine controls media access functions via the MAF API (Media Access Function Application Program Interface). For example, as part of this control, the presentation engine issues instructions for creating and discarding pipelines and requests for media acquisition. The presentation engine also controls buffer management via the Buffer API. For example, as part of this control, the presentation engine specifies buffers. Furthermore, the presentation engine acquires various data stored in buffers and performs rendering, etc.

[0037] The Media Access Function (MAF) performs processing (hereinafter also referred to as content processing) related to media (content data) in accordance with the control of the presentation engine. For example, the Media Access Function retrieves various media data requested by the presentation engine from the cloud, local storage, etc. The Media Access Function also creates or discards pipelines for processing media in accordance with the instructions of the presentation engine. Furthermore, the Media Access Function supplies various media data (encoded data) retrieved in accordance with the instructions of the presentation engine to those pipelines.

[0038] The pipeline decodes various data (encoded data) from the supplied media through pipeline processing and supplies the decoded results to a buffer specified by the presentation engine. The buffer stores the various data (decoded results) from the supplied media.

[0039] <Application of Timed Media> In recent years, as shown in Non-Patent Document 2, for example, there has been consideration to extend glTF2.0 in MPEG-I scene descriptions and apply timed media as 3D object content. Timed media is media data that changes along the time axis, like moving images in two-dimensional images.

[0040] glTF could only be used with still image data as media data (3D object content). In other words, glTF did not support moving image media data. When 3D objects were to be moved, animation (a method of switching between still images along a time axis) was used.

[0041] In MPEG-I scene descriptions, glTF 2.0 is applied, JSON format files are used as scene descriptions, and further extensions to glTF are being considered to allow handling of timed media (e.g., video data) as media data. The following extensions are being considered to handle timed media, for example:

[0042] Figure 8 illustrates an extension for handling timed media. In the example in Figure 8, the MPEG media object (MPEG_media) is a glTF extension and is an object that specifies attributes of MPEG media such as video data, such as uri, track, and startTime.

[0043] Further, as shown in FIG. 8, an MPEG texture video object (MPEG_texture_video) is provided as an extension object (extensions) of a texture object (texture). Information on an accessor corresponding to a buffer object to be accessed is stored in the MPEG texture video object. That is, the MPEG texture video object is an object that specifies an index of an accessor (accessor) corresponding to a buffer (buffer) in which texture media (texture media) specified by an MPEG media object (MPEG_media) is decoded and stored.

[0044] FIG. 9 is a diagram showing a description example of an MPEG media object (MPEG_media) and an MPEG texture video object (MPEG_texture_video) in a scene description for explaining an extension for handling time domain media. In the example of FIG. 9, in the second line from the top, an MPEG texture video object (MPEG_texture_video) is set as an extension object (extensions) of a texture object (texture) as follows. And as the value of the MPEG video texture object, an accessor index (in this example, "2") is specified.

[0045] "texture":[{"sampler":0, "source":1, "extensions":{"MPEG_texture_video ":"accessor":2}}],

[0046] Further, in the example of FIG. 9, in the seventh to sixteenth lines from the top, an MPEG media object (MPEG_media) is set as an extension object (extensions) of glTF as follows. And as the value of the MPEG media object, various information regarding the MPEG media object, such as encoding of the MPEG media object and URI, etc., is stored.

[0047] "MPEG_media": { "media": [ { "name": "source_1", "startTime": 9.0, "loop": "true", "controls": "false", "alternatives": [ { "mimeType": "video / mp4;codecs=\"avc1.42E01E\"", "uri": "video1.mp4", "tracks": [ { "track": ""#track_ID=1"} ]} ]} ]}

[0048] Also, each frame data is decoded and sequentially stored in the buffer. However, since its position and the like vary, the scene description is provided with a mechanism for storing the varying information so that the renderer can read the data. For example, as shown in FIG. 8, an MPEG buffer circular object (MPEG_buffer_circular) is provided as an extension object of the buffer object (buffer). Information for dynamically storing data in the buffer object is stored in the MPEG buffer circular object. For example, information such as access information to the MPEG media and information indicating the number of frames is stored in this MPEG buffer circular object.

[0049] Furthermore, as shown in Figure 8, an MPEG accessor timed object (MPEG_accessor_timed) is provided as an extension object (extension) of the accessor object (accessor). In this case, since the media data is video, the buffer view object (bufferView) referenced in the time direction may change (its position may fluctuate). Therefore, information indicating the referenced buffer view object is stored in this MPEG accessor timed object. For example, the MPEG accessor timed object stores information indicating a reference to the buffer view object (bufferView) in which a timedAccessor information header is described. The timedAccessor information header is, for example, header information that stores information within the dynamically changing accessor object and buffer view object.

[0050] <V-PCC Support> MPEG-I scene descriptions support V-PCC (Video-based Point Cloud Compression), an encoding method for point clouds, which are a type of 3D data. Non-patent document 2 discloses two methods for reconstructing a 3D model (point cloud): MAF reconstruction and PE reconstruction, as shown in Figure 10. MAF reconstruction is a method for reconstructing a 3D model in MAF. PE reconstruction is a method for reconstructing a 3D model in PE.

[0051] In the case of MAF reconstruction processing, the point cloud can be reconstructed using the CPU (Central Processing Unit). In this case, the buffer stores the geometry data (position) and attribute data (color) of the reconstructed 3D data. In other words, the reconstructed 3D data is supplied to the PE. The scene description is configured as shown in Figure 11. In the position property of the attributes object in mesh.primitives, an accessor to the buffer that stores the position information of the points is specified. Similarly, in the color property of the attributes object, an accessor to the buffer that stores the color information of the points is specified.

[0052] In the case of PE reconstruction processing, the point cloud can be reconstructed and rendered using the GPU (Graphics Processing Unit). In this case, the buffer stores the V-PCC data before reconstruction (video data (decoded video) and atlas information (patch info) such as geometry, attributes, and occupancy). In other words, the V-PCC data before reconstruction is supplied to the PE. The scene description is configured as shown in Figure 12. The attributes object in mesh.primitives has properties such as _MPEG_V3C_AVD, _MPEG_V3C_OVD_MAPS, _MPEG_V3C_GVD_MAPS, _MPEG_V3C_AD, and _MPEG_V3C_CONFIG, which specify accessors to the buffers that store the V-PCC data before reconstruction. Each buffer can be accessed via these accessors.

[0053] <Buffer Format> Each buffer (data structure) is defined in the buffer format of each property. The _MPEG_V3C_AD object has a buffer format (buffer_format) as shown in the example in Figure 13A. In this buffer format, three types of data structures (baseline, extended, and miv) are defined as shown in the example in Figure 13B.

[0054] The baseline buffer format defines the data structure of the buffer that stores basic atlas information. This buffer format also defines configuration data such as patch_count, the 2D position and size of each patch. The buffer data structure when a baseline is specified is defined as shown in the example in Figure 14.

[0055] In the extended buffer format, a data structure for the buffer that stores the extended atlas information is defined. The extended atlas information may include, for example, atlas information with application-specific data for PROJECTED patch types, which includes common atlas parameters and PLR information, application-specific data for EOM patch types, and application-specific data for RAW patch types. In addition, this buffer format defines configuration data such as, for example, patch_count, the 2D position (2D_pos_x,y) and size (2D_size_x,y) of each patch, and the PLR ​​level (plrd_level). Furthermore, the data structure of the buffer when extended is specified is defined as shown in the example in Figure 15.

[0056] The MIV buffer format defines a data structure for a buffer that stores atlas information for the MIV. This atlas information may include, for example, common atlas parameters and patch parameters for the MIV. Furthermore, this buffer format defines configuration data such as patch_count, the 2D position (2D_pos_x,y) and size (2D_size_x,y) of each patch, and patch_view_index. The buffer data structure when an MIV is specified is defined as shown in the example in Figure 16.

[0057] <V-DMC Support> Patent Document 1 proposes a technology for transmitting V-DMC (Video-based Dynamic Mesh Coding) data using MPEG-I scene descriptions. V-DMC data is a data format for transmitting mesh data, which is a type of 3D data. A mesh is 3D data that represents the three-dimensional shape of the surface of a 3D object by forming polygons with vertices and connections (also called edges). V-DMC data consists of a base mesh, displacement vectors, attribute map, and atlas information. The base mesh is a simplified version of the three-dimensional structure of the original mesh (i.e., with a reduced number of vertices and connections). Displacement vectors are vector information that shows the displacement of vertices between the subdivided base mesh and the original mesh. Subdividing is a process that increases the number of vertices and connections of a mesh (i.e., increases the number of polygons in the mesh). An attribute map is map information in which the attributes applied to each polygon of the mesh are arranged on a plane. Atlas information is information used when reconstructing the mesh. For example, the atlas information may include the correspondence between the base mesh and displacement maps or attribute maps (such as UV maps), as well as quantized values ​​of displacement vectors.

[0058] In the case of applying such V-DMC data in MPEG-I scene descriptions, MAF reconstruction and PE reconstruction are available as methods for reconstructing the 3D model (mesh). As shown in Figure 17, in the case of MAF reconstruction, decoding and reconstruction are performed in MAF, so the buffer format is the same as in the case of the V-PCC pipeline. The configuration of the scene description is the same as in the example in Figure 11.

[0059] In contrast, in the case of PE reconstruction processing, as shown in Figure 17, processes such as demultiplexing of V-DMC data, decoding of various data, and subdivision of the base mesh are performed in the MAF, and various data such as the subdivided base mesh, displacement vectors, attribute maps, and atlas information are transmitted to the PE via buffers. Therefore, a new buffer for storing the subdivided base mesh is defined, which did not exist in the case of V-PCC. The scene description is configured as shown in Figure 18. The attributes object in mesh.primitives has properties such as _MPEG_V3C_AVD, _MPEG_V3C_GVD, _MPEG_V3C_AD, and _MPEG_V3C_MESH, and accessors to the buffers that store the V-PCC data before reconstruction are specified. Each buffer can be accessed via these accessors. Also, the data structure of the buffer when V-DMC is specified is defined as shown in the example in Figure 19.

[0060] <V3C Codec Support> Incidentally, as shown in Non-Patent Documents 3 to 5, a codec (also called a V3C codec) using Visual Volumetric Video-Based Coding, a technology that compresses 3D objects generated by camera capture using a video codec, has been proposed. In other words, encoding methods such as V-PCC, MIV, and V-DMC are collectively referred to as the V3C codec. To put it another way, the V3C codec may include encoding methods such as V-PCC, MIV, and V-DMC.

[0061] By applying this V3C codec, it is possible to transmit 3D objects with a higher compression ratio than conventional methods, and support for this V3C codec is being advanced in MPEG-I scene descriptions as one of the 3D object encoding technologies that make up a scene.

[0062] In other words, V3C data (V3C bitstream) representing 3D objects in a visual scene is used as the content data to be transmitted (e.g., distributed), and the development of transmission control (distribution control) using MPEG-I scene descriptions that represent that visual scene is underway. In this specification, content data refers to data that constitutes content, such as visual information data or audio information data. Content data may be either encoded data or decoded data (unencoded data).

[0063] Furthermore, in this specification, data for the V3C codec is also referred to as V3C data. The encoded data of that V3C data is also referred to as the V3C bitstream. The aforementioned sub-bitstreams are the encoded data of each of the multiple data that constitute the V3C data. V3C data may include, for example, V-PCC data, MIV data, V-DMC data, etc. V-PCC data is 3D data to be encoded in the V-PCC codec, and consists of data such as geometry data, attribute data, and atlas information. MIV data is 3D data to be encoded in the MIV codec. V-DMC data is 3D data to be encoded in the V-DMC codec, and consists of data such as base mesh, displacement vector, attribute map, and atlas information.

[0064] The V3C bitstream may include, for example, the V-PCC bitstream, MIV bitstream, and V-DMC bitstream. The V-PCC bitstream is the encoded data for V-PCC data, and is multiplexed with sub-bitstreams such as the geometry bitstream, attribute bitstream, and atlas information bitstream. The geometry bitstream is the encoded data for geometry data. The attribute bitstream is the encoded data for attribute data. The atlas information bitstream is the encoded data for atlas information. The MIV bitstream is the encoded data for MIV data. The V-DMC bitstream is the encoded data for V-DMC data, and is multiplexed with sub-bitstreams such as the base mesh bitstream, displacement vector bitstream, attribute bitstream, and atlas information bitstream. The base mesh bitstream is the encoded data for the base mesh. The displacement vector bitstream is the encoded data for the displacement vector. The attribute bitstream is the encoded data for the attribute map. The atlas information bitstream is the encoded data for atlas information.

[0065] Incidentally, the V3C codec provides quality control information used in the 3D model reconstruction process. Quality control information is information about the quality of the 3D model after reconstruction. For example, this quality control information includes information that specifies the quality of the reconstructed 3D model. This quality control information is generated, for example, by content creators. By using this quality control information in the 3D model reconstruction process, the 3D model can be reconstructed with the quality specified by that quality control information (i.e., the quality intended by the content creators). In other words, by reconstructing the 3D model using this quality control information, it is possible to suppress, for example, a decrease in the quality of the reconstructed 3D model or an increase in the load of the reconstruction process. For example, it is possible to improve the quality of the 3D model. To put it another way, quality control information can also be said to be control information for controlling the quality of the reconstructed 3D model (the 3D model generated by the reconstruction process). Note that quality control information may be configured as SEI (Supplemental Enhancement Information), for example.

[0066] An example of a decoding block diagram for V3C data is shown in Figure 20. As shown in Figure 20, the V3C bitstream is divided into subbitstreams, each of which is decoded by its respective decoder.

[0067] The decoded data is input into a nominal format conversion process, where it is reconstructed and a volumetric frame is generated. During this process, the atlas information may include the aforementioned quality control information (SEI). This quality control information is defined for each codec (e.g., V-PCC, MIV, V-DMC).

[0068] <Quality Control Information for V-PCC> Non-Patent Literature 3 defines SEI messages such as "Attribute transformation parameters," "Occupancy synthesis," "Geometry smoothing," and "Attribute smoothing" as quality control information for V-PCC, as shown in Figure 21. Of these, SEI messages with a NAL Type of "NAL_PREFIX_ESEI" are essential information (information that must not be ignored). In contrast, SEI messages with a NAL Type of "NAL_PREFIX_NSEI" are not essential information. The payload type is shown in the "General SEI message syntax" of Non-Patent Literature 3.

[0069] Non-patent document 3 shows examples of the syntax for each SEI message. An example of the syntax for the "Attribute transformation parameters" SEI message is shown in Figure 22A. An example of the syntax for the "Occupancy synthesis" SEI message is shown in Figure 22B. An example of the syntax for the "Geometry smoothing" SEI message is shown in Figure 23A. An example of the syntax for the "Attribute smoothing" SEI message is shown in Figure 23B.

[0070] <Quality control information for MIV> Non-patent document 4 defines SEI messages such as "Viewing space," "Viewing space handling," "Geometry upscaling parameters," "Atlas view enabled," "OMAF v1 compatible," and "Geometry assistance" as quality control information for MIV, as shown in Figure 24. Of these, information with a NAL Type of "NAL_PREFIX_NSEI" is not mandatory. The payload type is shown in the "General SEI message syntax" of Non-patent document 3.

[0071] Non-patent document 4 shows examples of the syntax for each SEI message. An example of the syntax for the "Viewing space" SEI message is shown in Figure 25A. An example of the syntax for the "Viewing space handling" SEI message is shown in Figure 25B. An example of the syntax for the "Geometry upscaling parameters" SEI message is shown in Figure 26A. An example of the syntax for the "Atlas view enabled" SEI message is shown in Figure 26B. An example of the syntax for the "OMAF v1 compatible" SEI message is shown in Figure 27. An example of the syntax for the "Geometry assistance" SEI message is shown in Figure 28.

[0072] <Quality Control Information for V-DMC> Non-Patent Document 5 defines SEI messages such as "Zippering" and "Submesh distortion indication" as quality control information for V-DMC, as shown in Figure 29. Of these, SEI messages with a NAL Type of "NAL_PREFIX_ESEI" are essential information (information that must not be ignored). In contrast, SEI messages with a NAL Type of "NAL_PREFIX_NSEI" are not essential information. The payload type is shown in the "General SEI message syntax" of Non-Patent Document 3.

[0073] Non-patent document 5 shows examples of the syntax for each SEI message. An example of the syntax for the "Submesh distortion indication" SEI message is shown in Figure 30. Examples of the syntax for the "Zippering" SEI message are shown in Figures 31 and 32.

[0074] <Utilization of Quality Control Information in Reconstruction Processing> In the case of MAF reconstruction processing, the reconstruction of the 3D model is performed in MAF as described above, so quality control information included in the atlas information can be used, as shown in Figure 33. Therefore, in this case, it is possible to generate a 3D model as intended by the creator.

[0075] In contrast, in the case of PE reconstruction processing, the reconstruction of the 3D model is performed in PE as described above. Therefore, in this case, as shown in Figure 34, in order to perform 3D model reconstruction using the quality control information contained in the atlas information, it is necessary to transmit that quality control information from MAF to PE via a buffer.

[0076] However, because the atlas information buffer only contains patch data, it was difficult to store the quality control information included in the atlas information. In other words, it was difficult to transmit quality control information from MAF to PE. This meant that it was not possible to perform 3D model reconstruction using the quality control information included in the atlas information, making it difficult to generate 3D models of a quality that matched the content creator's intentions, for example. In other words, it was not possible to control the quality of the reconstructed 3D model (the 3D model generated from the PE reconstruction process). For example, it was difficult to suppress the reduction in quality of the reconstructed 3D model or the increase in the load of the reconstruction process. For example, it was difficult to improve the quality of the 3D model.

[0077] <3. Use of Quality Control Information in PE Reconstruction Processing> <Method 1> Therefore, as shown in the top row of the table in Figure 35, a buffer for storing quality control information is set using the scene description (Method 1).

[0078] For example, the first information processing device may include a scene description generation unit that generates a scene description including a quality control information buffer format that defines a buffer for storing quality control information regarding the quality of the reconstructed 3D model.

[0079] For example, the first information processing method executed by the first information processing device may include generating a scene description that includes a quality control information buffer format that defines a buffer for storing quality control information regarding the quality of the reconstructed 3D model.

[0080] For example, the first program may cause a first information processing device (computer) to perform a process that includes generating a scene description including a quality control information buffer format that defines a buffer for storing quality control information regarding the quality of the reconstructed 3D model.

[0081] By generating a scene description that includes such a quality control information buffer format, the first information processing device can transmit quality control information from the MAF to the PE. Therefore, the first information processing device can apply the quality control information to the reconstruction process performed in the PE. Consequently, even in the case of PE reconstruction, the first information processing device can generate a 3D model with the quality indicated by the quality control information (for example, quality that matches the intent of the content creator). In other words, even in the case of PE reconstruction, the first information processing device can control the quality of the reconstructed 3D model. Therefore, the first information processing device can, for example, suppress a decrease in the quality of the reconstructed 3D model or suppress an increase in the load of the reconstruction process. Furthermore, the first information processing device can improve the quality of the 3D model.

[0082] For example, the second information processing device may include a content processing unit that acquires content data based on a scene description including a quality control information buffer format that defines a buffer for storing quality control information regarding the quality of the reconstructed 3D model, and stores the quality control information contained in the acquired content data in the buffer, and a reconstruction unit that reads the quality control information from the buffer based on the scene description, and uses the content data to reconstruct a 3D model with a quality corresponding to the read quality control information.

[0083] For example, a second information processing method executed by a second information processing device may include acquiring content data based on a scene description that includes a quality control information buffer format defining a buffer for storing quality control information relating to the quality of the reconstructed 3D model, storing the quality control information contained in the acquired content data in the buffer, and reading the quality control information from the buffer based on the scene description, and using the content data to reconstruct a 3D model of a quality corresponding to the read quality control information.

[0084] For example, the second program may cause a second information processing device (computer) to perform a process that includes acquiring content data based on a scene description including a quality control information buffer format that defines a buffer for storing quality control information regarding the quality of the reconstructed 3D model, storing the quality control information contained in the acquired content data in the buffer, and reading the quality control information from the buffer based on the scene description, and using the content data to reconstruct a 3D model with a quality corresponding to the read quality control information.

[0085] By performing client processing based on a scene description containing such a quality control information buffer format, the second information processing device can transmit quality control information from the MAF to the PE via the buffer. Therefore, the second information processing device can apply the quality control information to the reconstruction process performed in the PE. Consequently, even in the case of PE reconstruction processing, the second information processing device can generate a 3D model with the quality indicated by the quality control information (for example, quality in line with the content creator's intentions). In other words, even in the case of PE reconstruction processing, the second information processing device can control the quality of the reconstructed 3D model. Therefore, the second information processing device can, for example, suppress a decrease in the quality of the reconstructed 3D model or suppress an increase in the load of the reconstruction process. Furthermore, the second information processing device can improve the quality of the 3D model.

[0086] In this specification, "quality control information buffer format" refers to a buffer format that defines a buffer for storing quality control information.

[0087] Furthermore, this quality control information buffer format may contain any information. For example, the quality control information buffer format may include information indicating the number of quality control information items. In this way, the quality control information buffer format can define a buffer equal to the number of quality control information items, regardless of the actual number of items. Therefore, the first information processing device can store any number of quality control information items in the buffer based on this information. Similarly, the second information processing device can store any number of quality control information items in the buffer based on this information.

[0088] Furthermore, the quality control information buffer format may include information specifying the payload type of the quality control information. In this way, the quality control information buffer format can define a buffer for storing quality control information of any payload type. In other words, the quality control information buffer format can associate the buffer with any payload type. Therefore, the first information processing device can store quality control information of any payload type in the buffer based on this information. Similarly, the second information processing device can store quality control information of any payload type in the buffer based on this information.

[0089] Furthermore, the quality control information buffer format may include information specifying the payload type when the quality control information is a codec-registered SEI. In this way, the quality control information buffer format can define a buffer that stores quality control information of any payload type, even when the quality control information is a codec-registered SEI. In other words, the quality control information buffer format can make the buffer correspond to any payload type, even when the quality control information is a codec-registered SEI. Therefore, based on this information, the first information processing device can store quality control information of any payload type in the buffer, even when the quality control information is a codec-registered SEI. Similarly, based on this information, the second information processing device can store quality control information of any payload type in the buffer, even when the quality control information is a codec-registered SEI.

[0090] Furthermore, the quality control information buffer format may include information indicating whether processing is mandatory (or not mandatory). In this way, the quality control information buffer format can indicate whether the definition of the buffer is mandatory or not. Therefore, the first information processing device can set the necessary buffers as appropriate based on this information. In other words, the first information processing device can suppress the setting of unnecessary buffers based on this information. That is, the first information processing device can suppress an increase in the load on client processing. Furthermore, the second information processing device can set the necessary buffers as appropriate based on this information. In other words, the second information processing device can suppress the setting of unnecessary buffers based on this information. That is, the second information processing device can suppress an increase in the load on client processing.

[0091] Furthermore, the quality control information buffer format may include information indicating the data length of the quality control information. By doing so, the quality control information buffer format can define a buffer of a size capable of storing quality control information. Therefore, the first information processing device can store quality control information of any data length in the buffer based on this information. Similarly, the second information processing device can store quality control information of any data length in the buffer based on this information.

[0092] Furthermore, the quality control information buffer format may include information indicating quality control information. In this way, the quality control information buffer format can specify the quality control information to be stored in the buffer. Therefore, the first information processing device can store the desired quality control information in the buffer based on this information. Similarly, the second information processing device can store the desired quality control information in the buffer based on this information.

[0093] <Method 1-1> When Method 1 is applied, a new buffer format for transmitting quality control information as part of the atlas information may be defined in the scene description, as shown in the second row from the top of the table in Figure 35 (Method 1-1). In other words, the quality control information buffer format may be newly defined as a buffer format different from the existing buffer format defined in the existing standard. The existing buffer format defined in the existing standard is a buffer format that defines a buffer for storing content data that does not contain quality control information.

[0094] In other words, the quality control information buffer format when Method 1 is applied may be a different buffer format from the buffer format that defines the buffer for storing content data that does not contain quality control information. By doing so, the same effect as Method 1 can be obtained. Furthermore, the quality control information buffer format can be defined without changing the definition of the existing buffer format. In addition, since Method 1-1 is backward compatible, it can be used in conjunction with existing methods.

[0095] <Method 1-1-1> When Method 1-1 is applied, a new buffer format for transmitting atlas information and quality control information may be defined, as shown in the third row from the top of the table in Figure 35 (Method 1-1-1).

[0096] In other words, the quality control information buffer format when Method 1-1 is applied may define a buffer for storing atlas information and quality control information. For example, the quality control information buffer format may be added as a new buffer format to the buffer format of the _MPEG_V3C_AD object. For example, as shown in Figure 36, "baseline_with_supplement_info", "extended_with_supplement_info", "miv_with_supplement_info", and "vdmc_with_supplement_info" may be defined as quality control information buffer formats. By doing so, the same effect as Method 1-1 can be obtained.

[0097] For example, Figure 37 shows an example of the buffer data structure when "baseline_with_supplement_info" is specified in the buffer format of the _MPEG_V3C_AD object. In this figure, the gray area shows an example of the data structure related to quality control information. In other words, "baseline_with_supplement_info" is a buffer format in which a data structure related to quality control information is added to the data structure of the existing baseline buffer format. In this way, the quality control information buffer format may define a buffer that stores basic atlas information and quality control information. The existing "baseline buffer format" may be replaced with this "baseline_with_supplement_info".

[0098] As shown in Figure 37, in "baseline_with_supplement_info", the following information is defined as a data structure related to quality control information. For example, "supplemental_info_count" is defined as information indicating the number of quality control information items. Also, "type" is defined as information specifying the payload type of the quality control information. Also, "sub-type" is defined as information specifying the payload type when the quality control information is a codec-registered SEI. Also, "essential" is defined as information indicating whether processing is mandatory. Also, "data_length" is defined as information indicating the data length of the quality control information. Also, "data" is defined as information indicating the quality control information.

[0099] For example, Figure 38 shows an example of the buffer data structure when "extended_with_supplement_info" is specified in the buffer format of the _MPEG_V3C_AD object. In this figure, the gray area shows an example of the data structure related to quality control information. In other words, "extended_with_supplement_info" is a buffer format in which a data structure related to quality control information is added to the data structure of the existing extended buffer format. In this way, the quality control information buffer format may define a buffer that stores both extended atlas information and quality control information. The existing "extended buffer format" may be replaced with this "extended_with_supplement_info".

[0100] As shown in Figure 38, in "extended_with_supplement_info", the following information is defined as a data structure related to quality control information. For example, "supplemental_info_count" is defined as information indicating the number of quality control information items. Also, "type" is defined as information specifying the payload type of the quality control information. Also, "sub-type" is defined as information specifying the payload type when the quality control information is a codec-registered SEI. Also, "essential" is defined as information indicating whether processing is mandatory. Also, "data_length" is defined as information indicating the data length of the quality control information. Also, "data" is defined as information indicating the quality control information.

[0101] For example, Figure 39 shows an example of the buffer data structure when "miv_with_supplement_info" is specified in the buffer format of the _MPEG_V3C_AD object. In this figure, the gray area shows an example of the data structure related to quality control information. In other words, "miv_with_supplement_info" is a buffer format that adds a data structure related to quality control information to the data structure of an existing MIV buffer format (miv buffer format). In this way, the quality control information buffer format may define a buffer that stores both MIV atlas information and quality control information. The existing "miv buffer format" may be replaced with this "miv_with_supplement_info".

[0102] As shown in Figure 39, in "miv_with_supplement_info", the following information is defined as a data structure related to quality control information. For example, "supplemental_info_count" is defined as information indicating the number of quality control information items. Also, "type" is defined as information specifying the payload type of the quality control information. Also, "sub-type" is defined as information specifying the payload type when the quality control information is a codec-registered SEI. Also, "essential" is defined as information indicating whether processing is mandatory. Also, "data_length" is defined as information indicating the data length of the quality control information. Also, "data" is defined as information indicating the quality control information.

[0103] For example, Figure 40 shows an example of the buffer data structure when "vdmc_with_supplement_info" is specified in the buffer format of the _MPEG_V3C_AD object. In this figure, the gray area shows an example of the data structure related to quality control information. In other words, "vdmc_with_supplement_info" is a buffer format in which a data structure related to quality control information is added to the data structure of the existing V-DMC buffer format (vdmc buffer format). Thus, the quality control information buffer format may define a buffer that stores both the V-DMC atlas information and the quality control information. The existing "vdmc buffer format" may be replaced with this "vdmc_with_supplement_info".

[0104] As shown in Figure 40, in "vdmc_with_supplement_info", the following information is defined as a data structure related to quality control information. For example, "supplemental_info_count" is defined as information indicating the number of quality control information items. Also, "type" is defined as information specifying the payload type of the quality control information. Also, "sub-type" is defined as information specifying the payload type when the quality control information is a codec-registered SEI. Also, "essential" is defined as information indicating whether processing is required. Also, "data_length" is defined as information indicating the data length of the quality control information. Also, "data" is defined as information indicating the quality control information.

[0105] Examples of the "type," "sub-type," and "essential" for each of the quality control information described above are shown in Figure 41. As shown in Figure 41, "type," "sub-type," and "essential" are defined for each SEI of quality control information, making it easy to expand the system to accommodate the addition of new SEIs of quality control information.

[0106] <Method 1-1-2> The buffer for storing atlas information and the buffer for storing quality control information may be separated. In other words, the buffer format may be defined so that atlas information and quality control information are stored in different buffers. In that case, the buffer for storing atlas information may be defined using an existing buffer format. In other words, a new buffer for storing only quality control information may be defined. That is, when Method 1-1 is applied, a new buffer format for transmitting quality control information may be defined that is different from the buffer format for transmitting atlas information, as shown in the fourth row from the top of the table in Figure 35 (Method 1-1-2).

[0107] In other words, the quality control information buffer format when Method 1-1 is applied may define a buffer that stores only quality control information. By doing so, the same effect as Method 1-1 can be obtained.

[0108] For example, a quality control information buffer format may be added as a new buffer format to the buffer format of the _MPEG_V3C_AD object. For example, as shown in Figure 42A, "supplement_info" may be defined as the quality control information buffer format. For example, Figure 42B shows an example of the buffer data structure when "supplement_info" is specified in the buffer format of the _MPEG_V3C_AD object. In other words, "supplement_info" is a buffer format that contains only the data structure related to quality control information.

[0109] As shown in Figure 42B, the following information is defined in "supplement_info" as a data structure related to quality control information. For example, "supplemental_info_count" is defined as information indicating the number of quality control information items. Also, "type" is defined as information specifying the payload type of the quality control information. Also, "sub-type" is defined as information specifying the payload type when the quality control information is a codec-registered SEI. Also, "essential" is defined as information indicating whether processing is mandatory. Also, "data_length" is defined as information indicating the data length of the quality control information. Also, "data" is defined as information indicating the quality control information.

[0110] This "supplement_info" is used in conjunction with the existing buffer format that defines the buffer for storing atlas information. In other words, as shown in Figure 43, in addition to the buffer for storing atlas information, a new buffer (a buffer for transmitting quality control information) is set up, and the quality control information is stored in that buffer for transmitting quality control information. In other words, the scene description in this case will have the configuration shown in Figure 44. That is, in Primitives' MPEG_primitive_V3C, there are _MPEG_V3C_AD objects used for transmitting patch information and _MPEG_V3C_AD objects used for transmitting quality control information.

[0111] Although Figures 43 and 44 show the case of V-PCC, a buffer for transmitting quality control information is also added in the case of MIV and V-DMC.

[0112] <Method 1-2> Although Method 1-1 describes defining a new buffer format, an existing buffer format may be extended to define a buffer capable of storing quality control information. In other words, when Method 1 is applied, as shown in the fifth row from the top of the table in Figure 35, the buffer format for transmitting atlas information in the scene description may be extended to a buffer format capable of transmitting quality control information as well (Method 1-2).

[0113] In other words, the quality control information buffer format to which Method 1 is applied may be a buffer format that further extends a buffer format that defines a buffer for storing atlas information of content data that does not contain quality control information, in order to store quality control information.

[0114] In this case, no new buffer format is added to the _MPEG_V3C_AD object. In the buffer format of this _MPEG_V3C_AD object, three types of data structures (baseline, extended, and miv) are defined, similar to the example in Figure 13B. Whenever any of these data structures is specified, the buffer format is extended and a data structure related to quality control information is added. The added data structure is the same as in the case of method 1-1 described above. That is, for example, if the baseline buffer format is specified, the same data structure as the buffer when "baseline_with_supplement_info" is specified is applied. Also, if the extended buffer format is specified, the same data structure as the buffer when "extended_with_supplement_info" is specified is applied. Also, if the MIV buffer format is specified, the same data structure as the buffer when "miv_with_supplement_info" is specified is applied. In the case of V-DMC, the same extension should be made to the defined data structure.

[0115] By doing so, the same effect as Method 1 can be obtained. Furthermore, since Method 1-2 is backward compatible, it can be used in conjunction with existing methods.

[0116] <Method 1-3> Alternatively, instead of adding a new buffer format to a property as in Methods 1-1 and 1-2 described above, a new property having a new buffer format may be added. In other words, when Method 1 is applied, as shown in the bottom row of the table in Figure 35, a new property for transmitting quality control information may be defined in the MPEG_primitive_V3C property in the scene description, which is different from the property for transmitting atlas information (Method 1-3).

[0117] In other words, the scene description when Method 1 is applied may include a quality control information property that includes a quality control information buffer format, which is different from the property that includes a buffer format that defines a buffer for storing atlas information of content data that does not include quality control information. For example, an _MPEG_V3C_SID object having properties as shown in Figure 45B may be defined. In this case, the configuration of the scene description will be as shown in the example in Figure 46. That is, in Primitives' MPEG_primitive_V3C, an _MPEG_V3C_SID object used for transmitting quality control information is provided.

[0118] In this case, the buffer data structure includes only the data structure related to quality control information, as shown in Figure 45A.

[0119] As shown in Figure 45A, in this buffer format, the following information is defined as a data structure for quality control information. For example, "supplemental_info_count" is defined as information indicating the number of quality control information items. Also, "type" is defined as information specifying the payload type of the quality control information. Also, "sub-type" is defined as information specifying the payload type when the quality control information is a codec-registered SEI. Also, "essential" is defined as information indicating whether processing is required. Also, "data_length" is defined as information indicating the data length of the quality control information. Also, "data" is defined as information indicating the quality control information.

[0120] By doing so, the same effect as Method 1 can be obtained. Furthermore, Methods 1-3 are backward compatible and can be used in conjunction with existing methods.

[0121] <Combinations> Each of the methods described above may be applied in combination with any other method, as long as no contradiction arises. Three or more methods may be applied in combination. Furthermore, the combinatorial methods may include not only those shown in the table in Figure 35 as "methods," but all of the elements described above. In addition, each of the methods described above may be applied in combination with other methods not mentioned above.

[0122] In this specification, descriptions given for higher-level methods also apply to lower-level methods belonging to that method, provided that they do not create a contradiction. For example, if it is stated that "Method 1 may be applied," then one or more of Methods 1-1 through 1-3 may be applied. Alternatively, one or more of Methods 1-1-1 and 1-1-2 may be applied.

[0123] <4. First Embodiment> <File Generation Device> The technology described above can be applied to any device. Figure 47 is a block diagram showing an example of the configuration of a file generation device, which is one embodiment of an information processing device to which the technology is applied. The file generation device 300 (first information processing device) shown in Figure 47 is a device that generates a content file for distribution by converting some 3D data into V3C data, encoding it, and storing it in a file container such as ISOBMFF. The file generation device 300 also generates a scene description used for distribution.

[0124] Note that Figure 47 shows the main components such as the processing unit and data flow, and does not necessarily represent everything. In other words, the file generation device 300 may have processing units that are not shown as blocks in Figure 47, or processes and data flows that are not shown as arrows or other symbols in Figure 47.

[0125] As shown in Figure 47, the file generation device 300 (first information processing device) includes a control unit 301, a file generation processing unit 302, and an input unit 303. The control unit 301 performs processing related to the control of the file generation processing unit 302. The file generation processing unit 302 performs processing related to the generation of content files, etc., which store content data. The input unit 303 has an input device and accepts information input from external sources such as users or other devices, and supplies the input information to the control unit 301. The input device may include, for example, a keyboard, mouse, touch panel, physical switch, physical button, input terminal, etc. The control unit 301 may perform processing based on the information supplied from the input unit 303.

[0126] The file generation processing unit 302 includes a V3C data generation unit 311, a V3C encoding unit 312, a file generation unit 313, a scene description generation unit 314, a storage unit 315, and a supply unit 316.

[0127] The V3C data generation unit 311 performs processing related to the generation of V3C data. For example, the V3C data generation unit 311 may acquire 3D data representing the three-dimensional structure of a 3D object, which is supplied from outside the file generation device 300. The V3C data generation unit 311 may convert the 3D data into V3C data. The V3C data generation unit 311 may supply the generated V3C data to the V3C encoding unit 312.

[0128] The V3C encoding unit 312 performs processing related to the encoding of V3C data. For example, the V3C encoding unit 312 may acquire V3C data supplied from the V3C data generation unit 311. The V3C encoding unit 312 may encode the V3C data and generate a V3C bitstream. The V3C encoding unit 312 may supply the generated V3C bitstream to the file generation unit 313.

[0129] The file generation unit 313 performs processing related to the generation of a content file that stores content data. For example, the file generation unit 313 may obtain a V3C bitstream supplied from the V3C encoding unit 312. The file generation unit 313 may generate a content file that stores the V3C bitstream. This content file may have any specifications, for example, it may be ISOBMFF (International Organization for Standardization Base Media File Format). The file generation unit 313 may supply the generated content file to the scene description generation unit 314. The file generation unit 313 may supply the generated content file to the storage unit 315.

[0130] The scene description generation unit 314 performs processing related to the generation of a scene description. For example, the scene description generation unit 314 may acquire 3D data supplied from outside the file generation device 300. The scene description generation unit 314 may acquire a content file supplied from the file generation unit 313. Based on this information, the scene description generation unit 314 generates a scene description representing a scene composed of 3D objects that represent a three-dimensional structure of V3C data. In other words, this scene description corresponds to the V3C bitstream (i.e., V3C data) stored in the content file. The scene description generation unit 314 generates a scene description file to store the scene description it has generated. The scene description generation unit 314 may supply the scene description file it has generated to the storage unit 315.

[0131] The storage unit 315 has a storage area and performs processing related to the storage of information. For example, the storage unit 315 may acquire a content file supplied from the file generation unit 313. The storage unit 315 may store the content file in its storage area. The storage unit 315 may also acquire a scene description file supplied from the scene description generation unit 314. The storage unit 315 may store the scene description file in its storage area.

[0132] The supply unit 316 performs processing related to the supply of information. For example, the supply unit 316 may acquire scene description files and content files stored in the storage unit 315. The supply unit 316 may also supply the acquired scene description files and content files to an external device (e.g., a distribution server or playback device) outside the file generation device 300.

[0133] In other words, the first information processing device (for example, the file generation device 300) may further include a supply unit (for example, a supply unit 316) that supplies the generated scene description. With such a configuration, the first information processing device can supply the generated scene description to an external device (for example, another device such as a distribution server or client device).

[0134] <Application of this technology> The file generation device 300 having the above configuration may be used as the first information processing device, and the various methods described above (this technology) may be applied in <3. Use of quality control information in PE reconstruction processing>.

[0135] For example, the file generation device 300 may include a scene description generation unit 314 that generates a scene description including a quality control information buffer format that defines a buffer for storing quality control information regarding the quality of the reconstructed 3D model.

[0136] Furthermore, this quality control information buffer format may contain any information. For example, the quality control information buffer format may contain information indicating the number of quality control information items. The quality control information buffer format may also contain information specifying the payload type of the quality control information. The quality control information buffer format may also contain information specifying the payload type when the quality control information is a codec-registered SEI. The quality control information buffer format may also contain information indicating whether processing is mandatory (or not). The quality control information buffer format may also contain information indicating the data length of the quality control information. The quality control information buffer format may also contain information indicating the quality control information.

[0137] Furthermore, the quality control information buffer format may be a different buffer format from the buffer format that defines a buffer for storing content data that does not contain quality control information. Also, the quality control information buffer format may define a buffer for storing atlas information and quality control information. For example, the quality control information buffer format may define a buffer for storing basic atlas information and quality control information. Alternatively, the quality control information buffer format may define a buffer for storing extended atlas information and quality control information. Furthermore, the quality control information buffer format may define a buffer for storing atlas information and quality control information for MIV. Also, the quality control information buffer format may define a buffer for storing atlas information and quality control information for V-DMC.

[0138] Furthermore, the quality control information buffer format may define a buffer that stores only quality control information. Alternatively, the quality control information buffer format may be an extension of a buffer format that defines a buffer that stores atlas information for content data that does not contain quality control information, to include the storage of quality control information. Additionally, the scene description may include a quality control information property that includes a quality control information buffer format, which is different from a property that includes a buffer format that defines a buffer that stores atlas information for content data that does not contain quality control information.

[0139] By having such a configuration, the file generation device 300 can obtain the effects described above in <3. Utilization of quality control information in PE reconstruction processing>.

[0140] <File Generation Process Flow> An example of the file generation process flow executed by such a file generation device 300 will be explained with reference to the flowchart in Figure 48.

[0141] In this case, when the file generation process is started, the V3C data generation unit 311 of the file generation device 300 acquires 3D data in step S301 and generates V3C data from that 3D data.

[0142] In step S302, the V3C encoding unit 312 encodes the V3C data and generates a V3C bitstream.

[0143] In step S303, the file generation unit 313 generates a content file and stores its V3C bitstream as content data in the content file.

[0144] In step S304, the scene description generation unit 314 generates a scene description corresponding to the V3C bitstream (content file) and generates a scene description file to store the scene description.

[0145] In step S305, the scene description generation unit 314 sets a buffer format for storing quality control information in the scene description. That is, the scene description generation unit 314 generates a scene description (scene description file) that includes a quality control information buffer format that defines a buffer for storing quality control information related to the quality of the reconstructed 3D model.

[0146] In step S306, the storage unit 315 stores the generated scene description file and content file.

[0147] In step S307, the supply unit 316 supplies the scene description file and content file to an external device (such as a distribution server or playback device) outside the file generation device 300.

[0148] The file generation process ends when the processing in step S307 is completed.

[0149] By executing each process as described above, the file generation device 300 can transmit quality control information from the MAF to the PE in the device that performs the client processing. Therefore, the file generation device 300 can apply the quality control information to the reconstruction process performed in the PE. Consequently, even in the case of PE reconstruction processing, the file generation device 300 can generate a 3D model with the quality indicated by the quality control information (for example, quality that matches the intent of the content creator). In other words, even in the case of PE reconstruction processing, the file generation device 300 can control the quality of the reconstructed 3D model. Therefore, the file generation device 300 can, for example, suppress a decrease in the quality of the reconstructed 3D model or suppress an increase in the load of the reconstruction process. Furthermore, the file generation device 300 can improve the quality of the 3D model.

[0150] <5. Second Embodiment> <Playback Device> The technology described above can be applied to any device. Figure 49 is a block diagram showing an example of the configuration of a playback device, which is one embodiment of an information processing device to which the technology is applied. The playback device 400 (second information processing device) shown in Figure 49 is a playback device that performs playback processing of V3C data. For example, the playback device 400 acquires a scene description file corresponding to the content data to be supplied (content file), acquires the content file based on the scene description file, decodes the content data (V3C bitstream) stored in the content file, and plays back the V3C data. In other words, the playback device 400 reconstructs a 3D model using the V3C data, renders it to generate a display image, and displays the display image. Note that this scene description file may be generated by, for example, the file generation device 300. Also, this content file may be generated by, for example, the file generation device 300.

[0151] Note that Figure 49 shows the main components such as the processing unit and data flow, and does not necessarily represent everything. In other words, the playback device 400 may have processing units that are not shown as blocks in Figure 49, or processes and data flows that are not shown as arrows or other symbols in Figure 49.

[0152] As shown in Figure 49, the playback device 400 (second information processing device) has a Media Access Function (MAF) 401, a buffer 402, and a Presentation Engine (PE) 403. This playback device 400 applies PE reconstruction processing. That is, the playback device 400 reconstructs the 3D model in the PE 403. Therefore, the MAF 401 has a content acquisition unit 411 and a V3C decoding unit 412. The PE 403 also has a scene description acquisition unit 421, a control unit 422, a reconstruction unit 423, and a display processing unit 424.

[0153] The content acquisition unit 411 performs processing related to the acquisition of content files. For example, the content acquisition unit 411 may acquire a specified content file in accordance with the control of the PE 403 (or its control unit 422). Alternatively, the content acquisition unit 411 may acquire a V3C bitstream from the content file. The content acquisition unit 411 may supply the V3C bitstream to the V3C decoding unit 412.

[0154] The V3C decoding unit 412 performs processing related to the decoding of the V3C bitstream. For example, the V3C decoding unit 412 may acquire the V3C bitstream supplied from the content acquisition unit 411 under the control of the PE 403 (control unit 422). The V3C decoding unit 412 may decode the acquired V3C bitstream and generate V3C data under the control of the PE 403 (control unit 422). The V3C decoding unit 412 may store the V3C data in a designated area of ​​the buffer 402 under the control of the PE 403 (control unit 422).

[0155] The scene description acquisition unit 421 performs processing related to the acquisition of scene descriptions. For example, the scene description acquisition unit 421 may acquire a scene description file generated by the file generation device 300. In other words, the playback device 400 may further include a scene description acquisition unit that acquires scene descriptions. The scene description acquisition unit 421 may supply the acquired scene description file to the control unit 422.

[0156] The control unit 422 performs processing related to the control of client processing. For example, the control unit 422 may acquire a scene description file supplied from the scene description acquisition unit 421. The control unit 422 may control the MAF 401 based on the scene description stored in the scene description file. The control unit 422 may also control the reconstruction unit 423 and the display processing unit 424 based on the scene description.

[0157] The reconstruction unit 423 performs processing related to the reconstruction of the 3D model. For example, the reconstruction unit 423 may acquire V3C data stored in the buffer 402 in accordance with the control unit 422. Alternatively, the reconstruction unit 423 may reconstruct the 3D model from the acquired V3C data in accordance with the control unit 422. In this case, the reconstruction unit 423 may acquire quality control information stored in the buffer 402 and apply it to the reconstruction of the 3D model. For example, the reconstruction unit 423 may generate a 3D model with the quality indicated by the quality control information (quality in line with the content creator's intentions) through its reconstruction. That is, the reconstruction unit 423 may control the quality of the reconstructed 3D model using the quality control information. The reconstruction unit 423 may supply the reconstructed 3D model (3D data representing the 3D model) to the display processing unit 424.

[0158] The display processing unit 424 performs processing related to the display of content. For example, the display processing unit 424 may acquire a reconstructed 3D model (3D data representing the 3D model) supplied from the reconstruction unit 423. The display processing unit 424 may generate content display information using the acquired 3D data. For example, the display processing unit 424 may generate the display information by rendering the 3D data. Here, display information refers to information that is displayed. For example, a display image (an image to be displayed) may be included in this display information. The display processing unit 424 may supply the generated display information to a display device (not shown) and have it displayed.

[0159] Buffer 402 has a memory area and performs processing related to the storage of information. For example, buffer 402 may store data supplied from MAF 401 (such as V3C data and quality control information) in its own memory area. Alternatively, buffer 402 may supply the data stored in its own memory area to PE 403. In other words, MAF 401 may supply content data and quality control information to PE 403 via buffer 402.

[0160] A regeneration device 400 with such a configuration may be used as a second information processing device, and the various methods described above (this technology) may be applied in <3. Use of quality control information in PE reconstruction processing>.

[0161] For example, the playback device 400 may include a MAF 401 that acquires content data based on a scene description including a quality control information buffer format that defines a buffer 402 for storing quality control information regarding the quality of the reconstructed 3D model, and stores the quality control information contained in the acquired content data in the buffer 402, and a reconstruction unit 423 that reads the quality control information from the buffer 402 based on the scene description and reconstructs a 3D model of a quality corresponding to the read quality control information using the content data.

[0162] Furthermore, this quality control information buffer format may contain any information. For example, the quality control information buffer format may contain information indicating the number of quality control information items. The quality control information buffer format may also contain information specifying the payload type of the quality control information. The quality control information buffer format may also contain information specifying the payload type when the quality control information is a codec-registered SEI. The quality control information buffer format may also contain information indicating whether processing is mandatory (or not). The quality control information buffer format may also contain information indicating the data length of the quality control information. The quality control information buffer format may also contain information indicating the quality control information.

[0163] Furthermore, the quality control information buffer format may be a different buffer format from the buffer format that defines a buffer for storing content data that does not contain quality control information. Also, the quality control information buffer format may define a buffer for storing atlas information and quality control information. For example, the quality control information buffer format may define a buffer for storing basic atlas information and quality control information. Alternatively, the quality control information buffer format may define a buffer for storing extended atlas information and quality control information. Furthermore, the quality control information buffer format may define a buffer for storing atlas information and quality control information for MIV. Also, the quality control information buffer format may define a buffer for storing atlas information and quality control information for V-DMC.

[0164] Furthermore, the quality control information buffer format may define a buffer that stores only quality control information. Alternatively, the quality control information buffer format may be an extension of a buffer format that defines a buffer that stores atlas information for content data that does not contain quality control information, to include the storage of quality control information. Additionally, the scene description may include a quality control information property that includes a quality control information buffer format, which is different from a property that includes a buffer format that defines a buffer that stores atlas information for content data that does not contain quality control information.

[0165] With this configuration, the regeneration device 400 can obtain the effects described above in <3. Utilization of quality control information in PE reconstruction processing>.

[0166] <Flow of the regeneration process> An example of the flow of the regeneration process performed by the regeneration device 400 having the above configuration will be explained with reference to the flowchart in Figure 50.

[0167] When playback processing begins, the scene description acquisition unit 421 of the playback device 400 acquires a scene description file in step S401.

[0168] In step S402, the content acquisition unit 411 acquires a content file based on the scene description stored in the scene description file. In other words, the MAF 401 acquires content data based on a scene description that includes a quality control information buffer format that defines a buffer for storing quality control information regarding the quality of the reconstructed 3D model. The content acquisition unit 411 acquires the content data (V3C bitstream) stored in the acquired content file.

[0169] In step S403, the V3C decoding unit 412 decodes the V3C bitstream based on the scene description and generates V3C data.

[0170] In step S404, the V3C decoding unit 412 decodes the V3C bitstream based on the scene description and stores the generated V3C data as content data in the buffer 402. The V3C decoding unit 412 also extracts quality control information contained in the atlas information of the V3C data based on the scene description and stores the extracted quality control information in the buffer 402. In other words, the MAF 401 stores the quality control information contained in the acquired content data in the buffer 402 based on the scene description, which includes a quality control information buffer format that defines a buffer for storing quality control information related to the quality of the reconstructed 3D model.

[0171] In step S405, the reconstruction unit 423 may read and acquire V3C data from the buffer 402 based on the scene description. Alternatively, the reconstruction unit 423 may read and acquire quality control information from the buffer 402 based on the scene description. In other words, the reconstruction unit 423 reads quality control information from the buffer 402 based on a scene description that includes a quality control information buffer format that defines a buffer for storing quality control information related to the quality of the reconstructed 3D model.

[0172] In step S406, the reconstruction unit 423 reconstructs a 3D model from the V3C data based on the quality control information and generates 3D data that represents that 3D model. In other words, the reconstruction unit 423 uses the content data to reconstruct a 3D model with a quality corresponding to the quality control information it has read.

[0173] In step S407, the display processing unit 424 renders the generated 3D data and generates display information. The display processing unit 424 supplies the generated display information to a display device and displays it.

[0174] When the process in step S407 is completed, the playback process ends.

[0175] By performing each process as described above, the playback device 400 can transmit quality control information from the MAF to the PE. Therefore, the playback device 400 can apply the quality control information to the reconstruction process performed in the PE. Consequently, even in the case of PE reconstruction, the playback device 400 can generate a 3D model with the quality indicated by the quality control information (for example, quality that matches the intent of the content creator). In other words, even in the case of PE reconstruction, the playback device 400 can control the quality of the reconstructed 3D model. Therefore, the playback device 400 can, for example, suppress a decrease in the quality of the reconstructed 3D model or suppress an increase in the load of the reconstruction process. Furthermore, the playback device 400 can improve the quality of the 3D model.

[0176] <6. Third Embodiment> <Content Viewing System 1> In the second embodiment, client processing was described as being performed in the playback device 400 (client device), but client processing is not limited to the playback device 400 and can be performed in any device or configuration. For example, client processing may be performed on a server (cloud server, etc.).

[0177] The content viewing system 500 shown in Figure 51 is a system in which a server provides content for a client's user to view. As shown in Figure 51, the content viewing system 500 includes a CDN (Contents Delivery Network) 501, a MAF instance 502, a PE instance 503, and a client 504. In Figure 51, the configuration above the dotted line (i.e., from CDN 501 to PE instance 503) represents the server configuration, and the configuration below the dotted line (i.e., the client 504) represents the local (client) configuration.

[0178] In the content viewing system 500, the server configuration can be anything. For example, this server may consist of a single physical server or multiple physical servers. Alternatively, this server may consist of virtual servers. Furthermore, this server may be configured as a so-called cloud server.

[0179] CDN 501 is a network that has the function of distributing content. CDN 501 can have advanced distribution functions, such as storing the content to be distributed in a cache close to the client. In the content viewing system 500, CDN 501 may supply the V3C bitstream (V3C Content), which is the content data to be distributed, to MAF instance 502. CDN 501 may also supply the scene description corresponding to the V3C bitstream to PE instance 503.

[0180] MAF instance 502 is an instance that implements the MAF functions described above. PE instance 503 is an instance that implements the PE functions described above. PE instance 503 can control MAF instance 502 via the MAF API. For example, PE instance 503 can control MAF instance 502 based on a scene description. Furthermore, MAF instance 502 can supply content data, etc., to PE instance 503 via a buffer (not shown).

[0181] Client 504 is a terminal device operated by a user who is viewing content. The server supplies the content that the user is viewing to client 504. As described above, the server of this content viewing system 500 has functions that execute client processing, such as MAF instance 502 and PE instance 503. Therefore, this server reconstructs and renders the 3D model, generates a display image of the 3D model, and supplies that display image to client 504.

[0182] At that time, the server executes PE reconstruction processing and reconstructs the 3D model in PE instance 503. PE instance 503 then uses quality control information to reconstruct the 3D model and generates a 3D model with the quality specified by the quality control information (for example, quality consistent with the content creator's intentions).

[0183] For example, when user information from client 504 is uploaded to the server, PE instance 503 supplies that user information to MAF instance 502. PE instance 503 also obtains a scene description from CDN 501 and controls MAF instance 502 based on that scene description. MAF instance 502, following the control of PE instance 503, obtains content data (V3C bitstream) from CDN 501 and decodes it. MAF instance 502 supplies the decoded V3C data to PE instance 503. MAF instance 502 also supplies quality control information for the V3C data to PE instance 503. Based on this quality control information, PE instance 503 reconstructs a 3D model of the desired quality from the V3C data. Furthermore, PE instance 503 performs rendering to generate a display image of the 3D model. PE instance 503 supplies this display image to client 504. Client 504 obtains and displays the display image.

[0184] By distributing the displayed images in this manner, it is possible to suppress the increase in processing load on client 504 due to processes such as reconstruction and rendering.

[0185] In this server processing, data transmission from MAF instance 502 to PE instance 503 is performed via a buffer (not shown), similar to the case of the regeneration device 400 in the second embodiment.

[0186] Therefore, the present technology may be applied in the same way as in the case of the playback device 400 of the second embodiment, so that quality control information can be transmitted from the MAF instance 502 to the PE instance 503. In other words, in the server, the MAF instance 502 may acquire content data based on a scene description that includes a quality control information buffer format that defines a buffer for storing quality control information regarding the quality of the reconstructed 3D model, and store the quality control information contained in the acquired content data in the buffer. The PE instance 503 may then read the quality control information from the buffer based on the scene description and reconstruct a 3D model with a quality corresponding to the read quality control information using the content data. By doing so, the server (MAF instance 502 and PE instance 503) can obtain the effects described above in <3. Use of Quality Control Information in PE Reconstruction Processing>, in the same way as in the case of the playback device 400.

[0187] Furthermore, by performing the generation of display images on the server in this manner, content can be easily delivered (while suppressing processing failures) even if the client 504 is a thin client with low processing power, such as an HMD (Head Mounted Display). In addition, it is possible to select server-side instances individually for MAF and PE, and to select instances specialized for CPU or GPU processing.

[0188] <Content Viewing System 2> In addition, as shown in the example in Figure 52, multiple PE instances may be provided. The content viewing system 600 shown in Figure 52 is a system similar to the content viewing system 500 in Figure 51, where the server provides the client with content for the client's users to view. The CDN 601 shown in Figure 52 is a network (a network for distributing content) similar to the CDN 501. The MAF instance 602 is an instance with the same functionality as the MAF instance 502. In the example in Figure 52, the server has multiple PE instances, such as PE instance 603-1, PE instance 603-2, ... When there is no need to distinguish each PE instance from one another, they are referred to as PE instance 603. Each PE instance 603 is an instance with the same functionality as PE instance 503. Also, in the example in Figure 52, multiple clients are connected to the server, such as client 604-1, client 604-2, ... When there is no need to distinguish each client from one another, they are referred to as client 604. Each client 604 has the same functionality as client 504.

[0189] In other words, in this example, the server distributes content to multiple clients 604. A PE instance 603 may be provided for each of these clients 604, for example. By doing so, the processing load of the PE instance 603, such as 3D model reconstruction and rendering, can be distributed, thereby suppressing failures in client processing. In other words, the processing of the MAF instance 602 for multiple PE instances 603 can be standardized. Furthermore, the same effects as in the content viewing system 500 can be obtained by applying this technology to the content viewing system 600.

[0190] <7. Addendum> <Computer> The series of processes described above may be executed by hardware or by software. When the series of processes are executed by software, the programs that make up the software are installed on a computer. Here, a computer includes computers built into dedicated hardware, as well as general-purpose personal computers, for example, that can perform various functions by installing various programs.

[0191] Figure 53 is a block diagram showing an example of the hardware configuration of a computer that executes the series of processes described above using a program.

[0192] In the computer 900 shown in Figure 53, the CPU (Central Processing Unit) 901, ROM (Read Only Memory) 902, and RAM (Random Access Memory) 903 are interconnected via a bus 904.

[0193] An input / output interface 910 is also connected to the bus 904. An input / output interface 910 is connected to an input unit 911, an output unit 912, a storage unit 913, a communication unit 914, and a drive 915.

[0194] The input unit 911 may consist of, for example, a keyboard, mouse, microphone, touch panel, input terminal, etc. The output unit 912 may consist of, for example, a display, speaker, output terminal, etc. The storage unit 913 may consist of, for example, a hard disk, RAM disk, non-volatile memory, etc. The communication unit 914 may consist of, for example, a network interface, etc. The drive 915 may drive removable media 921 such as a magnetic disk, optical disk, magneto-optical disk, or semiconductor memory.

[0195] In a computer configured as described above, the CPU 901 loads, for example, a program stored in the memory unit 913 into the RAM 903 via the input / output interface 910 and the bus 904, and executes it, thereby performing the series of processes described above. The RAM 903 also appropriately stores data necessary for the CPU 901 to perform various processes.

[0196] The program executed by the computer may be recorded and applied on a removable medium 921, such as a package medium. In that case, the program may be installed in the storage unit 913 via the input / output interface 910 when the removable medium 921 is mounted on the drive 915.

[0197] Furthermore, this program may be provided via wired or wireless transmission media such as a local area network, the internet, or digital satellite broadcasting. In that case, the program may be received by the communication unit 914 and installed in the storage unit 913.

[0198] In addition, this program may be pre-installed in ROM 902 or storage unit 913.

[0199] <Applicable Subjects of This Technology> This technology can be applied to any encoding and decoding scheme.

[0200] Furthermore, this technology can be applied to any configuration. For example, it can be applied to various electronic devices.

[0201] Furthermore, this technology can also be implemented as part of a device, such as a processor as a system LSI (Large Scale Integration) (e.g., a video processor), a module using multiple processors (e.g., a video module), a unit using multiple modules (e.g., a video unit), or a set with additional functions added to a unit (e.g., a video set).

[0202] Furthermore, this technology can also be applied to network systems composed of multiple devices. For example, this technology may be implemented as cloud computing, where multiple devices share and collaborate on processing via a network. For example, this technology may be implemented in a cloud service that provides image (video) related services to any terminal such as computers, AV (Audio Visual) equipment, portable information processing terminals, and IoT (Internet of Things) devices.

[0203] In this specification, a system refers to a collection of multiple components (devices, modules (parts), etc.), regardless of whether all components are located in the same enclosure. Therefore, multiple devices housed in separate enclosures and connected via a network, and a single device containing multiple modules within a single enclosure, are both considered systems.

[0204] <Applicable Fields and Applications of This Technology> Systems, devices, and processing units incorporating this technology can be used in any field, such as transportation, medicine, security, agriculture, livestock farming, mining, beauty, factories, home appliances, weather, and nature monitoring. Furthermore, the applications are entirely arbitrary.

[0205] For example, this technology can be applied to systems and devices used to provide entertainment content. Furthermore, for example, this technology can be applied to systems and devices used for traffic management, such as traffic condition monitoring and automated driving control. In addition, for example, this technology can be applied to systems and devices used for security. Furthermore, for example, this technology can be applied to systems and devices used for automatic control of machinery, etc. Furthermore, for example, this technology can be applied to systems and devices used for agriculture and livestock farming. Furthermore, for example, this technology can be applied to systems and devices that monitor natural conditions such as volcanoes, forests, and oceans, as well as wildlife. Furthermore, for example, this technology can be applied to systems and devices used for sports.

[0206] <Other> In this specification, "flag" refers to information used to identify multiple states, and includes not only information used to identify two states, true (1) or false (0), but also information capable of identifying three or more states. Therefore, the values ​​that this "flag" can take are, for example, two values, 1 / 0, or three or more values. In other words, the number of bits that constitute this "flag" is arbitrary, and can be 1 bit or multiple bits. Furthermore, identification information (including flags) is envisioned not only in the form of including the identification information itself in the bitstream, but also in the form of including difference information of the identification information relative to a certain reference information in the bitstream. Therefore, in this specification, "flag" and "identification information" include not only the information itself, but also difference information relative to the reference information.

[0207] Furthermore, various types of information (metadata, etc.) related to encoded data (bitstream) may be transmitted or recorded in any form, as long as they are associated with the encoded data. Here, the term "associate" means, for example, making it possible to use (link) one piece of data when processing the other. In other words, associated data may be combined into a single piece of data, or they may be individual pieces of data. For example, information associated with encoded data (image) may be transmitted on a different transmission path than the encoded data (image). Also, for example, information associated with encoded data (image) may be recorded on a different recording medium (or a different recording area on the same recording medium) than the encoded data (image). Note that this "association" may not apply to the entire data, but only to a part of it. For example, an image and the information corresponding to that image may be associated with each other in any unit, such as multiple frames, one frame, or a part within a frame.

[0208] In this specification, terms such as "combine," "multiplex," "add," "integrate," "include," "store," "insert," "insert," and "place" mean combining multiple things into one, such as combining encoded data and metadata into a single data, and represent one method of "associating" as described above.

[0209] Furthermore, the embodiments of this technology are not limited to those described above, and various modifications are possible without departing from the spirit of this technology.

[0210] For example, the configuration described as a single device (or processing unit) may be divided and configured as multiple devices (or processing units). Conversely, the configurations described above as multiple devices (or processing units) may be combined and configured as a single device (or processing unit). Furthermore, it is also possible to add configurations other than those described above to the configuration of each device (or each processing unit). In addition, if the overall system configuration and operation are substantially the same, a part of the configuration of one device (or processing unit) may be included in the configuration of another device (or other processing unit).

[0211] Furthermore, for example, the program described above may be executed on any device. In that case, the device should have the necessary functions (such as functional blocks) and be able to obtain the necessary information.

[0212] Furthermore, for example, each step of a flowchart may be executed by one device, or it may be divided among multiple devices. Additionally, if a single step includes multiple processes, these processes may be executed by one device, or they may be divided among multiple devices. In other words, multiple processes included in a single step can be executed as multiple steps. Conversely, processes described as multiple steps can be combined and executed as a single step.

[0213] Furthermore, for example, a program executed by a computer may be structured so that the steps of the program are executed chronologically in the order described herein, or they may be executed in parallel or individually at necessary times, such as when a call is made. In other words, the steps may be executed in an order different from the order described above, as long as no inconsistencies arise. Moreover, the steps of this program may be executed in parallel with the processing of other programs, or in combination with the processing of other programs.

[0214] Furthermore, for example, multiple technologies relating to this technology can be implemented independently, as long as they do not create a contradiction. Of course, any multiple technologies can also be implemented in combination. For example, some or all of the technologies described in one embodiment can be implemented in combination with some or all of the technologies described in another embodiment. Also, some or all of the above-mentioned technologies can be implemented in combination with other technologies not mentioned above.

[0215] Furthermore, this technology can also be configured as follows: (1) An information processing device comprising a scene description generation unit that generates a scene description including a quality control information buffer format that defines a buffer for storing quality control information regarding the quality of the reconstructed 3D model. (2) The information processing device according to (1), wherein the quality control information buffer format includes information indicating the number of quality control pieces. (3) The information processing device according to (1) or (2), wherein the quality control information buffer format includes information specifying the payload type of the quality control pieces. (4) The information processing device according to any one of (1) to (3), wherein the quality control information buffer format includes information specifying the payload type when the quality control pieces are codec-registered SEI (Supplemental Enhancement Information). (5) The information processing device according to any one of (1) to (4), wherein the quality control information buffer format includes information indicating whether processing is mandatory. (6) The information processing device according to any one of (1) to (5), wherein the quality control information buffer format includes information indicating the data length of the quality control pieces. (7) The information processing device according to any one of (1) to (6), wherein the quality control information buffer format includes information indicating the quality control information. (8) The information processing device according to any one of (1) to (7), wherein the quality control information buffer format is a buffer format different from a buffer format that defines a buffer for storing content data that does not include the quality control information. (9) The information processing device according to (8), wherein the quality control information buffer format defines the buffer for storing atlas information and the quality control information. (10) The information processing device according to (9), wherein the quality control information buffer format defines the buffer for storing basic atlas information and the quality control information. (11) The information processing device according to (9) or (10), wherein the quality control information buffer format defines the buffer for storing extended atlas information and the quality control information.(12) The information processing device according to any one of (9) to (11), wherein the quality control information buffer format defines the buffer for storing the atlas information and quality control information for MIV (Moving Picture Experts Group Immersive Video). (13) The information processing device according to any one of (9) to (12), wherein the quality control information buffer format defines the buffer for storing the atlas information and quality control information for V-DMC (Video-based Dynamic Mesh Coding). (14) The information processing device according to (8), wherein the quality control information buffer format defines the buffer for storing only the quality control information. (15) The information processing device according to (1), wherein the quality control information buffer format is a buffer format that further extends a buffer format that defines a buffer for storing atlas information of content data that does not include the quality control information, to include the quality control information. (16) The information processing device according to (1), wherein the scene description includes a quality control information property that includes the quality control information buffer format, which is different from a property that includes a buffer format that includes a buffer for storing atlas information of content data that does not include the quality control information. (17) The information processing apparatus according to any one of (1) to (16), further comprising a supply unit for supplying the generated scene description. (18) An information processing method for generating a scene description including a quality control information buffer format that defines a buffer for storing quality control information relating to the quality of a reconstructed 3D model. (19) A program for causing a computer to perform a process of generating a scene description including a quality control information buffer format that defines a buffer for storing quality control information relating to the quality of a reconstructed 3D model.

[0216] (31) An information processing device comprising: a content processing unit that acquires content data based on a scene description including a quality control information buffer format that defines a buffer for storing quality control information relating to the quality of the reconstructed 3D model, and stores the quality control information contained in the acquired content data in the buffer; and a reconstruction unit that reads the quality control information from the buffer based on the scene description, and reconstructs the 3D model with a quality corresponding to the read quality control information using the content data. (32) The information processing device according to (31), wherein the quality control information buffer format includes information indicating the number of quality control pieces. (33) The information processing device according to (31) or (32), wherein the quality control information buffer format includes information specifying the payload type of the quality control information. (34) The information processing device according to any one of (31) to (33), wherein the quality control information buffer format includes information specifying the payload type when the quality control information is codec-registered SEI (Supplemental Enhancement Information). (35) The information processing device according to any one of (31) to (34), wherein the quality control information buffer format includes information indicating whether processing is mandatory. (36) The information processing device according to any one of (31) to (35), wherein the quality control information buffer format includes information indicating the data length of the quality control information. (37) The information processing device according to any one of (31) to (36), wherein the quality control information buffer format includes information indicating the quality control information. (38) The information processing device according to any one of (31) to (37), wherein the quality control information buffer format is a buffer format different from the buffer format that defines a buffer for storing content data that does not include the quality control information. (39) The information processing device according to (38), wherein the quality control information buffer format defines the buffer for storing atlas information and the quality control information.(40) The information processing device according to (39), wherein the quality control information buffer format defines the buffer for storing the basic atlas information and the quality control information. (41) The information processing device according to (39) or (40), wherein the quality control information buffer format defines the buffer for storing the extended atlas information and the quality control information. (42) The information processing device according to any one of (39) to (41), wherein the quality control information buffer format defines the buffer for storing the atlas information and the quality control information for MIV (Moving Picture Experts Group Immersive Video). (43) The information processing device according to any one of (39) to (42), wherein the quality control information buffer format defines the buffer for storing the atlas information and the quality control information for V-DMC (Video-based Dynamic Mesh Coding). (44) The information processing device according to (38), wherein the quality control information buffer format defines the buffer for storing only the quality control information. (45) The information processing device according to (31), wherein the quality control information buffer format is a buffer format that further extends a buffer format that defines a buffer for storing atlas information of content data that does not contain the quality control information, to store the quality control information. (46) The information processing device according to (31), wherein the scene description includes a quality control information property that includes the quality control information buffer format, which is different from a property that includes a buffer format that defines a buffer for storing atlas information of content data that does not contain the quality control information. (47) The information processing device according to any one of (31) to (46), further comprising a scene description acquisition unit for acquiring the scene description.(48) An information processing method comprising: acquiring content data based on a scene description including a quality control information buffer format that defines a buffer for storing quality control information relating to the quality of a reconstructed 3D model; storing the quality control information contained in the acquired content data in the buffer; and reading the quality control information from the buffer based on the scene description and reconstructing the 3D model with a quality corresponding to the read quality control information using the content data. (49) A program for causing a computer to perform a process comprising: acquiring content data based on a scene description including a quality control information buffer format that defines a buffer for storing quality control information relating to the quality of a reconstructed 3D model; storing the quality control information contained in the acquired content data in the buffer; and reading the quality control information from the buffer based on the scene description and reconstructing the 3D model with a quality corresponding to the read quality control information using the content data.

[0217] 300 File generation device, 301 Control unit, 302 File generation processing unit, 303 Input unit, 311 V3C data generation unit, 312 V3C encoding unit, 313 File generation unit, 314 Scene description generation unit, 315 Storage unit, 316 Supply unit, 400 Playback device, 401 MAF, 402 Buffer, 403 PE, 411 Content acquisition unit, 412 V3C decoding unit, 421 Scene description acquisition unit, 422 Control unit, 423 Reconstruction unit, 424 Display processing unit, 500 Content viewing system, 501 CDN, 502 MAF instance, 503 PE instance, 504 Client, 600 Content viewing system, 601 CDN, 602 MAF instance, 603 PE instance, 604 Client, 900 computers

Claims

1. An information processing device comprising a scene description generation unit that generates a scene description including a quality control information buffer format that defines a buffer for storing quality control information regarding the quality of the reconstructed 3D model.

2. The information processing apparatus according to claim 1, wherein the quality control information buffer format includes information indicating the number of quality control pieces of information.

3. The information processing apparatus according to claim 1, wherein the quality control information buffer format includes information specifying the payload type of the quality control information.

4. The information processing apparatus according to claim 1, wherein the quality control information buffer format includes information specifying the payload type when the quality control information is codec-registered SEI (Supplemental Enhancement Information).

5. The information processing apparatus according to claim 1, wherein the quality control information buffer format includes information indicating whether processing is essential.

6. The information processing apparatus according to claim 1, wherein the quality control information buffer format includes information indicating the data length of the quality control information.

7. The information processing apparatus according to claim 1, wherein the quality control information buffer format includes information indicating the quality control information.

8. The information processing apparatus according to claim 1, wherein the quality control information buffer format is a buffer format different from the buffer format that defines a buffer for storing content data that does not contain the quality control information.

9. The information processing apparatus according to claim 8, wherein the quality control information buffer format defines the buffer for storing atlas information and the quality control information.

10. The information processing apparatus according to claim 9, wherein the quality control information buffer format defines the buffer for storing the basic atlas information and the quality control information.

11. The information processing apparatus according to claim 9, wherein the quality control information buffer format defines the buffer for storing the extended atlas information and the quality control information.

12. The information processing apparatus according to claim 9, wherein the quality control information buffer format defines the buffer for storing the atlas information and the quality control information for MIV (Moving Picture Experts Group Immersive Video).

13. The information processing apparatus according to claim 9, wherein the quality control information buffer format defines the buffer for storing the atlas information and the quality control information for V-DMC (Video-based Dynamic Mesh Coding).

14. The information processing apparatus according to claim 8, wherein the quality control information buffer format defines the buffer that stores only the quality control information.

15. The information processing apparatus according to claim 1, wherein the quality control information buffer format is a buffer format that further extends a buffer format that defines a buffer for storing atlas information of content data that does not include the quality control information, to store the quality control information.

16. The information processing apparatus according to claim 1, wherein the scene description includes a quality control information property that includes the quality control information buffer format, which is different from a property that includes a buffer format that defines a buffer for storing atlas information of content data that does not include the quality control information.

17. The information processing apparatus according to claim 1, further comprising a supply unit for supplying the generated scene description.

18. An information processing method for generating a scene description that includes a quality control information buffer format that defines a buffer for storing quality control information regarding the quality of the reconstructed 3D model.

19. Information processing apparatus comprising: a content processing unit that acquires content data based on a scene description including a quality control information buffer format that defines a buffer for storing quality control information relating to the quality of the reconstructed 3D model, and stores the quality control information contained in the acquired content data in the buffer; and a reconstruction unit that reads the quality control information from the buffer based on the scene description, and reconstructs the 3D model with a quality corresponding to the read quality control information using the content data.

20. An information processing method comprising: acquiring content data based on a scene description including a quality control information buffer format that defines a buffer for storing quality control information relating to the quality of a reconstructed 3D model; storing the quality control information contained in the acquired content data in the buffer; and reading the quality control information from the buffer based on the scene description; and reconstructing the 3D model with a quality corresponding to the read quality control information using the content data.

Images