Encoding device and method, and decoding device and method

Abstract

The present disclosure relates to an encoding device and method, and a decoding device and method, with which it is possible, in adaptive edge segmentation for a base mesh, to suppress the difference between subdivision of triangles in UV unwrap data and subdivision of triangles at 3D vertices. When adaptive edge segmentation is applied to 3D data in which a base mesh is subdivided after decoding, a constraint that each of a plurality of vertices of the base mesh corresponding to a single texture position is at the same three-dimensional vertex position is applied. The present disclosure can be applied to, for example, an encoding device, a decoding device, an information processing device, an electronic apparatus, an encoding method, a decoding method, an information processing method, a program, or the like.

Description

Encoding apparatus and method, and decoding apparatus and method

[0001] This disclosure relates to an encoding apparatus and method, and a decoding apparatus and method, and more particularly to an encoding apparatus and method, and a decoding apparatus and method, that can suppress the occurrence of differences between the subdivision of triangles in UV unwrapping data and the subdivision of triangles at 3D vertices in adaptive edge segmentation for a base mesh.

[0002] Conventionally, V-DMC (Video-based Dynamic Mesh Coding) has been used as a method for encoding meshes, which are 3D data that represent the three-dimensional structure of an object using vertices and connections (see, for example, Non-Patent Document 1). In V-DMC, the mesh to be encoded (original mesh) is represented by a coarse (i.e., low-resolution) base mesh and displacement vectors of division points obtained by subdividing the base mesh, and the base mesh and displacement vectors are encoded. The displacement vectors are stored (packed) in a two-dimensional image and encoded as an image. Since objects can change in the time direction (they are dynamic), the mesh (i.e., the base mesh and displacement vectors) are also dynamic. Therefore, the displacement vectors are encoded as a moving image (displacement video) with the two-dimensional image as the frame.

[0003] During decoding, the bitstream is decoded using a decoding method corresponding to the encoding method, and the base mesh and displacement vectors are restored (generated). Then, the base mesh is subdivided, and the displacement vectors are applied to each subdivision point, thereby restoring (generating) a mesh equivalent to the original mesh. Adaptive edge segmentation, which performs edge subdivision adaptively according to the edge length, could be applied to the base mesh subdivision. In this case, edge subdivision control was performed not only for triangles at 3D vertices but also for triangles in the UV unwrapped data. In controlling edge subdivision for triangles in the UV unwrapped data, the edge length was represented using the three-dimensional position of the 3D vertex corresponding to each UV vertex.

[0004] Khaled Mammou, Jungsun Kim, Alexis Tourapis, Dimitri Podborski, Krasimir Kolarov, "[V-CG] Apple’s Dynamic Mesh Coding CfP Response", ISO / IEC JTC 1 / SC 29 / WG 7 m59281, April 2022

[0005] However, depending on the UV unwrapping algorithm, a plurality of vertices of the base mesh located at different three-dimensional positions may sometimes correspond to a single texture position. In such a case, the correspondence between UV vertices and 3D vertices cannot be correctly established, and there is a risk that a difference may occur between the triangle subdivision result in the UV unwrapping data and the triangle subdivision result for the 3D vertices.

[0006] The present disclosure has been made in view of such circumstances, and is intended to enable suppression of the occurrence of a difference between triangle subdivision in UV unwrapping data and triangle subdivision of 3D vertices in adaptive edge segmentation for a base mesh.

[0007] An encoding device according to one aspect of the present technology includes a constraint processing unit that executes processing related to bitstream constraints including: obtaining a base mesh obtained by thinning out vertices from an original mesh to be encoded, the original mesh being configured by vertices and connections representing a three-dimensional structure of an object; obtaining UV unwrapping data representing texture coordinates of the base mesh; and, when adaptive edge segmentation that adaptively divides edges included in the base mesh is applied, checking whether or not 3D data including the base mesh and the UV unwrapping data complies with the constraint that a plurality of vertices of the base mesh each corresponding to a single texture position are at the same three-dimensional vertex position; and an encoding unit that encodes the 3D data in accordance with a result of the check that the 3D data complies with the constraint. The encoding device is configured as described above.

[0008] One aspect of this technology is an encoding method that includes: obtaining a base mesh from which vertices have been thinned out from an original mesh to be encoded, which is composed of vertices and connections representing the three-dimensional structure of an object; obtaining UV unwrapping data representing the texture coordinates of the base mesh; and performing bitstream constraint processing, which includes checking whether the 3D data, including the base mesh and the UV unwrapping data, conforms to the constraint that, when adaptive edge segmentation which adaptively divides the edges included in the base mesh is applied, multiple vertices of the base mesh corresponding to a single texture position are at the same three-dimensional vertex position; and encoding the 3D data according to the result of the check that the 3D data conforms to the constraint.

[0009] Another aspect of this technology is a decoding device comprising: a decoding unit that decodes a bitstream that conforms to the constraint that when adaptive edge segmentation, which adaptively divides the edges contained in the base mesh, is applied, multiple vertices of the base mesh corresponding to a single texture position are at the same three-dimensional vertex position, and generates the base mesh and UV unwrapping data representing the texture coordinates of the base mesh; and a subdivision unit that subdivides the base mesh using the UV unwrapping data.

[0010] Another aspect of this technology involves decoding a bitstream that conforms to the constraint that, when adaptive edge segmentation is applied to adaptively divide the edges contained in the base mesh, multiple vertices of the base mesh corresponding to a single texture location are at the same three-dimensional vertex location; generating the base mesh and UV unwrapped data representing the texture coordinates of the base mesh; and subdividing the base mesh using the UV unwrapped data.

[0011] A decoding device in yet another aspect of this technology comprises: a decoding unit that decodes a bitstream and generates a base mesh obtained by thinning out vertices from an original mesh to be encoded, which is composed of vertices and connections representing the three-dimensional structure of an object, and UV unwrapping data representing the texture coordinates of the base mesh; a constraint processing unit that performs bitstream constraint processing, which includes obtaining the generated base mesh, obtaining the generated UV unwrapping data, and checking whether the 3D data including the base mesh and the UV unwrapping data conforms to the constraint that, when adaptive edge segmentation which adaptively divides the edges included in the base mesh is applied, multiple vertices of the base mesh corresponding to a single texture position are at the same three-dimensional vertex position; and a subdivision unit that subdivides the base mesh using the UV unwrapping data according to the result of the confirmation that the 3D data conforms to the constraint.

[0012] A decoding method for yet another aspect of this technology is a decoding method that includes decoding a bitstream to generate a base mesh obtained by thinning out vertices from the original mesh to be encoded, which is composed of vertices and connections representing the three-dimensional structure of an object, and UV unwrap data representing the texture coordinates of the base mesh; obtaining the generated base mesh; obtaining the generated UV unwrap data; and performing processing on bitstream constraints, which includes checking whether the 3D data, including the base mesh and the UV unwrap data, conforms to the constraint that, when adaptive edge segmentation which adaptively divides the edges included in the base mesh is applied, multiple vertices of the base mesh corresponding to a single texture position are at the same three-dimensional vertex position; and, depending on the result of the check that the 3D data conforms to the constraint, subdividing the base mesh using the UV unwrap data.

[0013] In one aspect of this technology, the encoding device and method include the following steps: obtaining a base mesh from which vertices have been thinned from the original mesh to be encoded, which is composed of vertices and connections representing the three-dimensional structure of an object; obtaining UV unwrapping data representing the texture coordinates of the base mesh; and, if adaptive edge segmentation is applied to adaptively divide the edges contained in the base mesh, verifying whether the 3D data, including the base mesh and UV unwrapping data, conforms to the constraint that multiple vertices of the base mesh corresponding to a single texture position are at the same three-dimensional vertex position; and, depending on the result of verifying that the 3D data conforms to the constraint, encoding the 3D data.

[0014] In the decoding apparatus and methods of other aspects of this technology, when adaptive edge segmentation is applied to adaptively divide the edges contained in the base mesh, a bitstream is decoded that conforms to the constraint that multiple vertices of the base mesh corresponding to a single texture location are at the same 3D vertex location, and a process is performed which includes generating a base mesh and UV unwrapped data representing the texture coordinates of the base mesh, and then subdividing the base mesh using the UV unwrapped data.

[0015] In other aspects of the decoding apparatus and method of this technology, the bitstream is decoded to generate a base mesh from which vertices have been thinned from the original mesh to be encoded, which is composed of vertices and connections representing the three-dimensional structure of an object, and UV unwrap data representing the texture coordinates of the base mesh; the generated base mesh is acquired; the generated UV unwrap data is acquired; and processing is performed on the bitstream constraints, including checking whether the 3D data, including the base mesh and UV unwrap data, conforms to the constraint that, if adaptive edge segmentation which adaptively divides the edges included in the base mesh is applied, multiple vertices of the base mesh corresponding to a single texture position are at the same three-dimensional vertex position; and depending on the result of the check that the 3D data conforms to the constraint, the base mesh is subdivided using the UV unwrap data.

[0016] This is a diagram to explain mesh. This is a diagram to explain V-DMC. This is a diagram showing an example of adaptive subdivision. This is a diagram showing an example of UV unwrapping. This is a diagram showing an example of edge length transformation. This is a diagram to explain an example of 3D vertex triangle subdivision. This is a diagram to explain an example of UV vertex triangle subdivision. This is a diagram showing an example of UV vertex triangle subdivision. This is a diagram showing an example of bitstream constraints and a method of edge subdivision control based on those constraints. This is a diagram showing an example of bitstream constraints. This is a diagram showing an example of checking constraints. This is a diagram showing an example of updating vertex indices. This is a diagram showing an example of updating vertex indices. This is a diagram showing an example of changing the subdivision method. This is a block diagram showing an example of the main configuration of the encoding device. This is a block diagram showing an example of the main configuration of the V-DMC encoding unit. This is a flowchart explaining an example of the encoding process flow. This is a flowchart explaining an example of the constraint process flow. This is a flowchart explaining an example of the constraint process flow. This is a block diagram explaining an example of the constraint process flow. This is a block diagram explaining other application examples of this technology. This is a block diagram showing an example of the main configuration of the decoding device. This is a flowchart explaining an example of the decoding process flow. This is a block diagram showing an example of the main configuration of the encoding device. This is a flowchart explaining an example of the encoding process flow. This is a block diagram showing an example of the main configuration of the decoding device. This is a flowchart illustrating an example of the decoding process flow. This is a flowchart illustrating an example of the constraint processing flow. This is a flowchart illustrating an example of the constraint processing flow. This is a flowchart illustrating an example of the constraint processing flow. This is a block diagram illustrating an example of the main configuration of the encoding device. This is a block diagram illustrating an example of the main configuration of the V-DMC encoding unit. This is a flowchart illustrating an example of the encoding process flow. This is a flowchart illustrating an example of the V-DMC encoding process flow. This is a block diagram illustrating an example of the main configuration of the decoding device. This is a block diagram illustrating an example of the decoding process flow. This is a block diagram illustrating an example of the main configuration of the computer.

[0017] The following describes the embodiments for implementing this disclosure. The explanation will be given in the following order: 1. Supporting documents, etc., for technical content and technical terms 2. Subdivision processing 3. Introduction of bitstream constraints 4. First embodiment (Method 1) 5. Second embodiment (Method 2) 6. Third embodiment (Method 3) 7. Appendix

[0018] <1. Supporting Documents for Technical Content and Terminology> The scope disclosed in this technology includes not only the contents described in the embodiments, but also the contents described in the following non-patent documents that were publicly known at the time of filing, as well as the contents of other documents referenced in the following non-patent documents.

[0019] Non-patent document 1: (mentioned above)

[0020] In other words, the content described in the aforementioned non-patent literature, as well as the content of other documents referenced in the aforementioned non-patent literature, can also serve as a basis for determining the support requirements.

[0021] <2. Subdivision Processing> <V-DMC> Conventionally, as 3D data representing the three-dimensional structure of a three-dimensional structure (an object with a three-dimensional shape), there was a mesh that represented the three-dimensional shape of the object's surface by forming polygons with vertices and connections (also called edges).

[0022] As shown in the upper left of Figure 1, in the mesh, polygonal planes are formed by vertices 11 and connections 12 that connect these vertices 11. These polygons (also called faces) represent the surface of a three-dimensional object, that is, the three-dimensional shape of the object. A texture 13 can be applied to each face of this mesh.

[0023] The mesh data is composed of information such as that shown in the lower part of Figure 1. The vertex information 14, shown first from the left in the lower part of Figure 1, is information indicating the three-dimensional position (three-dimensional coordinates (X, Y, Z)) of each vertex 11 that makes up the mesh. The connection information 15, shown second from the left in the lower part of Figure 1, is information indicating each connection (edge) 12 that makes up the mesh. The texture image 16, shown third from the left in the lower part of Figure 1, is map information of the texture 13 that is applied to each face. The UV map 17, shown fourth from the left in the lower part of Figure 1, is information indicating the correspondence between the vertices 11 and the texture 13. In the UV map 17, the coordinates (UV coordinates) of each vertex 11 in the texture image 16 are shown.

[0024] One method for encoding such meshes is V-DMC (Video-based Dynamic Mesh Coding), as disclosed in Non-Patent Document 1.

[0025] In V-DMC, the mesh to be encoded (referred to as the original mesh in this specification) is represented by a base mesh with a lower resolution (i.e., coarser) than the original mesh, and displacement vectors of the division points obtained by subdividing the base mesh, and the base mesh and displacement vectors are encoded.

[0026] For example, suppose we have an original mesh as shown in the top row of Figure 2. In Figure 2, the black dots represent vertices, and the lines connecting the black dots represent connections (edges). As mentioned above, a mesh is originally formed by polygons consisting of vertices and edges, but here, for the sake of explanation, it is described as a group of vertices connected linearly (in series).

[0027] By decimating some of the vertices in the original mesh, a coarse (low-resolution) mesh is formed, as shown in the second row from the top in Figure 2. This will be used as the base mesh.

[0028] By subdividing each polygon of this base mesh, vertices and edges are added, as shown in the third row from the top in Figure 2. Here, it is assumed that this subdivision adds the same number of vertices as the original mesh. This results in a mesh with the same number of vertices as the original mesh. In this specification, these added vertices are also referred to as subdivision points.

[0029] However, when vertices of the original mesh are thinned, the connections are updated, and the division points are formed on these updated connections (edges). Therefore, the shape of this subdivided base mesh differs from that of the original mesh. More specifically, as shown in the bottom row of Figure 2, the positions of the division points (on the dotted lines) differ from those of the original mesh. In this specification, the difference between the positions of these division points and the positions of the vertices of the original mesh is referred to as the displacement vector.

[0030] By representing the original mesh as a base mesh and displacement vectors in this way, the number of polygons (i.e., the number of vertices and edges) is reduced. Therefore, by encoding the base mesh and displacement vectors instead of encoding the original mesh, the reduction in encoding efficiency (increase in the amount of encoding) can be suppressed.

[0031] In other words, in V-DMC, the base mesh, displacement vectors, attribute maps (where attributes such as textures are unfolded on the UV plane), and atlas information used to map geometry to attributes are each encoded and combined into a single bitstream. In this specification, 3D data including such a base mesh, displacement vectors, attribute maps, and atlas information (i.e., the 3D data to be encoded) is also referred to as V-DMC data. Furthermore, encoded data obtained by encoding this V-DMC data (i.e., a bitstream in which the base mesh, displacement vectors, attribute maps, and atlas information are each encoded and multiplexed) is also referred to as a V-DMC bitstream.

[0032] During decoding, the V-DMC bitstream is decoded using a decoding method corresponding to the encoding method, and V-DMC data such as the base mesh, displacement vectors, attribute maps, and atlas information are restored (generated). Then, the base mesh is subdivided, and the displacement vectors are applied to each subdivision point, thereby restoring (generating) the original mesh. In reality, the restored mesh may contain encoding distortions, etc., so it may not perfectly match the original mesh before encoding. However, for the sake of explanation, encoding distortions, etc., will not be considered below, and the explanation will assume that the original mesh is restored by decoding. Also, in the following, polygons will be assumed to be triangular in shape. Therefore, in the following, polygons (faces) will also be referred to as triangles.

[0033] <Subdivision of the Base Mesh> Next, we will explain the subdivision of the base mesh in this decoding process. As mentioned above, the base mesh has a lower resolution (fewer triangles) than the original mesh, so the triangles are divided until its resolution becomes equivalent to that of the original mesh (that is, vertices and edges are added to divide the triangles). In this specification, the process of dividing the triangles of such a mesh is also referred to as "subdivision".

[0034] In other words, in V-DMC, the base mesh is subdivided to generate a mesh with the same level of detail as the original mesh. This subdivision method is arbitrary, or it can be a method of recursively repeating the subdivision on the base mesh. That is, instead of generating a mesh with the same level of detail as the original mesh in a single subdivision, the subdivision of triangles is repeated recursively multiple times, gradually increasing the number of triangles. Using this method, the base mesh can be subdivided hierarchically. Therefore, by controlling the number of repetitions, a mesh with the desired level of detail can be generated. In this specification, each of these recursively repeated subdivisions is also referred to as an "iteration".

[0035] For example, in a method called midpoint subdivision, as shown in Figure 3A, subdivision points (vertices) are added to divide each edge of the triangle on the left of the figure into two equal parts, and as shown on the right of the figure, the triangle is divided into four parts. By recursively repeating this subdivision, a mesh with the desired level of detail is restored (generated).

[0036] However, in a method that involves repeating subdivision a predetermined number of times, the triangles may become unnecessarily small. In that case, the reconstruction and rendering processes would have to process a large number of small triangles, potentially increasing the processing load unnecessarily compared to the improvement in subjective quality.

[0037] Therefore, adaptive edge segmentation, which adaptively divides the edges included in the base mesh, was considered. For example, a minimum limit (lower limit) was set for the edge length, and the segmentation was permitted only if the edge length did not fall below that minimum limit after subdivision. For example, in Figure 3B, the triangle on the left in the figure has edge lengths of "6" on the left and right sides and an edge length of "1" on the bottom side. For example, if the minimum limit is set to "1", the bottom edge cannot be segmented, and only the left and right sides are segmented, as shown on the right side of the figure. With this kind of control, it is possible to guarantee that the length of each edge in the triangle after subdivision is greater than or equal to the minimum limit. Therefore, it is possible to suppress the increase in the unnecessary processing load as described above.

[0038] By the way, for the base mesh in 3D space, vertex information, edge information (connection information), and triangle information (polygon information) are represented and managed using the index of each vertex (also called a 3D vertex) in 3D space. In vertex information, the 3D position (3D coordinate) of each 3D vertex is managed. In edge information, each edge is managed using the 3D vertices that make up its ends. In triangle information, each triangle is managed using the 3D vertices that make up that triangle.

[0039] Furthermore, the triangles formed by each vertex of the base mesh and the edges between each vertex are unfolded onto the UV plane and managed as UV unfolded data in order to associate them with the texture unfolded on the UV plane. In this UV unfolded data, vertex information, edge information, and triangle information are managed, as in the case of the 3D space described above. However, since UV unfolding is performed in patches, there may be cases where, for example, one 3D vertex is divided into multiple parts and UV unfolded, and the number of vertices may differ between the 3D space and the UV plane. Therefore, in the UV unfolded data, this information is represented using the index of each vertex (also called a UV vertex) in the UV unfolded state (on the UV plane).

[0040] As described above, the base mesh is subdivided not only for triangles in 3D space but also for UV-unwrapped triangles. In other words, when applying the adaptive edge segmentation described above and controlling edge division based on the edge length limit (lower limit), the edge division control is applied to UV-unwrapped triangles in the same way as for triangles in 3D space. In this case, since the shape (edge ​​length) of UV-unwrapped triangles may differ from that of triangles in 3D space, each edge length is derived using the coordinates (3D position) of the 3D vertices. Therefore, if each UV vertex corresponds to one 3D vertex, UV-unwrapped triangles can be subdivided in the same way as triangles in 3D space.

[0041] However, depending on the UV unwrapping algorithm, a plurality of vertices of a base mesh located at mutually different three-dimensional positions may correspond to a single texture coordinate. In other words, there could be a case where a plurality of 3D vertices correspond to one UV vertex. For example, assume that a triangle in 3D space shown in A of FIG. 4 is UV-unwrapped (arranged on a UV plane) as shown in B of FIG. 4. In this example, 3D vertex Va and 3D vertex Vb, which are vertices at mutually different positions in 3D space, are arranged at a single position on the UV plane. That is, on the UV plane, 3D vertex Va and 3D vertex Vb are represented as one UV vertex UVa. In other words, UV vertex UVa corresponds to two 3D vertices (3D vertex Va and 3D vertex Vb).

[0042] In such a case, in edge split control, the correspondence between UV vertices and 3D vertices cannot be correctly established, and there is a risk that a difference occurs between the triangulation subdivision result in the UV unwrapping data and the triangulation subdivision result for the 3D vertices.

[0043] As described above, since this edge split control is performed using the 3D coordinates of 3D vertices even for UV-unwrapped triangles, it is necessary to obtain (the 3D coordinates of) the 3D vertex corresponding to each UV vertex. For example, in an algorithm as shown in FIG. 5, the edge length is evaluated by referring to the position information "positionTexRef[]" of a 3D vertex corresponding to UV coordinates. In FIG. 5, "TriRef[]" indicates triangle information storing indices of 3D coordinates, and "PositionRef[]" indicates information storing the 3D coordinates.

[0044] For example, when controlling subdivision (edge splitting) of triangles in 3D space, 3D triangle information for managing triangles in 3D space as shown in FIG. 6 is referred to. In this 3D triangle information, indices of 3D vertices (e.g., Va, Vb, etc.) are used, and their 3D coordinates can be uniquely obtained.

[0045] In contrast, for the subdivision (edge splitting) of UV-unfolded triangles, as shown in A of FIG. 7, UV triangle information that manages UV-unfolded triangles is referenced. However, since this UV triangle information uses indices of UV vertices, 3D vertices corresponding to the UV vertices are referenced by an algorithm like the example in FIG. 5 using the 3D triangle information in FIG. 6.

[0046] Therefore, for example, when the 3D coordinates of the 3D vertex corresponding to the UV vertex (UVa) of "t=0, i=0" is obtained, the 3D coordinates of 3D vertex Va are obtained. Furthermore, when the 3D coordinates of the 3D vertex corresponding to the UV vertex (UVa) of "t=4, i=1" is obtained, the 3D coordinates of 3D vertex Vb are obtained. That is, the 3D coordinates of UV vertex UVa (the 3D coordinates of the 3D vertex corresponding to UV vertex UVa) are updated from the coordinates of 3D vertex Va to the coordinates of 3D vertex Vb.

[0047] In other words, as shown in the UV triangle information in A of FIG. 7, UV vertex UVa is used as both the UV vertex of "t=0, i=0" and "t=4, i=1", so as in the 3D triangle information shown in B of FIG. 7, both of these UV vertices are end up being associated with 3D vertex Vb. That is, the 3D coordinates of 3D vertex Vb are incorrectly applied as the 3D coordinates of the UV vertex "t=0, i=0", which should originally correspond to 3D vertex Va.

[0048] As described above, the failure to correctly establish the correspondence between UV vertices and 3D vertices may cause, for example, as shown in FIG. 8, the edge length of the UV-unfolded triangle t0 to differ from that of triangle t0 in 3D space. For this reason, there was a risk that a difference would occur between the triangle subdivision result in the UV unfolding data and the triangle subdivision result for the 3D vertices. When such a difference occurs, for example, the correspondence between geometry and attributes cannot be correctly established, which may reduce the quality of 3D data obtained through reconstruction.

[0049] <3. Introduction of Bitstream Constraints> <Bitstream Constraints> Therefore, as shown in the top row of the table in Figure 9, a constraint is imposed on the bitstream that "when adaptive edge segmentation, which adaptively divides the edges included in the base mesh, is applied (or when the enable flag is true), multiple vertices of the base mesh corresponding to a single texture position are at the same 3D vertex position."

[0050] For example, the V-DMC encoding standard imposes a constraint as shown in Figure 10: (If adaptive edge segmentation is applied, all vertices of the base mesh, each of which corresponds to a single (same) texture coordinate, shall be at the same 3D vertex position.) In other words, when adaptive edge segmentation is enabled, all vertices corresponding to a single texture coordinate must be at the same 3D position.

[0051] This ensures that only V-DMC data conforming to this constraint is encoded. In other words, it is guaranteed that cases like the example in Figure 4 do not exist in the V-DMC data obtained by decoding the V-DMC bitstream. That is, a single UV vertex will not correspond to multiple 3D vertices with different 3D positions. Therefore, it is possible to suppress the occurrence of discrepancies between the triangle subdivision results in the UV unwrapped data and the triangle subdivision results of the 3D vertices, as described above. In other words, it is possible to suppress the occurrence of differences between the triangle subdivision in the UV unwrapped data and the triangle subdivision of the 3D vertices in adaptive edge segmentation for the base mesh. Therefore, for example, it is possible to suppress the reduction in the quality of 3D data obtained by reconstruction.

[0052] Furthermore, in the constraints described above, the expression "when the edge-based segmentation enable flag is true" may be used instead of "when adaptive edge segmentation is applied." The edge-based segmentation enable flag being true means that the adaptive edge segmentation described above is applied. In other words, "when the edge-based segmentation enable flag is true" has the same meaning as "when adaptive edge segmentation is applied."

[0053] <Method 1> In order to comply with the constraints described above, for example, as shown in the second row from the top of the table in Figure 9, it is possible to check whether the V-DMC data conforms to these constraints and perform V-DMC encoding if it does (Method 1). In other words, when encoding the V-DMC data, it is possible to check whether the V-DMC data conforms to these constraints.

[0054] For example, the first encoding device may include a constraint processing unit that performs bitstream constraint processing, which includes obtaining a base mesh obtained by thinning out vertices from an original mesh to be encoded, which is composed of vertices and connections representing the three-dimensional structure of an object; obtaining UV unwrapping data representing the texture coordinates of the base mesh; and checking whether the 3D data, including the base mesh and UV unwrapping data, conforms to the constraint that, if adaptive edge segmentation which adaptively divides the edges included in the base mesh is applied, multiple vertices of the base mesh corresponding to a single texture position are at the same three-dimensional vertex position; and an encoding unit that encodes the 3D data according to the result of the check that the 3D data conforms to the constraint.

[0055] Furthermore, the first encoding method performed by the first encoding device may include processing related to bitstream constraints, which includes obtaining a base mesh from which vertices have been thinned out from the original mesh to be encoded, which is composed of vertices and connections representing the three-dimensional structure of an object; obtaining UV unwrapping data representing the texture coordinates of the base mesh; and, if adaptive edge segmentation is applied to adaptively divide the edges contained in the base mesh, checking whether the 3D data, including the base mesh and UV unwrapping data, conforms to the constraint that multiple vertices of the base mesh corresponding to a single texture position are at the same three-dimensional vertex position; and encoding the 3D data according to the result of the check that the 3D data conforms to the constraint.

[0056] The first program may also be a program that causes a computer to perform a process that includes obtaining a base mesh from which vertices have been thinned out from the original mesh to be encoded, which is composed of vertices and connections representing the three-dimensional structure of an object; obtaining UV unwrapping data representing the texture coordinates of the base mesh; and checking whether the 3D data, including the base mesh and UV unwrapping data, conforms to the constraint that, if adaptive edge segmentation which adaptively divides the edges included in the base mesh is applied, multiple vertices of the base mesh corresponding to a single texture position are at the same three-dimensional vertex position; and encoding the 3D data according to the result of the check that the 3D data conforms to the constraint.

[0057] For example, this check may be performed using the algorithm shown in Figure 11. In the example in Figure 11, when referencing the 3D coordinates of a 3D vertex to be applied as the coordinates of a UV vertex, if a 3D coordinate has not yet been assigned to that UV vertex, the referenced 3D coordinate will be set as follows: if(table[texIdx][0] == -inf.){ table[texIdx] = baseSubmeshFrame.verCoordsArray[verIdx]

[0058] Furthermore, if the referenced 3D coordinates have already been assigned to those UV vertices, the 3D coordinates will be determined to match, as shown below. In this case, the V-DMC data will be encoded.} else if(table[texIdx] == baseSubmeshFrame.verCoordsArray[verIdx]){ / / 3D positions are same.

[0059] Furthermore, if a different 3D coordinate is already assigned to that UV vertex, it will be determined that the 3D coordinates do not match, as shown below. In this case, a different process will be performed.}else{ / / 3D positions are NOT same.

[0060] By performing such verification, V-DMC data that conforms to the bitstream constraints described above (Figure 10) can be encoded. Therefore, in adaptive edge segmentation for the base mesh, the occurrence of differences between the triangle subdivision in the UV unwrapped data and the triangle subdivision of the 3D vertices can be suppressed. Consequently, for example, the reduction in the quality of 3D data obtained by reconstruction can be suppressed.

[0061] <Determination based on the enable flag> Whether or not adaptive edge segmentation is applied may be determined based on the enable flag for edge-based segmentation. In other words, if this enable flag is true, it may be determined that "adaptive edge segmentation is applied". For example, in the first encoding device, the constraint processing unit may check whether the V-DMC data conforms to the bitstream constraints (Figure 10) described above if the enable flag for edge-based segmentation is true.

[0062] By making this determination based on the enable flag, it becomes easier to determine whether or not adaptive edge segmentation is applied.

[0063] <Transmission of Verification Results> Information indicating the result of the verification of compliance with the bitstream constraints described above may also be included in the V-DMC bitstream and provided to the decoding side. For example, flag information indicating the result of this verification may be stored in the V-DMC bitstream. For example, in the first encoding device, the constraint processing unit may generate a flag indicating the result of the verification and include it in the 3D data. By doing so, the decoding side can easily determine whether or not the V-DMC bitstream complies with the constraints in Figure 10.

[0064] <Decoding of V-DMC bitstream conforming to bitstream constraints> Alternatively, for example, the first decoding device may include a decoding unit that decodes a bitstream conforming to the constraint that, when adaptive edge segmentation which adaptively divides the edges included in the base mesh is applied, multiple vertices of the base mesh corresponding to a single texture position are at the same three-dimensional vertex position, and generates a base mesh and UV unwrap data representing the texture coordinates of the base mesh, and a subdivision unit that subdivides the base mesh using the UV unwrap data.

[0065] Furthermore, if the first decoding method performed by the first decoding device is applied to adaptive edge segmentation that adaptively divides the edges included in the base mesh, the first decoding device may include decoding a bitstream that conforms to the constraint that multiple vertices of the base mesh corresponding to a single texture position are at the same three-dimensional vertex position, generating a base mesh and UV unwrapped data representing the texture coordinates of the base mesh, and subdividing the base mesh using the UV unwrapped data.

[0066] Alternatively, the second program may be a program that causes a computer to perform a process that includes decoding a bitstream that conforms to the constraint that, when adaptive edge segmentation is applied to adaptively divide the edges contained in the base mesh, multiple vertices of the base mesh corresponding to a single texture position are at the same three-dimensional vertex position, generating a base mesh and UV unwrapped data representing the texture coordinates of the base mesh, and subdividing the base mesh using the UV unwrapped data.

[0067] By decoding the bitstream after the above-described verification, V-DMC data conforming to the above-described bitstream constraints (Figure 10) can be encoded. Therefore, in adaptive edge segmentation for the base mesh, the occurrence of differences between the triangle subdivision in the UV unwrapped data and the triangle subdivision of the 3D vertices can be suppressed. Consequently, for example, the reduction in the quality of the 3D data obtained by reconstruction can be suppressed.

[0068] <Method 1-1> When Method 1 described above is applied, for example, as shown in the third row from the top of Figure 9, the encoding of the V-DMC data may be processed as an error when the V-DMC data does not conform to this constraint (Figure 10) (Method 1-1). For example, in the first encoding device, the constraint processing unit may process the encoding of the 3D data as an error when the 3D data does not conform to the constraint.

[0069] The error handling process can be anything as long as it does not involve encoding the V-DMC data. For example, it could be to cancel (terminate) the V-DMC data encoding process, to temporarily suspend the V-DMC data encoding process and display a warning, or to perform any other action.

[0070] By doing so, it is possible to control the encoding of V-DMC data that does not conform to the bitstream constraints described above (Figure 10). Therefore, in adaptive edge segmentation for the base mesh, it is possible to suppress the occurrence of differences between the triangle subdivision in the UV unwrapped data and the triangle subdivision of the 3D vertices. Therefore, for example, it is possible to suppress the reduction in the quality of the 3D data obtained by reconstruction.

[0071] <Method 1-2> Furthermore, when Method 1 described above is applied, for example, as shown in the fourth row from the top of Figure 9, if the V-DMC data does not conform to this constraint (Figure 10), the UV vertex index may be updated so that the V-DMC data conforms to this constraint, and the updated V-DMC data may be encoded (Method 1-2). For example, in the first encoding device, if the constraint processing unit does not conform to the constraint, the vertex index in the UV unwrapped data may be updated so that multiple vertices of the base mesh corresponding to a single texture position correspond to different vertices in the UV unwrapped data.

[0072] For example, as shown in Figure 12A, if 3D vertices "2" and "4" located at different positions (3D positions) in 3D space (3D) are placed at a single position (2D position) in the UV plane (2D), then the UV vertices corresponding to those 3D vertices are assigned a common index. In other words, regardless of the corresponding 3D vertices, UV vertices existing at a single position are treated as a single UV vertex. Consequently, this no longer conforms to the bitstream constraints mentioned above.

[0073] Therefore, we add an index to the duplicate UV vertices. In other words, we identify the UV vertex for each corresponding 3D vertex and assign a different index to it. That is, we assume that multiple UV vertices are located in a single position. In the example of Figure 12A, the index of the UV vertex corresponding to 3D vertex "2" remains "2", and the index of the UV vertex corresponding to 3D vertex "4" is set to "6". Here, the updated (added) index is represented by a circled number. Consequently, the 2D triangle information for triangles in the UV plane is updated as shown in Figure 12B (the 3D triangle information for triangles in 3D space is not updated). Also, the 2D vertex information for UV vertices is updated as shown in Figure 13A (the 3D vertex information for 3D vertices is not updated). Also, the 2D edge information for edges in the UV plane is updated as shown in Figure 13B (the 3D edge information for edges in 3D space is not updated).

[0074] In this way, UV vertices corresponding to different 3D vertices located at a single position on the UV plane are managed as multiple UV vertices. Therefore, they comply with the bitstream constraints described above. In other words, V-DMC data that complies with the bitstream constraints described above (Figure 10) can be encoded. Therefore, in adaptive edge segmentation of the base mesh, the occurrence of differences between the triangle subdivision in the UV unwrapped data and the triangle subdivision of the 3D vertices can be suppressed. Therefore, for example, the reduction in the quality of 3D data obtained by reconstruction can be suppressed.

[0075] <Method 1-3> Furthermore, when Method 1 described above is applied, for example, as shown in the fifth row from the top of Figure 9, if the V-DMC data does not conform to this constraint (Figure 10), the UV unwrapping may be updated so that the V-DMC data conforms to this constraint, and the updated V-DMC data may be encoded (Method 1-3). For example, in the first encoding device, if the constraint processing unit does not conform to the constraint, the UV unwrapping data may be updated so that multiple vertices of the base mesh corresponding to a single texture position correspond to different positions in the UV unwrapping data.

[0076] In other words, the UV unwrapping is redone so that 3D vertices at different positions in 3D space (3D positions) are placed at different positions on the UV plane. By doing so, the model conforms to the bitstream constraints mentioned above. That is, V-DMC data conforming to the bitstream constraints (Figure 10) can be encoded. Therefore, in adaptive edge segmentation of the base mesh, the occurrence of discrepancies between the triangle subdivision in the UV unwrapping data and the triangle subdivision of the 3D vertices can be suppressed. Consequently, for example, the reduction in the quality of 3D data obtained by reconstruction can be suppressed.

[0077] <Examples of Configuration Applications> For example, a base mesh may be generated from the original mesh, and the above-mentioned verification may be performed on that base mesh. For example, the first encoding device may further include a base mesh generation unit that generates a base mesh by thinning out vertices from the original mesh. The constraint processing unit may then perform the above-mentioned verification on the 3D data including the generated base mesh.

[0078] Furthermore, the V-DMC bitstream may be checked to see if it complies with the above-mentioned constraints. In that case, the V-DMC bitstream may be decoded to generate V-DMC data, and the check may be performed using the base mesh and its UV unwrapping data contained in the V-DMC data. For example, the first encoding device may further include a decoding unit that decodes the bitstream and generates 3D data. The constraint processing unit may then perform the above-mentioned check on the generated 3D data.

[0079] <Method 2> Alternatively, the decoding side that decodes the V-DMC bitstream may check whether the decoded V-DMC data conforms to the bitstream constraints described above (Figure 10). In other words, as shown in the sixth row from the top of the table in Figure 9, the decoding V-DMC data may be checked to see if it conforms to these constraints, and if it does, the base mesh may be subdivided (Method 2).

[0080] For example, the second decoding device may include a decoding unit that decodes a bitstream and generates a base mesh obtained by thinning out vertices from the original mesh to be encoded, which is composed of vertices and connections representing the three-dimensional structure of an object, and UV unwrap data representing the texture coordinates of the base mesh; a constraint processing unit that performs bitstream constraint processing, which includes obtaining the generated base mesh, obtaining the generated UV unwrap data, and checking whether the 3D data including the base mesh and UV unwrap data conforms to the constraint that, if adaptive edge segmentation which adaptively divides the edges included in the base mesh is applied, multiple vertices of the base mesh corresponding to a single texture position are at the same three-dimensional vertex position; and a subdivision unit that subdivides the base mesh using the UV unwrap data according to the result of the check that the 3D data conforms to the constraint.

[0081] For example, the second decoding method performed by the second decoding device may include decoding the bitstream and generating a base mesh from which vertices have been thinned out from the original mesh to be encoded, which is composed of vertices and connections representing the three-dimensional structure of an object, and UV unwrap data representing the texture coordinates of the base mesh; obtaining the generated base mesh; obtaining the generated UV unwrap data; and performing processing on the bitstream constraints, including checking whether the 3D data, including the base mesh and UV unwrap data, conforms to the constraint that, if adaptive edge segmentation which adaptively divides the edges included in the base mesh is applied, multiple vertices of the base mesh corresponding to a single texture position are at the same three-dimensional vertex position; and depending on the result of the check that the 3D data conforms to the constraint, subdividing the base mesh using the UV unwrap data.

[0082] For example, the third program may be a program that causes a computer to perform a bitstream constraint process that includes decoding the bitstream and generating a base mesh from which vertices have been thinned out from the original mesh to be encoded, which is composed of vertices and connections representing the three-dimensional structure of an object, and UV unwrapping data representing the texture coordinates of the base mesh; obtaining the generated base mesh; obtaining the generated UV unwrapping data; and checking whether the 3D data, including the base mesh and UV unwrapping data, conforms to the constraint that, if adaptive edge segmentation which adaptively divides the edges included in the base mesh is applied, multiple vertices of the base mesh corresponding to a single texture position are at the same three-dimensional vertex position; and depending on the result of the check that the 3D data conforms to the constraint, subdividing the base mesh using the UV unwrapping data.

[0083] For example, this verification may be performed using an algorithm such as the one shown in Figure 11, similar to the verification during encoding described above. By performing such a verification, the base mesh can be subdivided only if the decoded V-DMC data conforms to the bitstream constraints described above (Figure 10). Therefore, in adaptive edge segmentation of the base mesh, it is possible to suppress the occurrence of differences between the subdivision of triangles in the UV unwrapped data and the subdivision of triangles at 3D vertices. Consequently, for example, it is possible to suppress the reduction in the quality of 3D data obtained by reconstruction.

[0084] <Determination based on the enable flag> Whether or not adaptive edge segmentation is applied may be determined based on the enable flag for edge-based segmentation. In other words, if this enable flag is true, it may be determined that "adaptive edge segmentation is applied". For example, in the second decoder, the constraint processing unit may check whether the V-DMC data conforms to the bitstream constraints (Figure 10) described above if the enable flag for edge-based segmentation is true.

[0085] By making this determination based on the enable flag, it becomes easier to determine whether or not adaptive edge segmentation is applied.

[0086] <Transmission of Verification Results> Furthermore, if a check is performed during the encoding of V-DMC data to ensure compliance with the bitstream constraints described above, information indicating the result (e.g., a flag) may be included in the V-DMC bitstream and provided to the decoding side. If such information exists, the decoding side of the V-DMC bitstream may perform a check based on that information to determine whether the V-DMC data complies with the bitstream constraints. For example, in the second decoding device, the constraint processing unit may perform the check based on a flag indicating the result of the check extracted from the bitstream. By doing so, it is possible to easily determine whether the V-DMC bitstream complies with the constraints in Figure 10.

[0087] <Method 2-1> When Method 2 described above is applied, for example, as shown in the seventh row from the top of Figure 9, the base mesh subdivision may be processed as an error when the decoded V-DMC data does not conform to this constraint (Figure 10) (Method 2-1). For example, in the second decoding device, the constraint processing unit may process the base mesh subdivision as an error when the 3D data does not conform to the constraint.

[0088] The error handling method can be anything as long as it does not subdivide the base mesh. For example, it could be to cancel (terminate) the base mesh subdivision process (i.e., the reconstruction of the V-DMC data), or to temporarily stop the base mesh subdivision process and display a warning, or any other action.

[0089] By doing so, it is possible to control the base mesh of V-DMC data that does not conform to the bitstream constraints (Figure 10) described above, so as not to subdivide it. Therefore, in adaptive edge segmentation of the base mesh, it is possible to suppress the occurrence of differences between the subdivision of triangles in the UV unwrapped data and the subdivision of triangles at 3D vertices. Therefore, for example, it is possible to suppress the reduction in the quality of 3D data obtained by reconstruction.

[0090] <Method 2-2> Furthermore, when Method 2 described above is applied, for example, as shown in the eighth row from the top of Figure 9, if the decoded V-DMC data does not conform to this constraint (Figure 10), the UV vertex index may be updated so that the V-DMC data conforms to this constraint, and the base mesh of the updated V-DMC data may be subdivided (Method 2-2). For example, in the second decoding device, if the constraint processing unit does not conform to the constraint, the vertex index in the UV unwrapped data may be updated so that multiple vertices of the base mesh corresponding to a single texture position correspond to different vertices in the UV unwrapped data. The method for updating this index is the same as in the case of encoding described above.

[0091] In this way, UV vertices corresponding to different 3D vertices located at a single position on the UV plane are managed as multiple UV vertices. Therefore, they comply with the bitstream constraints described above. In other words, the base mesh of the V-DMC data can be subdivided to comply with the bitstream constraints described above (Figure 10). Therefore, in adaptive edge segmentation of the base mesh, the occurrence of differences between the subdivision of triangles in the UV unwrapped data and the subdivision of triangles in the 3D vertices can be suppressed. Therefore, for example, the reduction in the quality of 3D data obtained by reconstruction can be suppressed.

[0092] <Method 2-3> Furthermore, when Method 2 described above is applied, for example, as shown in the ninth row from the top of Figure 9, if the decoded V-DMC data does not conform to this constraint (Figure 10), the UV unwrapping may be updated so that the V-DMC data conforms to this constraint, and the base mesh of the updated V-DMC data may be subdivided (Method 2-3). For example, in the second decoding device, if the constraint processing unit does not conform to the constraint, the UV unwrapping data may be updated so that multiple vertices of the base mesh corresponding to a single texture position correspond to different positions in the UV unwrapping data.

[0093] In other words, the UV unwrapping is redone so that 3D vertices at different positions in 3D space (3D positions) are placed at different positions on the UV plane. By doing so, the model conforms to the bitstream constraints mentioned above. That is, the base mesh of the V-DMC data can be subdivided to conform to the bitstream constraints mentioned above (Figure 10). Therefore, in adaptive edge segmentation of the base mesh, the occurrence of differences between the subdivision of triangles in the UV unwrapped data and the subdivision of triangles at 3D vertices can be suppressed. Consequently, for example, the reduction in the quality of 3D data obtained by reconstruction can be suppressed.

[0094] <Examples of Configuration Applications> For example, displacement vectors may be applied to a subdivided base mesh. For example, the second decoding device may further include a displacement vector decoding unit that decodes a bitstream and generates displacement vectors, which are vector information indicating the displacement of the vertices of the subdivided base mesh, and a displacement vector application unit that applies the generated displacement vectors to the subdivided base mesh.

[0095] Furthermore, additional attributes may be applied. For example, the second decoding device may further include an attribute decoding unit that decodes the bitstream and generates an attribute map, which is a two-dimensional region in which the texture of the original mesh is packed, and an attribute application unit that further applies the attributes contained in the generated attribute map to the subdivided base mesh to which the displacement vector has been applied.

[0096] <Method 3> In adaptive edge segmentation of the base mesh, the subdivision method may be changed so that no difference occurs between the subdivision of triangles in the UV unwrapped data and the subdivision of triangles at 3D vertices. For example, as explained with reference to Figure 7, in the conventional subdivision method, UV triangle information was referenced when controlling the edge division of UV-unwrapped triangles, but 3D triangle information may be referenced instead of this UV triangle information. In other words, as shown in the bottom row of the table in Figure 9, the base mesh may be subdivided by referencing 3D triangle information (Method 3).

[0097] For example, the third decoding device may include a decoding unit that decodes a bitstream and generates a base mesh obtained by thinning out vertices from the original mesh to be encoded, which is composed of vertices and connections representing the three-dimensional structure of an object, and 3D triangle information which is information about the triangles of the base mesh and includes the index of each vertex constituting the triangle, and a subdivision unit that subdivides the base mesh by referring to the 3D triangle information.

[0098] Furthermore, the third decoding method performed by the third decoding device may include decoding the bitstream and generating a base mesh obtained by thinning out vertices from the original mesh to be encoded, which is composed of vertices and connections representing the three-dimensional structure of the object, and 3D triangle information, which is information about the triangles of the base mesh and includes the index of each vertex constituting the triangle, and subdividing the base mesh by referring to the 3D triangle information.

[0099] Alternatively, the fourth program may be a program that causes the computer to perform the following processes: decode the bitstream and generate a base mesh from which vertices have been thinned out from the original mesh to be encoded, which consists of vertices and connections representing the three-dimensional structure of the object; generate 3D triangle information, which is information about the triangles of the base mesh and includes the index of each vertex constituting the triangle; and subdivide the base mesh by referring to the 3D triangle information.

[0100] For example, when controlling the edge subdivision of a UV-unwrapped triangle as in the conventional method, referring to UV triangle information only allows for a one-to-one correspondence between UV vertices and 3D vertices, as shown in Figure 14A. Therefore, as mentioned above, there may be cases where it is not possible to correctly associate 3D vertices with UV vertices.

[0101] Therefore, by referring to 3D triangle information instead of UV triangle information, the index of the 3D vertex can be directly determined, and in effect, multiple 3D vertices can be associated with a single UV vertex, as shown in Figure 14B. Consequently, in adaptive edge segmentation of the base mesh, the occurrence of discrepancies between the triangle subdivision in the UV unwrapped data and the triangle subdivision of the 3D vertices can be suppressed. Consequently, for example, the reduction in the quality of the 3D data obtained by reconstruction can be suppressed.

[0102] <Examples of Configuration Applications> For example, displacement vectors may be applied to a subdivided base mesh. For example, the third decoding device may further include a displacement vector decoding unit that decodes a bitstream and generates displacement vectors, which are vector information indicating the displacement of the vertices of the subdivided base mesh, and a displacement vector application unit that applies the generated displacement vectors to the subdivided base mesh.

[0103] Furthermore, additional attributes may be applied. For example, the third decoding device may further include an attribute decoding unit that decodes the bitstream and generates an attribute map, which is a two-dimensional region in which the texture of the original mesh is packed, and an attribute application unit that further applies the attributes contained in the generated attribute map to the subdivided base mesh to which the displacement vector has been applied.

[0104] Furthermore, this technique (Method 3) can also be applied to encoding devices that encode V-DMC when reconstructing the mesh. The method is the same as in the case of the third decoding device described above. In other words, when the encoding device subdivides the base mesh to reconstruct the mesh, it can refer to 3D triangle information instead of UV triangle information to control the edge division of the UV-unwrapped triangles.

[0105] <4. First Embodiment (Method 1)> <First Encoding Device> This technology can be applied to any device. For example, this technology can be applied to an encoding device that encodes a mesh and generates a bitstream. Figure 15 is a block diagram showing an example of the configuration of an encoding device, which is one aspect of an information processing device (first encoding device) to which Method 1 of this technology is applied. The encoding device 300 shown in Figure 15 is a device that encodes a mesh and generates its bitstream. Therefore, the encoding device 300 can also be called a bitstream generation device that generates a bitstream.

[0106] Figure 15 shows the main components such as the processing unit and data flow, but it does not necessarily represent everything. In other words, the encoding device 300 may have processing units that are not shown as blocks in Figure 15, or processes and data flows that are not shown as arrows or other symbols in Figure 15.

[0107] The encoding device 300 encodes the mesh in essentially the same way as the V-DMC described in the above-mentioned non-patent literature, except that it applies this technology. For example, the encoding device 300 obtains the original mesh to be encoded and an attribute map containing the texture corresponding to that original mesh. This original mesh includes not only information about the mesh geometry but also information indicating the correspondence with the attribute map (e.g., a UV list). The encoding device 300 converts the original mesh and attribute map into V-DMC data, encodes it, generates a V-DMC bitstream, and outputs it.

[0108] As shown in Figure 15, the encoding device 300 (first encoding device) has a pre-processing unit 311 and a V-DMC encoding unit 312. The pre-processing unit 311 performs processing related to pre-processing before encoding. As shown in Figure 15, the pre-processing unit 311 has a base mesh generation unit 321, an atlas information generation unit 322, a constraint processing unit 323, and a displacement vector generation unit 324.

[0109] The base mesh generation unit 321 performs processing related to the generation of a base mesh. For example, the base mesh generation unit 321 may acquire the original mesh input to the encoding device 300. Alternatively, the base mesh generation unit 321 may perform decimation processing (vertex thinning) on ​​the original mesh to generate a base mesh. The base mesh generation unit 321 may supply the generated base mesh together with the original mesh to the atlas information generation unit 322.

[0110] The atlas information generation unit 322 performs processing related to the generation of atlas information corresponding to the base mesh. For example, the atlas information generation unit 322 may acquire the base mesh or original mesh supplied from the base mesh generation unit 321. The atlas information generation unit 322 may generate atlas information by performing UV unwrapping on the base mesh, etc. In other words, this atlas information also includes UV unwrapping data such as vertex information, edge information, and triangle information. The atlas information generation unit 322 may exchange information with the constraint processing unit 323. The atlas information generation unit 322 may supply the generated atlas information, along with the base mesh, etc., to the displacement vector generation unit 324.

[0111] The constraint processing unit 323 performs processing related to bitstream constraints. In doing so, the constraint processing unit 323 may exchange information with the atlas information generation unit 322.

[0112] The displacement vector generation unit 324 performs processing related to the generation of displacement vectors. For example, the displacement vector generation unit 324 may acquire the base mesh and atlas information supplied from the atlas information generation unit 322. Alternatively, the displacement vector generation unit 324 may acquire the original mesh input to the encoding device 300. Furthermore, the displacement vector generation unit 324 may use this information to generate displacement vectors that displace the vertices of the subdivided base mesh. The displacement vector generation unit 324 may supply the generated displacement vectors together with the base mesh and atlas information to the V-DMC encoding unit 312.

[0113] The V-DMC encoding unit 312 performs processing related to the encoding of V-DMC data. For example, the V-DMC encoding unit 312 may acquire the original mesh input to the encoding device 300. The V-DMC encoding unit 312 may also acquire the base mesh, displacement vector, atlas information, etc., supplied from the displacement vector generation unit 324. The V-DMC encoding unit 312 may also acquire the attribute map input to the encoding device 300. The V-DMC encoding unit 312 may encode the atlas information, base mesh, displacement vector, and attribute map, respectively, and generate encoded data for each. Therefore, the V-DMC encoding unit 312 can also be called an encoding unit. The V-DMC encoding unit 312 may also multiplex these encoded data as substreams to generate a single bitstream. This bitstream is also called a V-DMC bitstream. Therefore, the V-DMC encoding unit 312 can also be called a bitstream generation unit (or V-DMC bitstream generation unit). The V-DMC encoding unit 312 may output the generated V-DMC bitstream to the outside of the encoding device 300.

[0114] <V-DMC Encoding Unit> Figure 16 is a block diagram showing an example of the main configuration of the V-DMC encoding unit 312. Figure 16 shows the main components such as the processing unit and data flow, but it does not necessarily show everything. In other words, the V-DMC encoding unit 312 may have processing units that are not shown as blocks in Figure 16, or processes and data flows that are not shown as arrows, etc., in Figure 16.

[0115] As shown in Figure 16, the V-DMC coding unit 312 includes an atlas information coding unit 351, a base mesh coding unit 352, a displacement vector correction unit 353, a displacement vector coding unit 354, a mesh reconstruction unit 355, an attribute map conversion unit 356, an attribute coding unit 357, and a multiplexing unit 358.

[0116] The Atlas information coding unit 351 performs processing related to coding the Atlas information. For example, the Atlas information coding unit 351 may acquire Atlas information supplied from the displacement vector generation unit 324. Alternatively, the Atlas information coding unit 351 may code the acquired Atlas information using a predetermined coding scheme to generate coded Atlas information data. The Atlas information coding unit 351 may supply the generated coded Atlas information data to the multiplexing unit 358.

[0117] The base mesh coding unit 352 performs processing related to the coding of the base mesh. For example, the base mesh coding unit 352 may acquire the base mesh supplied from the displacement vector generation unit 324. The base mesh coding unit 352 may acquire atlas information supplied from the displacement vector generation unit 324. The base mesh coding unit 352 may quantize the acquired base mesh, code it using a predetermined coding method (e.g., Draco), and generate coded base mesh data. In this case, the base mesh coding unit 352 may perform coding of the base mesh based on the acquired atlas information. The base mesh coding unit 352 may supply the coded base mesh data it has generated to the displacement vector correction unit 353. The base mesh coding unit 352 may also supply the coded base mesh data it has generated to the multiplexing unit 358.

[0118] The displacement vector correction unit 353 performs processing related to the correction of the displacement vector. For example, the displacement vector correction unit 353 may acquire the base mesh and displacement vector supplied from the displacement vector generation unit 324. Alternatively, the displacement vector correction unit 353 may acquire the encoded data of the base mesh supplied from the base mesh encoding unit 352. Based on this information, the displacement vector correction unit 353 may correct the displacement vector. For example, the displacement vector correction unit 353 may decode the acquired encoded data of the base mesh, compare the base mesh before and after encoding to determine the encoding distortion of the base mesh, and correct the displacement vector according to that encoding distortion. The displacement vector correction unit 353 may supply the corrected displacement vector to the displacement vector encoding unit 354. Alternatively, the displacement vector correction unit 353 may dequantize the decoded base mesh and supply it to the mesh reconstruction unit 355.

[0119] The displacement vector coding unit 354 performs processing related to the coding of the displacement vector. For example, the displacement vector coding unit 354 may acquire the displacement vector supplied from the displacement vector correction unit 353. Alternatively, the displacement vector coding unit 354 may generate a displacement map by performing a wavelet transform on the displacement vector, quantizing it, and packing it into a two-dimensional region. The displacement vector coding unit 354 may also generate a displacement video using the displacement map as the frame image. In other words, the displacement video is a moving image in which the displacement map, which is a two-dimensional region in which the displacement vector is packed, is used as the frame image. The displacement vector coding unit 354 may code the generated displacement video using a predetermined coding scheme for 2D moving images to generate coded data of the displacement vector (displacement video). The displacement vector coding unit 354 may supply the coded data of the displacement vector thus generated to the multiplexing unit 358. Alternatively, the displacement vector coding unit 354 may decode the generated coded data, unpack the displacement vector from the displacement map, and dequantize the displacement vector. The displacement vector coding unit 354 may supply its inversely quantized displacement vector to the mesh reconstruction unit 355.

[0120] The displacement vector coding unit 354 may also generate encoded data of the displacement vector by arithmetic coding the displacement vector. In that case, the displacement vector coding unit 354 may generate the displacement vector by arithmetic decoding the encoded data and supply it to the mesh reconstruction unit 355.

[0121] Alternatively, the displacement vector coding unit 354 may acquire atlas information supplied from the displacement vector generation unit 324 and perform displacement vector coding based on the acquired atlas information.

[0122] The mesh reconstruction unit 355 performs processing related to mesh reconstruction. For example, the mesh reconstruction unit 355 may acquire a base mesh supplied from the displacement vector correction unit 353. Alternatively, the mesh reconstruction unit 355 may acquire displacement vectors supplied from the displacement vector encoding unit 354. The mesh reconstruction unit 355 may reconstruct the mesh using these. The mesh reconstruction unit 355 may supply the reconstructed mesh to the attribute map conversion unit 356.

[0123] The attribute map conversion unit 356 performs processing related to the conversion of the attribute map. For example, the attribute map conversion unit 356 may acquire the reconstructed mesh supplied from the mesh reconstruction unit 355. Alternatively, the attribute map conversion unit 356 may acquire atlas information supplied from the displacement vector generation unit 324. Furthermore, the attribute map conversion unit 356 may acquire the original mesh and attribute map input to the encoding device 300. The attribute map conversion unit 356 may convert the acquired attribute map based on other acquired information. For example, the attribute map conversion unit 356 may convert the attribute map to correspond to the reconstructed mesh based on atlas information, the original mesh, etc. In other words, the attribute map conversion unit 356 can also be said to generate the converted attribute map. Therefore, the attribute map conversion unit 356 can also be called an attribute map generation unit. The attribute map conversion unit 356 may supply the converted attribute map to the attribute encoding unit 357.

[0124] The attribute coding unit 357 performs processing related to attribute coding. For example, the attribute coding unit 357 may acquire an attribute map supplied from the attribute map conversion unit 356. Alternatively, the attribute coding unit 357 may generate an attribute video using the acquired attribute map as a frame image. Furthermore, the attribute coding unit 357 may code the generated attribute video using a predetermined coding scheme for 2D moving images to generate coded attribute data. The attribute coding unit 357 may supply the generated coded attribute data to the multiplexing unit 358.

[0125] The multiplexing unit 358 performs processing related to the multiplexing of encoded data (substreams). For example, the multiplexing unit 358 may acquire encoded data of atlas information supplied from the atlas information encoding unit 351. The multiplexing unit 358 may also acquire encoded data of base mesh supplied from the base mesh encoding unit 352. The multiplexing unit 358 may also acquire encoded data of displacement vectors supplied from the displacement vector encoding unit 354. The multiplexing unit 358 may also acquire encoded data of attributes supplied from the attribute encoding unit 357. The multiplexing unit 358 may multiplex these encoded data as substreams to generate a V-DMC bitstream. Therefore, the multiplexing unit 358 can also be called a bitstream generation unit (or V-DMC bitstream generation unit). The multiplexing unit 358 may output the generated V-DMC bitstream to the outside of the encoding device 300. For example, the multiplexing unit 358 may supply its V-DMC bitstream to the decoding device 400, which will be described later. Therefore, the multiplexing unit 358 can also be said to be a supply unit (provider) of its V-DMC bitstream.

[0126] <Application of this technology> This technology (Method 1) may be applied to an encoding device 300 having such a configuration. For example, the constraint processing unit 323 may perform bitstream constraint processing, which includes obtaining a base mesh from which vertices have been thinned out from the original mesh to be encoded, which is composed of vertices and connections representing the three-dimensional structure of an object; obtaining UV unwrapping data representing the texture coordinates of the base mesh; and, if adaptive edge segmentation that adaptively divides the edges included in the base mesh is applied, checking whether the 3D data, including the base mesh and UV unwrapping data, conforms to the constraint that multiple vertices of the base mesh corresponding to a single texture position are at the same three-dimensional vertex position. Alternatively, the V-DMC encoding unit 312 may encode the 3D data according to the result of the check that the 3D data conforms to the constraint.

[0127] With this configuration, the encoding device 300 can encode V-DMC data that conforms to the bitstream constraints described above (Figure 10). Therefore, the encoding device 300 can suppress the occurrence of differences between the triangle subdivision in the UV unwrapped data and the triangle subdivision of the 3D vertices in adaptive edge segmentation for the base mesh. Consequently, the encoding device 300 can suppress the reduction in the quality of the 3D data obtained by reconstruction, for example.

[0128] Furthermore, if the edge-based segmentation enable flag is true, the constraint processing unit 323 may perform a check to determine whether or not the above constraints are met. By doing so, the constraint processing unit 323 can easily determine whether or not adaptive edge segmentation is applied.

[0129] Furthermore, the constraint processing unit 323 may generate a flag indicating the result of the check to see whether or not the data conforms to the above constraints, and include it in the 3D data. In other words, the atlas information generation unit 322 may include the flag in the atlas information, and the V-DMC encoding unit 312 (atlas information encoding unit 351) may encode the atlas information (including the flag). By doing so, the flag generated by the constraint processing unit 323, indicating the result of the check to see whether or not the data conforms to the above constraints, can be stored in the V-DMC bitstream. In other words, it can be transmitted to the decoding side that decodes the V-DMC bitstream.

[0130] Furthermore, Method 1-1 may be applied to the encoding device 300. For example, if the constraint processing unit 323 does not conform to the above-mentioned constraints, it may perform error processing on the encoding of the 3D data. The content of this error processing can be anything as long as it does not encode the V-DMC data. For example, the encoding process of the V-DMC data may be stopped (forcibly terminated), the encoding process of the V-DMC data may be temporarily stopped and a warning may be issued, or other processing may be performed. By doing so, it is possible to control the encoding of V-DMC data that does not conform to the above-mentioned bitstream constraints (Figure 10).

[0131] Furthermore, method 1-2 may be applied to this encoding device 300. For example, if the constraint processing unit 323 determines that the 3D data does not conform to the above constraints, it may update the vertex indices in the UV unwrapped data so that multiple vertices of the base mesh corresponding to a single texture position correspond to different vertices in the UV unwrapped data. In other words, a UV vertex is identified for each corresponding 3D vertex, and a different index is assigned. That is, multiple UV vertices are assumed to be located at a single position. By doing so, UV vertices corresponding to different 3D vertices located at a single position on the UV plane are managed as multiple UV vertices and conform to the above-mentioned bitstream constraints. In other words, V-DMC data conforming to the above-mentioned bitstream constraints (Figure 10) can be encoded.

[0132] Furthermore, method 1-3 may be applied to this encoding device 300. For example, if the constraint processing unit 323 determines that the 3D data does not conform to the above constraints, it may update the UV unwrapped data so that multiple vertices of the base mesh corresponding to a single texture position correspond to different positions in the UV unwrapped data. In other words, the UV unwrapping is redone so that 3D vertices at different positions in 3D space (3D positions) are placed at different positions on the UV plane. By doing so, the data conforms to the above-mentioned bitstream constraints.

[0133] <Encoding Process Flow> An example of the encoding process flow performed by this encoding device 300 will be explained with reference to the flowchart in Figure 17.

[0134] When the encoding process is started, the base mesh generation unit 321 of the encoding device 300 decimates the original mesh to be encoded in step S301 and generates a base mesh.

[0135] In step S302, the atlas information generation unit 322 generates atlas information for its base mesh.

[0136] In step S303, the constraint processing unit 323 applies the method 1 described above to perform constraint processing and makes the V-DMC data to be encoded conform to the above constraints.

[0137] In step S304, the displacement vector generation unit 324 generates a displacement vector.

[0138] In step S305, the V-DMC encoding unit 312 performs V-DMC encoding processing and encodes the V-DMC data generated as described above to generate a V-DMC bitstream.

[0139] The encoding process ends when the processing in step S305 is completed. The encoding device 300 performs this encoding process for each frame of the original mesh.

[0140] <Constraint Processing Flow 1> Next, an example of the constraint processing flow executed in step S303 of Figure 17 when method 1-1 is applied will be explained with reference to the flowchart in Figure 18.

[0141] When constraint processing is initiated, in step S321, the constraint processing unit 323 obtains the base mesh and atlas information (including UV unwrapping data) from the atlas information generation unit 322.

[0142] In step S322, the constraint processing unit 323 checks whether the V-DMC data, including the base mesh and UV unwrapping data, conforms to the constraint that "when adaptive edge segmentation is applied, multiple vertices of the base mesh corresponding to each single texture position are at the same 3D vertex position." In other words, the constraint processing unit 323 checks whether, when adaptive edge segmentation, which adaptively divides the edges included in the base mesh, is applied, multiple vertices of the base mesh corresponding to each single texture position are at the same 3D vertex position.

[0143] In step S323, the constraint processing unit 323 determines whether the V-DMC data conforms to the constraint based on the result of the verification. If it is determined that the V-DMC data does not conform to the constraint, the process proceeds to step S324.

[0144] In step S324, the constraint processing unit 323 performs error processing on the encoding of the V-DMC data. For example, the constraint processing unit 323 may abort (terminate) the encoding of the V-DMC data. Alternatively, the constraint processing unit 323 may temporarily stop the encoding of the V-DMC data and issue a warning. Once the processing in step S324 is completed, the constraint processing is finished, and the process returns to Figure 17.

[0145] Furthermore, if it is determined in step S323 that the V-DMC data conforms to the constraint, the constraint processing ends and the process returns to Figure 17.

[0146] <Constraint Processing Flow 2> Next, an example of the constraint processing flow performed in step S303 of Figure 17 when method 1-2 is applied will be explained with reference to the flowchart in Figure 19.

[0147] When constraint processing is initiated, in step S341, the constraint processing unit 323 obtains the base mesh and atlas information (including UV unwrapping data) from the atlas information generation unit 322.

[0148] In step S342, the constraint processing unit 323 checks whether the V-DMC data, including the base mesh and UV unwrapping data, conforms to the constraint that "when adaptive edge segmentation is applied, multiple vertices of the base mesh corresponding to each single texture position are at the same 3D vertex position." In other words, the constraint processing unit 323 checks whether, when adaptive edge segmentation, which adaptively divides the edges included in the base mesh, is applied, multiple vertices of the base mesh corresponding to each single texture position are at the same 3D vertex position.

[0149] In step S343, the constraint processing unit 323 determines, based on the result of the verification, whether the V-DMC data conforms to the constraint. If it is determined that the V-DMC data conforms to the constraint, the process proceeds to step S344.

[0150] In step S344, the constraint processing unit 323 updates the UV vertex index so that the V-DMC data conforms to the constraint. That is, it assigns a different index to each of multiple UV vertices whose corresponding 3D vertices are different from each other, even if they are located at the same UV position. In this way, the V-DMC data conforms to the above constraint. When the processing in step S344 is completed, the constraint processing is finished, and the process returns to Figure 17.

[0151] Furthermore, if it is determined in step S343 that the V-DMC data conforms to the constraint, the constraint processing ends and the process returns to Figure 17.

[0152] <Constraint Processing Flow 3> Next, an example of the constraint processing flow executed in step S303 of Figure 17 when method 1-3 is applied will be explained with reference to the flowchart in Figure 20.

[0153] When constraint processing is initiated, in step S361, the constraint processing unit 323 obtains the base mesh and atlas information (including UV unwrapping data) from the atlas information generation unit 322.

[0154] In step S362, the constraint processing unit 323 checks whether the V-DMC data, including the base mesh and UV unwrapping data, conforms to the constraint that "when adaptive edge segmentation is applied, multiple vertices of the base mesh corresponding to each single texture location are at the same 3D vertex location." In other words, the constraint processing unit 323 checks whether, when adaptive edge segmentation, which adaptively divides the edges included in the base mesh, is applied, multiple vertices of the base mesh corresponding to each single texture location are at the same 3D vertex location.

[0155] In step S363, the constraint processing unit 323 determines whether the V-DMC data conforms to the constraint based on the result of the verification. If it is determined that the V-DMC data conforms to the constraint, the process proceeds to step S364.

[0156] In step S364, the constraint processing unit 323 updates the UV unwrapping so that the V-DMC data conforms to the constraints. In other words, it redoes the UV unwrapping so that multiple UV vertices are not placed at a single UV position. By doing so, the V-DMC data conforms to the above constraints. When the processing in step S364 is completed, the constraint processing is finished, and the process returns to Figure 17.

[0157] Furthermore, if it is determined in step S363 that the V-DMC data conforms to the constraint, the constraint processing ends and the process returns to Figure 17.

[0158] By performing each process as described above, the encoding device 300 can encode V-DMC data that conforms to the bitstream constraints (Figure 10) described above. Therefore, the encoding device 300 can suppress the occurrence of differences between the subdivision of triangles in the UV unwrapped data and the subdivision of triangles at 3D vertices in adaptive edge segmentation for the base mesh. Therefore, the encoding device 300 can suppress the reduction in the quality of 3D data obtained by reconstruction, for example.

[0159] <Examples of Configuration Applications> As shown in Figure 15, the encoding device 300 (first encoding device) may, for example, generate a base mesh from the original mesh, and the constraint processing unit 323 may perform the above-mentioned verification on the base mesh. For example, the encoding device 300 may further include a base mesh generation unit 321 that generates a base mesh by thinning out vertices from the original mesh. The constraint processing unit 323 may then perform the above-mentioned verification on the 3D data including the generated base mesh.

[0160] Furthermore, as shown in Figure 21A, the encoding device 370 (first encoding device) may be configured to receive V-DMC data from an external source and perform the above-mentioned verification on the supplied V-DMC data. In the example of Figure 21A, the encoding device 370 has a constraint processing unit 323 and a V-DMC encoding unit 312. The constraint processing unit 323 may perform constraint processing on the V-DMC data supplied from outside the encoding device 370 and supply V-DMC data compliant with the above-mentioned constraints to the V-DMC encoding unit 312. The V-DMC encoding unit 312 may encode the V-DMC data and generate a V-DMC bitstream. In other words, the processing units such as the base mesh generation unit 321 in Figure 15 may be omitted.

[0161] Furthermore, as shown in Figure 21B, for example, the constraint processing device 380 (first encoding device) may be configured to check whether the V-DMC bitstream conforms to the above-mentioned constraints. As shown in Figure 21B, the constraint processing device 380 includes a V-DMC decoding unit 381, a constraint processing unit 323, and a V-DMC encoding unit 312. The V-DMC decoding unit 381 may acquire a V-DMC bitstream supplied from outside the constraint processing device 380, decode it, generate V-DMC data, and supply it to the constraint processing unit 323. The constraint processing unit 323 may acquire the V-DMC data, perform constraint processing on the V-DMC data, and supply V-DMC data conforming to the above-mentioned constraints to the V-DMC encoding unit 312. The V-DMC encoding unit 312 may encode the V-DMC data and generate a V-DMC bitstream.

[0162] <First Decoder> This technology can be applied to a decoding device that decodes mesh encoded data. Figure 22 is a block diagram showing an example of the configuration of a decoding device, which is one aspect of an information processing device (first decoding device) to which Method 1 of this technology is applied. The decoding device 400 shown in Figure 22 is a device that decodes the mesh encoded data (V-DMC bitstream generated by the multiplexing unit 358 (Figure 16)) generated in the encoding device 300 (Figure 15) and reconstructs the decoded mesh.

[0163] Figure 22 shows the main components such as the processing unit and data flow, but it does not necessarily represent everything. In other words, the decoding device 400 may have processing units that are not shown as blocks in Figure 22, or processes and data flows that are not shown as arrows or other symbols in Figure 22.

[0164] The decoding device 400 decodes the encoded mesh data, which has been encoded in essentially the same way as the V-DMC described in the above-mentioned non-patent document, except for applying this technology, and reconstructs the decoded mesh. For example, the decoding device 400 acquires a V-DMC bitstream. This V-DMC bitstream may be generated, for example, by the encoding device 300. As part of the reconstruction process, the decoding device 400 decodes the V-DMC bitstream and reconstructs the mesh (also referred to as the decoded mesh). The decoding device 400 also applies a texture to the decoded mesh, generates a display image for displaying the decoded mesh, and outputs it to the outside of the decoding device 400. For example, the decoding device 400 supplies the display image to an external display device for display.

[0165] As shown in Figure 22, the decoding device 400 (first decoding device) includes a demultiplexing unit 411, an atlas information decoding unit 412, a base mesh decoding unit 413, a subdivision unit 414, a displacement vector decoding unit 415, a displacement vector application unit 416, an attribute decoding unit 417, an attribute application unit 418, and a display processing unit 419.

[0166] The demultiplexing unit 411 performs processing related to demultiplexing. For example, the demultiplexing unit 411 may acquire the V-DMC bitstream to be decoded, which is supplied to the decoding device 400. Alternatively, the demultiplexing unit 411 may demultiplex the acquired V-DMC bitstream and extract encoded data of atlas information, encoded data of base mesh, encoded data of displacement vectors, and encoded data of attributes. Therefore, the demultiplexing unit 411 can also be said to be an acquisition unit for the V-DMC bitstream or the various information contained in the V-DMC bitstream. The demultiplexing unit 411 may supply the extracted encoded data of atlas information to the atlas information decoding unit 412. Alternatively, the demultiplexing unit 411 may supply the extracted encoded data of base mesh to the base mesh decoding unit 413. Alternatively, the demultiplexing unit 411 may supply the extracted encoded data of displacement vectors to the displacement vector decoding unit 415. Furthermore, the demultiplexing unit 411 may supply the encoded data of the extracted attributes to the attribute decoding unit 417.

[0167] The atlas information decoding unit 412 performs processing related to decoding the atlas information. For example, the atlas information decoding unit 412 may acquire encoded data of the atlas information supplied from the demultiplexing unit 411. Alternatively, the atlas information decoding unit 412 may decode the acquired encoded data of the atlas information and generate (restore) the atlas information. The atlas information decoding unit 412 may supply the generated atlas information to the subdivision unit 414. Although the arrows are omitted in Figure 22, the atlas information decoding unit 412 may also supply the atlas information to one or more of the base mesh decoding unit 413, subdivision unit 414, displacement vector decoding unit 415, displacement vector application unit 416, attribute decoding unit 417, attribute application unit 418, and display processing unit 419.

[0168] The base mesh decoding unit 413 performs processing related to the decoding of the base mesh. For example, the base mesh decoding unit 413 may acquire encoded data of the base mesh supplied from the demultiplexing unit 411. Alternatively, the base mesh decoding unit 413 may decode the acquired encoded data of the base mesh using a predetermined decoding method (e.g., Draco) to generate (restore) a base mesh (e.g., a vertex list or a triangle list). In this case, the base mesh decoding unit 413 may acquire atlas information supplied from the atlas information decoding unit 412 and decode the encoded data of the base mesh based on that atlas information. The base mesh decoding unit 413 may supply the generated base mesh to the subdivision unit 414.

[0169] The subdivision unit 414 performs processing related to the subdivision of the triangles of the base mesh. For example, the subdivision unit 414 may acquire the base mesh supplied from the base mesh decoding unit 413. The subdivision unit 414 may subdivide the base mesh (triangles) and generate division points. In this case, the subdivision unit 414 may acquire atlas information supplied from the atlas information decoding unit 412 and use it for the subdivision of the base mesh. The subdivision unit 414 may supply the subdivided base mesh to the displacement vector application unit 416.

[0170] The displacement vector decoding unit 415 performs processing related to the decoding of the displacement vector. For example, the displacement vector decoding unit 415 may acquire encoded data of the displacement vector (i.e., a displacement bitstream) supplied from the demultiplexing unit 411. The displacement vector decoding unit 415 may decode the encoded data of the displacement vector and generate (restore) the displacement vector. For example, if the displacement vector is encoded as a displacement video, the displacement vector decoding unit 415 may decode the encoded data of the displacement vector using a predetermined decoding method for 2D moving images, generate (restore) the displacement video, and unpack the displacement vector from the displacement map, which is a frame image of the displacement video. Alternatively, if the displacement video is arithmetic encoded, the displacement vector decoding unit 415 arithmetically decodes the encoded data of the displacement vector and generates the displacement vector. In this case, the displacement vector decoding unit 415 may acquire atlas information supplied from the atlas information decoding unit 412 and decode the displacement vector based on that atlas information. The displacement vector decoding unit 415 may supply the displacement vector obtained in this way to the displacement vector application unit 416.

[0171] The displacement vector application unit 416 performs processing related to the application of displacement vectors to the subdivided base mesh. For example, the displacement vector application unit 416 may acquire the subdivided base mesh supplied from the subdivision unit 414. The displacement vector application unit 416 may acquire the displacement vector supplied from the displacement vector decoding unit 415. The displacement vector application unit 416 may apply the displacement vector to the vertices of the subdivided base mesh. In other words, the displacement vector application unit 416 may generate a decoded mesh. In this case, the displacement vector application unit 416 may acquire atlas information supplied from the atlas information decoding unit 412 and apply the displacement vector to the vertices of the subdivided base mesh based on that atlas information. The displacement vector application unit 416 may supply the decoded mesh thus generated to the attribute application unit 418.

[0172] The attribute decoding unit 417 performs processing related to attribute decoding. For example, the attribute decoding unit 417 may acquire encoded attribute data supplied from the demultiplexing unit 411. Alternatively, the attribute decoding unit 417 may decode the acquired encoded attribute data using a predetermined decoding method for 2D moving images to generate (restore) an attribute video. In this case, the attribute decoding unit 417 may acquire atlas information supplied from the atlas information decoding unit 412 and decode the attributes based on that atlas information. The attribute decoding unit 417 may supply an attribute map, which is a frame image of the generated attribute video, to the attribute application unit 418.

[0173] The attribute application unit 418 performs processing related to the application of attributes to the decoded mesh. For example, the attribute application unit 418 may acquire the decoded mesh supplied from the displacement vector application unit 416. The attribute application unit 418 may acquire the attribute map supplied from the attribute decoding unit 417. The attribute application unit 418 may apply the attributes of the attribute map to the decoded mesh. In this case, the attribute application unit 418 may acquire atlas information supplied from the atlas information decoding unit 412 and apply attributes to the decoded mesh based on that atlas information. The attribute application unit 418 may supply the decoded mesh with the attributes applied in this way to the display processing unit 419.

[0174] The display processing unit 419 performs processing related to the display of the mesh. For example, the display processing unit 419 may acquire a decoded mesh to which attributes have been applied, supplied from the attribute application unit 418. The display processing unit 419 may render the acquired decoded mesh and generate a display image for displaying the decoded mesh. The display processing unit 419 may then supply the generated display image to an external device, such as another device, to display the display image.

[0175] <Application of this technology> This technology (Method 1) may be applied to a decoding device 400 with such a configuration. For example, the atlas information decoding unit 412 and the base mesh decoding unit 413 may decode a bitstream that conforms to the constraint that, when adaptive edge segmentation is applied to adaptively divide the edges included in the base mesh, multiple vertices of the base mesh corresponding to a single texture position are at the same three-dimensional vertex position, and generate a base mesh and UV unwrapped data representing the texture coordinates of the base mesh. Alternatively, the subdivision unit 414 may subdivide the base mesh using the UV unwrapped data.

[0176] With this configuration, the decoding device 400 can encode V-DMC data that conforms to the bitstream constraints described above (Figure 10). Therefore, the decoding device 400 can suppress the occurrence of differences between the triangle subdivision in the UV unwrapped data and the triangle subdivision of the 3D vertices in adaptive edge segmentation for the base mesh. Therefore, the decoding device 400 can suppress the reduction in the quality of the 3D data obtained by reconstruction, for example.

[0177] <Decryption Process Flow> An example of the decoding process flow performed by this decoding device 400 will be explained with reference to the flowchart in Figure 23.

[0178] When the decoding process is started, the demultiplexing unit 411 of the decoding device 400 demultiplexes the V-DMC bitstream in step S401.

[0179] In step S402, the Atlas information decoding unit 412 decodes the encoded data of the Atlas information and generates (restores) the Atlas information.

[0180] In step S403, the base mesh decoding unit 413 decodes the base mesh of the entire region.

[0181] In step S404, the subdivision unit 414 subdivides the base mesh.

[0182] In step S405, the displacement vector decoding unit 415 decodes the displacement vector.

[0183] In step S406, the displacement vector application unit 416 applies the displacement vector to the subdivided mesh.

[0184] In step S407, the attribute decoding unit 417 decodes the attribute bitstream and generates an attribute map.

[0185] In step S408, the attribute application unit 418 applies the attributes (textures, etc.) stored in the attribute map to the decoded mesh.

[0186] In step S409, the display processing unit 419 renders the decoded mesh to which the attributes have been applied to generate a display image.

[0187] When the process in step S409 is completed, the decoding process is finished. The decoding device 400 performs this decoding process for each frame of the original mesh.

[0188] By performing each process as described above, the decoding device 400 can encode V-DMC data that conforms to the bitstream constraints (Figure 10) described above. Therefore, the decoding device 400 can suppress the occurrence of differences between the subdivision of triangles in the UV unwrapped data and the subdivision of triangles at 3D vertices in adaptive edge segmentation for the base mesh. Consequently, the decoding device 400 can suppress the reduction in the quality of 3D data obtained by reconstruction, for example.

[0189] <5. Second Embodiment (Method 2)> <Second Encoding Device> Figure 24 is a block diagram showing an example of the configuration of an encoding device (second encoding device). The encoding device 500 shown in Figure 24 is basically the same as the encoding device 300, and encodes the mesh and generates its bitstream. Therefore, the encoding device 500 can also be called a bitstream generation device that generates a bitstream.

[0190] Figure 24 shows the main components such as the processing unit and data flow, but it does not necessarily represent everything. In other words, the encoding device 500 may have processing units that are not shown as blocks in Figure 24, or processes and data flows that are not shown as arrows or other symbols in Figure 24.

[0191] The encoding device 500 encodes the mesh in essentially the same way as the V-DMC described in the non-patent literature mentioned above. For example, the encoding device 500 obtains the original mesh to be encoded and an attribute map containing the texture corresponding to that original mesh. This original mesh includes not only information about the mesh geometry but also information indicating the correspondence with the attribute map (e.g., a UV list). The encoding device 500 converts the original mesh and attribute map into V-DMC data, encodes it, generates a V-DMC bitstream, and outputs it.

[0192] As shown in Figure 24, the encoding device 500 (second encoding device) has a pre-processing unit 511 and a V-DMC encoding unit 512. The pre-processing unit 511 is basically the same as the pre-processing unit 311, except that it does not have a constraint processing unit, and performs processing related to pre-processing before encoding. As shown in Figure 24, the pre-processing unit 511 has a base mesh generation unit 521, an atlas information generation unit 522, and a displacement vector generation unit 523. The base mesh generation unit 521, the atlas information generation unit 522, and the displacement vector generation unit 523 are the same processing units as the base mesh generation unit 321, the atlas information generation unit 322, and the displacement vector generation unit 324 of the encoding device 300, respectively, and perform the same processing. The V-DMC encoding unit 512 is the same processing unit as the V-DMC encoding unit 312 and performs the same processing.

[0193] <Encoding Process Flow> An example of the encoding process flow performed by this encoding device 500 will be explained with reference to the flowchart in Figure 25.

[0194] Once the encoding process begins, each of the processes from step S501 to step S504 is executed in the same manner as the processes in steps S301, S302, S304, and S305 in Figure 17.

[0195] The encoding process ends when the processing in step S504 is completed. The encoding device 500 performs this encoding process for each frame of the original mesh.

[0196] <Second Decoder> Furthermore, this technology can be applied to a decoding device that decodes mesh encoded data. Figure 26 is a block diagram showing an example of the configuration of a decoding device, which is one aspect of an information processing device (second decoding device) to which Method 2 of this technology is applied. The decoding device 600 shown in Figure 26 is a device that decodes mesh encoded data (V-DMC bitstream) generated in, for example, the encoding device 300 (Figure 15) or the encoding device 500 (Figure 24), and reconstructs the decoded mesh.

[0197] Figure 26 shows the main components such as the processing unit and data flow, but it does not necessarily represent everything. In other words, the decoding device 600 may have processing units that are not shown as blocks in Figure 26, or processes and data flows that are not shown as arrows or other symbols in Figure 26.

[0198] The decoding device 600 decodes the encoded mesh data, which has been encoded in essentially the same way as the V-DMC described in the above-mentioned non-patent literature, except for applying this technology, and reconstructs the decoded mesh. For example, the decoding device 600 acquires a V-DMC bitstream. This V-DMC bitstream may be generated, for example, by the encoding device 300 or encoding device 500. As a reconstruction process, the decoding device 600 decodes the V-DMC bitstream and reconstructs the mesh (also referred to as the decoded mesh). The decoding device 600 also applies a texture to the decoded mesh, generates a display image for displaying the decoded mesh, and outputs it to the outside of the decoding device 600. For example, the decoding device 600 supplies the display image to an external display device for display.

[0199] As shown in Figure 26, the decoding device 600 (second decoding device) includes a demultiplexing unit 611, an atlas information decoding unit 612, a base mesh decoding unit 613, a constraint processing unit 614, a subdivision unit 615, a displacement vector decoding unit 616, a displacement vector application unit 617, an attribute decoding unit 618, an attribute application unit 619, and a display processing unit 620.

[0200] The demultiplexing unit 611 is a processing unit similar to the demultiplexing unit 411 of the decoding device 400 and performs the same processing. The atlas information decoding unit 612 is a processing unit similar to the atlas information decoding unit 412 of the decoding device 400 and performs the same processing. The base mesh decoding unit 613 is a processing unit similar to the base mesh decoding unit 413 of the decoding device 400 and performs the same processing.

[0201] The constraint processing unit 614 is a processing unit similar to the constraint processing unit 323, and performs processing related to bitstream constraints. For example, the constraint processing unit 614 may acquire atlas information (UV unfolding data) supplied from the atlas information decoding unit 612, acquire the base mesh supplied from the base mesh decoding unit 613, and use them to check whether the V-DMC data conforms to the bitstream constraints. Alternatively, the constraint processing unit 614 may supply the base mesh and UV unfolding data that conform to the bitstream constraints to the subdivision unit 615.

[0202] The subdivision unit 615 is a processing unit similar to the subdivision unit 414 of the decoding device 400 and performs the same processing. The displacement vector decoding unit 616 is a processing unit similar to the displacement vector decoding unit 415 of the decoding device 400 and performs the same processing. The displacement vector application unit 617 is a processing unit similar to the displacement vector application unit 416 of the decoding device 400 and performs the same processing. The attribute decoding unit 618 is a processing unit similar to the attribute decoding unit 417 of the decoding device 400 and performs the same processing. The attribute application unit 619 is a processing unit similar to the attribute application unit 418 of the decoding device 400 and performs the same processing. The display processing unit 620 is a processing unit similar to the display processing unit 419 of the decoding device 400 and performs the same processing.

[0203] <Application of this technology> This technology (Method 2) may be applied to a decoding device 600 with such a configuration. For example, the atlas information decoding unit 612 and the base mesh decoding unit 613 may decode the bitstream and generate a base mesh from which vertices have been thinned out from the original mesh to be encoded, which is composed of vertices and connections representing the three-dimensional structure of an object, and UV unwrapping data representing the texture coordinates of the base mesh. The constraint processing unit 614 may also perform bitstream constraint processing, which includes obtaining the generated base mesh, obtaining the generated UV unwrapping data, and, if adaptive edge segmentation is applied to adaptively divide the edges included in the base mesh, checking whether the 3D data including the base mesh and UV unwrapping data conforms to the constraint that multiple vertices of the base mesh corresponding to a single texture position are at the same three-dimensional vertex position. The subdivision unit 615 may also subdivide the base mesh using the UV unwrapping data depending on the result of the check that the 3D data conforms to the constraint.

[0204] With this configuration, the decoding device 600 can subdivide the base mesh only if the decoded V-DMC data conforms to the bitstream constraints described above (Figure 10). Therefore, the decoding device 600 can suppress the occurrence of differences between the subdivision of triangles in the UV unwrapped data and the subdivision of triangles at 3D vertices in adaptive edge segmentation of the base mesh. Consequently, the decoding device 600 can suppress, for example, a reduction in the quality of the 3D data obtained by reconstruction.

[0205] Furthermore, if the edge-based segmentation enable flag is true, the constraint processing unit 614 may perform a check to determine whether the above constraints are met. By doing so, the constraint processing unit 614 can easily determine whether adaptive edge segmentation is applied or not.

[0206] Furthermore, the constraint processing unit 614 may perform the verification based on a flag extracted from the bitstream that indicates the result of the verification. In other words, if information (e.g., a flag) indicating the result of the bitstream constraint verification performed during the encoding of the V-DMC data is transmitted, the constraint processing unit 614 may determine whether or not the bitstream conforms to the constraints based on that information. In this way, the constraint processing unit 614 can easily determine whether or not the V-DMC bitstream conforms to the constraints in Figure 10.

[0207] Furthermore, Method 2-1 may be applied to the decoding device 600. For example, if the constraint processing unit 614 does not conform to the above-mentioned constraints, it may perform error processing on the base mesh subdivision. The content of this error processing can be anything as long as it does not subdivide the base mesh. For example, the base mesh subdivision process may be stopped (forcibly terminated), the base mesh subdivision process may be temporarily stopped and a warning may be displayed, or other processing may be performed. In this way, it is possible to control the process so that the base mesh of V-DMC data that does not conform to the above-mentioned bitstream constraints (Figure 10) is not subdivided.

[0208] Furthermore, Method 2-2 may be applied to the decoding device 600. For example, if the constraint processing unit 614 determines that the 3D data does not conform to the above constraints, it may update the vertex indices in the UV unwrapped data so that multiple vertices of the base mesh corresponding to a single texture position correspond to different vertices in the UV unwrapped data. In other words, a UV vertex is identified for each corresponding 3D vertex, and a different index is assigned. That is, multiple UV vertices are assumed to be located at a single position. By doing so, UV vertices corresponding to different 3D vertices located at a single position on the UV plane are managed as multiple UV vertices and conform to the above bitstream constraints. In other words, the base mesh of the V-DMC data conforming to the above bitstream constraints (Figure 10) can be subdivided.

[0209] Furthermore, method 2-3 may be applied to this decoding device 600. For example, if the constraint processing unit 614 determines that the 3D data does not conform to the above constraints, it may update the UV unwrapped data so that multiple vertices of the base mesh corresponding to a single texture position correspond to different positions in the UV unwrapped data. In other words, the UV unwrap is redone so that 3D vertices at different positions in 3D space (3D positions) are placed at different positions on the UV plane. By doing so, the data conforms to the above-mentioned bitstream constraints. In other words, the base mesh of the V-DMC data conforming to the above-mentioned bitstream constraints (Figure 10) can be subdivided.

[0210] As described above, the displacement vector decoding unit 616 of the decoding device 600 may decode the bitstream and generate displacement vectors, which are vector information indicating the displacement of the vertices of the subdivided base mesh. Alternatively, the displacement vector application unit 617 of the decoding device 600 may apply the generated displacement vectors to the subdivided base mesh.

[0211] Alternatively, the attribute decoding unit 618 of the decoding device 600 may decode the bitstream and generate an attribute map, which is a two-dimensional region in which the texture of the original mesh is packed. Then, the attribute application unit 619 may further apply the attributes contained in the generated attribute map to the subdivided base mesh to which the displacement vector has been applied.

[0212] <Decryption Process Flow> An example of the decoding process flow performed by this decoding device 600 will be explained with reference to the flowchart in Figure 27.

[0213] Once the decryption process begins, each of the processes from step S601 to step S603 is executed in the same manner as the processes from step S401 to step S403 in Figure 23.

[0214] In step S604, the constraint processing unit 614 applies the method 2 described above to perform constraint processing and makes the V-DMC data to be encoded conform to the above constraints.

[0215] Once the processing in step S604 is completed, the processes from step S605 to step S610 are executed in the same manner as the processes from step S404 to step S409 in Figure 23. Once the processing in step S610 is completed, the decoding process is finished. The decoding device 600 performs this decoding process for each frame of the original mesh.

[0216] <Constraint Processing Flow 4> Next, an example of the constraint processing flow performed in step S604 of Figure 27 when method 2-1 is applied will be explained with reference to the flowchart in Figure 28.

[0217] When constraint processing is initiated, in step S631, the constraint processing unit 614 acquires atlas information (including UV unwrapping data) supplied from the atlas information decoding unit 612 and the base mesh from the base mesh decoding unit 613.

[0218] In step S632, the constraint processing unit 614 checks whether the V-DMC data, including the base mesh and UV unwrapping data, conforms to the constraint that "when adaptive edge segmentation is applied, multiple vertices of the base mesh corresponding to each single texture location are at the same 3D vertex location." In other words, the constraint processing unit 614 checks whether, when adaptive edge segmentation, which adaptively divides the edges included in the base mesh, is applied, multiple vertices of the base mesh corresponding to each single texture location are at the same 3D vertex location.

[0219] In step S633, the constraint processing unit 614 determines whether the V-DMC data conforms to the constraint based on the result of the verification. If it is determined that the V-DMC data does not conform to the constraint, the process proceeds to step S634.

[0220] In step S634, the constraint processing unit 614 handles the base mesh subdivision with error processing. For example, the constraint processing unit 614 may cancel (terminate) the base mesh subdivision. Alternatively, the constraint processing unit 614 may temporarily stop the base mesh subdivision and display a warning or the like. Once the processing in step S634 is completed, the constraint processing is finished, and the process returns to Figure 27.

[0221] Furthermore, if it is determined in step S633 that the V-DMC data conforms to the constraint, the constraint processing ends and the process returns to Figure 27.

[0222] <Constraint Processing Flow 5> Next, an example of the constraint processing flow performed in step S604 of Figure 27 when method 2-2 is applied will be explained with reference to the flowchart in Figure 29.

[0223] When constraint processing is initiated, in step S651, the constraint processing unit 614 acquires atlas information (including UV unwrapping data) supplied from the atlas information decoding unit 612 and the base mesh from the base mesh decoding unit 613.

[0224] In step S652, the constraint processing unit 614 checks whether the V-DMC data, including the base mesh and UV unwrapping data, conforms to the constraint that "when adaptive edge segmentation is applied, multiple vertices of the base mesh corresponding to each single texture location are at the same three-dimensional vertex location." In other words, the constraint processing unit 614 checks whether, when adaptive edge segmentation, which adaptively divides the edges included in the base mesh, is applied, multiple vertices of the base mesh corresponding to each single texture location are at the same three-dimensional vertex location.

[0225] In step S653, the constraint processing unit 614 determines whether the V-DMC data conforms to the constraint based on the result of the verification. If it is determined that the V-DMC data conforms to the constraint, the process proceeds to step S654.

[0226] In step S654, the constraint processing unit 614 updates the UV vertex index so that the V-DMC data conforms to the constraint. That is, even if they are placed at the same UV position, different indices are assigned to each of the multiple UV vertices whose corresponding 3D vertices are different from each other. In this way, the V-DMC data conforms to the above constraint. When the processing in step S654 is completed, the constraint processing is finished, and the process returns to Figure 27.

[0227] Furthermore, if it is determined in step S653 that the V-DMC data conforms to the constraint, the constraint processing ends and the process returns to Figure 27.

[0228] <Constraint Processing Flow 6> Next, an example of the constraint processing flow performed in step S604 of Figure 27 when method 2-3 is applied will be explained with reference to the flowchart in Figure 30.

[0229] When constraint processing is initiated, in step S671, the constraint processing unit 614 acquires atlas information (including UV unfolding data) supplied from the atlas information decoding unit 612 and the base mesh from the base mesh decoding unit 613.

[0230] In step S672, the constraint processing unit 614 checks whether the V-DMC data, including the base mesh and UV unwrapping data, conforms to the constraint that "when adaptive edge segmentation is applied, multiple vertices of the base mesh corresponding to each single texture location are at the same 3D vertex location." In other words, the constraint processing unit 614 checks whether, when adaptive edge segmentation, which adaptively divides the edges included in the base mesh, is applied, multiple vertices of the base mesh corresponding to each single texture location are at the same 3D vertex location.

[0231] In step S673, the constraint processing unit 614 determines whether the V-DMC data conforms to the constraint based on the result of the verification. If it is determined that the V-DMC data conforms to the constraint, the process proceeds to step S674.

[0232] In step S674, the constraint processing unit 614 updates the UV unwrapping so that the V-DMC data conforms to the constraints. In other words, it redoes the UV unwrapping so that multiple UV vertices are not placed at a single UV position. By doing so, the V-DMC data conforms to the above constraints. When the processing in step S674 is completed, the constraint processing is finished, and the process returns to Figure 27.

[0233] Furthermore, if it is determined in step S673 that the V-DMC data conforms to the constraint, the constraint processing ends and the process returns to Figure 27.

[0234] By performing each process as described above, the decoding device 600 can subdivide the base mesh of the V-DMC data in accordance with the bitstream constraints (Figure 10) described above. Therefore, the decoding device 600 can suppress the occurrence of differences between the subdivision of triangles in the UV unwrapped data and the subdivision of triangles at 3D vertices in adaptive edge segmentation of the base mesh. Consequently, the decoding device 600 can suppress a reduction in the quality of the 3D data obtained by reconstruction, for example.

[0235] <6. Third Embodiment (Method 3)> <Third Encoding Device> This technology (Method 3) can also be applied to an encoding device that encodes a mesh and generates a bitstream. Figure 31 is a block diagram showing an example of the configuration of an encoding device, which is one aspect of an information processing device (third encoding device) to which Method 3 of this technology is applied. The encoding device 700 shown in Figure 31 is basically the same as the encoding device 300 and the encoding device 500, and encodes a mesh and generates its bitstream. Therefore, the encoding device 700 can also be called a bitstream generation device that generates a bitstream.

[0236] Figure 31 shows the main components such as the processing unit and data flow, but it does not necessarily represent everything. In other words, the encoding device 700 may have processing units that are not shown as blocks in Figure 31, or processes and data flows that are not shown as arrows or other symbols in Figure 31.

[0237] The encoding device 700, like the encoding devices 300 and 500, encodes the mesh in essentially the same way as the V-DMC described in the above-mentioned non-patent literature, except that it applies this technology.

[0238] As shown in Figure 31, the encoding device 700 (third encoding device) has a pre-processing unit 711 and a V-DMC encoding unit 712. The pre-processing unit 711 is a processing unit similar to the pre-processing unit 311 of the encoding device 300 and performs processing related to pre-processing before encoding. The V-DMC encoding unit 712 is a processing unit similar to the V-DMC encoding unit 312 of the encoding device 300 and performs processing related to encoding V-DMC data.

[0239] As shown in Figure 31, the preprocessing unit 711 includes a base mesh generation unit 721, an atlas information generation unit 722, and a displacement vector generation unit 723. The base mesh generation unit 721 is a processing unit similar to the base mesh generation unit 321 of the encoding device 300 and performs the same processing. The atlas information generation unit 722 is a processing unit similar to the atlas information generation unit 322 of the encoding device 300 and performs the same processing. The displacement vector generation unit 723 is a processing unit similar to the displacement vector generation unit 324 of the encoding device 300 and performs the same processing.

[0240] <V-DMC Encoding Unit> Figure 32 is a block diagram showing an example of the main configuration of the V-DMC encoding unit 712. Figure 32 shows the main components such as the processing unit and data flow, but it does not necessarily show everything. In other words, the V-DMC encoding unit 712 may have processing units that are not shown as blocks in Figure 32, or processes and data flows that are not shown as arrows, etc., in Figure 32.

[0241] As shown in Figure 32, the V-DMC coding unit 712 includes an atlas information coding unit 751, a base mesh coding unit 752, a displacement vector correction unit 753, a displacement vector coding unit 754, a mesh reconstruction unit 755, an attribute map conversion unit 756, an attribute coding unit 757, and a multiplexing unit 758.

[0242] The Atlas Information Encoding Unit 751 is a processing unit similar to the Atlas Information Encoding Unit 351 of the Encoding Device 300 and performs the same processing. The Base Mesh Encoding Unit 752 is a processing unit similar to the Base Mesh Encoding Unit 352 of the Encoding Device 300 and performs the same processing. The Displacement Vector Correction Unit 753 is a processing unit similar to the Displacement Vector Correction Unit 353 of the Encoding Device 300 and performs the same processing. The Displacement Vector Encoding Unit 754 is a processing unit similar to the Displacement Vector Encoding Unit 354 of the Encoding Device 300 and performs the same processing. The Mesh Reconstruction Unit 755 is a processing unit similar to the Mesh Reconstruction Unit 355 of the Encoding Device 300 and performs the same processing. The Attribute Map Conversion Unit 756 is a processing unit similar to the Attribute Map Conversion Unit 356 of the Encoding Device 300 and performs the same processing. The Attribute Encoding Unit 757 is a processing unit similar to the Attribute Encoding Unit 357 of the Encoding Device 300 and performs the same processing. The multiplexing unit 758 is a processing unit similar to the multiplexing unit 358 of the encoding device 300, and performs the same processing.

[0243] <Application of this technology> This technology (Method 3) may be applied to an encoding device 700 having such a configuration. For example, the mesh reconstruction unit 755 subdivides the base mesh by referring to 3D triangle information and reconstructs the mesh.

[0244] With this configuration, the encoding device 700 can refer to 3D triangle information instead of UV triangle information and directly obtain the index of a 3D vertex. Therefore, the encoding device 700 can substantially associate multiple 3D vertices with one UV vertex. Thus, the encoding device 700 can suppress the occurrence of discrepancies between the subdivision of triangles in the UV unwrapped data and the subdivision of triangles in the 3D vertices during adaptive edge segmentation of the base mesh. Therefore, the encoding device 700 can suppress the reduction in the quality of 3D data obtained by reconstruction, for example. <Encoding process flow> An example of the encoding process flow performed by this encoding device 700 will be explained with reference to the flowchart in Figure 33.

[0245] Once the encoding process begins, steps S701 to S704 are executed in the same manner as steps S301, S302, S304, and S305 in Figure 17.

[0246] The encoding process ends when the processing in step S704 is completed. The encoding device 700 performs this encoding process for each frame of the original mesh.

[0247] <Flow of V-DMC encoding process> Next, with reference to the flowchart in Figure 34, an example of the flow of the V-DMC encoding process performed in step S704 in Figure 33 will be explained.

[0248] When the V-DMC encoding process is started, the atlas information encoding unit 751 encodes the atlas information in step S721.

[0249] In step S722, the base mesh coding unit 752 encodes the base mesh.

[0250] In step S723, the displacement vector correction unit 753 corrects the displacement vector.

[0251] In step S724, the displacement vector encoding unit 754 encodes the corrected displacement vector. For example, the displacement vector encoding unit 754 may pack the displacement vector into a displacement video and encode it using a 2D encoding method. Alternatively, the displacement vector encoding unit 754 may arithmetically encode the displacement vector.

[0252] In step S725, the mesh reconstruction unit 755 subdivides the base mesh by referring to the 3D triangle information and reconstructs the mesh.

[0253] In step S726, the attribute map conversion unit 756 converts the attribute map.

[0254] In step S727, the attribute encoding unit 757 encodes an attribute video in which the attribute map is used as a frame image.

[0255] In step S728, the multiplexing unit 758 multiplexes encoded data of atlas information (including subdivision setting information), encoded data of base mesh, encoded data of displacement vectors, and encoded data of attributes to generate a V-DMC bitstream.

[0256] Once the process in step S728 is completed, the V-DMC encoding process ends, and the process returns to Figure 33.

[0257] By performing each process as described above, the encoding device 700 can refer to 3D triangle information instead of UV triangle information and directly obtain the index of a 3D vertex. Therefore, the encoding device 700 can substantially associate multiple 3D vertices with a single UV vertex. Consequently, the encoding device 700 can suppress the occurrence of discrepancies between the triangle subdivision in the UV unwrapped data and the triangle subdivision of the 3D vertices in adaptive edge segmentation for the base mesh. Consequently, the encoding device 700 can suppress the reduction in the quality of 3D data obtained by reconstruction, for example.

[0258] <Third Decoder> This technology (Method 3) can be applied to a decoding device that decodes mesh encoded data. Figure 35 is a block diagram showing an example of the configuration of a decoding device, which is one aspect of an information processing device (third decoding device) to which Method 3 of this technology is applied. The decoding device 800 shown in Figure 35 is basically the same as the decoding device 400 and the decoding device 600, and performs the same processing. For example, the decoding device 800 is a device that decodes the V-DMC bitstream generated by the encoding device 700 (Figure 31) and reconstructs the decoded mesh.

[0259] Figure 35 shows the main components such as the processing unit and data flow, but it does not necessarily represent everything. In other words, the decoding device 800 may have processing units that are not shown as blocks in Figure 35, or processes and data flows that are not shown as arrows or other symbols in Figure 35.

[0260] The decoding device 800 decodes the encoded mesh data, which has been encoded in essentially the same way as the V-DMC described in the above-mentioned non-patent document, except for applying this technology, and reconstructs the decoded mesh.

[0261] As shown in Figure 35, the decoding device 800 (third decoding device) includes a demultiplexing unit 811, an atlas information decoding unit 812, a base mesh decoding unit 813, a subdivision unit 814, a displacement vector decoding unit 815, a displacement vector application unit 816, an attribute decoding unit 817, an attribute application unit 818, and a display processing unit 819.

[0262] The demultiplexing unit 811 is a processing unit similar to the demultiplexing unit 411 of the decoding device 400 and performs the same processing. The atlas information decoding unit 812 is a processing unit similar to the atlas information decoding unit 412 of the decoding device 400 and performs the same processing. The base mesh decoding unit 813 is a processing unit similar to the base mesh decoding unit 413 of the decoding device 400 and performs the same processing. The subdivision unit 814 is a processing unit similar to the subdivision unit 414 of the decoding device 400 and performs the same processing. The displacement vector decoding unit 815 is a processing unit similar to the displacement vector decoding unit 415 of the decoding device 400 and performs the same processing. The displacement vector application unit 816 is a processing unit similar to the displacement vector application unit 416 of the decoding device 400 and performs the same processing. The attribute decoding unit 817 is a processing unit similar to the attribute decoding unit 417 of the decoding device 400 and performs the same processing. The attribute application unit 818 is a processing unit similar to the attribute application unit 418 of the decoding device 400 and performs the same processing. The display processing unit 819 is a processing unit similar to the display processing unit 419 of the decoding device 400 and performs the same processing.

[0263] <Application of this technology> This technology (Method 3) may be applied to a decoding device 800 having such a configuration. For example, the atlas information decoding unit 812 and the base mesh decoding unit 813 may decode the bitstream and generate a base mesh obtained by thinning out vertices from the original mesh to be encoded, which is composed of vertices and connections representing the three-dimensional structure of an object, and 3D triangle information which is information about the triangles of the base mesh and includes the index of each vertex constituting the triangle. The subdivision unit 814 may then subdivide the base mesh by referring to the 3D triangle information.

[0264] With this configuration, the decoder 800 can directly determine the index of a 3D vertex by referring to 3D triangle information instead of UV triangle information. Therefore, the decoder 800 can substantially associate multiple 3D vertices with a single UV vertex. Consequently, the decoder 800 can suppress the occurrence of discrepancies between the triangle subdivision in the UV unwrapped data and the triangle subdivision of the 3D vertices in adaptive edge segmentation of the base mesh. Therefore, the decoder 800 can suppress the reduction in the quality of the 3D data obtained by reconstruction, for example.

[0265] The displacement vector decoding unit 815 of the decoding device 800 may decode the bitstream and generate displacement vectors, which are vector information indicating the displacement of the vertices of the subdivided base mesh. Alternatively, the displacement vector application unit 816 may apply the generated displacement vectors to the subdivided base mesh.

[0266] Alternatively, the attribute decoding unit 817 of the decoding device 800 may decode the bitstream and generate an attribute map, which is a two-dimensional region in which the texture of the original mesh is packed. Then, the attribute application unit 818 may further apply the attributes contained in the generated attribute map to the subdivided base mesh to which the displacement vector has been applied.

[0267] <Decryption Process Flow> An example of the decoding process flow performed by this decoding device 800 will be explained with reference to the flowchart in Figure 36.

[0268] Once the decryption process begins, each of the processes from step S801 to step S603 is executed in the same manner as the processes from step S401 to step S403 in Figure 23.

[0269] In step S804, the subdivision unit 814 subdivides the base mesh by referring to the 3D triangle information using the method 3 described above.

[0270] Once step S804 is completed, steps S805 to S809 are executed in the same manner as steps S405 to S409 in Figure 23. Once step S809 is completed, the decoding process is finished. The decoding device 800 performs this decoding process for each frame of the original mesh.

[0271] By performing each process as described above, the decoder 800 can directly determine the index of a 3D vertex by referring to 3D triangle information instead of UV triangle information. Therefore, the decoder 800 can substantially associate multiple 3D vertices with a single UV vertex. Consequently, the decoder 800 can suppress the occurrence of discrepancies between the triangle subdivision in the UV unwrapped data and the triangle subdivision of the 3D vertices in adaptive edge segmentation of the base mesh. Therefore, the decoder 800 can suppress the reduction in the quality of 3D data obtained by reconstruction, for example.

[0272] <7. Notes> <Polygon Shape> In the above explanation, the polygon shape was described as a triangle, but this shape is just one example. The polygon shape can be any polygon.

[0273] <Encoding Method> In the above explanation, V-DMC was used as an example of an encoding method to which this technology can be applied. However, this technology is not limited to this example and can be applied to any encoding method that encodes the base mesh, displacement vectors, attribute maps including textures, and atlas information, or information equivalent thereto.

[0274] <Computer> The series of processes described above can be executed by hardware or by software. When the series of processes are executed by software, the programs that make up that software are installed on the computer. Here, "computer" includes computers built into dedicated hardware, as well as general-purpose personal computers that can perform various functions by installing various programs.

[0275] Figure 37 is a block diagram showing an example of the hardware configuration of a computer that executes the series of processes described above using a program.

[0276] In the computer 900 shown in Figure 37, the CPU (Central Processing Unit) 901, ROM (Read Only Memory) 902, and RAM (Random Access Memory) 903 are interconnected via a bus 904.

[0277] An input / output interface 910 is also connected to the bus 904. An input / output interface 910 is connected to an input unit 911, an output unit 912, a storage unit 913, a communication unit 914, and a drive 915.

[0278] The input unit 911 consists of, for example, a keyboard, mouse, microphone, touch panel, and input terminals. The output unit 912 consists of, for example, a display, speaker, and output terminals. The storage unit 913 consists of, for example, a hard disk, RAM disk, and non-volatile memory. The communication unit 914 consists of, for example, a network interface. The drive 915 drives removable media 921 such as a magnetic disk, optical disk, magneto-optical disk, or semiconductor memory.

[0279] In a computer configured as described above, the CPU 901 loads, for example, a program stored in the memory unit 913 into the RAM 903 via the input / output interface 910 and the bus 904, and executes it, thereby performing the series of processes described above. The RAM 903 also appropriately stores data necessary for the CPU 901 to perform various processes.

[0280] The program executed by the computer can be recorded and applied, for example, on removable media 921 such as a package medium. In this case, the program can be installed in the storage unit 913 via the input / output interface 910 by inserting the removable media 921 into the drive 915.

[0281] Furthermore, this program can also be provided via wired or wireless transmission media such as a local area network, the internet, or digital satellite broadcasting. In that case, the program can be received by the communication unit 914 and installed in the storage unit 913.

[0282] In addition, this program can be pre-installed in ROM 902 or memory unit 913.

[0283] <Applications of this technology> This technology can be applied to any configuration. For example, this technology can be applied to various electronic devices.

[0284] Furthermore, this technology can also be implemented as part of a device, such as a processor as a system LSI (Large Scale Integration) (e.g., a video processor), a module using multiple processors (e.g., a video module), a unit using multiple modules (e.g., a video unit), or a set with additional functions added to a unit (e.g., a video set).

[0285] Furthermore, this technology can also be applied to network systems composed of multiple devices. For example, this technology may be implemented as cloud computing, where multiple devices share and collaborate on processing via a network. For example, this technology may be implemented in a cloud service that provides image (video) related services to any terminal such as computers, AV (Audio Visual) equipment, portable information processing terminals, and IoT (Internet of Things) devices.

[0286] In this specification, a system refers to a collection of multiple components (devices, modules (parts), etc.), regardless of whether all components are located in the same enclosure. Therefore, multiple devices housed in separate enclosures and connected via a network, and a single device containing multiple modules within a single enclosure, are both considered systems.

[0287] <Applicable Fields and Applications of This Technology> Systems, devices, and processing units incorporating this technology can be used in any field, such as transportation, medical care, security, agriculture, livestock farming, mining, beauty, factories, home appliances, weather, and nature monitoring. Furthermore, the applications are entirely arbitrary.

[0288] <Other> In this specification, "flag" refers to information used to identify multiple states, and includes not only information used to identify two states, true (1) or false (0), but also information capable of identifying three or more states. Therefore, the values ​​that this "flag" can take are, for example, two values, 1 / 0, or three or more values. In other words, the number of bits that constitute this "flag" is arbitrary, and can be 1 bit or multiple bits. Furthermore, identification information (including flags) is envisioned not only in the form of including the identification information itself in the bitstream, but also in the form of including difference information of the identification information relative to a certain reference information in the bitstream. Therefore, in this specification, "flag" and "identification information" include not only the information itself, but also difference information relative to the reference information.

[0289] Furthermore, various types of information (metadata, etc.) related to encoded data (bitstream) may be transmitted or recorded in any form as long as they are associated with the encoded data. Here, the term "associate" means, for example, making it possible to use (link) one data when processing the other. In other words, associated data may be combined into a single data, or they may be individual data. For example, information associated with encoded data (image) may be transmitted on a different transmission path than the encoded data (image). Also, for example, information associated with encoded data (image) may be recorded on a different recording medium (or a different recording area on the same recording medium) than the encoded data (image). Note that this "association" may not apply to the entire data, but only to a part of it. For example, an image and the information corresponding to that image may be associated with each other in any unit, such as multiple frames, one frame, or a part within a frame.

[0290] In this specification, terms such as "combine," "multiplex," "add," "integrate," "include," "store," "insert," "insert," and "place" mean combining multiple things into one, such as combining encoded data and metadata into a single data, and represent one method of "associating" as described above.

[0291] Furthermore, the embodiments of this technology are not limited to those described above, and various modifications are possible without departing from the gist of this technology.

[0292] For example, the configuration described as a single device (or processing unit) may be divided and configured as multiple devices (or processing units). Conversely, the configurations described above as multiple devices (or processing units) may be combined and configured as a single device (or processing unit). Furthermore, it is also possible to add configurations other than those described above to the configuration of each device (or each processing unit). In addition, if the overall system configuration and operation are substantially the same, a part of the configuration of one device (or processing unit) may be included in the configuration of another device (or other processing unit).

[0293] Furthermore, for example, the program described above may be executed on any device. In that case, the device should have the necessary functions (such as functional blocks) and be able to obtain the necessary information.

[0294] Furthermore, for example, each step of a flowchart may be executed by one device, or it may be divided among multiple devices. Additionally, if a single step includes multiple processes, these processes may be executed by one device, or they may be divided among multiple devices. In other words, multiple processes included in a single step can be executed as multiple steps. Conversely, processes described as multiple steps can be combined and executed as a single step.

[0295] Furthermore, for example, a program executed by a computer may be structured so that the steps of the program are executed chronologically in the order described herein, or they may be executed in parallel or individually at necessary times, such as when a call is made. In other words, the steps may be executed in an order different from the order described above, as long as no inconsistencies arise. Moreover, the steps of this program may be executed in parallel with the processing of other programs, or in combination with the processing of other programs.

[0296] Furthermore, for example, multiple technologies relating to this technology can be implemented independently, as long as they do not create a contradiction. Of course, any multiple technologies can also be implemented in combination. For example, some or all of the technologies described in one embodiment can be implemented in combination with some or all of the technologies described in another embodiment. Also, some or all of the above-mentioned technologies can be implemented in combination with other technologies not mentioned above.

[0297] Furthermore, this technology can also take the following configurations: (1) An encoding device comprising: a constraint processing unit that performs bitstream constraint processing, which includes obtaining a base mesh obtained by thinning out vertices from an original mesh to be encoded, which is composed of vertices and connections representing the three-dimensional structure of an object; obtaining UV unwrapping data representing the texture coordinates of the base mesh; and checking whether the 3D data, including the base mesh and the UV unwrapping data, conforms to the constraint that, when adaptive edge segmentation which adaptively divides the edges included in the base mesh is applied, multiple vertices of the base mesh corresponding to a single texture position are at the same three-dimensional vertex position; and an encoding unit that encodes the 3D data according to the result of the confirmation that the 3D data conforms to the constraint. (2) The encoding device according to (1), wherein the constraint processing unit performs the confirmation when the edge-based subdivision enable flag is true. (3) The encoding device according to (1) or (2), wherein the constraint processing unit performs error processing on the encoding of the 3D data if the 3D data does not conform to the constraint. (4) The encoding device according to (1) or (2), wherein the constraint processing unit updates the vertex indices in the UV unwrapped data so that the plurality of vertices of the base mesh corresponding to each of the single texture positions correspond to different vertices in the UV unwrapped data if the 3D data does not conform to the constraints. (5) The encoding device according to (1) or (2), wherein the constraint processing unit updates the UV unwrapped data so that the plurality of vertices of the base mesh corresponding to each of the single texture positions correspond to different positions in the UV unwrapped data if the 3D data does not conform to the constraints. (6) The encoding device according to any one of (1) to (5), wherein the constraint processing unit generates a flag indicating the result of the verification and includes it in the 3D data.(7) The encoding device according to any one of (1) to (6), further comprising a base mesh generation unit that generates the base mesh by thinning out vertices from the original mesh, wherein the constraint processing unit performs the verification on the 3D data including the generated base mesh. (8) The encoding device according to any one of (1) to (7), further comprising a decoding unit that decodes a bitstream and generates the 3D data, wherein the constraint processing unit performs the verification on the generated 3D data. (9) The encoding method, which includes: obtaining a base mesh obtained by thinning out vertices from an original mesh to be encoded, which is composed of vertices and connections representing the three-dimensional structure of an object; obtaining UV unwrapping data representing the texture coordinates of the base mesh; and performing processing on a bitstream constraint, which includes checking whether the 3D data including the base mesh and the UV unwrapping data conforms to the constraint that, when adaptive edge segmentation which adaptively divides the edges included in the base mesh is applied, multiple vertices of the base mesh corresponding to a single texture position are at the same three-dimensional vertex position. (10) A program for causing a computer to perform a bitstream constraint process, which includes: obtaining a base mesh obtained by thinning out vertices from an original mesh to be encoded, which is composed of vertices and connections representing the three-dimensional structure of an object; obtaining UV unwrapping data representing the texture coordinates of the base mesh; checking whether the 3D data, including the base mesh and the UV unwrapping data, conforms to the constraint that, when adaptive edge segmentation which adaptively divides the edges included in the base mesh is applied, multiple vertices of the base mesh corresponding to a single texture position are at the same three-dimensional vertex position; and encoding the 3D data according to the result of the check that the 3D data conforms to the constraint.

[0298] (21) A decoding device comprising: a decoding unit that decodes a bitstream that conforms to the constraint that, when adaptive edge segmentation which adaptively divides the edges included in the base mesh is applied, multiple vertices of the base mesh corresponding to each single texture position are at the same three-dimensional vertex position, and generates the base mesh and UV unwrapped data representing the texture coordinates of the base mesh; and a subdivision unit that subdivides the base mesh using the UV unwrapped data. (22) A decoding method comprising: decodes a bitstream that conforms to the constraint that, when adaptive edge segmentation which adaptively divides the edges included in the base mesh is applied, multiple vertices of the base mesh corresponding to each single texture position are at the same three-dimensional vertex position, and generates the base mesh and UV unwrapped data representing the texture coordinates of the base mesh; and subdivisions the base mesh using the UV unwrapped data. (23) A program for causing a computer to perform a process that includes decoding a bitstream that conforms to the constraint that, when adaptive edge segmentation is applied which adaptively divides the edges contained in the base mesh, multiple vertices of the base mesh corresponding to a single texture position are at the same three-dimensional vertex position, generating the base mesh and UV unwrapping data representing the texture coordinates of the base mesh, and subdividing the base mesh using the UV unwrapping data.

[0299] (41) A decoding device comprising: a decoding unit that decodes a bitstream and generates a base mesh obtained by thinning out vertices from an original mesh to be encoded, which is composed of vertices and connections representing the three-dimensional structure of an object, and UV unwrapping data representing the texture coordinates of the base mesh; a constraint processing unit that performs bitstream constraint processing, which includes obtaining the generated base mesh, obtaining the generated UV unwrapping data, and checking whether the 3D data including the base mesh and the UV unwrapping data conforms to the constraint that, when adaptive edge segmentation which adaptively divides the edges included in the base mesh is applied, multiple vertices of the base mesh corresponding to a single texture position are at the same three-dimensional vertex position; and a subdivision unit that subdivides the base mesh using the UV unwrapping data according to the result of the confirmation that the 3D data conforms to the constraint. (42) The decoding device according to (41), wherein the constraint processing unit performs the confirmation when the enable flag for edge-based subdivision is true. (43) The decoding apparatus according to (41) or (42), wherein the constraint processing unit performs error processing on the subdivision of the base mesh if the 3D data does not conform to the constraints. (44) The decoding apparatus according to (41) or (42), wherein the constraint processing unit updates the vertex indices in the UV unwrapped data so that the plurality of vertices of the base mesh corresponding to each of the single texture positions correspond to different vertices in the UV unwrapped data if the 3D data does not conform to the constraints. (45) The decoding apparatus according to (41) or (42), wherein the constraint processing unit updates the UV unwrapped data so that the plurality of vertices of the base mesh corresponding to each of the single texture positions correspond to different positions in the UV unwrapped data if the 3D data does not conform to the constraints. (46) The decoding apparatus according to any one of (41) to (45), wherein the constraint processing unit performs the verification based on a flag indicating the result of the verification extracted from the bitstream.(47) A decoding device according to any one of (41) to (46), further comprising: a displacement vector decoding unit that decodes the bitstream and generates displacement vectors which are vector information indicating the displacement of the vertices of the subdivided base mesh; and a displacement vector application unit that applies the generated displacement vectors to the subdivided base mesh. (48) A decoding device according to (47), further comprising: an attribute decoding unit that decodes the bitstream and generates an attribute map which is a two-dimensional region on which the texture of the original mesh is packed; and an attribute application unit that further applies the attributes included in the generated attribute map to the subdivided base mesh to which the displacement vectors have been applied. (49) A decoding method comprising: decoding a bitstream to generate a base mesh obtained by thinning out vertices from an original mesh to be encoded, which is composed of vertices and connections representing the three-dimensional structure of an object, and UV unwrapping data representing the texture coordinates of the base mesh; obtaining the generated base mesh; obtaining the generated UV unwrapping data; and performing processing on a bitstream constraint, which includes checking whether the 3D data, including the base mesh and the UV unwrapping data, conforms to the constraint that, when adaptive edge segmentation is applied to adaptively divide the edges included in the base mesh, multiple vertices of the base mesh corresponding to a single texture position are at the same three-dimensional vertex position; and, depending on the result of the check that the 3D data conforms to the constraint, subdividing the base mesh using the UV unwrapping data.(50) A program for causing a computer to perform a bitstream constraint process, which includes: decoding a bitstream to generate a base mesh obtained by thinning out vertices from an original mesh to be encoded, which is composed of vertices and connections representing the three-dimensional structure of an object, and UV unwrapping data representing the texture coordinates of the base mesh; obtaining the generated base mesh; obtaining the generated UV unwrapping data; checking whether the 3D data, including the base mesh and the UV unwrapping data, conforms to the constraint that, when adaptive edge segmentation is applied to adaptively divide the edges included in the base mesh, multiple vertices of the base mesh corresponding to a single texture position are at the same three-dimensional vertex position; and, depending on the result of the check that the 3D data conforms to the constraint, subdividing the base mesh using the UV unwrapping data.

[0300] (61) A decoding device comprising: a decoding unit that decodes a bitstream and generates a base mesh obtained by thinning out vertices from an original mesh to be encoded, which is composed of vertices and connections representing the three-dimensional structure of an object, and 3D triangle information which is information relating to the triangles of the base mesh and includes indices of each vertex constituting the triangle; and a subdivision unit that subdivides the base mesh by referring to the 3D triangle information. (62) The decoding device according to (61), further comprising: a displacement vector decoding unit that decodes the bitstream and generates displacement vectors which are vector information indicating the displacement of the vertices of the subdivided base mesh; and a displacement vector application unit that applies the generated displacement vectors to the subdivided base mesh. (63) The decoding device according to (62), further comprising: an attribute decoding unit that decodes the bitstream and generates an attribute map which is a two-dimensional region on which the texture of the original mesh is packed; and an attribute application unit that further applies the attributes included in the generated attribute map to the base mesh which has been subdivided and to which the displacement vectors have been applied. (64) A decoding method comprising: decoding a bitstream to generate a base mesh obtained by thinning out vertices from an original mesh to be encoded, which is composed of vertices and connections representing the three-dimensional structure of an object; and 3D triangle information relating to the triangles of the base mesh, which includes the index of each vertex constituting the triangle; and subdividing the base mesh by referring to the 3D triangle information. (65) A program for causing a computer to perform a process comprising: decoding a bitstream to generate a base mesh obtained by thinning out vertices from an original mesh to be encoded, which is composed of vertices and connections representing the three-dimensional structure of an object; and 3D triangle information relating to the triangles of the base mesh, which includes the index of each vertex constituting the triangle; and subdividing the base mesh by referring to the 3D triangle information.

[0301] 300 Encoding device, 311 Preprocessing unit, 312 V-DMC encoding unit, 321 Base mesh generation unit, 322 Atlas information generation unit, 323 Constraint processing unit, 324 Displacement vector generation unit, 351 Atlas information encoding unit, 352 Base mesh encoding unit, 353 Displacement vector correction unit, 354 Displacement vector encoding unit, 355 Mesh reconstruction unit, 356 Attribute map conversion unit, 357 Attribute encoding unit, 358 Multiplexing unit, 370 Encoding device, 380 Constraint processing unit, 381 V-DMC decoding unit, 400 Decoding device, 411 Demultiplexing unit, 412 Atlas information decoding unit, 413 Base mesh decoding unit, 414 Subdivision unit, 415 Displacement vector decoding unit, 416 Displacement vector application unit, 417 Attribute decoding unit, 418 Attribute application unit, 419 Display processing unit, 500 Encoding device, 511 Preprocessing unit, 512 V-DMC encoding unit, 521 Base mesh generation unit, 522 Atlas information generation unit, 523 Displacement vector generation unit, 600 Decoding device, 611 Demultiplexing unit, 612 Atlas information decoding unit, 613 Base mesh decoding unit, 614 Constraint processing unit, 615 Subdivision unit, 616 Displacement vector decoding unit, 617 Displacement vector application unit, 618 Attribute decoding unit, 619 Attribute application unit, 620 Display processing unit, 700 Encoding device, 711 Preprocessing unit, 712 V-DMC encoding unit, 721 Base mesh generation unit, 722 Atlas information generation unit, 723 Displacement vector generation unit, 751 Atlas information coding unit, 752 Base mesh coding unit, 753 Displacement vector correction unit, 754 Displacement vector coding unit, 755 Mesh reconstruction unit, 756 Attribute map conversion unit, 757 Attribute coding unit, 758 Multiplexing unit, 800 Decoding device, 811 Demultiplexing unit, 812 Atlas information decoding unit, 813 Base mesh decoding unit, 814 Subdivision unit, 815 Displacement vector decoding unit, 816 Displacement vector application unit, 817 Attribute decoding unit, 818 Attribute application unit, 819 Display processing unit, 900 Computer

Claims

1. An encoding device comprising: a constraint processing unit that performs bitstream constraint processing, which includes obtaining a base mesh obtained by thinning out vertices from an original mesh to be encoded, which is composed of vertices and connections representing the three-dimensional structure of an object; obtaining UV unwrapping data representing the texture coordinates of the base mesh; and checking whether the 3D data, including the base mesh and the UV unwrapping data, conforms to the constraint that, when adaptive edge segmentation which adaptively divides the edges included in the base mesh is applied, multiple vertices of the base mesh corresponding to a single texture position are at the same three-dimensional vertex position; and an encoding unit that encodes the 3D data according to the result of the check that the 3D data conforms to the constraint.

2. The encoding apparatus according to claim 1, wherein the constraint processing unit performs the check if the edge-based subdivision enable flag is true.

3. The encoding device according to claim 1, wherein the constraint processing unit processes the encoding of the 3D data with an error if the 3D data does not conform to the constraint.

4. The encoding device according to claim 1, wherein the constraint processing unit updates the vertex index in the UV unwrapping data such that, if the 3D data does not conform to the constraint, the plurality of vertices of the base mesh corresponding to each of the single texture positions correspond to different vertices in the UV unwrapping data.

5. The encoding device according to claim 1, wherein the constraint processing unit updates the UV unwrapped data if the 3D data does not conform to the constraints, such that the plurality of vertices of the base mesh corresponding to each of the single texture positions correspond to different positions in the UV unwrapped data.

6. The encoding device according to claim 1, further comprising a base mesh generation unit that generates the base mesh by thinning out vertices from the original mesh, wherein the constraint processing unit performs the verification on the 3D data including the generated base mesh.

7. The encoding device according to claim 1, further comprising a decoding unit that decodes a bitstream and generates the 3D data, wherein the constraint processing unit performs the verification on the generated 3D data.

8. An encoding method comprising: obtaining a base mesh obtained by thinning out vertices from an original mesh to be encoded, which is composed of vertices and connections representing the three-dimensional structure of an object; obtaining UV unwrapping data representing the texture coordinates of the base mesh; and performing processing related to bitstream constraints, including checking whether the 3D data, including the base mesh and the UV unwrapping data, conforms to the constraint that, when adaptive edge segmentation is applied to adaptively divide the edges included in the base mesh, multiple vertices of the base mesh corresponding to a single texture position are at the same three-dimensional vertex position; and encoding the 3D data according to the result of the check that the 3D data conforms to the constraint.

9. A decoding device comprising: a decoding unit that decodes a bitstream that conforms to the constraint that when adaptive edge segmentation is applied, which adaptively divides the edges contained in the base mesh, multiple vertices of the base mesh corresponding to a single texture position are at the same three-dimensional vertex position, and generates the base mesh and UV unwrapping data representing the texture coordinates of the base mesh; and a subdivision unit that subdivides the base mesh using the UV unwrapping data.

10. A decoding method comprising: decoding a bitstream that conforms to the constraint that, when adaptive edge segmentation is applied to adaptively divide the edges contained in the base mesh, multiple vertices of the base mesh corresponding to a single texture position are at the same three-dimensional vertex position; generating the base mesh and UV unwrapping data representing the texture coordinates of the base mesh; and subdividing the base mesh using the UV unwrapping data.

11. A decoding device comprising: a decoding unit that decodes a bitstream and generates a base mesh obtained by thinning out vertices from an original mesh to be encoded, which is composed of vertices and connections representing the three-dimensional structure of an object, and UV unwrapping data representing the texture coordinates of the base mesh; a constraint processing unit that performs bitstream constraint processing, which includes obtaining the generated base mesh, obtaining the generated UV unwrapping data, and checking whether the 3D data including the base mesh and the UV unwrapping data conforms to the constraint that, when adaptive edge segmentation which adaptively divides the edges included in the base mesh is applied, multiple vertices of the base mesh corresponding to a single texture position are at the same three-dimensional vertex position; and a subdivision unit that subdivides the base mesh using the UV unwrapping data according to the result of the check that the 3D data conforms to the constraint.

12. The decoding device according to claim 11, wherein the constraint processing unit performs the check if the edge-based subdivision enable flag is true.

13. The decoding device according to claim 11, wherein the constraint processing unit processes the base mesh subdivision as an error if the 3D data does not conform to the constraints.

14. The decoding device according to claim 11, wherein the constraint processing unit updates the vertex indices in the UV unwrapping data such that, if the 3D data does not conform to the constraints, the plurality of vertices of the base mesh corresponding to each of the single texture positions correspond to different vertices in the UV unwrapping data.

15. The decoding device according to claim 11, wherein the constraint processing unit updates the UV unwrapped data if the 3D data does not conform to the constraints, such that the plurality of vertices of the base mesh corresponding to each of the single texture positions correspond to different positions in the UV unwrapped data.

16. The decoding device according to claim 11, further comprising: a displacement vector decoding unit that decodes the bitstream and generates displacement vectors which are vector information indicating the displacement of the vertices of the subdivided base mesh; and a displacement vector application unit that applies the generated displacement vectors to the subdivided base mesh.

17. The decoding device according to claim 16, further comprising: an attribute decoding unit that decodes the bitstream and generates an attribute map which is a two-dimensional region on which the texture of the original mesh is packed; and an attribute application unit that further applies the attributes included in the generated attribute map to the base mesh which has been subdivided and to which the displacement vector has been applied.

18. A decoding method comprising: decoding a bitstream to generate a base mesh obtained by thinning out vertices from an original mesh to be encoded, which is composed of vertices and connections representing the three-dimensional structure of an object, and UV unwrapping data representing the texture coordinates of the base mesh; obtaining the generated base mesh; obtaining the generated UV unwrapping data; and performing processing on bitstream constraints, including checking whether the 3D data, including the base mesh and the UV unwrapping data, conforms to the constraint that, when adaptive edge segmentation is applied to adaptively divide the edges included in the base mesh, multiple vertices of the base mesh corresponding to a single texture position are at the same three-dimensional vertex position; and, depending on the result of the check that the 3D data conforms to the constraint, subdividing the base mesh using the UV unwrapping data.

Images