Information processing device and method

Abstract

The present disclosure relates to an information processing device and method that make it possible to suppress a reduction in coding efficiency while suppressing a reduction in the shape quality of a decoded mesh. Encoded data of a base mesh and encoded data of a displacement vector are decoded, the base mesh is subdivided by recursively dividing a face on the basis of a gradient score that represents the complexity of the three-dimensional structure of an object, and the displacement vector is applied. Alternatively, the base mesh is subdivided by recursively dividing the face on the basis of the gradient score, a displacement vector is generated on the basis of an original mesh and the subdivided base mesh, and the base mesh and the displacement vector are encoded. The present disclosure can be applied to, e.g., an information processing device, an encoding device, a decoding device, an electronic apparatus, an information processing method, an encoding method, a decoding method, a program, or the like.

Description

Information Processing Apparatus and Method

[0001] The present disclosure relates to an information processing apparatus and method, and more particularly to an information processing apparatus and method capable of suppressing a reduction in coding efficiency while suppressing a reduction in the shape quality of a decoded mesh.

[0002] Conventionally, as a method for coding a mesh, which is 3D data representing the three-dimensional structure of an object by connection of vertices, there has been V-DMC (Video-based Dynamic Mesh Coding) (see, for example, Non-Patent Document 1). In V-DMC, a mesh to be coded (original mesh) is represented by a coarse (i.e., low in fineness) base mesh and displacement vectors of division points obtained by subdividing the base mesh, and the base mesh and the displacement vectors are coded. The displacement vectors are stored (packed) in a two-dimensional image and coded as an image. Since the object can change in the time direction (is dynamic), the mesh (i.e., the base mesh and the displacement vectors) is also dynamic. Therefore, the displacement vectors are coded as a moving image (displacement video) having the two-dimensional image as a frame.

[0003] At the time of decoding, the bitstream is decoded by a decoding method corresponding to the coding method, and the base mesh and the displacement vectors are restored (generated). Then, the base mesh is subdivided, and by applying the displacement vectors to each division point, a mesh equivalent to the original mesh is restored (generated). Midpoint subdivision was applied to the subdivision of the base mesh.

[0004] Khaled Mammou, Jungsun Kim, Alexis Tourapis, Dimitri Podborski, Krasimir Kolarov, "[V-CG] Apple’s Dynamic Mesh Coding CfP Response", ISO / IEC JTC 1 / SC 29 / WG 7 m59281, April 2022

[0005] However, in the case of midpoint subdivision, each face of the base mesh was uniformly subdivided regardless of the complexity of the shape (3D structure) in the object (original mesh). Therefore, it was difficult to suppress the reduction in encoding efficiency while suppressing the reduction in the shape quality of the decoded mesh.

[0006] This disclosure is made in view of these circumstances and aims to suppress a reduction in coding efficiency while suppressing a reduction in the shape quality of the decoded mesh.

[0007] An information processing device, representing one aspect of this technology, comprises a decoding unit that decodes encoded data of a base mesh to generate the base mesh and decodes encoded data of a displacement vector to generate the displacement vector, and a subdivided displacement vector application unit that subdivides the generated base mesh by recursively dividing faces based on a gradient score representing the complexity of the three-dimensional structure of an object, and applies the generated displacement vector, wherein the base mesh is a mesh obtained by thinning out vertices from an original mesh to be encoded, which is composed of vertices and connections representing the three-dimensional structure of the object, and the displacement vector is information indicating the difference in the positions of vertices between the subdivided base mesh and the original mesh.

[0008] One aspect of this technology is an information processing method which includes decoding encoded data of a base mesh to generate the base mesh, decoding encoded data of a displacement vector to generate the displacement vector, recursively dividing faces based on a gradient score representing the complexity of the three-dimensional structure of an object to subdivide the generated base mesh and applying the generated displacement vector, wherein the base mesh is a mesh obtained by thinning out vertices from the original mesh to be encoded, which is composed of vertices and connections representing the three-dimensional structure of the object, and the displacement vector is information indicating the difference in vertex positions between the subdivided base mesh and the original mesh.

[0009] The information processing device for another aspect of this technology comprises: a subdivision unit that subdivides a base mesh by recursively dividing faces based on a gradient score representing the complexity of the three-dimensional structure of an object; a displacement vector generation unit that generates a displacement vector based on an original mesh to be encoded, which is composed of vertices and connections representing the three-dimensional structure of the object, and the subdivided base mesh; and an encoding unit that encodes the base mesh and the displacement vector, wherein the base mesh is a mesh from which vertices have been thinned out from the original mesh, and the displacement vector is information indicating the difference in the positions of vertices between the subdivided base mesh and the original mesh.

[0010] Another aspect of this technology is an information processing method which includes: subdividing a base mesh by recursively dividing faces based on a gradient score representing the complexity of the three-dimensional structure of an object; generating a displacement vector based on an original mesh to be encoded, which is composed of vertices and connections representing the three-dimensional structure of the object, and the subdivided base mesh; and encoding the base mesh and the displacement vector, wherein the base mesh is a mesh from which vertices have been thinned out from the original mesh, and the displacement vector is information indicating the difference in vertex positions between the subdivided base mesh and the original mesh.

[0011] In one aspect of this technology, the information processing device and method are configured such that the encoded data of the base mesh is decoded to generate the base mesh, the encoded data of the displacement vector is decoded to generate the displacement vector, and the faces are recursively subdivided based on a gradient score representing the complexity of the three-dimensional structure of the object, thereby refining the generated base mesh and applying the generated displacement vector.

[0012] In other aspects of the information processing device and method of this technology, the base mesh is subdivided by recursively dividing faces based on gradient scores representing the complexity of the object's three-dimensional structure. A displacement vector is generated based on the original mesh to be encoded, which consists of vertices and connections representing the object's three-dimensional structure, and the subdivided base mesh, and the base mesh and displacement vector are encoded.

[0013] This is a diagram to explain mesh. This is a diagram to explain V-DMC. This is a diagram to explain an example of the complexity bias of object shapes. This is a diagram showing an example of how subdivision is truncated. This is a diagram showing an example of how subdivision is controlled. This is a diagram showing an example of how subdivision is truncated. This is a diagram to explain an example of how gradient scores are derived. This is a diagram to explain an example of how gradient scores are derived. This is a diagram to explain the selection of subdivision patterns. This is a diagram showing an example of syntax. This is a diagram showing an example of syntax. This is a diagram showing an example of syntax. This is a diagram showing an example of syntax. This is a diagram showing an example of syntax. This is a block diagram showing an example of the main configuration of the encoding device. This is a block diagram showing an example of the main configuration of the V-DMC encoding unit. This is a flowchart explaining an example of the encoding process flow. This is a flowchart explaining an example of the subdivision process flow. This is a flowchart explaining an example of the subdivision process flow. This is a flowchart explaining an example of the subdivision process flow. This is a flowchart explaining an example of the subdivision process flow. This is a flowchart explaining an example of the subdivision process flow. This is a flowchart explaining an example of the V-DMC encoding process flow. This is a block diagram showing an example of the main configuration of the decoding device. This is a flowchart illustrating an example of the decoding process flow. This is a flowchart illustrating an example of the subdivided displacement vector application process flow. This is a flowchart illustrating an example of the subdivided displacement vector application process flow. This is a flowchart illustrating an example of the subdivided displacement vector application process flow. This is a flowchart illustrating an example of the subdivided displacement vector application process flow. This is a flowchart illustrating an example of the subdivided displacement vector application process flow. This is a block diagram illustrating an example of the main configuration of a computer.

[0014] The following describes the embodiments for implementing this disclosure. The description will be in the following order: 1. Supporting literature, etc., for technical content and technical terms 2. Subdivision processing 3. Subdivision control using gradient scores 4. First embodiment (encoding device) 5. Second embodiment (decoding device) 6. Appendix

[0015] <1. Supporting Documents for Technical Content and Terminology> The scope disclosed in this technology includes not only the contents described in the embodiments, but also the contents described in the following non-patent documents that were publicly known at the time of filing, as well as the contents of other documents referenced in the following non-patent documents.

[0016] Non-patent document 1: (mentioned above)

[0017] In other words, the content described in the aforementioned non-patent literature, as well as the content of other documents referenced in the aforementioned non-patent literature, can also serve as a basis for determining the support requirements.

[0018] <2. Subdivision Processing> <V-DMC> Conventionally, as 3D data representing the three-dimensional structure of a three-dimensional structure (an object with a three-dimensional shape), there was a mesh that represented the three-dimensional shape of the object's surface by forming polygons with vertices and connections (also called edges).

[0019] As shown in the upper left of Figure 1, in the mesh, polygonal planes are formed by vertices 11 and connections 12 that connect these vertices 11. These polygons (also called faces) represent the surface of a three-dimensional object, that is, the three-dimensional shape of the object. A texture 13 can be applied to each face of this mesh.

[0020] The mesh data is composed of information such as that shown in the lower part of Figure 1. The vertex information 14, shown first from the left in the lower part of Figure 1, is information indicating the three-dimensional position (three-dimensional coordinates (X, Y, Z)) of each vertex 11 that makes up the mesh. The connection information 15, shown second from the left in the lower part of Figure 1, is information indicating each connection (edge) 12 that makes up the mesh. The texture image 16, shown third from the left in the lower part of Figure 1, is map information of the texture 13 that is applied to each face. The UV map 17, shown fourth from the left in the lower part of Figure 1, is information indicating the correspondence between the vertices 11 and the texture 13. In the UV map 17, the coordinates (UV coordinates) of each vertex 11 in the texture image 16 are shown.

[0021] One method for encoding such meshes is V-DMC (Video-based Dynamic Mesh Coding), as disclosed in Non-Patent Document 1.

[0022] In V-DMC, the mesh to be encoded (referred to as the original mesh in this specification) is represented by a base mesh with a lower resolution (i.e., coarser) than the original mesh, and displacement vectors of the division points obtained by subdividing the base mesh, and the base mesh and displacement vectors are encoded.

[0023] For example, suppose we have an original mesh as shown in the top row of Figure 2. The original mesh is a mesh composed of vertices and connections that represent the three-dimensional structure of an object, and it is the object to be encoded. For example, the original mesh is generated from an image of an object in real space (by camera capture). In Figure 2, the black dots represent vertices, and the lines connecting the black dots represent connections (edges). As mentioned above, a mesh is originally formed by polygons made up of vertices and edges, but here, for the sake of explanation, it is described as a group of vertices connected linearly (in series).

[0024] By simplifying the original mesh, a coarse (low-resolution) mesh is formed, as shown in the second row from the top of Figure 2. This will be the base mesh. One simplification method is to thin out some of the vertices (decimate). In other words, the base mesh is a mesh with lower resolution than the original mesh, generated by thinning out vertices from the original mesh (i.e., simplifying the original mesh).

[0025] By subdividing each polygon of this base mesh, vertices and edges are added, as shown in the third row from the top in Figure 2. The degree of subdivision is arbitrary; that is, the number of vertices and edges added is arbitrary. For example, this subdivision can add the same number of vertices as the original mesh, by thinning it out. In other words, subdivision can make the number of vertices the same as the original mesh. In this specification, these added vertices are also called subdivision points. This subdivision can also be repeated recursively. For example, in the midpoint method, the process of adding vertices to the midpoints of edges (subdivision) is repeated recursively. In other words, by recursively repeating the subdivision, the number of vertices increases and the mesh resolution improves. In this way, for example, it is possible to perform subdivision up to an arbitrary resolution (i.e., control the resolution of the subdivided mesh). In other words, the subdivided mesh can be hierarchized according to its resolution. That is, it can be said to be a hierarchy of the subdivision process and the vertices (subdivision points) and edges obtained by that subdivision process.

[0026] However, the base mesh's connections are updated when vertices are thinned from the original mesh. Therefore, the subdivision points obtained through subdivision are formed on these updated connections (edges). Consequently, the shape of the subdivided base mesh differs from that of the original mesh. More specifically, as shown in the bottom row of Figure 2, the positions of the subdivision points (on the dotted lines) differ from those of the original mesh. It is also possible that the positions of the vertices in the base mesh differ from those of the original mesh.

[0027] In other words, by moving the vertices of the subdivided base mesh (vertices and division points of the base mesh) closer to the vertex positions of the original mesh, the difference in shape between the subdivided base mesh and the original mesh can be reduced. In this specification, such movement of the vertices of the subdivided base mesh (vertices and division points of the base mesh) is also referred to as displacement. Furthermore, the amount and direction of this displacement, expressed as a vector, is also referred to as a displacement vector. Ideally, by displacing each vertex of the subdivided base mesh, the shape of the subdivided base mesh can be made to match the shape of the original mesh. In other words, the original mesh can be represented as the base mesh and the displacement vector.

[0028] In V-DMC, instead of the original mesh (geometry), a base mesh and displacement vectors are encoded. By encoding the base mesh and displacement vectors in this way, encoding can be performed with a reduced polygon count (i.e., number of vertices and edges) compared to encoding the original mesh. Generally, this allows for a reduction in the amount of encoding required for equivalent quality, thus improving encoding efficiency.

[0029] During decoding, as described above, the base mesh is subdivided, and a displacement vector is applied to each vertex of the subdivided base mesh to displace it, thereby restoring (generating) the mesh. In this specification, this mesh will also be referred to as the decoded mesh. Attributes are then applied to the decoded mesh, and the decoded mesh with the applied attributes is rendered. Note that the shape of a polygon (face) can be any polygon, but in the following explanation, polygons will be described as triangular. Therefore, in the following explanation, polygons (faces) will also be referred to as triangles.

[0030] <V-DMC Data Encoding and Decoding> In the case of V-DMC, mesh data consists of a base mesh, displacement vectors, attributes, and atlas information. These data sets are also referred to as V-DMC data. The base mesh consists of information indicating vertices and connections, and is encoded using existing mesh encoding methods such as Draco. Note that the base mesh can be encoded not only intra-encoded but also inter-encoded. When inter-encoded, the base mesh is encoded as motion vectors between frames.

[0031] Displacement vectors are either arithmetically encoded or packed into a two-dimensional plane and encoded as video data using a 2D video encoding method. For example, when encoded as video data, the displacement vector is converted into displacement coefficients in a predetermined way. These displacement coefficients are then placed as pixel values ​​in a two-dimensional region (also called a displacement map). This placement (mapping) of displacement coefficients is also called packing. A video (also called a displacement video) using this displacement map as frame images is then encoded using a 2D video encoding method. In other words, the displacement coefficients are scalar values ​​corresponding to the displacement vectors. A displacement map is map information (which can also be called image data) that stores the displacement coefficients as pixel values. A displacement video is video data that uses the displacement map as frame images.

[0032] Attributes are non-geometric information applied to a mesh (geometry), which is 3D data. For example, attributes may include textures applied to the faces of the mesh (geometry). Attributes (e.g., textures) are divided into multiple sub-regions, projected in a predetermined projection direction for each sub-region, and these projected images (patches) are placed in a two-dimensional region (also called an attribute map). In other words, attribute patches are packed into an attribute map. A moving image (also called an attribute video) using this attribute map as the frame image is encoded using an encoding method for 2D moving images. In short, an attribute map is map information (also called image data) that stores patches (projected textures) as pixel values. An attribute video is moving image data that uses an attribute map as the frame image.

[0033] Atlas information is used when reconstructing a mesh. For example, atlas information may include the correspondence between the base mesh and displacement maps or attribute maps (such as UV maps), as well as quantized values ​​of displacement vectors. This atlas information is encoded using a predetermined encoding method.

[0034] The encoded data (bitstream) for each data set is decoded using the decoding method corresponding to its encoding method. In other words, by decoding the encoded data (bitstream), various types of information such as the base mesh, displacement vectors, attributes, and atlas information are restored (generated).

[0035] In reality, the reconstructed mesh may contain encoding distortions, etc., and therefore may not perfectly match the original mesh before encoding. However, for the sake of explanation, in the following, we will not consider encoding distortions, etc., and will assume that the original mesh is reconstructed through decoding.

[0036] <Subdivision of the Base Mesh> Next, we will explain the subdivision of the base mesh in this decoding process. As mentioned above, the base mesh has a lower resolution (fewer triangles) than the original mesh, so the triangles are divided until its resolution becomes equivalent to that of the original mesh (that is, vertices and edges are added to divide the triangles). In this specification, the process of dividing the triangles of such a mesh is also referred to as "subdivision".

[0037] In other words, in V-DMC, the base mesh is subdivided to generate a mesh with the same level of detail as the original mesh. This subdivision method is arbitrary, or it can be a method of recursively repeating the subdivision on the base mesh. That is, instead of generating a mesh with the same level of detail as the original mesh in a single subdivision, the subdivision of triangles is repeated recursively multiple times, gradually increasing the number of triangles. Using this method, the base mesh can be subdivided hierarchically. Therefore, by controlling the number of repetitions, a mesh with the desired level of detail can be generated. In this specification, each of these recursively repeated subdivisions is also referred to as an "iteration".

[0038] For example, in a method called midpoint subdivision, a triangle is divided into four by adding subdivision points (vertices) that bisect the edges. By recursively repeating this subdivision, a mesh with the desired level of detail is restored (generated). The following explanation will use this midpoint subdivision as an example.

[0039] In this subdivision process, three types of information are managed: vertex information, edge information (connection information), and triangle information (polygon information). Vertex information (vertex(float3[])) includes the identification information (also called the vertex ID) and coordinates of each vertex. Edge information (edges(int2[])) includes the identification information (also called the edge ID) of each edge and the identification information of the vertices located at both ends of that edge. Triangle information (triangles(int3[])) includes the identification information (also called the triangle ID) of each polygon (triangle) and the identification information (vertex ID) of each vertex of that polygon.

[0040] As described above, in mesh subdivision, the level of detail is updated hierarchically through recursive processing. In other words, vertex information, edge information, and triangle information are also updated hierarchically. In this specification, the level of detail hierarchy (also referred to as LoD (Level Of Detail)) is described with the base mesh (LoD0) as the lowest layer, and each time a triangle is subdivided, it moves up one layer. In other words, the higher the level of detail (i.e., the smaller the triangle), the higher the hierarchy (the higher the layer).

[0041] For example, let's say the level of detail of the base mesh is LoD0. In LoD0, the vertices of the base mesh are considered LoD0 vertices, and information about those vertices is stored in the vertex information. Also, the polygons (triangles) formed by connecting these vertices are considered LoD0 triangles, and information about those triangles is stored in the triangle information. Note that no edges are formed in LoD0, so no information about LoD0 edges is stored in the edge information.

[0042] By dividing each triangle of this base mesh once (into four parts), a mesh of one upper level (LoD1) is formed. For example, the edges connecting the vertices of LoD0 become the edges of LoD1, and the information regarding those edges is stored in the edge information. Also, the midpoints of the edges of LoD1 become the vertices of LoD1, and the information regarding those vertices is stored (added) in the vertex information. Further, the polygons (triangles) formed by connecting the vertices between LoD0 and LoD1 become the triangles of LoD1, and the information regarding those triangles is stored in the triangle information (the triangle information is updated).

[0043] Similarly, by dividing each polygon of the LoD1 mesh once (into four parts), a mesh of one upper level (LoD2) is formed. For example, the edges connecting the vertices of LoD1 become the edges of LoD2, and the information regarding those edges is stored (added) in the edge information. Also, the midpoints of the edges of LoD2 become the vertices of LoD2, and the information regarding those vertices is stored (added) in the vertex information. Further, the polygons (triangles) formed by connecting the vertices from LoD0 to LoD2 become the triangles of LoD2, and the information regarding those triangles is stored in the triangle information (the triangle information is updated).

[0044] <Bias in Shape Complexity> Generally, the complexity of the shape (3D structure) of an object (or its original mesh) is not uniform, and there is a bias in the complexity depending on the part.

[0045] For example, A in FIG. 3 shows an example of the state of the original mesh of object 31. This object 31 is a person, and in A of FIG. 3, its upper body is shown. In such an original mesh, generally, the shapes of the regions 32 corresponding to the face part of the person and the regions 33 corresponding to the hand parts of the person are relatively complex. In contrast, the shape of the region 34 corresponding to the arm part of the person is relatively flat.

[0046] However, when this object 31 is converted into a base mesh, vertices and faces are decimated, and each part becomes a relatively flat shape as shown in B of FIG. 3. When this base mesh is subdivided by midpoint subdivision, each face of the base mesh is uniformly subdivided regardless of the complexity of the shape (three-dimensional structure) in the object (original mesh). That is, all of region 32, region 33, and region 34 are uniformly subdivided.

[0047] Therefore, for example, when subdivision is performed to reproduce the shape of a relatively complex part (for example, region 32 or region 33 of object 31) with sufficient accuracy, the number of faces may increase unnecessarily in a relatively flat part (for example, region 34 of object 31). That is, more faces than necessary for reproducing its shape with sufficient accuracy may be generated, which may result in redundancy. When the number of faces increases, the number of displacement vectors increases. Therefore, an unnecessary increase in the number of faces may cause an unnecessary increase in the amount of information of the displacement vectors, and there is a risk of reducing the coding efficiency of the V-DMC bitstream. In addition, an unnecessary increase in the number of faces may cause an unnecessary increase in the load of the subdivision process and the rendering process.

[0048] Also, when subdivision is performed to reproduce the shape of a relatively flat part (for example, region 34 of object 31) with sufficient accuracy, the number of faces may be insufficient in a relatively complex part (for example, region 32 or region 33 of object 31), and it may be difficult to reproduce the shape with sufficient accuracy. That is, there is a risk of reducing the shape quality (reproduction accuracy of the three-dimensional structure) of the decoded mesh. In addition, there is a risk of reducing the subjective quality of the rendering image of the decoded mesh.

[0049] For example, when the base mesh (LoD0) shown in the upper part of Figure 4 is subdivided by midpoint subdivision, the edges are divided as shown in the upper to middle and middle to lower parts of Figure 4. For example, if the subdivision process is stopped at 22 edges to suppress the increase in processing load, then, as shown in the lower part of Figure 4 (LoD2), relatively flat parts of the shape are unnecessarily subdivided, and relatively complex parts of the shape are not sufficiently subdivided, which may reduce the efficiency of reproducing the shape of the object relative to the amount of information.

[0050] In other words, it was difficult to suppress the reduction in encoding efficiency while suppressing the reduction in the shape quality of the decoded mesh (the subjective quality of the rendered image). Similarly, it was difficult to suppress the increase in the load of subdivision and rendering processes while suppressing the reduction in the shape quality of the decoded mesh (the subjective quality of the rendered image).

[0051] <3. Subdivision Control Using Gradient Scores> <Method 1> Subdivision is controlled based on gradient scores, as shown in the top row of the table in Figure 5 (Method 1). In this specification, "gradient score" is a parameter that represents the complexity of the object's shape (three-dimensional structure). In other words, the "gradient score" is a parameter that represents the magnitude of the angle (also referred to as "gradient" in this specification) between adjacent faces (or between the normals of faces). In a mesh, generally, the more complex the shape of the object, the greater this "gradient" becomes. Or, the larger the portion of the "gradient" becomes. Conversely, the flatter the shape, the smaller this "gradient" becomes. Or, the larger the portion of the "gradient" becomes. The "gradient score" represents the complexity (or flatness) of the object's shape by representing the magnitude of such "gradients" according to its value.

[0052] The "gradient score" may represent the value of the "gradient" (i.e., the angle), or it may represent a relationship of magnitude. In other words, the "gradient score" and the value of the "gradient" do not have to be directly proportional. Also, the relationship of magnitude between the "gradient score" and the "gradient (angle)" may or may not be the same. For example, the smaller the "gradient score," the larger the "gradient."

[0053] The control of subdivision based on such gradient scores can be any type of control. For example, the gradient score can be used to control whether or not to subdivide faces. For example, the gradient score can be used to prioritize subdividing faces in more complex areas and decrease the priority of subdividing faces in flatter areas. By controlling in this way, for example, it is possible to make the faces in the more complex areas of the decoded mesh smaller (more faces) and the faces in the flatter areas larger (fewer faces).

[0054] For example, in Figure 6, edges (faces) with a greater gradient are preferentially subdivided. Therefore, even if the subdivision process is stopped at 22 edges, as in the example in Figure 4, more faces are allocated to areas with a greater gradient than in the example in Figure 4.

[0055] Therefore, it is possible to suppress the reduction in the efficiency of reproducing the object's shape relative to the amount of information in the decoded mesh. In other words, it is possible to suppress the reduction in encoding efficiency while suppressing the reduction in the shape quality of the decoded mesh (subjective quality of the rendered image). Similarly, it is possible to suppress the increase in the load of subdivision processing and rendering processing while suppressing the reduction in the shape quality of the decoded mesh (subjective quality of the rendered image).

[0056] This gradient score may be used by both the encoder and the decoder. For example, the encoder may subdivide the base mesh based on the gradient score and use the subdivided base mesh to generate displacement vectors.

[0057] For example, the first information processing device comprises a subdivision unit that subdivides a base mesh by recursively dividing faces based on a gradient score representing the complexity of the object's three-dimensional structure; a displacement vector generation unit that generates a displacement vector based on the original mesh to be encoded, which is composed of vertices and connections representing the object's three-dimensional structure, and the subdivided base mesh; and an encoding unit that encodes the base mesh and the displacement vector.

[0058] Furthermore, the first information processing method executed by the first information processing device includes subdividing the base mesh by recursively dividing faces based on gradient scores representing the complexity of the object's three-dimensional structure, generating a displacement vector based on the original mesh to be encoded, which is composed of vertices and connections representing the object's three-dimensional structure, and the subdivided base mesh, and encoding the base mesh and the displacement vector.

[0059] Furthermore, the first program causes the first information processing device to perform the following processes: subdivide the base mesh by recursively dividing faces based on gradient scores representing the complexity of the object's three-dimensional structure; generate displacement vectors based on the original mesh to be encoded, which is composed of vertices and connections representing the object's three-dimensional structure, and the subdivided base mesh; and encode the base mesh and the displacement vectors.

[0060] The base mesh may be a mesh obtained by thinning out vertices from the original mesh. Furthermore, the displacement vector may be information indicating the difference in vertex positions between the refined base mesh and the original mesh.

[0061] By doing so, the first information processing device can suppress the reduction in the efficiency of reproducing the shape of an object relative to the amount of information in the decoded mesh, as described above. In other words, the first information processing device can suppress the reduction in encoding efficiency while suppressing the reduction in the shape quality of the decoded mesh (subjective quality of the rendered image). Similarly, the first information processing device can suppress the increase in the load of subdivision processing and rendering processing while suppressing the reduction in the shape quality of the decoded mesh (subjective quality of the rendered image).

[0062] The decoder also subdivides the base mesh based on the gradient score and applies displacement vectors.

[0063] For example, the second information processing device includes a decoding unit that decodes encoded data of a base mesh to generate the base mesh and decodes encoded data of a displacement vector to generate the displacement vector, and a subdivided displacement vector application unit that subdivides the generated base mesh by recursively dividing faces based on a gradient score that represents the complexity of the three-dimensional structure of an object, and applies the generated displacement vector.

[0064] Furthermore, the second information processing method executed by the second information processing device includes decoding the encoded data of the base mesh to generate the base mesh, decoding the encoded data of the displacement vector to generate the displacement vector, and recursively dividing the faces based on a gradient score that represents the complexity of the three-dimensional structure of the object, thereby subdividing the generated base mesh and applying the generated displacement vector.

[0065] Furthermore, the second program causes the second information processing device to perform the following processes: decode the encoded data of the base mesh to generate the base mesh; decode the encoded data of the displacement vector to generate the displacement vector; and subdivide the generated base mesh by recursively dividing the faces based on a gradient score that represents the complexity of the object's three-dimensional structure, and then apply the generated displacement vector.

[0066] The base mesh may be a mesh obtained by thinning out vertices from the original mesh. Furthermore, the displacement vector may be information indicating the difference in vertex positions between the refined base mesh and the original mesh.

[0067] By doing so, the second information processing device can suppress the reduction in the efficiency of reproducing the object's shape relative to the amount of information in the decoded mesh, as described above. In other words, the second information processing device can suppress the reduction in encoding efficiency while suppressing the reduction in the shape quality of the decoded mesh (subjective quality of the rendered image). Similarly, the second information processing device can suppress the increase in the load of subdivision processing and rendering processing while suppressing the reduction in the shape quality of the decoded mesh (subjective quality of the rendered image).

[0068] <Method 1-1> When Method 1 is applied, the subdivision may be controlled based on the gradient score per face, as shown in the second row from the top of the table in Figure 5 (Method 1-1). For example, in the first information processing device, the subdivision unit may divide the faces based on the gradient score per face of the mesh. Alternatively, in the second information processing device, the subdivision displacement vector application unit may divide the faces based on the gradient score per face of the mesh.

[0069] For example, if the gradient score of the target face satisfies a predetermined condition, the target face may be divided into four parts, similar to the case of midpoint subdivision. If the gradient score does not satisfy that condition, the division of the target face may be omitted. Note that the condition for the gradient score can be anything. Examples of this condition will be discussed later.

[0070] This face-specific gradient score can be any parameter, as long as it is determined on a face-by-face basis. For example, this gradient score may include a value based on the dot product of the normal of the target face and the normals of the adjacent faces adjacent to the target face. For example, as shown in Figure 7A, the gradient score of the target face f0 may be derived based on the dot product of the normal of the target face f0 and the normals of the adjacent faces (f1, f2, f3) adjacent to the target face f0.

[0071] For example, this gradient score may include the minimum, maximum, mean, or variance of the dot product. For instance, the minimum value of the dot product of the normals shown in Figure 7A (the dot product of the normals between f0 and f1, the dot product of the normals between f0 and f2, and the dot product of the normals between f0 and f3) may be used as the gradient score of the target face f0. Alternatively, the maximum value of these dot products may be used as the gradient score of the target face f0. Furthermore, the mean value of these dot products may be used as the gradient score of the target face f0. Finally, the variance of these dot products may be used as the gradient score of the target face f0. Of course, other statistical values ​​may also be used.

[0072] The normal vector used in the dot product may be a unit vector, or it may be a weighted vector based on the size of the area of ​​the face corresponding to that normal vector.

[0073] Furthermore, this face-level gradient score may include a value based on the dot product of the normals of the vertices of the target face. For example, as shown in Figure 7B, the gradient score of the target face f0 may be derived based on the dot product of the normals of each vertex (v1, v2, v3) of the target face f0.

[0074] For example, this gradient score may include the minimum, maximum, mean, or variance of the dot product. For instance, the minimum value of the dot product of the normals shown in Figure 7B (the dot product of the normals between v1 and v2, the dot product of the normals between v2 and v3, and the dot product of the normals between v3 and v1) may be used as the gradient score of the target face f0. Alternatively, the maximum value of these dot products may be used as the gradient score of the target face f0. Furthermore, the mean value of these dot products may be used as the gradient score of the target face f0. Finally, the variance of these dot products may be used as the gradient score of the target face f0. Of course, other statistical values ​​may also be used.

[0075] In this case as well, the normal vector used in the dot product may be a unit vector, or it may be a weighted vector based on the size of the area of ​​the face corresponding to that normal vector.

[0076] For example, in the example in Figure 7A or Figure 7B, if the gradient score of face (f0, f1, f2) satisfies the condition but the gradient score of face (f3) does not, then, as shown in Figure 7C, face (f0, f1, f2) may be divided into four parts each, while face (f3) may not be divided. Alternatively, in this case, face (f3) may be divided, as shown in the example in Figure 7D. When face (f0) is divided into four parts as described above, the edge shared by face (f0) and face (f3) is divided into two. In other words, a vertex is added to this edge. As shown in Figure 7D, face (f3) may be divided by adding an edge to face (f3) that ends at this added vertex. By dividing face (f3) in accordance with the division of face (f0) in this way, it is possible to suppress the occurrence of cracks due to the position of this added vertex moving for some reason (the occurrence of a gap between the divided face (f0) and face (f3)).

[0077] <Method 1-2> When Method 1 is applied, the subdivision may be controlled based on the gradient score per edge, as shown in the third row from the top of the table in Figure 5 (Method 1-2). For example, in the first information processing device, the subdivision unit may divide the faces based on the gradient score per edge of the mesh. Alternatively, in the second information processing device, the subdivision displacement vector application unit may divide the faces based on the gradient score per edge of the mesh.

[0078] For example, if the gradient score of a target edge satisfies a predetermined condition, the target edge may be divided into two, or the faces sharing that target edge may be divided into two or three. Conversely, if the gradient score does not satisfy that condition, the division of the faces sharing that target edge may be omitted. Note that the condition for the gradient score can be anything. Examples of this condition will be discussed later.

[0079] This edge-specific gradient score can be any parameter, as long as it is determined on an edge-by-edge basis. For example, this gradient score may include a value based on the dot product of the normals of the faces that share the target edge. For example, as shown in Figure 8A, the gradient score of the target face f0 may be derived based on the dot product of the normals of the faces (f0, f1) that supply the target edge e1.

[0080] In this case as well, the normal vector used in the dot product may be a unit vector, or it may be a weighted vector based on the size of the area of ​​the face corresponding to that normal vector.

[0081] Furthermore, this edge-level gradient score may include a value based on the dot product of the normals of the vertices at both ends of the target edge. For example, as shown in Figure 8B, the gradient score of the target edge e1 may be derived based on the dot product of the normals of the vertices (v1, v2) at both ends of the target edge e1.

[0082] In this case as well, the normal vector used in the dot product may be a unit vector, or it may be a weighted vector based on the size of the area of ​​the face corresponding to that normal vector.

[0083] For example, in the example of Figure 8A or Figure 8B, if the gradient score of edge (e1) satisfies the condition, but the gradient scores of edges (e2, e3) do not, then, as shown in Figure 8C, faces (f0, f1) may be divided into two such that edge e1 is divided into two, while faces (f2, f3) may not be divided.

[0084] Furthermore, the above-described methods 1-1 and 1-2 may be applied in combination. That is, in some parts of the mesh, the subdivision may be controlled using face-level gradient scores, while in other parts, the subdivision may be controlled using edge-level gradient scores.

[0085] Furthermore, method 1-1 described above may be used in combination with "subdivision control based on edge length limit." Furthermore, method 1-2 described above may be used in combination with "subdivision control based on edge length limit." Furthermore, methods 1-1 and 1-2 described above may be used in combination with "subdivision control based on edge length limit." "Subdivision control based on edge length limit" is a control method in which, for example, a minimum value (lower limit) of the edge length for which division is permitted is set, and edge divisions where the edge length after division is less than or equal to that minimum value are prohibited (i.e., subdivision is stopped).

[0086] <Method 1-2-1> When Method 1-2 is applied, the subdivision pattern may be set as shown in the fourth row from the top of the table in Figure 5 (Method 1-2-1). For example, when dividing a face into three parts, the subdivision patterns shown in Figure 9A and Figure 9B are possible. In Figure 9A and Figure 9B, the new edges generated by subdivision are shown by dotted lines. In the case of the subdivision pattern shown in Figure 9A, a new edge passing through vertex b is formed by subdivision. In the case of the subdivision pattern shown in Figure 9B, a new edge passing through vertex c is formed by subdivision. When a subdivision pattern that divides one triangle into three parts (also called subdivision pattern 2) is selected, one of these subdivision patterns will be applied.

[0087] In this case, you may choose which subdivision pattern to apply. This selection method can be anything. For example, the subdivision pattern may be selected so that the length of the divided edges generated by the subdivision is longer. By doing so, the number of subdivision iterations can be increased. Therefore, insufficient subdivision can be suppressed, and a higher-resolution mesh can be generated. Alternatively, the subdivision pattern may be selected so that the length of the divided edges generated by the subdivision is shorter. By doing so, the number of subdivision iterations can be reduced. Therefore, the increase in the load of the subdivision process can be suppressed. In addition, since the increase in the number of vertices and edges can be suppressed, the increase in the amount of mesh data can be suppressed. Furthermore, the subdivision pattern may be selected based on the order of the vertices. For example, the subdivision pattern may be selected so that new edges are formed on vertices with a lower order. By selecting the subdivision pattern in this way, the calculation of edge lengths becomes unnecessary, so the increase in the load of the subdivision process can be suppressed.

[0088] Alternatively, a subdivision pattern may be selected in which the dot product of the normals of adjacent faces having generated edges is larger. Alternatively, a subdivision pattern may be selected in which the dot product of the normals of adjacent faces having generated edges is smaller.

[0089] <Method 1-3> When Method 1 is applied, the gradient score may be derived in the lowest level of LoD and carried over to the higher level of LoD, as shown in the fifth row from the top of the table in Figure 5 (Method 1-3). For example, in the first information processing device, the subdivision unit may derive a gradient score based on the base mesh, and in each iteration of subdivision, divide the face based on that gradient score. Alternatively, in the second information processing device, the subdivision displacement vector application unit may derive a gradient score based on the base mesh, divide the face based on that gradient score in each iteration of subdivision, and apply the displacement vector after the completion of subdivision. By doing so, the subdivision process can be simplified and the increase in load can be suppressed.

[0090] <Method 1-4> When Method 1 is applied, a gradient score may be derived for each LoD, as shown in the sixth row from the top of the table in Figure 5, and the displacement vector may be reflected in the mesh for each LoD (Method 1-4). For example, in the first information processing device, the subdivision unit may derive a gradient score in each iteration of subdivision using the mesh to which the displacement vector has been applied, and divide the faces based on the derived gradient score. Alternatively, in the second information processing device, the subdivision displacement vector application unit may derive a gradient score in each iteration of subdivision using the mesh to which the displacement vector has been applied, divide the faces based on the derived gradient score, and apply the displacement vector. By doing so, the subdivision can be controlled in a way that more accurately reflects the object shape.

[0091] <Method 1-5> The "condition for gradient score" described above may be changed to "the number of faces must not exceed the upper limit when the subdivision order is rearranged based on the gradient score." That is, when Method 1 is applied, the subdivision order may be rearranged based on the gradient score and the subdivision may be limited using the upper limit of the number of faces (Method 1-5). For example, in the first information processing device, the subdivision unit may rearrange the processing order of faces based on the gradient score and divide each face in the rearranged order until the upper limit of the number of faces is reached. Alternatively, in the second information processing device, the subdivision displacement vector application unit may rearrange the processing order of faces based on the gradient score and divide each face in the rearranged order until the upper limit of the number of faces is reached. In other words, subdivision proceeds in the rearranged order and is terminated when the number of faces reaches the upper limit. That is, the condition is that the number of faces does not exceed the upper limit.

[0092] Note that sorting may be omitted for data units that are not likely to be truncated (e.g., Level of Data). In this case, the data units for which sorting is performed (or omitted) should be the same for both the encoder and the decoder.

[0093] Furthermore, sorting control information for controlling such sorting may be transmitted. For example, in the first information processing device, the encoding unit may further encode sorting control information for controlling the sorting of the processing order. In addition, in the second information processing device, the decoding unit may further decode the encoded data of sorting control information for controlling the sorting of the processing order and generate sorting control information. This sorting control information can be any information that controls the sorting of the processing order of subdivision. For example, this sorting control information may include a sorting flag that indicates whether to sort the processing order at the target data unit level. In addition, this sorting control information may include sorting application start specification information that specifies the iteration at which the application of sorting of the processing order begins. This sorting control information may be transmitted at any data unit level. For example, it may be at the sequence unit level, the frame unit level, the mesh patch unit level, the submesh unit level, or the LoD unit level. Furthermore, sorting control information for different data units may be used in combination. For example, if sorting control information for a higher data unit level exists, it may be basically inherited, and if sorting control information for a lower data unit level exists, it may be updated by the lower data unit level.

[0094] For example, in the example of sequence-based syntax shown in Figure 10, "asve_subdivision_gradient_reorder_flag" is a reorder flag that indicates whether to reorder the processing order in the current sequence. Also, "asve_lod_subdivision_gradient_reorder_flag [i]" is reorder application start specification information that specifies the iteration (LoD [i]) in which the application of the reordered processing order should begin in the current sequence.

[0095] For example, in the frame-based syntax example shown in Figure 11, "afve_subdivision_gradient_reorder_flag" is a reorder flag indicating whether to reorder the processing order in the current frame. "afve_lod_subdivision_gradient_reorder_flag [i]" is reorder application start information that specifies the iteration (LoD [i]) in which the reordering of the processing order should begin in the current frame.

[0096] For example, in the example syntax for each mesh patch shown in Figure 12, "mdu_subdivision_gradient_reorder_flag" is a reorder flag that indicates whether to reorder the processing order in the current mesh patch. Also, "mdu_lod_subdivision_gradient_reorder_flag [i]" is reorder application start specification information that specifies the iteration (LoD [i]) in which the application of the reordered processing order should begin in the current mesh patch.

[0097] Such sorting control information may be transmitted from the encoding side to the decoding side. This allows the encoder to control the sorting of the decoder, making it easier for the decoder to perform the same sorting as the encoder.

[0098] <Method 1-5-1> The encoder may pack all displacement vectors in the order they were derived, as in the conventional method. However, in this case, the decoder will need to rearrange the processing order, which also necessitates rearranging the displacement vectors. Therefore, the encoder may rearrange the displacement vectors in the order they are used before packing them. In other words, when Method 1-5 is applied, the displacement vectors may be rearranged in the order they are used before packing them, as shown in the eighth row from the top of the table in Figure 5 (Method 1-5-1). For example, in the first information processing device, the displacement vector generation unit may rearrange the generated displacement vectors based on the gradient score before packing them. The encoding unit may then encode the packed displacement vectors. Alternatively, in the second information processing device, the decoding unit may decode the encoded data of the rearranged and packed displacement vectors based on the gradient score and generate the displacement vectors in the rearranged order. By doing so, the decoder can omit rearranging the displacement vectors.

[0099] Furthermore, in this process, only some of the displacement vectors may be packed. For example, each displacement vector may be packed preferentially in the order it is sorted, and packing may be stopped midway. That is, some of the later (lower-order) displacement vectors may be deleted after sorting and then packed. For example, the packing range (number of displacement vectors to be packed) may be controlled according to the upper limit of the number of faces (decoder performance), etc. For example, in the first information processing device, the displacement vector generation unit may pack the displacement vectors from the beginning up to a predetermined rank in the sorted order. Alternatively, in the second information processing device, the decoding unit may decode the encoded data of the displacement vectors that have been packed from the beginning up to a predetermined rank in the sorted order. By doing so, the increase in the amount of information in the displacement vectors can be suppressed.

[0100] <Method 1-6> Alternatively, the "condition for the gradient score" may be defined as "the result of comparing the gradient score with a predetermined threshold (also called the gradient threshold) being 'false'." In other words, whether or not to subdivide may be controlled based on the comparison result between the gradient score and the gradient threshold. That is, when Method 1 is applied, as shown in the ninth row from the top of the table in Figure 5, it may be selected whether or not to subdivide based on the comparison result between the gradient score and the gradient threshold (Method 1-6). For example, in the first information processing device, the subdivision unit may decide whether or not to subdivide the target face based on the comparison result between the gradient score of the target face and a predetermined gradient threshold. Alternatively, in the second information processing device, the subdivision displacement vector application unit may decide whether or not to subdivide the target face based on the comparison result between the gradient score of the target face and a predetermined gradient threshold.

[0101] For example, if the gradient score is below the gradient threshold, the target face (or target edge) may be split. Alternatively, if the gradient score is below the gradient threshold, the splitting of the target face (or target edge) may be omitted. How this is controlled depends on how the gradient score and gradient threshold are derived (what these parameters mean), etc.

[0102] In this case, rearranging the processing order is unnecessary and may be omitted. Also, the gradient threshold to be compared can be a common value used by both the encoder and the decoder. For example, this gradient threshold may be set in the encoder and transmitted to the decoder for sharing. For example, in the first information processing device, the encoding unit may further encode the gradient threshold. Alternatively, in the second information processing device, the decoding unit may further decode the encoded data of the gradient threshold to generate the gradient threshold.

[0103] The gradient threshold may be a value that can be directly compared with the gradient score, or it may be a value that can be compared indirectly (through some calculation), such as an angle.

[0104] The gradient scores of all faces and edges may be compared with a gradient threshold, or only some faces and edges may be compared with a gradient threshold. For example, it may be possible to control which gradient scores are compared with the gradient threshold. For example, an encoder may perform this control. For example, an encoder may supply threshold control information to a decoder that controls the application of the gradient threshold (comparison of the gradient flag with the gradient threshold). For example, in the first information processing device, an encoding unit may further encode threshold control information that controls the application of the gradient threshold. Alternatively, in the second information processing device, a decoding unit may further decode the encoded data of threshold control information that controls the application of the gradient threshold and generate that threshold control information.

[0105] This threshold control information can be any information that controls the application of gradient thresholds. For example, the threshold control information may include a threshold application flag indicating whether to apply gradient thresholds to the target data unit. The threshold control information may also include threshold application start specification information that specifies the iteration at which to start applying gradient thresholds. This threshold control information can be transmitted in any data unit. For example, it may be transmitted in sequence units, frame units, mesh patch units, submesh units, or LoD units. Threshold control information from different data units may also be used in combination. For example, if threshold control information for a higher data unit exists, it may be inherited, and if threshold control information for a lower data unit exists, it may be updated by that lower data unit's threshold control information.

[0106] For example, in the example of sequence-based syntax shown in Figure 13, "asve_gradient_based_subdivision_flag" is a threshold application flag indicating whether to apply a gradient threshold in the current sequence. "asve_lod_gradient_based_subdivision_flag [i]" is threshold application start information specifying the iteration (LoD [i]) in which the application of the gradient threshold begins in the current sequence. "asve_subdivision_min_gradient" indicates the gradient threshold applied in the current sequence.

[0107] For example, in the frame-based syntax example shown in Figure 14, "afve_gradient_based_subdivision_flag" is a threshold application flag indicating whether to apply a gradient threshold in the current frame. "afve_lod_gradient_based_subdivision_flag [i]" is threshold application start information specifying the iteration (LoD [i]) in which gradient threshold application should begin in the current frame. "afve_subdivision_min_gradient" indicates the gradient threshold applied in the current frame.

[0108] For example, in the example syntax for each mesh patch shown in Figure 15, "mdu_gradient_based_subdivision_flag" is a threshold application flag indicating whether to apply a gradient threshold to the current mesh patch. "mdu_lod_gradient_based_subdivision_flag [i]" is threshold application start information specifying the iteration (LoD [i]) at which gradient threshold application begins in the current mesh patch. "mdu_subdivision_min_gradient" indicates the gradient threshold applied to the current mesh patch.

[0109] Threshold control information like this may be transmitted from the encoding side to the decoding side. This allows the encoder to control the application of the decoder's gradient threshold, making it easier for the decoder to perform the same comparison and determination of gradient score and gradient threshold as the encoder.

[0110] Furthermore, methods 1-5 and 1-6 described above may be applied in combination. That is, a portion of the gradient score may be rearranged and a portion of it may be compared with the gradient threshold.

[0111] <Method 1-7> When Method 1 is applied, the writing location for triangle and edge information may be determined for each LoD, as shown in the 10th row from the top of the table in Figure 5 (Method 1-7). Any method may be used to set the storage location for edge information (first storage location) and the storage location for triangle information (second storage location). For example, the storage location for edge information may be set based on the number of divided edges. For example, in the first information processing device, the storage location setting unit may set the first storage location for the edge information of a divided edge based on the number of divided edges, which are the edges after division in the processing of the current iteration. By setting the storage location for divided edges based on the number of divided edges, the first information processing device can set an appropriate storage location according to the number of divided edges. For example, the first information processing device can suppress the generation of unnecessary free space in the buffer and suppress the increase in buffer capacity required to store edge information and triangle information.

[0112] Alternatively, the number of divisible edges may be determined based on an edge length limit in subdivision (i.e., a lower limit on edge length), and the number of divided edges obtained by dividing these divisible edges may be determined. For example, in the first information processing device, the storage location setting unit may derive the divided edges based on the edge length limit and set a first storage location for the edge information of the divided edges based on the cumulative sum of the number of divided edges. By doing so, the first information processing device can more easily derive the number of divided edges. In other words, the first information processing device can more easily set a first storage location for the edge information of the divided edges.

[0113] Furthermore, the storage location for the edge information of the new edge may be set based on the cumulative sum of the number of divided edges and the cumulative sum of the number of triangles after subdivision. For example, in the first information processing device, the storage location setting unit may set a first storage location for the edge information of the new edge, which is an edge generated by the processing of the current iteration, based on the cumulative sum of the number of divided edges and the cumulative sum of the number of triangles after subdivision in the processing of the current iteration. By doing so, the first information processing device can more easily set the first storage location for the edge information of the new edge.

[0114] Furthermore, the storage location for triangle information may be set based on the number of triangles. For example, in the first information processing device, the storage location setting unit may set a second storage location based on the number of triangles after subdivision in the processing of the current iteration. By doing so, the first information processing device can set an appropriate storage location according to the number of triangles. For example, the first information processing device can suppress the occurrence of unnecessary free space in the buffer and suppress the increase in buffer capacity required to store edge information and triangle information.

[0115] Alternatively, the triangle subdivision pattern may be determined based on the edge length limit in subdivision (i.e., the lower limit of the edge length), and the number of triangles after subdivision may be determined. For example, in the first information processing device, the storage position setting unit may derive the triangle subdivision pattern based on the edge length limit, and set the second storage position based on the cumulative sum of the number of triangles after subdivision in the processing of the current iteration to which that subdivision pattern is applied. By doing so, the first information processing device can derive the number of triangles more easily. In other words, the first information processing device can set the second storage position more easily.

[0116] For example, let "edges[]" be the buffer that stores the edge information of the mesh after subdivision in the current iteration, and let its size be "EdgeCount". Also, let "triangles[]" be the buffer that stores the triangle information of the mesh after subdivision in the current iteration, and let its size be "TriangleCount". Furthermore, let "NextEdges[]" be the buffer that stores the edge information of the mesh after subdivision in the next iteration, and let its size be "NextEdgeCount". Furthermore, let "NextTriangles[]" be the buffer that stores the triangle information of the mesh after subdivision in the next iteration, and let its size be "NextTriangleCount". In addition, let "SubdivEdge[e]" be a parameter that indicates whether the e-th edge is subdivided after subdivision. For example, if it is subdivided, the value of SubdivEdge[e] is "1", and if it is not subdivided, the value of SubdivEdge[e] is "0".

[0117] And let's assume that the parameter "NextOffsetEdge[]" is defined as shown in equation (1) below.

[0118] ... (1)

[0119] In this case, when the e-th edge is divided, the destination for writing the edge information of the generated divided edge can be expressed as shown in equation (2) below, when SubdivEdge[e] > 0. Note that when SubdivEdge[e] == 0, the edge is not divided, so no edge information is written.

[0120] ... (2)

[0121] The parameter "SubdivPattern[t]" represents the number of triangles generated when the t-th triangle is subdivided. For example, in the case of subdivision pattern N, the value of this parameter "SubdivPattern[t]" will be "N+1".

[0122] And let's assume that the parameter "NextOffsetTri[]" is defined as shown in equation (3) below.

[0123] ... (3)

[0124] In this case, the destination for writing the triangle information of the triangle generated when the t-th triangle is divided can be expressed as shown in equation (4) below.

[0125] ... (4)

[0126] Then, the destination for writing the edge information of the new edge generated when the t-th triangle is divided can be expressed as shown in equations (5) to (7) below.

[0127] ... (5) ... (6) ... (7)

[0128] Through the calculations described above, the first and second storage locations can be set. The cumulative sum can be calculated by sequentially adding the number of triangles added by the division of each triangle. Cumulative sum calculations on a GPU can also be implemented using existing methods.

[0129] <Method 1-8> The gradient score is used in both the encoder and the decoder. The encoder and decoder may each derive this gradient score, or the gradient score derived by the encoder may be transmitted to the decoder. In other words, when Method 1 is applied, the gradient score may be transmitted as shown in the bottom row of the table in Figure 5 (Method 1-8). For example, in the first information processing device, the encoding unit may further encode the gradient score. In the second information processing device, the decoding unit may further decode the encoded data of the gradient score to generate the gradient score. The subdivided displacement vector application unit may then subdivide the base mesh based on the generated gradient score. In this way, the calculation process for deriving the gradient score by the decoder can be omitted.

[0130] For example, in the first information processing device, the encoding unit may encode the gradient score of the base mesh as an attribute of each face of the base mesh. Alternatively, the encoding unit may encode the gradient score of the base mesh as metadata. Alternatively, the encoding unit may encode the gradient score of a base mesh in which the faces have been divided one or more times (i.e., a mesh with LoD 1 or higher) as metadata. For example, in the second information processing device, the decoding unit may decode the encoded data of the gradient score of the base mesh that has been encoded as an attribute of each face of the base mesh. Alternatively, the decoding unit may decode the encoded data of the gradient score of the base mesh that has been encoded as metadata. Alternatively, the decoding unit may decode the encoded data of the gradient score of a base mesh in which the faces have been divided one or more times (i.e., a mesh with LoD 1 or higher) that has been encoded as metadata.

[0131] <Combinations> Each of the methods described above may be applied in combination with any other method, as long as no contradiction arises. Three or more methods may be applied in combination. Furthermore, the combinatorial methods may include not only those shown in the table in Figure 5 as "methods," but all elements described herein. In addition, each of the methods described above may be applied in combination with other methods not described above.

[0132] <4. First Embodiment> <Encoding Device> This technology can be applied to any device. For example, this technology can be applied to an encoding device that encodes a mesh and generates a bitstream. Figure 16 is a block diagram showing an example of the configuration of an encoding device, which is one embodiment of an information processing device to which this technology is applied. The encoding device 300 (first information processing device) shown in Figure 16 is a device that encodes a mesh and generates its bitstream. Therefore, the encoding device 300 can also be called a bitstream generation device that generates a bitstream.

[0133] Figure 16 shows the main components such as the processing unit and data flow, but it does not necessarily represent everything. In other words, the encoding device 300 may have processing units that are not shown as blocks in Figure 16, or processes and data flows that are not shown as arrows or other symbols in Figure 16.

[0134] The encoding device 300 encodes the mesh in essentially the same way as the V-DMC described in the above-mentioned non-patent literature, except that it applies this technology. For example, the encoding device 300 obtains the original mesh to be encoded and an attribute map containing the texture corresponding to that original mesh. This original mesh includes not only information about the mesh geometry but also information indicating the correspondence with the attribute map (e.g., a UV list). The encoding device 300 encodes the original mesh and attribute map using the V-DMC method, generates a V-DMC bitstream, and outputs it.

[0135] As shown in Figure 16, the encoding device 300 has a pre-processing unit 311 and a V-DMC encoding unit 312. The pre-processing unit 311 performs processing related to pre-processing before encoding. As shown in Figure 16, the pre-processing unit 311 has a base mesh generation unit 321, a subdivision unit 322, a displacement vector generation unit 323, and an atlas information generation unit 324.

[0136] The base mesh generation unit 321 performs processing related to the generation of a base mesh. For example, the base mesh generation unit 321 may acquire the original mesh input to the encoding device 300. Alternatively, the base mesh generation unit 321 may perform decimation processing (vertex thinning) on ​​the original mesh to generate a base mesh. The base mesh generation unit 321 may supply the generated base mesh to the subdivision unit 322.

[0137] The subdivision unit 322 performs processing related to the subdivision of the base mesh. For example, the subdivision unit 322 may acquire the base mesh supplied from the base mesh generation unit 321. Alternatively, the subdivision unit 322 may acquire the original mesh input to the encoding device 300. The subdivision unit 322 may subdivide the base mesh and generate division points. In this case, the subdivision unit 322 may perform the subdivision by applying, for example, the present technology (Method 1, etc.) described above. The subdivision unit 322 may supply the subdivided base mesh to the displacement vector generation unit 323.

[0138] The displacement vector generation unit 323 performs processing related to the generation of displacement vectors. For example, the displacement vector generation unit 323 may acquire the subdivided base mesh supplied from the subdivision unit 322. Alternatively, the displacement vector generation unit 323 may acquire the original mesh input to the encoding device 300. Furthermore, the displacement vector generation unit 323 may use this information to generate displacement vectors that displace the vertices of the subdivided base mesh. The displacement vector generation unit 323 may also supply the necessary information to the atlas information generation unit 324 to generate atlas information corresponding to the base mesh and displacement vectors. The displacement vector generation unit 323 may then acquire the atlas information generated by the atlas information generation unit 324. The displacement vector generation unit 323 may also supply the generated displacement vectors, base mesh, and atlas information to the V-DMC encoding unit 312.

[0139] The atlas information generation unit 324 performs processing related to the generation of atlas information corresponding to the base mesh and displacement vectors. For example, the atlas information generation unit 324 may acquire information supplied from the displacement vector generation unit 323. Based on that information, the atlas information generation unit 324 may generate atlas information corresponding to the base mesh and displacement vectors. The atlas information generation unit 324 may supply the generated atlas information to the displacement vector generation unit 323.

[0140] The V-DMC encoding unit 312 performs processing related to the encoding of V-DMC data. For example, the V-DMC encoding unit 312 may acquire the original mesh input to the encoding device 300. The V-DMC encoding unit 312 may also acquire the base mesh, displacement vector, atlas information, etc., supplied from the displacement vector generation unit 323. The V-DMC encoding unit 312 may also acquire the attribute map input to the encoding device 300. The V-DMC encoding unit 312 may encode the acquired atlas information, base mesh, displacement vector, and attribute map, respectively, and generate encoded data for each. Therefore, the V-DMC encoding unit 312 can also be called an encoding unit. The V-DMC encoding unit 312 may also multiplex these encoded data as substreams to generate a single bitstream. This bitstream is also called a V-DMC bitstream. Therefore, the V-DMC encoding unit 312 can also be called a bitstream generation unit (or V-DMC bitstream generation unit). The V-DMC encoding unit 312 may output the generated V-DMC bitstream to the outside of the encoding device 300.

[0141] <V-DMC Encoding Unit> Figure 17 is a block diagram showing an example of the main configuration of the V-DMC encoding unit 312. Figure 17 shows the main components such as the processing unit and data flow, but it does not necessarily show everything. In other words, the V-DMC encoding unit 312 may have processing units that are not shown as blocks in Figure 17, or processes and data flows that are not shown as arrows, etc., in Figure 17.

[0142] As shown in Figure 17, the V-DMC coding unit 312 includes an atlas information coding unit 351, a base mesh coding unit 352, a displacement vector correction unit 353, a displacement vector coding unit 354, a mesh reconstruction unit 355, an attribute map conversion unit 356, an attribute coding unit 357, and a multiplexing unit 358.

[0143] The Atlas information coding unit 351 performs processing related to coding the Atlas information. For example, the Atlas information coding unit 351 may acquire Atlas information supplied from the displacement vector generation unit 323. The Atlas information coding unit 351 may code the acquired Atlas information using a predetermined coding scheme to generate coded Atlas information data. The Atlas information coding unit 351 may supply the generated coded Atlas information data to the multiplexing unit 358.

[0144] The base mesh coding unit 352 performs processing related to the coding of the base mesh. For example, the base mesh coding unit 352 may acquire the base mesh supplied from the displacement vector generation unit 323. The base mesh coding unit 352 may acquire atlas information supplied from the displacement vector generation unit 323. The base mesh coding unit 352 may quantize the acquired base mesh, code it using a predetermined coding method (e.g., Draco), and generate coded base mesh data. In this case, the base mesh coding unit 352 may perform coding of the base mesh based on the acquired atlas information. The base mesh coding unit 352 may supply the coded base mesh data it has generated to the displacement vector correction unit 353. The base mesh coding unit 352 may also supply the coded base mesh data it has generated to the multiplexing unit 358.

[0145] The displacement vector correction unit 353 performs processing related to the correction of the displacement vector. For example, the displacement vector correction unit 353 may acquire the base mesh and displacement vector supplied from the displacement vector generation unit 323. Alternatively, the displacement vector correction unit 353 may acquire the encoded data of the base mesh supplied from the base mesh encoding unit 352. Based on this information, the displacement vector correction unit 353 may correct the displacement vector. For example, the displacement vector correction unit 353 may decode the acquired encoded data of the base mesh, compare the base mesh before and after encoding to determine the encoding distortion of the base mesh, and correct the displacement vector according to that encoding distortion. The displacement vector correction unit 353 may supply the corrected displacement vector to the displacement vector encoding unit 354. Alternatively, the displacement vector correction unit 353 may dequantize the decoded base mesh and supply it to the mesh reconstruction unit 355.

[0146] The displacement vector coding unit 354 performs processing related to the coding of the displacement vector. For example, the displacement vector coding unit 354 may acquire the displacement vector supplied from the displacement vector correction unit 353. Alternatively, the displacement vector coding unit 354 may generate a displacement map by performing a wavelet transform on the displacement vector, quantizing it, and packing it into a two-dimensional region. The displacement vector coding unit 354 may also generate a displacement video using the displacement map as the frame image. In other words, the displacement video is a moving image in which the displacement map, which is a two-dimensional region in which the displacement vector is packed, is used as the frame image. The displacement vector coding unit 354 may code the generated displacement video using a predetermined coding scheme for 2D moving images to generate coded data of the displacement vector (displacement video). The displacement vector coding unit 354 may supply the coded data of the displacement vector thus generated to the multiplexing unit 358. Alternatively, the displacement vector coding unit 354 may decode the generated coded data, unpack the displacement vector from the displacement map, and dequantize the displacement vector. The displacement vector coding unit 354 may supply its inversely quantized displacement vector to the mesh reconstruction unit 355.

[0147] The displacement vector coding unit 354 may also generate encoded data of the displacement vector by arithmetic coding the displacement vector. In that case, the displacement vector coding unit 354 may generate the displacement vector by arithmetic decoding the encoded data and supply it to the mesh reconstruction unit 355.

[0148] Alternatively, the displacement vector coding unit 354 may acquire atlas information supplied from the displacement vector generation unit 323 and perform encoding of the displacement vector based on the acquired atlas information.

[0149] The mesh reconstruction unit 355 performs processing related to mesh reconstruction. For example, the mesh reconstruction unit 355 may acquire a base mesh supplied from the displacement vector correction unit 353. Alternatively, the mesh reconstruction unit 355 may acquire displacement vectors supplied from the displacement vector encoding unit 354. The mesh reconstruction unit 355 may reconstruct the mesh using these. The mesh reconstruction unit 355 may supply the reconstructed mesh to the attribute map conversion unit 356.

[0150] The attribute map conversion unit 356 performs processing related to the conversion of the attribute map. For example, the attribute map conversion unit 356 may acquire the reconstructed mesh supplied from the mesh reconstruction unit 355. Alternatively, the attribute map conversion unit 356 may acquire atlas information supplied from the displacement vector generation unit 323. Furthermore, the attribute map conversion unit 356 may acquire the original mesh and attribute map input to the encoding device 300. The attribute map conversion unit 356 may convert the acquired attribute map based on other acquired information. For example, the attribute map conversion unit 356 may convert the attribute map to correspond to the reconstructed mesh based on atlas information, the original mesh, etc. In other words, the attribute map conversion unit 356 can also be said to generate the converted attribute map. Therefore, the attribute map conversion unit 356 can also be called an attribute map generation unit. The attribute map conversion unit 356 may supply the converted attribute map to the attribute encoding unit 357.

[0151] The attribute coding unit 357 performs processing related to attribute coding. For example, the attribute coding unit 357 may acquire an attribute map supplied from the attribute map conversion unit 356. Alternatively, the attribute coding unit 357 may generate an attribute video using the acquired attribute map as a frame image. Furthermore, the attribute coding unit 357 may code the generated attribute video using a predetermined coding scheme for 2D moving images to generate coded attribute data. The attribute coding unit 357 may supply the generated coded attribute data to the multiplexing unit 358.

[0152] The multiplexing unit 358 performs processing related to the multiplexing of encoded data (substreams). For example, the multiplexing unit 358 may acquire encoded data of atlas information supplied from the atlas information encoding unit 351. The multiplexing unit 358 may also acquire encoded data of base mesh supplied from the base mesh encoding unit 352. The multiplexing unit 358 may also acquire encoded data of displacement vectors supplied from the displacement vector encoding unit 354. The multiplexing unit 358 may also acquire encoded data of attributes supplied from the attribute encoding unit 357. The multiplexing unit 358 may multiplex these encoded data as substreams to generate a V-DMC bitstream. Therefore, the multiplexing unit 358 can also be called a bitstream generation unit (or V-DMC bitstream generation unit). The multiplexing unit 358 may output the generated V-DMC bitstream to the outside of the encoding device 300. For example, the multiplexing unit 358 may supply its V-DMC bitstream to the decoding device 500, which will be described later. Therefore, the multiplexing unit 358 can also be called a supply unit (provider) of its V-DMC bitstream.

[0153] <Application of this technology> The encoding device 300 having the above configuration may be used as the first information processing device, and this technology may be applied to it. For example, by applying Method 1, the encoding device 300 may include a subdivision unit 322 that subdivides the base mesh by recursively dividing faces based on a gradient score that represents the complexity of the three-dimensional structure of an object, a displacement vector generation unit 323 that generates a displacement vector based on the original mesh to be encoded and the subdivided base mesh, and a V-DMC encoding unit 312 that encodes the base mesh and the displacement vector.

[0154] With this configuration, the encoding device 300 can suppress a reduction in encoding efficiency while suppressing a reduction in the shape quality of the decoded mesh.

[0155] <Encoding Process Flow> An example of the encoding process flow performed by this encoding device 300 will be explained with reference to the flowchart in Figure 18.

[0156] When the encoding process is started, the base mesh generation unit 321 of the encoding device 300 decimates the original mesh to be encoded in step S301 and generates a base mesh.

[0157] In step S302, the subdivision unit 322 performs a subdivision process to subdivide the base mesh.

[0158] In step S303, the displacement vector generation unit 323 generates a displacement vector.

[0159] In step S304, the atlas information generation unit 324 generates atlas information.

[0160] In step S305, the V-DMC encoding unit 312 performs V-DMC encoding processing to encode V-DMC data and generate a V-DMC bitstream.

[0161] The encoding process ends when the processing in step S305 is completed. The encoding device 300 performs this encoding process for each frame of the original mesh.

[0162] <Subdivision Process Flow 1> Next, the subdivision process performed in step S302 of Figure 18 will be explained with reference to the flowcharts in Figures 19 to 24. The flowcharts in Figures 19 to 21 show an example of the subdivision process flow when the gradient score is derived only in LoD0 and the gradient score of LoD0 is carried over (reused) to the higher LoD (LoD1 and subsequent LoDs). The flowchart in Figure 19 shows an example of the subdivision process flow when the processing order of the divisions is rearranged in order of gradient score and the process is terminated at the upper limit of the number of faces.

[0163] When the subdivision process is started, the subdivision unit 322 derives all the gradient scores of LoD0 in step S321.

[0164] In step S322, the subdivision unit 322 rearranges the processing order of the divisions based on the gradient score.

[0165] In step S323, the subdivision unit 322 determines whether the current number of faces is less than the upper limit on the number of faces. If it is determined that the current number of faces is less than the upper limit, the process proceeds to step S324.

[0166] In step S324, the subdivision unit 322 divides the face to be processed.

[0167] In step S325, the subdivision unit 322 determines whether all gradient scores of the current LoD have been processed. If it is determined that there are unprocessed gradient scores, the processing target is moved to the next gradient score, and the process returns to step S323. In other words, each process from step S323 to step S325 is executed for each gradient score. If it is determined in step S325 that all gradient scores of the current LoD have been processed, the process proceeds to step S326.

[0168] In step S326, the subdivision unit 322 determines whether all LoDs have been processed. If it is determined that there are unprocessed LoDs, the processing target is moved to the next LoD, and the process returns to step S323. That is, each process from step S323 to step S326 is executed for each LoD. Then, in step S326, if it is determined that all LoDs have been processed, the subdivision process ends, and the process returns to Figure 18.

[0169] Furthermore, if it is determined in step S323 that the number of faces has reached the upper limit, the subdivision process is terminated and the process returns to Figure 18.

[0170] <Subdivision Processing Flow 2> The flowchart in Figure 20 shows an example of the subdivision processing flow when the division is controlled based on the comparison result of the gradient score and the gradient threshold.

[0171] When the subdivision process is started, the subdivision unit 322 derives all the gradient scores of LoD0 in step S331.

[0172] In step S332, the subdivision unit 322 compares the gradient score with the gradient threshold and determines whether the gradient score is less than or equal to the gradient threshold. If it is determined that the gradient score is less than or equal to the gradient threshold, the process proceeds to step S333.

[0173] In step S333, the subdivision unit 322 divides the face to be processed. When the processing in step S333 is completed, the process proceeds to step S334. Also, if it is determined in step S332 that the gradient score is greater than the gradient threshold, the processing in step S333 is omitted, and the process proceeds to step S334.

[0174] In step S334, the subdivision unit 322 determines whether all gradient scores of the current LoD have been processed. If it is determined that there are unprocessed gradient scores, the processing target is moved to the next gradient score, and the process returns to step S332. In other words, each process from step S332 to step S334 is executed for each gradient score. If it is determined in step S334 that all gradient scores of the current LoD have been processed, the process proceeds to step S335.

[0175] In step S335, the subdivision unit 322 determines whether all LoDs have been processed. If it is determined that there are unprocessed LoDs, the processing target is moved to the next LoD, and the process returns to step S332. That is, each process from step S332 to step S335 is executed for each LoD. Then, in step S335, if it is determined that all LoDs have been processed, the subdivision process ends, and the process returns to Figure 18.

[0176] <Subdivision Processing Flow 3> The flowchart in Figure 21 shows an example of the subdivision processing flow when both methods are used in combination: one that sorts the division processing order by gradient score and terminates at the upper limit of the number of faces, and another that controls the division based on the comparison result of the gradient score and the gradient threshold.

[0177] When the subdivision process is started, the subdivision unit 322 derives all the gradient scores of LoD0 in step S341.

[0178] In step S342, the subdivision unit 322 rearranges the processing order of the divisions based on the gradient score.

[0179] In step S343, the subdivision unit 322 determines whether the current number of faces is less than the upper limit on the number of faces. If it is determined that the current number of faces is less than the upper limit, the process proceeds to step S344.

[0180] In step S344, the subdivision unit 322 compares the gradient score with the gradient threshold and determines whether the gradient score is less than or equal to the gradient threshold. If it is determined that the gradient score is less than or equal to the gradient threshold, the process proceeds to step S345.

[0181] In step S345, the subdivision unit 322 divides the face to be processed. When the processing in step S345 is completed, the process proceeds to step S346. Also, if it is determined in step S344 that the gradient score is greater than the gradient threshold, the processing in step S345 is omitted, and the process proceeds to step S346.

[0182] In step S346, the subdivision unit 322 determines whether all gradient scores of the current LoD have been processed. If it is determined that there are unprocessed gradient scores, the processing target is moved to the next gradient score, and the process returns to step S343. In other words, each process from step S343 to step S346 is executed for each gradient score. If it is determined in step S346 that all gradient scores of the current LoD have been processed, the process proceeds to step S347.

[0183] In step S347, the subdivision unit 322 determines whether all LoDs have been processed. If it is determined that there are unprocessed LoDs, the processing target is moved to the next LoD, and the process returns to step S343. That is, each process from step S343 to step S347 is executed for each LoD. Then, in step S347, if it is determined that all LoDs have been processed, the subdivision process ends, and the process returns to Figure 18.

[0184] Furthermore, if it is determined in step S343 that the number of faces has reached the upper limit, the subdivision process is terminated and the process returns to Figure 18.

[0185] <Further Subdivision Process Flow 4> The flowcharts in Figures 22 to 24 show examples of the subdividement process flow when deriving gradient scores in each LoD. The flowchart in Figure 22 shows an example of the subdividement process flow when the processing order of the divisions is rearranged by gradient score and the process is terminated at the upper limit of the number of faces.

[0186] When the subdivision process is started, the subdivision unit 322 derives all gradient scores for the LoD to be processed (current LoD) in step S351.

[0187] In step S352, the subdivision unit 322 rearranges the processing order of the division based on its gradient score.

[0188] In step S353, the subdivision unit 322 determines whether the current number of faces is less than the upper limit on the number of faces. If it is determined that the current number of faces is less than the upper limit, the process proceeds to step S354.

[0189] In step S354, the subdivision unit 322 divides the face to be processed.

[0190] In step S355, the subdivision unit 322 determines whether all gradient scores of the current LoD have been processed. If it is determined that there are unprocessed gradient scores, the processing target is moved to the next gradient score, and the process returns to step S353. In other words, each process from step S353 to step S355 is executed for each gradient score. If it is determined in step S355 that all gradient scores of the current LoD have been processed, the process proceeds to step S356.

[0191] In step S356, the subdivision unit 322 applies displacement vectors to the vertices of the LoD to be processed. Once the processing in step S356 is complete, the process proceeds to step S357.

[0192] In step S357, the subdivision unit 322 determines whether all LoDs have been processed. If it is determined that there are unprocessed LoDs, the processing target is moved to the next LoD, and the process returns to step S351. That is, each process from step S351 to step S357 is executed for each LoD. Then, in step S357, if it is determined that all LoDs have been processed, the subdivision process ends, and the process returns to Figure 18.

[0193] Furthermore, if it is determined in step S353 that the number of faces has reached the upper limit, the subdivision process is terminated and the process returns to Figure 18.

[0194] <Further Subdivision Process Flow 5> The flowchart in Figure 23 shows an example of the further subdivision process flow when the division is controlled based on the comparison result of the gradient score and the gradient threshold.

[0195] When the subdivision process is started, the subdivision unit 322 derives the gradient score of the processing target (the processing target face or processing target edge) in step S361.

[0196] In step S362, the subdivision unit 322 compares the gradient score with the gradient threshold and determines whether the gradient score is less than or equal to the gradient threshold. If it is determined that the gradient score is less than or equal to the gradient threshold, the process proceeds to step S363.

[0197] In step S363, the subdivision unit 322 divides the face to be processed. When the processing in step S363 is completed, the process proceeds to step S364. Also, if it is determined in step S362 that the gradient score is greater than the gradient threshold, the processing in step S363 is omitted, and the process proceeds to step S364.

[0198] In step S364, the subdivision unit 322 determines whether all gradient scores of the current LoD have been processed. If it is determined that there are unprocessed gradient scores, the processing target is moved to the next gradient score, and the process returns to step S361. In other words, each process from step S361 to step S364 is executed for each gradient score. If it is determined in step S364 that all gradient scores of the current LoD have been processed, the process proceeds to step S365.

[0199] In step S365, the subdivision unit 322 applies displacement vectors to the vertices of the LoD to be processed. Once the processing in step S365 is complete, the process proceeds to step S366.

[0200] In step S366, the subdivision unit 322 determines whether all LoDs have been processed. If it is determined that there are unprocessed LoDs, the processing target is moved to the next LoD, and the process returns to step S361. That is, each process from step S361 to step S366 is executed for each LoD. Then, in step S366, if it is determined that all LoDs have been processed, the subdivision process ends, and the process returns to Figure 18.

[0201] <Subdivision Processing Flow 6> The flowchart in Figure 24 shows an example of the subdivision processing flow when both methods are used in combination: one that sorts the division processing order by gradient score and terminates at the upper limit of the number of faces, and another that controls the division based on the comparison result of the gradient score and the gradient threshold.

[0202] When the subdivision process is started, the subdivision unit 322 derives all gradient scores for the LoD to be processed (current LoD) in step S371.

[0203] In step S372, the subdivision unit 322 rearranges the processing order of the divisions based on the gradient score.

[0204] In step S373, the subdivision unit 322 determines whether the current number of faces is less than the upper limit on the number of faces. If it is determined that the current number of faces is less than the upper limit, the process proceeds to step S374.

[0205] In step S374, the subdivision unit 322 compares the gradient score with the gradient threshold and determines whether the gradient score is less than or equal to the gradient threshold. If it is determined that the gradient score is less than or equal to the gradient threshold, the process proceeds to step S375.

[0206] In step S375, the subdivision unit 322 divides the face to be processed. When the processing in step S375 is completed, the process proceeds to step S376. Also, if it is determined in step S374 that the gradient score is greater than the gradient threshold, the processing in step S375 is omitted, and the process proceeds to step S376.

[0207] In step S376, the subdivision unit 322 determines whether all gradient scores of the current LoD have been processed. If it is determined that there are unprocessed gradient scores, the processing target is moved to the next gradient score, and the process returns to step S373. In other words, each process from step S373 to step S376 is executed for each gradient score. If it is determined in step S376 that all gradient scores of the current LoD have been processed, the process proceeds to step S377.

[0208] In step S377, the subdivision unit 322 applies displacement vectors to the vertices of the LoD to be processed. Once the processing in step S377 is complete, the process proceeds to step S378.

[0209] In step S378, the subdivision unit 322 determines whether all LoDs have been processed. If it is determined that there are unprocessed LoDs, the processing target is moved to the next LoD, and the process returns to step S371. That is, each process from step S371 to step S378 is executed for each LoD. Then, in step S378, if it is determined that all LoDs have been processed, the subdivision process ends, and the process returns to Figure 18.

[0210] Furthermore, if it is determined in step S373 that the number of faces has reached the upper limit, the subdivision process is terminated and the process returns to Figure 18.

[0211] <Flow of V-DMC encoding process> Next, with reference to the flowchart in Figure 25, an example of the flow of the V-DMC encoding process performed in step S305 of Figure 18 will be explained.

[0212] When the V-DMC encoding process is started, the atlas information encoding unit 351 encodes the atlas information in step S391 by applying the method 1 described above.

[0213] In step S392, the base mesh coding unit 352 encodes the base mesh.

[0214] In step S393, the displacement vector correction unit 353 corrects the displacement vector.

[0215] In step S394, the displacement vector encoding unit 354 encodes the corrected displacement vector. For example, the displacement vector encoding unit 354 may pack the displacement vector into a displacement video and encode it using a 2D encoding method. Alternatively, the displacement vector encoding unit 354 may arithmetically encode the displacement vector.

[0216] In step S395, the mesh reconstruction unit 355 reconstructs the mesh.

[0217] In step S396, the attribute map conversion unit 356 converts the attribute map.

[0218] In step S397, the attribute encoding unit 357 encodes an attribute video in which the attribute map is used as a frame image.

[0219] In step S398, the multiplexing unit 358 multiplexes encoded data of atlas information (including subdivision setting information), encoded data of the base mesh, encoded data of the displacement vector, and encoded data of the attributes to generate a V-DMC bitstream.

[0220] Once the process in step S398 is completed, the V-DMC encoding process ends, and the process returns to Figure 18.

[0221] By performing each process as described above, the encoding device 300 can suppress a reduction in encoding efficiency while suppressing a reduction in the shape quality of the decoded mesh.

[0222] <5. Second Embodiment> <Decoding Device> This technology can be applied to a decoding device that decodes mesh encoded data. Figure 26 is a block diagram showing an example of the configuration of a decoding device, which is one embodiment of an information processing device to which this technology is applied. The decoding device 500 (second information processing device) shown in Figure 26 is a device that decodes the mesh encoded data (V-DMC bitstream generated by the multiplexing unit 358 (Figure 17)) generated in the encoding device 300 (Figure 16) and reconstructs the decoded mesh.

[0223] Figure 26 shows the main components such as the processing unit and data flow, but it does not necessarily represent everything. In other words, the decoding device 500 may have processing units that are not shown as blocks in Figure 26, or processes and data flows that are not shown as arrows or other symbols in Figure 26.

[0224] The decoding device 500 decodes the encoded mesh data, which has been encoded in essentially the same way as the V-DMC described in the above-mentioned non-patent document, except for applying this technology, and reconstructs the decoded mesh. For example, the decoding device 500 acquires a V-DMC bitstream. This V-DMC bitstream may be generated in the encoding device 300, for example. As a reconstruction process, the decoding device 500 decodes the V-DMC bitstream and reconstructs the mesh (also referred to as the decoded mesh). The decoding device 500 also applies a texture to the decoded mesh, generates a display image for displaying the decoded mesh, and outputs it to the outside of the decoding device 500. For example, the decoding device 500 supplies the display image to an external display device for display.

[0225] As shown in Figure 26, the decoding device 500 (second information processing device) includes a demultiplexing unit 511, an atlas information decoding unit 512, a base mesh decoding unit 513, a displacement vector decoding unit 514, an attribute decoding unit 515, a subdivision unit 516, a displacement vector application unit 517, an attribute application unit 518, and a display processing unit 519. The atlas information decoding unit 512, the base mesh decoding unit 513, the displacement vector decoding unit 514, and the attribute decoding unit 515 are collectively referred to as the decoding unit 521. In other words, these processing units may be integrated and configured as the decoding unit 521. The subdivision unit 516 and the displacement vector application unit 517 are collectively referred to as the subdivided displacement vector application unit 522. In other words, these processing units may be integrated and configured as the subdivided displacement vector application unit 522.

[0226] The demultiplexing unit 511 performs processing related to demultiplexing. For example, the demultiplexing unit 511 may acquire the V-DMC bitstream to be decoded, which is supplied to the decoding device 500. Alternatively, the demultiplexing unit 511 may demultiplex the acquired V-DMC bitstream and extract encoded data of atlas information, encoded data of base mesh, encoded data of displacement vectors, and encoded data of attributes. Therefore, the demultiplexing unit 511 can also be said to be an acquisition unit for the V-DMC bitstream or the various information contained in the V-DMC bitstream. The demultiplexing unit 511 may supply the extracted encoded data of atlas information to the atlas information decoding unit 512. Alternatively, the demultiplexing unit 511 may supply the extracted encoded data of base mesh to the base mesh decoding unit 513. Alternatively, the demultiplexing unit 511 may supply the extracted encoded data of displacement vectors to the displacement vector decoding unit 514. The demultiplexing unit 511 may also supply the encoded data of the extracted attributes to the attribute decoding unit 515.

[0227] The atlas information decoding unit 512 performs processing related to decoding the atlas information. For example, the atlas information decoding unit 512 may acquire encoded data of the atlas information supplied from the demultiplexing unit 511. Alternatively, the atlas information decoding unit 512 may decode the acquired encoded data of the atlas information and generate (restore) the atlas information. Although the arrows are omitted in Figure 26, the atlas information decoding unit 512 may supply the generated atlas information to one or more of the base mesh decoding unit 513, displacement vector decoding unit 514, attribute decoding unit 515, subdivision unit 516, displacement vector application unit 517, attribute application unit 518, and display processing unit 519.

[0228] The base mesh decoding unit 513 performs processing related to the decoding of the base mesh. For example, the base mesh decoding unit 513 may acquire encoded data of the base mesh supplied from the demultiplexing unit 511. Alternatively, the base mesh decoding unit 513 may decode the acquired encoded data of the base mesh using a predetermined decoding method (e.g., Draco) to generate (restore) a base mesh (e.g., a vertex list or a triangle list). In this case, the base mesh decoding unit 513 may acquire atlas information supplied from the atlas information decoding unit 512 and decode the encoded data of the base mesh based on that atlas information. The base mesh decoding unit 513 may supply the generated base mesh to the subdivision unit 516.

[0229] The displacement vector decoding unit 514 performs processing related to the decoding of the displacement vector. For example, the displacement vector decoding unit 514 may acquire encoded data of the displacement vector (i.e., a displacement bitstream) supplied from the demultiplexing unit 511. The displacement vector decoding unit 514 may decode the encoded data of the displacement vector and generate (restore) the displacement vector. For example, if the displacement vector is encoded as a displacement video, the displacement vector decoding unit 514 may decode the encoded data of the displacement vector using a predetermined decoding method for 2D moving images to generate (restore) the displacement video, and then unpack the displacement vector from the displacement map, which is a frame image of the displacement video. Alternatively, if the displacement video is arithmetic encoded, the displacement vector decoding unit 514 arithmetically decodes the encoded data of the displacement vector and generates the displacement vector. In this case, the displacement vector decoding unit 514 may acquire atlas information supplied from the atlas information decoding unit 512 and decode the displacement vector based on that atlas information. The displacement vector decoding unit 514 may supply the displacement vector obtained in this way to the displacement vector application unit 517.

[0230] The attribute decoding unit 515 performs processing related to the decoding of attributes. For example, the attribute decoding unit 515 may acquire encoded attribute data supplied from the demultiplexing unit 511. Alternatively, the attribute decoding unit 515 may decode the acquired encoded attribute data using a predetermined decoding method for 2D moving images to generate (restore) an attribute video. In this case, the attribute decoding unit 515 may acquire atlas information supplied from the atlas information decoding unit 512 and decode the attributes based on that atlas information. The attribute decoding unit 515 may also supply an attribute map, which is a frame image of the generated attribute video, to the attribute application unit 518.

[0231] In other words, the decoding unit 521 may decode one or more of the encoded data of the atlas information, the encoded data of the base mesh, the encoded data of the displacement vector, and the encoded data of the attributes.

[0232] The subdivision unit 516 performs processing related to the subdivision of triangles in the base mesh. For example, the subdivision unit 516 may acquire the base mesh supplied from the base mesh decoding unit 513. The subdivision unit 516 may subdivide the base mesh (triangles) and generate subdivision points. In this case, the subdivision unit 516 may acquire atlas information (including subdivision setting information) supplied from the atlas information decoding unit 512 and subdivide the base mesh based on the subdivision setting information. The subdivision unit 516 may supply the subdivided base mesh to the displacement vector application unit 517.

[0233] The displacement vector application unit 517 performs processing related to the application of displacement vectors to the subdivided base mesh. For example, the displacement vector application unit 517 may acquire the subdivided base mesh supplied from the subdivision unit 516. The displacement vector application unit 517 may acquire the displacement vector supplied from the displacement vector decoding unit 514. The displacement vector application unit 517 may apply the displacement vector to the vertices of the subdivided base mesh. In other words, the displacement vector application unit 517 may generate a decoded mesh. In this case, the displacement vector application unit 517 may acquire atlas information supplied from the atlas information decoding unit 512 and apply the displacement vector to the vertices of the subdivided base mesh based on that atlas information. The displacement vector application unit 517 may supply the decoded mesh generated in this way to the attribute application unit 518.

[0234] Furthermore, the subdivision unit 516 and the displacement vector application unit 517 can exchange information with each other and perform processing in cooperation with each other. That is, the subdivision displacement vector application unit 522 can perform a subdivision displacement vector application process, which includes subdividing the base mesh and applying displacement vectors to the subdivided base mesh, by coordinating these processing units.

[0235] The attribute application unit 518 performs processing related to the application of attributes to the decoded mesh. For example, the attribute application unit 518 may acquire the decoded mesh supplied from the displacement vector application unit 517. The attribute application unit 518 may acquire the attribute map supplied from the attribute decoding unit 515. The attribute application unit 518 may apply the attributes of the attribute map to the decoded mesh. In this case, the attribute application unit 518 may acquire atlas information supplied from the atlas information decoding unit 512 and apply attributes to the decoded mesh based on that atlas information. The attribute application unit 518 may supply the decoded mesh with the attributes applied in this way to the display processing unit 519.

[0236] The display processing unit 519 performs processing related to the display of the mesh. For example, the display processing unit 519 may acquire a decoded mesh to which attributes have been applied, supplied from the attribute application unit 518. The display processing unit 519 may render the acquired decoded mesh and generate a display image for displaying the decoded mesh. The display processing unit 519 may then supply the generated display image to an external device, such as another device, to display the display image.

[0237] <Application of this technology> The decoding device 500 having the above configuration may be used as a second information processing device, and this technology may be applied to it.

[0238] For example, the decoding device 500 (second information processing device) includes a decoding unit 521 that decodes encoded data of a base mesh to generate the base mesh and decodes encoded data of a displacement vector to generate the displacement vector, and a subdivided displacement vector application unit 522 that subdivides the generated base mesh by recursively dividing faces based on a gradient score that represents the complexity of the three-dimensional structure of an object, and applies the generated displacement vector.

[0239] With this configuration, the decoding device 500 can suppress a reduction in encoding efficiency while suppressing a reduction in the shape quality of the decoded mesh.

[0240] <Decryption Process Flow> An example of the decoding process flow performed by this decoding device 500 will be explained with reference to the flowchart in Figure 27.

[0241] When the decoding process is started, the demultiplexing unit 511 of the decoding device 500 demultiplexes the V-DMC bitstream in step S501.

[0242] In step S502, the Atlas information decoding unit 512 decodes the encoded data of the Atlas information and generates (restores) the Atlas information.

[0243] In step S503, the base mesh decoding unit 513 decodes the base mesh.

[0244] In step S504, the displacement vector decoding unit 514 decodes the displacement vector.

[0245] In step S505, the attribute decoding unit 515 decodes the attribute. That is, the attribute decoding unit 515 decodes the encoded data of the attribute and generates (restores) the attribute.

[0246] In step S506, the subdivided displacement vector application unit 522 performs a subdivided displacement vector application process, which subdivides the base mesh and applies the displacement vector to generate a decoded mesh.

[0247] In step S507, the attribute application unit 518 applies the attribute to the decoded mesh.

[0248] In step S508, the display processing unit 519 renders the decoded mesh to which the attributes have been applied to generate a display image.

[0249] When the process in step S509 is completed, the decoding process is finished. The decoding device 500 performs this decoding process for each frame of the original mesh.

[0250] <Flow of Subdivision Displacement Vector Application Process 1> Next, the subdivision displacement vector application process performed in step S506 of Figure 27 will be explained with reference to the flowcharts in Figures 28 to 33. The flowcharts in Figures 28 to 30 show an example of the flow of the subdivision displacement vector application process when the gradient score is derived only in LoD0 and the gradient score of LoD0 is carried over (reused) to the higher LoD (LoD1 and subsequent LoDs). The flowchart in Figure 28 shows an example of the flow of the subdivision displacement vector application process when the processing order of the divisions is rearranged in order of gradient score and the process is terminated at the upper limit of the number of faces.

[0251] When the subdivided displacement vector application process is started, the subdivided displacement vector application unit 522 derives all gradient scores of LoD0 in step S521.

[0252] In step S522, the subdivided displacement vector application unit 522 rearranges the processing order of the divisions based on the gradient score.

[0253] In step S523, the subdivided displacement vector application unit 522 determines whether the current number of faces is less than the upper limit for the number of faces. If it is determined that the current number of faces is less than the upper limit, the process proceeds to step S524.

[0254] In step S524, the subdivided displacement vector application unit 522 divides the face to be processed.

[0255] In step S525, the subdivided displacement vector application unit 522 determines whether all gradient scores of the current LoD have been processed. If it is determined that there are unprocessed gradient scores, the processing target is moved to the next gradient score, and the process returns to step S523. In other words, each process from step S523 to step S525 is executed for each gradient score. If it is determined in step S525 that all gradient scores of the current LoD have been processed, the process proceeds to step S526.

[0256] In step S526, the subdivided displacement vector application unit 522 determines whether all LoDs have been processed. If it is determined that there are unprocessed LoDs, the processing target is moved to the next LoD, and the process returns to step S523. That is, each process from step S523 to step S526 is executed for each LoD. If it is determined in step S526 that all LoDs have been processed, the process proceeds to step S527. Also, if it is determined in step S523 that the number of faces has reached the upper limit, the process proceeds to step S527.

[0257] In step S527, the subdivided displacement vector application unit 522 applies displacement vectors to all vertices of the Line of Direction (LoD). When the processing in step S527 is completed, the subdivided displacement vector application process is completed, and the process returns to Figure 27.

[0258] <Process Flow for Applying Subdivision Displacement Vectors 2> The flowchart in Figure 29 shows an example of the process flow for applying subdivision displacement vectors when the division is controlled based on the comparison result of the gradient score and the gradient threshold.

[0259] When the subdivided displacement vector application process is started, the subdivided displacement vector application unit 522 derives all gradient scores of LoD0 in step S531.

[0260] In step S532, the subdivided displacement vector application unit 522 compares the gradient score with the gradient threshold and determines whether the gradient score is less than or equal to the gradient threshold. If it is determined that the gradient score is less than or equal to the gradient threshold, the process proceeds to step S533.

[0261] In step S533, the subdivided displacement vector application unit 522 divides the face to be processed. When the processing in step S533 is completed, the process proceeds to step S534. Also, if it is determined in step S532 that the gradient score is greater than the gradient threshold, the processing in step S533 is omitted, and the process proceeds to step S534.

[0262] In step S334, the subdivided displacement vector application unit 522 determines whether all gradient scores of the current LoD have been processed. If it is determined that there are unprocessed gradient scores, the processing target is moved to the next gradient score, and the process returns to step S532. In other words, each process from step S532 to step S534 is executed for each gradient score. If it is determined in step S534 that all gradient scores of the current LoD have been processed, the process proceeds to step S535.

[0263] In step S535, the subdivided displacement vector application unit 522 determines whether all LoDs have been processed. If it is determined that there are unprocessed LoDs, the processing target is moved to the next LoD, and the process returns to step S532. That is, each process from step S532 to step S535 is executed for each LoD. If it is determined in step S535 that all LoDs have been processed, the process proceeds to step S536.

[0264] In step S536, the subdivided displacement vector application unit 522 applies displacement vectors to all vertices of the Line of Dance (LoD). When the processing in step S536 is completed, the subdivided displacement vector application process is completed, and the process returns to Figure 27.

[0265] <Process Flow for Applying Subdivision Displacement Vectors 3> The flowchart in Figure 30 shows an example of the process flow for applying subdivision displacement vectors when both methods are used in combination: one that sorts the processing order of the divisions by gradient score and terminates at the upper limit of the number of faces, and another that controls the divisions based on the comparison result of the gradient score and the gradient threshold.

[0266] When the subdivided displacement vector application process is started, the subdivided displacement vector application unit 522 derives all gradient scores of LoD0 in step S541.

[0267] In step S542, the subdivided displacement vector application unit 522 rearranges the processing order of the divisions based on the gradient score.

[0268] In step S543, the subdivided displacement vector application unit 522 determines whether the current number of faces is less than the upper limit for the number of faces. If it is determined that the current number of faces is less than the upper limit, the process proceeds to step S544.

[0269] In step S544, the subdivided displacement vector application unit 522 compares the gradient score with the gradient threshold and determines whether the gradient score is less than or equal to the gradient threshold. If it is determined that the gradient score is less than or equal to the gradient threshold, the process proceeds to step S545.

[0270] In step S545, the subdivided displacement vector application unit 522 divides the face to be processed. When the processing in step S545 is completed, the process proceeds to step S546. Also, if it is determined in step S544 that the gradient score is greater than the gradient threshold, the processing in step S545 is omitted, and the process proceeds to step S546.

[0271] In step S546, the subdivided displacement vector application unit 522 determines whether all gradient scores of the current LoD have been processed. If it is determined that there are unprocessed gradient scores, the processing target is moved to the next gradient score, and the process returns to step S543. In other words, each process from step S543 to step S546 is executed for each gradient score. If it is determined in step S546 that all gradient scores of the current LoD have been processed, the process proceeds to step S547.

[0272] In step S547, the subdivided displacement vector application unit 522 determines whether all LoDs have been processed. If it is determined that there are unprocessed LoDs, the processing target is moved to the next LoD, and the process returns to step S543. That is, each process from step S543 to step S547 is executed for each LoD.

[0273] Then, if it is determined in step S547 that all LoDs have been processed, the process proceeds to step S548. Also, if it is determined in step S543 that the number of faces has reached the upper limit, the process proceeds to step S548.

[0274] In step S548, the subdivided displacement vector application unit 522 applies displacement vectors to all vertices of the Line of Dance (LoD). When the processing in step S548 is completed, the subdivided displacement vector application process is completed, and the process returns to Figure 27.

[0275] <Process Flow for Applying Subdivision Displacement Vectors 4> The flowcharts in Figures 31 to 33 show examples of the process flow for applying subdivision displacement vectors when deriving gradient scores in each LoD. The flowchart in Figure 31 shows an example of the process flow for applying subdivision displacement vectors when the processing order of the divisions is rearranged in order of gradient scores and the process is terminated at the upper limit of the number of faces.

[0276] When the subdivided displacement vector application process is started, the subdivided displacement vector application unit 522 derives all gradient scores for the LoD (current LoD) to be processed in step S551.

[0277] In step S552, the subdivided displacement vector application unit 522 rearranges the processing order of the division based on its gradient score.

[0278] In step S553, the subdivided displacement vector application unit 522 determines whether the current number of faces is less than the upper limit for the number of faces. If it is determined that the current number of faces is less than the upper limit, the process proceeds to step S554.

[0279] In step S554, the subdivided displacement vector application unit 522 divides the face to be processed.

[0280] In step S555, the subdivided displacement vector application unit 522 determines whether all gradient scores of the current LoD have been processed. If it is determined that there are unprocessed gradient scores, the processing target is moved to the next gradient score, and the process returns to step S553. In other words, each process from step S553 to step S555 is executed for each gradient score. If it is determined in step S555 that all gradient scores of the current LoD have been processed, the process proceeds to step S556.

[0281] In step S556, the subdivided displacement vector application unit 522 applies displacement vectors to the vertices of the LoD to be processed. When the processing in step S556 is completed, the process proceeds to step S557.

[0282] In step S557, the subdivided displacement vector application unit 522 determines whether all LoDs have been processed. If it is determined that there are unprocessed LoDs, the processing target is moved to the next LoD, and the process returns to step S551. That is, each process from step S551 to step S557 is executed for each LoD. Then, in step S557, if it is determined that all LoDs have been processed, the subdivided displacement vector application process ends, and the process returns to Figure 27.

[0283] If it is determined in step S553 that the number of faces has reached the upper limit, the subdivision displacement vector application process is terminated. However, the process proceeds to step S556, where the displacement vector is applied to the vertices of the LoD to be processed. In step S557, it is determined that all LoDs have been processed, the subdivision displacement vector application process ends, and the process returns to Figure 27.

[0284] <Process Flow for Applying Subdivision Displacement Vectors 5> The flowchart in Figure 32 shows an example of the process flow for applying subdivision displacement vectors when the division is controlled based on the comparison result of the gradient score and the gradient threshold.

[0285] When the subdivision displacement vector application process is started, the subdivision displacement vector application unit 522 derives the gradient score of the processing target (the processing target face or processing target edge) in step S561.

[0286] In step S562, the subdivided displacement vector application unit 522 compares the gradient score with the gradient threshold and determines whether the gradient score is less than or equal to the gradient threshold. If it is determined that the gradient score is less than or equal to the gradient threshold, the process proceeds to step S563.

[0287] In step S563, the subdivided displacement vector application unit 522 divides the face to be processed. When the processing in step S563 is completed, the process proceeds to step S564. Also, if it is determined in step S562 that the gradient score is greater than the gradient threshold, the processing in step S563 is omitted, and the process proceeds to step S564.

[0288] In step S564, the subdivided displacement vector application unit 522 determines whether all gradient scores of the current LoD have been processed. If it is determined that there are unprocessed gradient scores, the processing target is moved to the next gradient score, and the process returns to step S561. In other words, each process from step S561 to step S564 is executed for each gradient score. If it is determined in step S564 that all gradient scores of the current LoD have been processed, the process proceeds to step S565.

[0289] In step S565, the subdivided displacement vector application unit 522 applies displacement vectors to the vertices of the LoD to be processed. When the processing in step S565 is completed, the process proceeds to step S566.

[0290] In step S566, the subdivided displacement vector application unit 522 determines whether all LoDs have been processed. If it is determined that there are unprocessed LoDs, the processing target is moved to the next LoD, and the process returns to step S561. That is, each process from step S561 to step S566 is executed for each LoD. Then, in step S566, if it is determined that all LoDs have been processed, the subdivided displacement vector application process ends, and the process returns to Figure 27.

[0291] <Process Flow for Applying Subdivision Displacement Vectors 6> The flowchart in Figure 33 shows an example of the process flow for applying subdivision displacement vectors when both methods are used in combination: one that sorts the processing order of the divisions by gradient score and terminates at the upper limit of the number of faces, and another that controls the divisions based on the comparison result of the gradient score and the gradient threshold.

[0292] When the subdivided displacement vector application process is started, the subdivided displacement vector application unit 522 derives all gradient scores of the LoD to be processed (current LoD) in step S571.

[0293] In step S572, the subdivided displacement vector application unit 522 rearranges the processing order of the divisions based on the gradient score.

[0294] In step S573, the subdivided displacement vector application unit 522 determines whether the current number of faces is less than the upper limit for the number of faces. If it is determined that the current number of faces is less than the upper limit, the process proceeds to step S574.

[0295] In step S574, the subdivided displacement vector application unit 522 compares the gradient score with the gradient threshold and determines whether the gradient score is less than or equal to the gradient threshold. If it is determined that the gradient score is less than or equal to the gradient threshold, the process proceeds to step S575.

[0296] In step S575, the subdivided displacement vector application unit 522 divides the target face. When the processing in step S575 is completed, the process proceeds to step S576. Also, if it is determined in step S574 that the gradient score is greater than the gradient threshold, the processing in step S575 is omitted, and the process proceeds to step S576.

[0297] In step S576, the subdivided displacement vector application unit 522 determines whether all gradient scores of the current LoD have been processed. If it is determined that there are unprocessed gradient scores, the processing target is moved to the next gradient score, and the process returns to step S573. In other words, each process from step S573 to step S576 is executed for each gradient score. If it is determined in step S576 that all gradient scores of the current LoD have been processed, the process proceeds to step S577.

[0298] In step S577, the subdivided displacement vector application unit 522 applies displacement vectors to the vertices of the LoD to be processed. When the processing in step S577 is completed, the process proceeds to step S578.

[0299] In step S578, the subdivided displacement vector application unit 522 determines whether all LoDs have been processed. If it is determined that there are unprocessed LoDs, the processing target is moved to the next LoD, and the process returns to step S571. That is, each process from step S571 to step S578 is executed for each LoD. Then, in step S578, if it is determined that all LoDs have been processed, the subdivided displacement vector application process ends, and the process returns to Figure 27.

[0300] If it is determined in step S573 that the number of faces has reached the upper limit, the subdivision displacement vector application process is terminated. However, the process proceeds to step S577, where the displacement vector is applied to the vertices of the LoD to be processed. In step S578, it is determined that all LoDs have been processed, the subdivision displacement vector application process ends, and the process returns to Figure 27.

[0301] By performing each process as described above, the decoding device 500 can suppress a reduction in encoding efficiency while suppressing a reduction in the shape quality of the decoded mesh.

[0302] <6. Notes> <Polygon Shape> In the above explanation, the polygon shape was described as a triangle, but this shape is just one example. The polygon shape can be any polygon.

[0303] <Encoding Method> In the above explanation, V-DMC was used as an example of an encoding method to which this technology can be applied. However, this technology is not limited to this example and can be applied to any encoding method that encodes the base mesh, displacement vectors, attribute maps including textures, and atlas information, or information equivalent thereto.

[0304] <Computer> The series of processes described above can be executed by hardware or by software. When the series of processes are executed by software, the programs that make up that software are installed on the computer. Here, "computer" includes computers built into dedicated hardware, as well as general-purpose personal computers that can perform various functions by installing various programs.

[0305] Figure 34 is a block diagram showing an example of the hardware configuration of a computer that executes the series of processes described above using a program.

[0306] In the computer 900 shown in Figure 34, the CPU (Central Processing Unit) 901, ROM (Read-Only Memory) 902, and RAM (Random Access Memory) 903 are interconnected via a bus 904.

[0307] An input / output interface 910 is also connected to the bus 904. An input / output interface 910 is connected to an input unit 911, an output unit 912, a storage unit 913, a communication unit 914, and a drive 915.

[0308] The input unit 911 consists of, for example, a keyboard, mouse, microphone, touch panel, and input terminals. The output unit 912 consists of, for example, a display, speaker, and output terminals. The storage unit 913 consists of, for example, a hard disk, RAM disk, and non-volatile memory. The communication unit 914 consists of, for example, a network interface. The drive 915 drives removable media 921 such as a magnetic disk, optical disk, magneto-optical disk, or semiconductor memory.

[0309] In a computer configured as described above, the CPU 901 loads, for example, a program stored in the memory unit 913 into the RAM 903 via the input / output interface 910 and the bus 904, and executes it, thereby performing the series of processes described above. The RAM 903 also appropriately stores data necessary for the CPU 901 to perform various processes.

[0310] The program executed by the computer can be recorded and applied, for example, on removable media 921 such as a package medium. In this case, the program can be installed in the storage unit 913 via the input / output interface 910 by inserting the removable media 921 into the drive 915.

[0311] Furthermore, this program can also be provided via wired or wireless transmission media such as a local area network, the internet, or digital satellite broadcasting. In that case, the program can be received by the communication unit 914 and installed in the storage unit 913.

[0312] In addition, this program can be pre-installed in ROM 902 or memory unit 913.

[0313] <Applications of this technology> This technology can be applied to any encoding / decoding scheme. Furthermore, this technology can be applied to any distribution scheme or file container.

[0314] Furthermore, this technology can be applied to any configuration. For example, it can be applied to various electronic devices.

[0315] Furthermore, this technology can also be implemented as part of a device, such as a processor as a system LSI (Large Scale Integration) (e.g., a video processor), a module using multiple processors (e.g., a video module), a unit using multiple modules (e.g., a video unit), or a set with additional functions added to a unit (e.g., a video set).

[0316] Furthermore, this technology can also be applied to network systems composed of multiple devices. For example, this technology may be implemented as cloud computing, where multiple devices share and collaborate on processing via a network. For example, this technology may be implemented in a cloud service that provides image (video) related services to any terminal such as computers, AV (Audio Visual) equipment, portable information processing terminals, and IoT (Internet of Things) devices.

[0317] In this specification, a system refers to a collection of multiple components (devices, modules (parts), etc.), regardless of whether all components are located in the same enclosure. Therefore, multiple devices housed in separate enclosures and connected via a network, and a single device containing multiple modules within a single enclosure, are both considered systems.

[0318] <Applicable Fields and Applications of This Technology> Systems, devices, and processing units incorporating this technology can be used in any field, such as transportation, medical care, security, agriculture, livestock farming, mining, beauty, factories, home appliances, weather, and nature monitoring. Furthermore, the applications are entirely arbitrary.

[0319] For example, this technology can be applied to systems and devices used to provide entertainment content. Furthermore, for example, this technology can be applied to systems and devices used for traffic management, such as traffic condition monitoring and automated driving control. In addition, for example, this technology can be applied to systems and devices used for security. Furthermore, for example, this technology can be applied to systems and devices used for automatic control of machinery, etc. Furthermore, for example, this technology can be applied to systems and devices used for agriculture and livestock farming. Furthermore, for example, this technology can be applied to systems and devices that monitor natural conditions such as volcanoes, forests, and oceans, as well as wildlife. Furthermore, for example, this technology can be applied to systems and devices used for sports.

[0320] <Other> In this specification, "flag" refers to information used to identify multiple states, and includes not only information used to identify two states, true (1) or false (0), but also information capable of identifying three or more states. Therefore, the values ​​that this "flag" can take are, for example, two values, 1 / 0, or three or more values. In other words, the number of bits that constitute this "flag" is arbitrary, and can be 1 bit or multiple bits. Furthermore, identification information (including flags) is envisioned not only in the form of including the identification information itself in the bitstream, but also in the form of including difference information of the identification information relative to a certain reference information in the bitstream. Therefore, in this specification, "flag" and "identification information" include not only the information itself, but also difference information relative to the reference information.

[0321] Furthermore, various types of information (metadata, etc.) related to encoded data (bitstream) may be transmitted or recorded in any form as long as they are associated with the encoded data. Here, the term "associate" means, for example, making it possible to use (link) one data when processing the other. In other words, associated data may be combined into a single data, or they may be individual data. For example, information associated with encoded data (image) may be transmitted on a different transmission path than the encoded data (image). Also, for example, information associated with encoded data (image) may be recorded on a different recording medium (or a different recording area on the same recording medium) than the encoded data (image). Note that this "association" may not apply to the entire data, but only to a part of it. For example, an image and the information corresponding to that image may be associated with each other in any unit, such as multiple frames, one frame, or a part within a frame.

[0322] In this specification, terms such as "combine," "multiplex," "add," "integrate," "include," "store," "insert," "insert," and "place" mean combining multiple things into one, such as combining encoded data and metadata into a single data, and represent one method of "associating" as described above.

[0323] Furthermore, the embodiments of this technology are not limited to those described above, and various modifications are possible without departing from the spirit of this technology.

[0324] For example, the configuration described as a single device (or processing unit) may be divided and configured as multiple devices (or processing units). Conversely, the configurations described above as multiple devices (or processing units) may be combined and configured as a single device (or processing unit). Furthermore, it is also possible to add configurations other than those described above to the configuration of each device (or each processing unit). In addition, if the overall system configuration and operation are substantially the same, a part of the configuration of one device (or processing unit) may be included in the configuration of another device (or other processing unit).

[0325] Furthermore, for example, the program described above may be executed on any device. In that case, the device should have the necessary functions (such as functional blocks) and be able to obtain the necessary information.

[0326] Furthermore, for example, each step of a flowchart may be executed by one device, or it may be divided among multiple devices. Additionally, if a single step includes multiple processes, these processes may be executed by one device, or they may be divided among multiple devices. In other words, multiple processes included in a single step can be executed as multiple steps. Conversely, processes described as multiple steps can be combined and executed as a single step.

[0327] Furthermore, for example, a program executed by a computer may be structured so that the steps of the program are executed chronologically in the order described herein, or they may be executed in parallel or individually at necessary times, such as when a call is made. In other words, the steps may be executed in an order different from the order described above, as long as no inconsistencies arise. Moreover, the steps of this program may be executed in parallel with the processing of other programs, or in combination with the processing of other programs.

[0328] Furthermore, for example, multiple technologies relating to this technology can be implemented independently, as long as they do not create a contradiction. Of course, any multiple technologies can also be implemented in combination. For example, some or all of the technologies described in one embodiment can be implemented in combination with some or all of the technologies described in another embodiment. Also, some or all of the above-mentioned technologies can be implemented in combination with other technologies not mentioned above.

[0329] Furthermore, this technology can also take the following configurations: (1) An information processing device comprising: a subdivision unit that subdivides a base mesh by recursively dividing faces based on a gradient score representing the complexity of the three-dimensional structure of an object; a displacement vector generation unit that generates a displacement vector based on an original mesh to be encoded, which is composed of vertices and connections representing the three-dimensional structure of the object, and the subdivided base mesh; and an encoding unit that encodes the base mesh and the displacement vector, wherein the base mesh is a mesh from which vertices have been thinned out from the original mesh, and the displacement vector is information indicating the difference in the position of vertices between the subdivided base mesh and the original mesh. (2) The information processing device according to (1), wherein the subdivision unit divides the faces based on the gradient score on a face-by-face basis of the mesh. (3) The information processing device according to (2), wherein the gradient score includes a value based on the dot product of the normal of the target face and the normal of an adjacent face adjacent to the target face. (4) The information processing device according to (3), wherein the gradient score includes the minimum value, maximum value, average value, or variance of the dot product. (5) The information processing device according to any one of (2) to (4), wherein the gradient score includes a value based on the dot product of the normals of the vertices of the target face. (6) The information processing device according to (5), wherein the gradient score includes the minimum, maximum, average, or variance of the dot product. (7) The information processing device according to any one of (1) to (6), wherein the subdivision unit divides the face based on the gradient score on an edge-by-edge basis of the mesh. (8) The information processing device according to (7), wherein the gradient score includes a value based on the dot product of the normals of the faces that share a target edge. (9) The information processing device according to (7) or (8), wherein the gradient score includes a value based on the dot product of the normals of the vertices at both ends of the target edge. (10) The information processing device according to any one of (1) to (9), wherein the subdivision unit derives the gradient score based on the base mesh and divides the face based on the gradient score in each iteration of the subdivision.(11) The subdivision unit derives the gradient score using the mesh to which the displacement vector is applied in each iteration of the subdivision, and divides the face based on the derived gradient score, as described in any of (1) to (10). (12) The subdivision unit rearranges the processing order of the faces based on the gradient score, and divides each face in the rearranged order until the upper limit of the number of faces is reached, as described in any of (1) to (11). (13) The encoding unit further encodes sorting control information that controls the rearrangement of the processing order, as described in (12). (14) The sorting control information includes a sorting flag indicating whether to perform the sorting of the processing order in the target data unit, as described in (13). (15) The sorting control information includes sorting application start designation information that specifies the iteration in which the application of the sorting of the processing order begins, as described in (13) or (14). (16) The information processing device according to any one of (12) to (15), wherein the displacement vector generation unit sorts and packs the generated displacement vectors based on the gradient score. (17) The information processing device according to (16), wherein the displacement vector generation unit packs the displacement vectors from the beginning up to a predetermined rank in the sorted order. (18) The information processing device according to any one of (1) to (17), wherein the subdivision unit determines whether to subdivide the target face based on the result of comparing the gradient score of the target face with a predetermined gradient threshold. (19) The information processing device according to (18), wherein the encoding unit further encodes the gradient threshold. (20) The information processing device according to (18) or (19), wherein the encoding unit further encodes threshold control information that controls the application of the gradient threshold. (21) The information processing device according to (20), wherein the threshold control information includes a threshold application flag indicating whether to apply the gradient threshold to the target data unit. (22) The information processing device according to (20) or (21), wherein the threshold control information includes threshold application start designation information that specifies the iteration for which the application of the gradient threshold is to be started.(23) The information processing device according to any one of (1) to (22), wherein the encoding unit further encodes the gradient score. (24) The information processing device according to (23), wherein the encoding unit encodes the gradient score of the base mesh as an attribute of each face of the base mesh. (25) The information processing device according to (23) or (24), wherein the encoding unit encodes the gradient score of the base mesh as metadata. (26) The information processing device according to any one of (23) to (25), wherein the encoding unit encodes the gradient score of the base mesh, in which the face has been divided one or more times, as metadata. (27) An information processing method comprising: subdividing a base mesh by recursively dividing faces based on a gradient score representing the complexity of the three-dimensional structure of an object; generating a displacement vector based on an original mesh to be encoded, which is composed of vertices and connections representing the three-dimensional structure of the object, and the subdivided base mesh; and encoding the base mesh and the displacement vector, wherein the base mesh is a mesh from which vertices have been thinned out from the original mesh, and the displacement vector is information indicating the difference in the positions of vertices between the subdivided base mesh and the original mesh. (28) A program for causing a computer to perform a process which includes: subdividing a base mesh by recursively dividing faces based on a gradient score representing the complexity of the three-dimensional structure of an object; generating a displacement vector based on an original mesh to be encoded, which is composed of vertices and connections representing the three-dimensional structure of the object, and the subdivided base mesh; and encoding the base mesh and the displacement vector, wherein the base mesh is a mesh from which vertices have been thinned out from the original mesh, and the displacement vector is information indicating the difference in the positions of vertices between the subdivided base mesh and the original mesh.

[0330] (41) An information processing device comprising: a decoding unit that decodes encoded data of a base mesh to generate the base mesh and decodes encoded data of a displacement vector to generate the displacement vector; and a subdivided displacement vector application unit that subdivides the generated base mesh by recursively dividing faces based on a gradient score representing the complexity of the three-dimensional structure of an object and applies the generated displacement vector, wherein the base mesh is a mesh obtained by thinning out vertices from an original mesh to be encoded, which is composed of vertices and connections representing the three-dimensional structure of the object, and the displacement vector is information indicating the difference in the position of vertices between the subdivided base mesh and the original mesh. (42) The information processing device according to (41), wherein the subdivided displacement vector application unit divides the faces based on the gradient score of the mesh on a face-by-face basis. (43) The information processing device according to (42), wherein the gradient score includes a value based on the dot product of the normal of the target face and the normal of an adjacent face adjacent to the target face. (44) The information processing device according to (43), wherein the gradient score includes the minimum, maximum, mean, or variance of the dot product. (45) The information processing device according to any one of (42) to (44), wherein the gradient score includes a value based on the dot product of the normals of the vertices of the target face. (46) The information processing device according to (45), wherein the gradient score includes the minimum, maximum, mean, or variance of the dot product. (47) The information processing device according to any one of (41) to (46), wherein the subdivision displacement vector application unit divides the face based on the gradient score on an edge-by-edge basis of the mesh. (48) The information processing device according to (47), wherein the gradient score includes a value based on the dot product of the normals of the faces that share the target edge. (49) The information processing device according to (47) or (48), wherein the gradient score includes a value based on the dot product of the normals of the vertices at both ends of the target edge.(50) The information processing device according to any one of (41) to (49), wherein the subdivided displacement vector application unit derives the gradient score based on the base mesh, divides the face based on the gradient score in each iteration of the subdivided process, and applies the displacement vector after the completion of the subdivided process. (51) The information processing device according to any one of (41) to (50), wherein the subdivided displacement vector application unit derives the gradient score using the mesh to which the displacement vector has been applied in each iteration of the subdivided process, divides the face based on the derived gradient score, and applies the displacement vector. (52) The information processing device according to any one of (41) to (51), wherein the subdivided displacement vector application unit rearranges the processing order of the faces based on the gradient score and divides each face in the rearranged order until the upper limit of the number of faces is reached. (53) The information processing device according to (52), wherein the decoding unit further decodes encoded data of sorting control information that controls the sorting of the processing order and generates the sorting control information. (54) The sorting control information includes a sorting flag indicating whether to sort the processing order at the target data unit level, as described in (53). (55) The sorting control information includes sorting application start designation information specifying the iteration at which to start applying the sorting of the processing order, as described in (53) or (54). (56) The decoding unit decodes the encoded data of the displacement vectors sorted and packed based on the gradient score and generates the displacement vectors in the sorted order, as described in any of (52) to (55). (57) The decoding unit decodes the encoded data of the displacement vectors packed from the beginning up to a predetermined rank in the sorted order, as described in (56). (58) The subdivided displacement vector application unit determines whether to subdivide the target face based on the comparison result between the gradient score of the target face and a predetermined gradient threshold, as described in any of (41) to (57).(59) The decoding unit further decodes the encoded data of the gradient threshold to generate the gradient threshold, as described in (58). (60) The decoding unit further decodes the encoded data of threshold control information that controls the application of the gradient threshold to generate the threshold control information, as described in (58) or (59). (61) The threshold control information includes a threshold application flag indicating whether to apply the gradient threshold to the target data unit, as described in (60). (62) The threshold control information includes threshold application start designation information that specifies the iteration for which to start applying the gradient threshold, as described in (60) or (61). (63) The decoding unit further decodes the encoded data of the gradient score to generate the gradient score, and the subdivision displacement vector application unit subdivides the base mesh based on the generated gradient score, as described in any of (41) to (62). (64) The information processing device according to (63), wherein the decoding unit decodes encoded data of the gradient score of the base mesh, which has been encoded as an attribute of each face of the base mesh. (65) The information processing device according to (63) or (64), wherein the decoding unit decodes encoded data of the gradient score of the base mesh, which has been encoded as metadata. (66) The information processing device according to any one of (63) to (65), wherein the decoding unit decodes encoded data of the gradient score of the base mesh, which has been divided one or more times, which has been encoded as metadata.(67) An information processing method comprising: decoding encoded data of a base mesh to generate the base mesh; decoding encoded data of a displacement vector to generate the displacement vector; recursively dividing faces based on a gradient score representing the complexity of the three-dimensional structure of an object to subdivide the generated base mesh and applying the generated displacement vector, wherein the base mesh is a mesh obtained by thinning out vertices from an original mesh to be encoded, which is composed of vertices and connections representing the three-dimensional structure of the object, and the displacement vector is information indicating the difference in the positions of vertices between the subdivided base mesh and the original mesh. (68) A program for causing a computer to perform a process which includes: decoding encoded data of a base mesh to generate the base mesh; decoding encoded data of a displacement vector to generate the displacement vector; recursively dividing faces based on a gradient score representing the complexity of the three-dimensional structure of an object to subdivide the generated base mesh and applying the generated displacement vector, wherein the base mesh is a mesh obtained by thinning out vertices from an original mesh to be encoded, which is composed of vertices and connections representing the three-dimensional structure of the object, and the displacement vector is information indicating the difference in the positions of vertices between the subdivided base mesh and the original mesh.

[0331] 300 Encoding device, 311 Preprocessing unit, 312 V-DMC encoding unit, 321 Base mesh generation unit, 322 Subdivision unit, 323 Displacement vector generation unit, 324 Atlas information generation unit, 351 Atlas information encoding unit, 352 Base mesh encoding unit, 353 Displacement vector correction unit, 354 Displacement vector encoding unit, 355 Mesh reconstruction unit, 356 Attribute map conversion unit, 357 Attribute encoding unit, 358 Multiplexing unit, 500 Decoding device, 511 Demultiplexing unit, 512 Atlas information decoding unit, 513 Base mesh decoding unit, 514 Displacement vector decoding unit, 515 Attribute decoding unit, 516 Subdivision unit, 517 Displacement vector application unit, 518 Attribute application unit, 519 Display processing unit, 521 Decoding unit, 522 Subdivision displacement vector application unit, 900 Computer

Claims

1. An information processing device comprising: a decoding unit that decodes encoded data of a base mesh to generate the base mesh and decodes encoded data of a displacement vector to generate the displacement vector; and a subdivided displacement vector application unit that subdivides the generated base mesh by recursively dividing faces based on a gradient score representing the complexity of the three-dimensional structure of an object and applies the generated displacement vector, wherein the base mesh is a mesh obtained by thinning out vertices from an original mesh to be encoded, which is composed of vertices and connections representing the three-dimensional structure of the object, and the displacement vector is information indicating the difference in the position of vertices between the subdivided base mesh and the original mesh.

2. The information processing apparatus according to claim 1, wherein the subdivided displacement vector application unit divides the faces based on the gradient score for each face of the mesh.

3. The information processing device according to claim 2, wherein the gradient score includes a value based on the dot product of the normal of the target face and the normal of an adjacent face adjacent to the target face.

4. The information processing apparatus according to claim 3, wherein the gradient score includes the minimum, maximum, mean, or variance of the dot product.

5. The information processing device according to claim 2, wherein the gradient score includes a value based on the dot product of the normals of the vertices of the target face.

6. The information processing apparatus according to claim 5, wherein the gradient score includes the minimum, maximum, mean, or variance of the inner product.

7. The information processing apparatus according to claim 1, wherein the subdivided displacement vector application unit divides the face based on the gradient score of the mesh edge unit.

8. The information processing device according to claim 7, wherein the gradient score includes a value based on the dot product of the normals of the faces that share the target edge.

9. The information processing device according to claim 7, wherein the gradient score includes a value based on the dot product of the normals of the vertices at both ends of the target edge.

10. The information processing apparatus according to claim 1, wherein the subdivided displacement vector application unit derives the gradient score based on the base mesh, divides the face based on the gradient score in each iteration of the subdivided portion, and applies the displacement vector after the completion of the subdivided portion.

11. The information processing apparatus according to claim 1, wherein the subdivided displacement vector application unit derives the gradient score using the mesh to which the displacement vector has been applied in each iteration of the subdivided portion, divides the face based on the derived gradient score, and applies the displacement vector.

12. The information processing apparatus according to claim 1, wherein the subdivided displacement vector application unit rearranges the processing order of the faces based on the gradient score and divides each face in the rearranged order until the upper limit of the number of faces is reached.

13. The information processing apparatus according to claim 12, wherein the decoding unit further decodes encoded data of sorting control information that controls the sorting of the processing order, and generates the sorting control information.

14. The information processing apparatus according to claim 12, wherein the decoding unit decodes the encoded data of the displacement vectors that have been sorted and packed based on the gradient score, and generates the displacement vectors in the sorted order.

15. The information processing apparatus according to claim 1, wherein the subdivided displacement vector application unit determines whether to subdivide the target face based on the result of comparing the gradient score of the target face with a predetermined gradient threshold.

16. The information processing apparatus according to claim 15, wherein the decoding unit further decodes the encoded data of the gradient threshold and generates the gradient threshold.

17. The information processing apparatus according to claim 15, wherein the decoding unit further decodes encoded data of threshold control information that controls the application of the gradient threshold, and generates the threshold control information.

18. An information processing method comprising: decoding encoded data of a base mesh to generate the base mesh; decoding encoded data of a displacement vector to generate the displacement vector; and recursively dividing faces based on a gradient score representing the complexity of the three-dimensional structure of an object to subdivide the generated base mesh and applying the generated displacement vector, wherein the base mesh is a mesh obtained by thinning out vertices from an original mesh to be encoded, which is composed of vertices and connections representing the three-dimensional structure of the object, and the displacement vector is information indicating the difference in vertex positions between the subdivided base mesh and the original mesh.

19. An information processing device comprising: a subdivision unit that subdivides a base mesh by recursively dividing faces based on a gradient score representing the complexity of the three-dimensional structure of an object; a displacement vector generation unit that generates a displacement vector based on an original mesh to be encoded, which is composed of vertices and connections representing the three-dimensional structure of the object, and the subdivided base mesh; and an encoding unit that encodes the base mesh and the displacement vector, wherein the base mesh is a mesh from which vertices have been thinned out from the original mesh, and the displacement vector is information indicating the difference in the positions of vertices between the subdivided base mesh and the original mesh.

20. An information processing method comprising: subdividing a base mesh by recursively dividing faces based on a gradient score representing the complexity of the three-dimensional structure of an object; generating a displacement vector based on an original mesh to be encoded, which is composed of vertices and connections representing the three-dimensional structure of the object, and the subdivided base mesh; and encoding the base mesh and the displacement vector, wherein the base mesh is a mesh from which vertices have been thinned out from the original mesh, and the displacement vector is information indicating the difference in vertex positions between the subdivided base mesh and the original mesh.

Images