Electronic apparatus and controlling method thereof

Abstract

An electronic apparatus for training an AI (Artificial Intelligence) model to generate an image of an object based on an input pose comprises at least one processor including processing circuitry and memory storing instructions. The instructions, when executed by the at least one processor individually or collectively, cause the electronic apparatus to obtain a plurality of images for training the AI model, obtain training images including an object from the plurality of images, obtain at least one parameter corresponding to each of the training images, obtain 3D geometry information of the object based on the at least one parameter, obtain a spherical neural texture mapped with the 3D geometry information, obtain warping prediction information for the spherical neural texture, update the spherical neural texture based on the warping prediction information, and train the AI model based on the updated spherical neural texture.

Description

[0001] [DESCRIPTION]

[0002] [Invention Title]

[0003] ELECTRONIC APPARATUS AND CONTROLLING METHOD THEREOF [Technical Field]

[0004] The present invention relates to the field of image processing, namely to obtaining an image of a subject in any pose and / or viewing angle, including those not present in the training frames.

[0005] [Background Art]

[0006] Currently, to create an image of a subject, a model of artificial intelligence called a neural network rendering system is trained.

[0007] A known method of neural network rendering from the prior art includes the following main stages:

[0008] 1) neural texture rendering, that is, overlaying the neural texture onto the 3D geometry of the subject, [2]),

[0009] 2) applying a rendering neural network to obtain the image.

[0010] For neural texture rendering for each frame depicting the subject, the posed 3D geometry of the subject and the common neural texture for all frames are used, and based on them, one rendered neural texture (hereinafter referred to as the rendered neural texture) is obtained using rendering for this frame.

[0011] At the stage of system training, neural texture rendering is performed for each posed 3D geometry of the subject extracted from the frame. For this, the posed 3D geometry of the subject, representing data about the shape and pose of the subject in the form of a polygonal mesh that repeats the shape and pose of the subject, is extracted from each training image of the subject from all frames of the video or from selected frames.

[0012] The rendered neural texture obtained for one frame is fed to the trainable rendering neural network, and an image of the subject is obtained. At each step of training the rendering neural network, the obtained image is compared with the frame for which the 3D geometry of the subject and the mentioned rendered neural texture were previously obtained.

[0013] Based on the comparison of the image obtained at the output of the trainable rendering neural network and the image of the frame for which the 3D geometry of the subject was extracted, the value of the loss function between the obtained image and the mentioned frame is determined. Then, based on the value of the loss function, the error is calculated and the weights of the trainable rendering neural network are updated using the backpropagation method.

[0014] During the operation of the system, when training is completed, neural texture rendering and subsequent obtaining of the final image are performed for the 3D geometry of the subject obtained for new poses and / or viewing angles in which it is necessary to display this subject. New poses are understood as arbitrary poses, not necessarily corresponding to the 3D geometries extracted from the training frames during training.

[0015] Known approaches in the existing state of the art use one common neural texture for all frames, storing data about the appearance of the subject for points on the surface of its 3D geometry regardless of the pose. However, known methods for extracting the 3D geometry of a subject from a frame can produce unstable results (that is, for example, for smoothly changing poses of a subject in input frames, sharply changing 3D geometries of the subject can be issued). In addition, existing neural network rendering approaches for constructing an image of a subject, including full-body, do not model the offset, for example, of clothing from frame to frame relative to the subject's body as it moves.

[0016] In known approaches, the peculiarities of forming input data for the neural network can lead to inconsistencies in training pairs consisting of an image in the frame and the 3D geometry of the subject. Such a problem can arise if the training of the neural texture parameters occurs with inconsistent training pairs, in this case, when training, information about the same fragment of the subject's appearance can be written to different places in the texture. Therefore, images generated using such a system will contain corresponding artifacts of blurring and / or repetition of image fragments, for example, due to the fact that different content fragments of the image from the training frame may correspond to the same part of the subject's 3D geometry.

[0017] The described problem is illustrated by the following example. Fig. 1 shows two different training frames (a) and (b), in which the subject assumes two different poses. The 3D geometries of the subject corresponding to these frames are divided into numbered quadrants, outlined and superimposed on the images for clarity. It can be seen that the image of the print (inscription) on the T-shirt in frame (b) is shifted to quadrants F5 and F6, while in frame (a) the image of the print on the T-shirt is absent in quadrants F5 and F6. That is, the images of part of the clothing in these two frames will be shifted relative to the same parts (F5 and A6) of the two 3D geometries of the subject extracted from these two frames. Both of these frames and the 3D geometries of the subject extracted from them are fed as training pairs during the training of the rendering neural network, therefore, due to the above-mentioned shift, the trainable neural network will receive contradictory and inconsistent training data, which will lead to implausible output images after training the neural network.

[0018] From document RU 2775825 Cl (publication date 11.07.2022) neural network rendering of three-dimensional human avatars is known. The known solution uses a deformable polygonal mesh that describes the geometry of the subject's body (3D geometry of the subject). Such 3D geometry of the subject can be deformed so as to repeat the pose that the subject assumes in the input image, that is, posed. Further, in the known solution, neural texture is applied to the mentioned posed 3D geometry of the subject (texture rendering), and the result of this operation is processed by the rendering neural network to form the final image of the subject. During training, the final image of the subject obtained is compared with the corresponding training frame extracted from the input video. The parameters of the neural texture and the rendering neural network are updated in accordance with the obtained difference based on the backpropagation method. During testing (operation) of the system, it is possible to perform neural texture rendering for the 3D geometry of the subject in an arbitrary pose and obtain the final image of this subject in this pose using the trained rendering neural network. The obtained image will transmit the appearance of the subject, on whose images the rendering neural network was trained. The disadvantage of the known solution is the ignoring of the inconsistency of the training data, which leads to unrealistic images containing artifacts.

[0019] From document US 11651540 B2 (publication date 16.05.2023), it is known to use neural texture to construct a full-body 3D image of a subject. According to the known solution, training frames are selected using a predetermined procedure that selects a small number of frames that show the subject from different viewing angles. The disadvantage of this method is the need to manually select the hyperparameters of this procedure, such as the number (percentage of the total number) of training frames used to update the neural texture parameters. With suboptimal hyperparameters, this method may be ineffective.

[0020] Also, in all solutions known from the prior art, the neural texture is “flat”, when rendering the neural texture, a 2D neural texture coordinate is applied to the 3D coordinates of the 3D mesh, which is similar to cutting out clothes from a flat canvas onto the volumetric body of the subject. In this regard, when applying (rendering) a flat neural texture to a volumetric 3D geometry of the subject, “structural seams” are obtained, and for a point lying on a “structural seam” in the 3D geometry of the subject, there are at least two corresponding points in the space of the 2D neural texture, which when applying warping of the neural texture can be inconsistently deformed, which can lead to breaks in the deformed neural texture in the area of the “structural seam” when applied to the 3D geometry of the subject. This, in turn, can lead to unrealistic effects in the final image.

[0021] Systems are also known from the prior art that include warping neural networks, however, such systems do not use neural texture applied to the 3D geometry of the subject, as proposed in the present invention. For example, instead of neural texture applied to the 3D geometry of the subject in [7, 13,14,15,16,17], a so-called implicit representation is used. Such known systems include predicting the warping of the 3D geometry of the subject itself, rather than the neural texture, as described in the present invention.

[0022] Thus, to eliminate the disadvantages of the prior art, a neural texture is needed that has frame-by-frame warping, and in addition, which can be mapped to the 3D geometry of the subject without "structural seams".

[0023] [Disclosure]

[0024] [Technical Solution]

[0025] In an embodiment, an electronic apparatus for training an AI(Artificial Intelligence) model to generate an image of an object based on an input pose comprises: at least one processor including processing circuitry, and memory storing instructions. The instructions, when executed by the at least one processor individually or collectively, may cause the electronic apparatus to obtain a plurality of images for training the Al model, obtain training images including an object from the plurality of images, obtain at least one parameter corresponding to each of the training images, obtain 3D geometry information of the object based on the at least one parameter, obtain a spherical neural texture mapped with the 3D geometry information, obtain warping prediction information for the spherical neural texture, update the spherical neural texture based on the warping prediction information, and train the Al model based on the updated spherical neural texture.

[0026] The instructions, when executed by the at least one processor individually or collectively, may cause the electronic apparatus to obtain first points indicating the obj ect based on the 3D geometry information, obtain second points on a surface of a canonical sphere, and obtain the spherical neural texture by mapping the points indicating the object to the second points.

[0027] The instructions, when executed by the at least one processor individually or collectively, may cause the electronic apparatus to identify the object capable of changing a pose from a plurality of objects included in the sequence of images.

[0028] The training images may be a sequence of images.

[0029] The parameters may be trainable parameters.

[0030] The warping prediction information may include information for deforming the spherical neural texture.

[0031] The instructions, when executed by the at least one processor individually or collectively, may cause the electronic apparatus to obtain an output image based on the updated spherical neural texture, obtain a loss value by comparing the output image with the training images, and train the Al model based on the loss value.

[0032] The instructions, when executed by the at least one processor individually or collectively, may cause the electronic apparatus to obtain the loss value by comparing the output image with the training images based on a loss function for the Al model, obtain error information for backpropagation to the Al model, and update weight information of the Al model based on the error information.

[0033] The weight information may correspond to the at least one parameter used to obtain the 3D geometry information of the object. The instructions, when executed by the at least one processor individually or collectively, may cause the electronic apparatus to obtain optimizable vector corresponding to each of the training images based on the at least one parameter, and obtain the 3D geometry information of the object based on the optimizable vector.

[0034] The instructions, when executed by the at least one processor individually or collectively, may cause the electronic apparatus to obtain a canonical sphere, obtain correspondence information between points on a surface of the canonical sphere and the 3D geometry information of the object, and obtain the spherical neural texture mapped with the 3D geometry information based on the correspondence information.

[0035] The instructions, when executed by the at least one processor individually or collectively, may cause the electronic apparatus to obtain a 3D mesh of the object based on the correspondence information, and obtain the spherical neural texture based on the 3D mesh.

[0036] The 3D mesh may include adjacent triangles sharing a common edge and two common vertices, and the two common vertices may correspond to points on the surface of the canonical sphere.

[0037] The instructions, when executed by the at least one processor individually or collectively, may cause the electronic apparatus to obtain the warping prediction information for the spherical neural texture by predicting coordinates of points on the 3D mesh.

[0038] In an embodiment, A controlling method of an electronic apparatus, the controlling method comprising obtaining a plurality of images for training the Al model, obtaining training images including an object from the plurality of images, obtaining at least one parameter corresponding to each of the training images, obtaining 3D geometry information of the object based on the at least one parameter, obtaining a spherical neural texture mapped with the 3D geometry information, obtaining warping prediction information for the spherical neural texture, updating the spherical neural texture based on the warping prediction information, and training the Al model based on the updated spherical neural texture.

[0039] [Description of Drawings]

[0040] The above-described and other features and advantages of the present invention are explained in the following description, illustrated by the drawings, which show the following:

[0041] FIG. 1 illustrates (a), (b) - two training frames in which the subject assumes two different poses, with marked parts of the corresponding 3D geometries.

[0042] FIG. 2 illustrates an example of the correspondence of (a) the area of the 3D geometry of the subject and (b) the area of the 3D mesh of the spherical neural texture. FIG. 3 illustrates: (a) a fragment of the surface of the 3D mesh of the subject, (b) the corresponding fragment of the surface of the 3D mesh of the spherical neural texture.

[0043] FIG. 4 schematically illustrates a system for creating an image of a subject in a desired pose according to the proposed invention and a method of its operation.

[0044] FIG. 5 schematically illustrates a method for training a system for creating an image of a subject in a desired pose according to the proposed invention.

[0045] FIG. 6 schematically illustrates the operation of the warping prediction module.

[0046] FIG. 7 schematically illustrates an electronic apparatus as an embodiment.

[0047] FIG. 8 schematically illustrates controlling method of an electronic apparatus as an embodiment.

[0048] [Mode for Invention]

[0049] A trainable system is proposed for creating an image of a subject in any desired pose and / or viewing angle based on frames extracted from an input video, or based on a photo.

[0050] Due to the proposed invention, the system for creating an image of a subject can generate an image of a given subject in desired poses and / or viewing angles, which may or may not be present in the frames of the input video or photo.

[0051] Also, thanks to the proposed invention, the negative effect of inconsistency of training data is reduced due to the warping of the neural texture used before applying the neural texture to the 3D geometry of the subject (before rendering), which will be described in detail below. Such warping is proposed to be calculated separately for each training frame so that different warpings can be obtained for different frames, and accordingly, the neural texture can be deformed differently in each training example. Due to the proposed warping, the mentioned neural texture will be shifted compared to its standard position. Thus, if there is an offset of similar image fragments relative to the same parts of the 3D geometry of the subject in the training examples, then the proposed warping of the neural texture shifts the neural texture so that similar parts of the neural texture are again on the same parts of the geometry for all training examples, thereby reducing the inconsistency of the training data. That is, the proposed warping models possible neural texture frame to frame shifts , which helps to reduce the negative effect of inconsistency of training pairs consisting of an image in the training frame and the 3D geometry of the subject, due to which the output image of the subject obtained by the trained system for creating the image of the subject looks sharp and natural, even when generating an image of this subject in a pose different from the pose of the subject in the original image.

[0052] Also, thanks to the proposed invention, the resulting image of the subject does not have artifacts, such as blurring and / or mismatch with the original image, which is achieved due to the representation of the neural texture, which is mapped to the 3D geometry of the image of the subject, without "structural seams", as will be described below.

[0053] The following terms and definitions are used in this description.

[0054] An image, which is understood as an RGB image or a four-channel image, where the first three channels correspond to the RGB channels, and the fourth represents the foreground mask. The proposed system, after training, generates an image of the subject in any desired pose and / or viewing angle from the 3D geometry of the subject corresponding to the desired pose and / or viewing angle.

[0055] The 3D geometry of the subject is a 3D polygonal mesh that describes the geometry of the subject's body, wherein the parameters of the 3D geometry of the subject are the parameters of the parametric model of the subject's body, (for example, such as SMPL-X [2]), namely the parameters of the 3D geometry of the subject, which are a set of numbers that set, for example, the shape of the body and the pose of the subject. The parameters of the 3D geometry of the subject are also the location (coordinates) of the vertices of the 3D geometry of the subject. As is known, the faces of the 3D geometry of the subject are usually triangles, wherein the 3D geometry of the subject for any shape of any subject is formed in such a way that the number of vertices and their connection by edges for all 3D geometries of the subject extracted from images of any subject is the same, that is, all such 3D geometries of the subject have the same standard structure [2], but correspond to different poses of the subject, that is, each 3D geometry of the subject contains data about the pose of the subject in the frame.

[0056] In this description, neural texture is defined as a function whose value can be calculated at any point on the surface of a sphere. Moreover, for each new subject, its own neural texture is determined and trained. By the value of the neural texture, we mean a vector of dimension L, which is an ordered set of L numbers (L-channel neural texture), which is data on the appearance of a given subject at each point of the image of this subject. The neural texture [1] is defined in the texture space, that is, the value of the neural texture can be obtained for points of the texture space. The domain of definition of the neural texture is a set of points of the texture space, for each of which the value of the neural texture can be obtained. For all frames with an image of the same subject, the same neural texture is used, and for images of different people, the neural textures will differ.

[0057] Neural texture formation is a process known from the prior art, for example, [1], the aforementioned documents RU 2775825 Cl, US 11651540 B2], etc. To obtain neural texture, it is necessary to:

[0058] 1) Specify a way to obtain the value of the neural texture at each point of the domain of definition of the neural texture in the texture space,

[0059] 2) Specify the texture correspondence, namely, the mapping of points of the 3D geometry of the subject to points of the texture space.

[0060] In the prior art, a "flat" neural texture is used, defined as an array of parameters and a standard existing texture correspondence of points on the surface of the 3D geometry of the subject and points on the plane [2], In the proposed invention, instead of a "flat" neural texture, it is proposed to use a spherical neural texture, which is defined by:

[0061] 1) A function for calculating the values of the neural texture at any point of the canonical sphere, including using trainable parameters, that is, the function for calculating the values of the neural texture can be parameterized by trainable numerical parameters, for example, according to the proposed invention, the calculation function can be specified by a neural network. The term "canonical sphere" refers to a sphere on whose surface the value of the proposed spherical neural texture can be calculated.

[0062] 2) Texture correspondence, which maps the points of the 3D geometry of the subject to points on the surface of the canonical sphere using the 3D mesh of the spherical neural texture created within the framework of this invention.

[0063] At the development stage of the proposed system for creating an image of a subject, only the function for calculating the neural texture is determined, its general form (in the proposed invention, it can be specified by a neural network), the set of parameters and how trainable parameters are used. The unlearned spherical neural texture modeled by the developer can be stored in an external source, for example, in the memory of an electronic apparatus, in a network, etc., from where it is received during the training and operation of the system by the pre-processing module.

[0064] The same untrained spherical neural texture is trained for the image of a new subject and used after training to construct the image of this subject in different poses. During training of the proposed system, the parameters of the neural texture are trained, including for each specific subject, which determine the specific values of the neural texture at its points. That is, the proposed system for creating an image of a subject in a desired pose must be retrained before starting its work for each new subject. During training of the entire system, the neural texture is trained to carry information, for example, about the color of hair, clothing, and other parameters of the subject at points on the surface of the 3D geometry of the subject. If the entire system is not yet trained, then the parameters of the neural texture are not configured, and then the texture values calculated at each point will not carry information about the appearance of the subject. As the training progresses, the parameters of the neural texture are updated so that the obtained neural texture values carry information about the appearance of a specific subject and, accordingly, so that the resulting images of the subject approach the images of the subject in the training frames without the occurrence of artifacts inherent in flat textures, as explained above.

[0065] To use a sphere as the domain of the neural texture, it is necessary to define a texture correspondence, namely, the correspondence of points in the structure of the 3D geometry of the subject and points of the canonical sphere of the neural texture. That is, for each point of the 3D geometry of the subject, the position of the corresponding point on the surface of the canonical sphere is set. To do this, establish the corresponding mapping (correspondence) between the three-dimensional space of the 3D geometry of the subject and the space of the spherical neural texture. To do this, a 3D mesh of the spherical neural texture is introduced, the vertices of which are located on the surface of the canonical sphere, representing triangles, each pair of which has one common edge and two common vertices, while the number of vertices and faces corresponds to the number of vertices and faces of the 3D geometry of the subject. Each of the vertices of the 3D mesh of the spherical neural texture lies on the surface of the canonical sphere of the neural texture and uniquely corresponds to one vertex of the 3D geometry of the subject, while the vertices of the 3D mesh of the spherical neural texture are connected in the same way as the vertices of the 3D geometry of the subject. Such a correspondence is possible because, as mentioned above, the 3D geometry of the subject for any shape of any subject has the same standard structure.

[0066] FIG. 2 illustrates an example of the correspondence of (a) the area of the 3D geometry of the subject and (b) the area of the 3D mesh of the spherical neural texture. Thus, each single point on the 3D mesh of the spherical neural texture uniquely corresponds to one point on the 3D geometry of the subject. Such a correspondence for all points is set using barycentric coordinates (in the case of triangles) relative to the vertices of the corresponding triangles.

[0067] The calculations of the described correspondence are as follows. FIG. 3 shows: (a) a fragment of the surface of the 3D geometry of the subject, (b) the corresponding fragment of the surface of the 3D mesh of the spherical neural texture. h^- coordinates of three vertices of the 3D geometry of the subject, connected by edges, and thus forming a triangle of the 3D geometry of the subject, i is an index from 1 to 3, curly braces mean a set of vertices. sf - coordinates of three corresponding vertices of the 3D mesh of the spherical neural texture, connected by edges, and thus forming a triangle on the surface of the neural texture, i is an index from 1 to 3, three vertices, curly braces mean a set of vertices.

[0068] Each of the vertices of the 3D geometry of the subject uniquely corresponds to the vertices of the 3D mesh of the spherical neural texture sf .

[0069] For any arbitrary point q on the surface of the 3D geometry of the subject, it is possible to determine the corresponding point p on the spherical neural texture, that is, on the surface of the 3D mesh of the spherical neural texture, using barycentric interpolation. To do this:

[0070] 1. Find the location (coordinates) of the vertices of the face on the 3D geometry of the subject, inside which point q is located: hf .

[0071] 2. Find the coordinates of point q relative to the vertices of the triangle of the surface of the 3D mesh of the subject, in which this point is located, they are denoted as and satisfy the following condition: q = wLhf, wL= 1

[0072] - coordinates of point q relative to the coordinates of the vertices of the 3D geometry of the subject (barycentric coordinates), - coordinates of the vertices (in the Cartesian coordinate system),

[0073] Wt are found, using known q and , using standard mathematical formulas for finding barycentric coordinates.

[0074] 3. Find the point p on the surface of the 3D mesh of the sphere p = sf

[0075] Based on this correspondence, for point q on the surface of the 3D geometry of the subject, find the corresponding point p on the surface of the 3D mesh of the spherical neural texture, and in it, calculate the value of the neural texture for this point of the 3D mesh of the spherical neural texture.

[0076] Also, to find the corresponding point on the surface of the canonical sphere, point p on the surface of the 3D mesh of the spherical neural texture can be projected using standard operations to a point on the surface of the canonical sphere. Calculating the value of each point of the spherical neural texture may consist in calculating the value of the function: t = T(p; 0) at point p, where t is the value of the neural texture at point p, p - coordinates of point p on the surface of the 3D mesh of the spherical neural texture, 0 - a set of trainable parameters of the neural texture, that is, a set of numerical parameters that are trained along with the rest of the system parameters using the backpropagation method.

[0077] As function T, one can use T(p)=NN(p, Enc(p)), where NN is a deep neural network, including a deep neural network, the input data of which are coordinates p, and / or the result of their encoding Enc(p) (for example [3,4,5]), including positional encoding [3], [4], hash encoding [5], or their combinations (for example, concatenations).

[0078] The optimizable vector of the training frame is a numerical vector that corresponds to a specific training frame and is a trainable parameter of the image of the subject and one of the inputs (input data) for the neural networks that make up the system. The optimizable vector of the training frame encodes information about the warping that needs to be fed to the warping neural network to obtain the warping of the spherical neural texture for this training frame. The optimizable vectors of the training frames are trainable parameters of the system and are trained using the backpropagation method along with the weights of the neural networks included in the system. The trained system for creating an image of a subject in a desired pose can operate by receiving only the 3D geometry corresponding to the pose in which it is required to display the subject as input, and not receiving new frames as input on which it was not trained. Therefore, when operating an already trained system, an optimized training frame vector is used, obtained as a result of training the entire system and representing a weighted sum of the optimized training frame vectors.

[0079] The deformed spherical neural texture is a spherical neural texture with a texture correspondence that has been changed (deformed) relative to the initially set one. neural texture rendering is performed by overlaying the neural texture onto the 3D geometry of the subject, which includes determining the points in the 3D geometry of the subject visible in this frame, determining the points in the texture space corresponding to them (using texture correspondence), and forming a rendered neural texture.

[0080] FIG. 4 illustrates the proposed system for creating an image of a subject in a desired pose, as well as the stages of operation of the proposed system according to the proposed invention.

[0081] The proposed system for creating an image of a subject includes: pre-processing module 1, which forms the 3D geometry 2 of the subject, warping prediction module 3, designed to store the trainable parameters of the image of the subject, and including a warping neural network, warping application module 4; neural texture rendering module 5, in which the rendered neural texture 5a is formed, rendering neural network 6, optimization module 7, including a loss calculation module and an optimizer.

[0082] FIG. 5 schematically illustrates the stages of training the proposed system for creating an image of a subject in a desired pose. Training the system for creating an image of a subject in a desired pose must be carried out each time for each new subject in the image / video.

[0083] System training is described with reference to FIG. 5 and is carried out as follows.

[0084] To obtain a training video of the subject, the subject turns around in front of the camera. A sequence of training frames of the training video, or a sequence of training frames, is an image of the subject in different poses and / or viewing angles and can be prepared in advance and stored in the memory of the electronic apparatus on which the proposed invention is used, or can be arbitrarily and independently selected by the user. The spherical texture, formed by the developer when creating the proposed system, can also be stored in the memory of the electronic apparatus.

[0085] A) In pre-processing module 1 :

[0086] A set of training frames with an image of the subject selected by the user in different poses is randomly selected from the sequence of training frames, which will be used for training, including for checking the training stop criterion.

[0087] Using methods known from the prior art, trainable image parameters are determined for each training frame, such as, for example, optimizable vectors of training frames, etc. Trainable image parameters are transmitted to the warping prediction module, where they are trained, and then, when operating, already trained image parameters are used.

[0088] Optionally, using methods known from the prior art (for example,

[0018] ), image segmentation and foreground mask calculation are performed for the image in each frame. Segmentation involves using ready-made segmentation neural network models trained to segment a specific type of subject, such as a person. As known from the prior art, the foreground mask stores information about which pixels of the image contain the subject.

[0089] Using methods known from the prior art (for example, [2]), the 3D geometry of the subject (denoted by position 2 in FIG. 4) is obtained for the image in each selected training frame, wherein the parameters of the 3D geometry of the subject contain information about the pose and shape of the subject in the frame, as well as the coordinates of the vertices of the 3D geometry of the subject. The number of vertices and faces of each obtained 3D geometry of the subject is the same for all selected frames, while the vertices of the 3D mesh of the spherical neural texture are connected in the same way as the vertices of the 3D geometry of the subject.

[0090] In addition, the spherical neural texture formed by the system developer is extracted from the memory of the electronic apparatus. As described above, the spherical neural texture is defined by known methods of obtaining the value of the neural texture at any point on the canonical sphere and texture correspondence that maps points of the 3D geometry of the subject to points on the surface of the canonical sphere using the 3D mesh of the spherical neural texture, which establishes a correspondence between points on the surface of the 3D geometry of the subject and points on the surface of the spherical neural texture. The parameters of the spherical neural texture are updated during training, and after training, the trained spherical neural texture is stored in the pre-processing module.

[0091] B) In the known prior art, after obtaining the 3D geometry of the subject and determining the corresponding neural texture, they are rendered, that is, the spherical neural texture is overlaid on the 3D geometry of the subject. In the present invention, it is proposed to first deform the spherical neural texture. Namely, for each frame from the selected training frames, deform the 3D mesh of the spherical neural texture, specified in step (A), to obtain its correspondence to the selected training frame. The operation is performed for all selected training frames. Such warping of the 3D mesh of the spherical neural texture is necessary to compensate for the inconsistency of the training pairs consisting of the 3D geometries of the subject obtained in step (A) and the corresponding training frames.

[0092] Warping here means changing the texture correspondence, that is, the way the spherical neural texture is overlaid onto the 3D geometry of the subject.

[0093] The warping operation is performed by the warping module, consisting of warping prediction module 3 and warping application module 4. Warping prediction module 3 includes warping neural network 3a, as well as trainable image parameters of the subject (transferred from the pre-processing module), for example, such as a set of optimizable vectors of training frames, in which each training frame corresponds to one optimizable vector from the set. The trainable parameters of the image of the subject are updated in accordance with the training data during the training of the entire system. Upon completion of training, the trained image parameters of the subject are stored in the warping prediction module, for example, optimized vectors of training frames.

[0094] Warping prediction module 3, including warping neural network 3a and obtained trainable image parameters, is trained to predict the change in coordinates (warping) of the spherical neural texture, namely the texture correspondence of the spherical neural texture, depending on which training frame is fed as input. The warping neural network during the training of the entire system is trained to predict the change in texture correspondence, which is determined by the coordinates of the vertices of the 3D mesh of the spherical neural texture.

[0095] The operation of warping prediction module 3 is shown in detail in FIG. 6 and described below.

[0096] The following are fed to warping prediction module 3 :

[0097] - trainable image parameters of the training frame from the selected training frames (in FIG. 6, the training frame K), wherein if the trainable image parameters are the optimizable vector of the frame, as trainable parameters, you can feed either the optimizable vector of the frame (in FIG. 6, the optimizable vector K corresponds to the training frame K, the remaining optimizable vectors correspond to other training frames), or only the number of the training frame, which is pre-assigned its own optimizable vector of this frame, which is fed to the warping neural network 3 a,

[0098] - the 3D geometry of the subject for the mentioned one training frame from the selected training frames, namely the parameters that include, for example, the shape of the subject, the pose of the subject,

[0099] - spherical neural texture.

[0100] Input data for the warping neural network 3a is formed from the data fed to warping prediction module 3. The input data for the warping neural network 3a of warping prediction module 3 are:

[0101] - trainable image parameters of the training frame from the selected training frames, for example, the optimizable vector for the training frame,

[0102] - parameters of the 3D geometry of the subject for the mentioned training frame,

[0103] - coordinates of the vertices of the 3D mesh of the spherical neural texture.

[0104] At the output of the warping neural network 3a, a prediction of the warping of the spherical neural texture is obtained, that is, for each vertex of the 3D mesh of the spherical neural texture, a prediction is made - to which point on the canonical sphere this vertex will shift for this training frame. In other words, it is predicted how the texture correspondence should change for this training frame.

[0105] Deep neural networks (for example, fully connected, convolutional neural networks, graph neural networks, as well as their combination, including using adaptive normalization [6]), known from the prior art, can be used as the warping neural network 3a.

[0106] For example, the first deep neural network is first applied to the optimizable vectors of the training frames and / or the parameters of the 3D geometry of the subject or their combination, and the result is fed to the second neural network as adaptive normalization parameters, as in [6], for example. The coordinates of the vertices of the 3D mesh of the spherical neural texture (that is, the original ones) and / or the result of their encoding, for example, positional encoding, known from the prior art, for example, [3,4], are also fed to the input of the second deep neural network. At the output of the second deep neural network, two numbers are obtained as a prediction for each point of the 3D mesh of the spherical neural texture, which are then used in the warping application module as shifts in the polar coordinates of the vertices of the 3D mesh of the spherical neural texture, which allows the vertices to remain on the surface of the sphere after the shift.

[0107] The warping prediction obtained from the warping neural network 3a, as well as the spherical neural texture, are fed to warping application module 4. Warping application module 4 implements a change in texture correspondence for the spherical neural texture, that is, the correspondence between points on the surface of the 3D geometry of the subject and points on the surface of the 3D mesh of the sphere, in accordance with the warping prediction. Such a change in texture correspondence is implemented by shifting the points of the 3D mesh of the spherical neural texture. After processing in warping application module 4, a deformed spherical neural texture is obtained, namely, a spherical neural texture with a deformed texture correspondence, that is, with a deformed 3D mesh of the spherical neural texture. All vertices of the deformed 3D mesh of the spherical neural texture remain on the surface of the canonical sphere, but change their coordinates.

[0108] Thus, when performing warping, the vertices of the 3D mesh of the spherical neural texture are shifted in accordance with this training frame, while all points remain, each on its face, but since the faces move (the vertices change coordinates), then the points also change coordinates accordingly, remaining each on its face. Warping of the 3D spherical neural texture is carried out depending on the training frame, that is, depending on the corresponding optimizable frame vector and / or the pose of the subject in the frame for which warping is calculated.

[0109] C) The obtained deformed spherical neural texture and the 3D geometry of the subject are fed to neural texture rendering module 5. Neural texture rendering module 5 is a software module and performs a standard operation of overlaying the deformed neural texture on the 3D geometry (part of the surface of the 3D geometry visible in this frame), that is, the rendering operation. At the output of the neural texture rendering module 5, a neural texture 5a that has undergone rendering (hereinafter referred to as the rendered neural texture) is obtained.

[0110] D) The obtained rendered neural texture 5a is fed to rendering neural network 6, at the output of which an image 6a of the subject is obtained, which is compared with image la in the selected training frame.

[0111] It is possible to obtain a four-channel image 6a, where the first three channels correspond to the RGB channels, and the fourth corresponds to the foreground mask. In this case, the obtained image is compared with the segmented image from the training frame.

[0112] E) In the loss calculation module of optimization module 7, based on the comparison of the image 6a of the subject obtained at the output of the trainable rendering neural network 6 and the image la of the subject in the selected training frame la, for which the 3D geometry of the subject was extracted, the values of the loss functions are determined between the obtained image 6a of the subject and image la in the mentioned training frame. This can be done (denoted in Fig. 5 by a thick double-sided arrow) using a discriminative network and, accordingly, a adversarial loss function (for example, [8,9]) by calculating the pixel-by-pixel difference directly between the images or calculating the value of the perceptual loss function (for example, [10,11]) using the segmentation loss function applied to the foreground mask in the obtained image and the frame image (for example,

[0012] ). In addition, a regularization loss function can be used, which is applied to the warping prediction.

[0113] F) Then, in the loss calculation module of optimization module 7, based on the values of the loss functions, the error is calculated as a weighted sum of the values of the loss function. Using the optimizer of optimization module 7, a standard backpropagation procedure is performed to tune the entire system, that is, to update the weights of the entire system using the backpropagation method. The following system parameters are updated:

[0114] - the parameters of the spherical neural texture,

[0115] - rendering neural network 6, - trainable parameters of warping prediction module 3, in particular, trainable parameters of the image of the subject, such as, for example, the optimizable frame vector, are updated for each new training frame, as well as the parameters of the warping neural network.

[0116] Steps (B) - (E) of system training are repeated until the average difference between the obtained images of the subject and the corresponding images of the subject in the training frames ceases to decrease, or until the specified number of iterations is performed.

[0117] Steps (B) - (F) can be performed in parallel for a subset of the set of selected training frames at once, since in known software tools that provide functionality for training neural networks (for example, [PyTorch Paszke, Adam, et al. "Automatic differentiation in pytorch." (2017)]) the possibility of parallel execution is assumed for several training examples. In this case, the values of the loss functions obtained in step E are averaged for the selected subset of training frames. It is also standard practice to calculate the loss functions as the average for several training frames before each update of the parameters of the trainable system.

[0118] At each iteration of training, it is possible to use a new subset of training frames, for example, randomly selected from the set of training frames, then obtain the corresponding images using the system, calculate the error for each pair of training frame-generated image. And then the error is considered as the average for all used pairs, and based on the calculated error, the system parameters are updated. All of the above is standard practice for training neural networks.

[0119] The training frames can be stored in the memory of the electronic apparatus on which the proposed invention is used, as well as selected by the user arbitrarily, or in a certain predetermined order.

[0120] In one embodiment of the proposed invention, the trained system for creating an image of a subject in a desired pose operates as follows. As shown in FIG. 4, when the trained system for creating an image of a subject in a desired pose and / or viewing angle is operating, posed 3D geometry of an arbitrary subject in the desired pose is fed to the input of the system for creating an image of a subject in a desired pose and / or viewing angle, trained on images of the subject selected by the user, wherein the pose and / or viewing angle parameters of the arbitrary subject in the desired pose correspond to the desired ones. The arbitrary subject can be either the subject from the images of the training frames, or any subject selected by the user. In one embodiment, the pre-processing module retrieves the posed 3D geometry of an arbitrary subject in the desired pose from the device memory, wherein the pose and viewing angle correspond to the pose and viewing angle desired by the user, in which it is necessary to display the subject selected during training. The number of vertices and faces of the 3D geometry of an arbitrary subject in the desired pose is equal to the number of vertices and faces of the mentioned 3D geometry of the subject used during training.

[0121] Such posed 3D geometry of an arbitrary subject in the desired pose can be obtained by methods known from the prior art:

[0122] - from an arbitrary image of the subject (see, for example, [2]), including from an image that does not belong to the sequence of training frames that were used for training,

[0123] - using a parametric model of the subject [2], for which numerical parameters can be set that determine the pose of the subject, and by other known methods.

[0124] The 3D geometry of an arbitrary subject in the desired pose can be fed to the input of the proposed system in any order and quantity. For example, you can submit one after the other 3D geometries from a sequence of 3D geometries that correspond to some movement, for example, when the arm is raised, or a person bends over, or turns, etc. Then, at the output of the system, a sequence of images will be obtained, which can be converted into a corresponding video with the specified movements.

[0125] In the pre-processing module, a weighted sum of the trained image parameters of the subject obtained after training the proposed system described above is calculated. In one embodiment, the trained image parameters of the subject are the optimized vectors of the training frames, obtained when updating the trainable image parameters of the subject, which are the optimizable vectors of the training frames after training the mentioned system.

[0126] Then, when the system is operating, the following are fed to the input of the warping neural network of warping prediction module 3 :

[0127] - a weighted sum of the trained image parameters of the subj ect, for example, a weighted sum of the optimized vectors of the training frames, obtained from the optimizable vectors of the training frames as a result of training the system;

[0128] - the 3D geometry of the arbitrary subject in the desired pose, selected by the user,

[0129] - the trained spherical neural texture.

[0130] At the output of the trained warping neural network, a prediction of the warping of the spherical neural texture is obtained.

[0131] At the output of the trained warping neural network 3a, a prediction is obtained for each vertex of the 3D mesh of the spherical neural texture - to which point on the canonical sphere this vertex will shift.

[0132] The warping prediction and the trained spherical neural texture are fed to warping application module 4. Warping application module 4 outputs a deformed spherical neural texture.

[0133] The obtained deformed spherical neural texture and the 3D geometry of the arbitrary subject in the desired pose or the original trained spherical neural texture and the 3D geometry of the arbitrary subject in the desired pose are fed to neural texture rendering module 5. A rendered neural texture 5a is obtained.

[0134] The obtained rendered neural texture 5a is fed to the trained rendering neural network 6, at the output of which an image 6a of the subject in the desired pose selected by the user is obtained.

[0135] The proposed invention can be used to obtain a video of the subject by feeding a sequence of 3D geometries corresponding to the changing pose of the subject in the video to the trained system. For example, you need a video where the subject raises his hand, then a set of 3D geometries of the subject is fed to the system, where in each subsequent 3D geometry of the subject, the hand will be slightly more raised compared to the previous 3D geometry of the subject. The proposed trained system for each 3D geometry of the subject fed to the input will output an image of the subject in the corresponding pose / vi ewing angle. If the sequence of 3D geometries fed to the input corresponds to the natural movement of the subject (that is, an ordered and sufficiently smooth change in geometry), then the sequence of obtained images will make up a video with such movement.

[0136] The proposed invention can also be used for virtual and augmented reality systems for natural display of user images in arbitrary poses and / or viewing angles. For example, for consistent display of the user's image and background, including for creating user animations that are naturally integrated into the virtual or augmented reality environment.

[0137] FIG. 7 schematically illustrates an electronic apparatus as an embodiment.

[0138] The electronic apparatus (100) may include at least one of a pre-processing module, a warping prediction module, a warping application module, a neural texture rendering module or an optimization module.

[0139] In an embodiment, a system for creating an image of a subject in a desired pose may include at least one of a pre-processing module, a warping prediction module, a warping application module, a neural texture rendering module or an optimization module. The pre-processing module may receive a spherical neural texture from the memory of an electronic device, store the trained spherical neural texture, wherein the spherical neural texture characterizes the mapping of points of the 3D geometry of the subject to points on the surface of a canonical sphere.

[0140] The pre-processing module may sample training frames with an image of the subject from a sequence of frames, during training.

[0141] The pre-processing module may obtain trainable parameters of the image of the subject during training.

[0142] The pre-processing module may obtain the 3D geometry of the subject from the image of the subject.

[0143] The warping prediction module may receive trainable parameters of the image from the pre-processing module and store trainable and trained parameters of the image of the subject.

[0144] The warping prediction module may include a warping neural network. The warping prediction module may obtain a prediction of the warping of the spherical neural texture, to which the 3D geometry of the subject is mapped.

[0145] The warping application module may obtain a deformed spherical neural texture based on the mentioned prediction.

[0146] The neural texture rendering module may render the mentioned deformed spherical neural texture and the corresponding 3D geometry of the subject.

[0147] The rendering neural network may obtain an image of the subject based on the rendered neural texture.

[0148] The optimization module may, during the system training, compare the obtained image of the subject with the image of the subject in the training frame.

[0149] The optimization module may determine the value of the loss function and calculate the error.

[0150] The optimization module may backpropagate the error to update the weights of the entire system.

[0151] The system, during training, the pre-processing module may receive a spherical neural texture from the memory of an electronic device and store the trained spherical neural texture.

[0152] The pre-processing module may sample training frames for further processing from a sequence of frames with an image of the subject in different poses.

[0153] For each frame from the selected training frames, the pre-processing module may obtain trainable parameters of the image of the subject.

[0154] The pre-processing module may obtain the 3D geometry of the subject, which is a polygonal 3D mesh corresponding to the pose and shape of the subject, while the number of vertices and faces of the 3D geometry of the subject is the same for the image of the subject in all selected frames, while the vertices of the 3D mesh of the spherical neural texture are connected in the same way as the vertices of the 3D geometry of the subject.

[0155] The points of the 3D geometry of the subject are mapped to points on the surface of the spherical neural texture using the 3D mesh of the spherical neural texture.

[0156] The warping prediction module may receive trainable parameters of the image from the pre-processing module and store trainable parameters of the image of the subject.

[0157] The warping neural network may obtain a prediction of the warping of the spherical neural texture based on the trainable parameters of the image of the subject, the 3D geometry of the subject and the spherical neural texture, wherein the prediction of the warping of the spherical neural texture is a modification of the parameters of the 3D mesh of the spherical neural texture for the corresponding training frame.

[0158] The warping application module may obtain a deformed spherical neural texture for the corresponding training frame based on the mentioned prediction. The neural texture rendering module may render the mentioned deformed spherical neural texture and the corresponding 3D geometry of the subject, and to obtain a rendered neural texture thereby.

[0159] The rendering neural network may obtain an image of the subject based on the rendered neural texture,

[0160] The optimization module may compare the obtained image of the subject with the image of the subject in the frame from the selected training frames.

[0161] The optimization module may determine the value of the loss function and calculate the error.

[0162] The optimization module may backpropagate the error to update the weights of the entire system. when the system is running, the pre-processing module may store the trained spherical neural texture, obtain the posed 3D geometry of an arbitrary subject in the desired pose (for example, extract from the device memory), wherein the pose and viewing angle correspond to the pose and viewing angle desired by the user, in which it is necessary to display the subject selected during training, while the number of vertices and faces of the 3D geometry of an arbitrary subject in the desired pose is equal to the number of vertices and faces of the mentioned 3D geometry of the subject used during training.

[0163] The warping prediction module may extract the trained spherical neural texture.

[0164] The warping neural network of the warping prediction module may obtain a prediction of the warping of the trained spherical neural texture, based on the trained parameters of the image of the subject, the 3D geometry of the subject and the trained spherical neural texture.

[0165] The warping application module may obtain a deformed trained spherical neural texture for the corresponding 3D geometry of the selected subject based on the mentioned prediction.

[0166] The neural texture rendering module may render the mentioned deformed trained spherical neural texture and the corresponding 3D geometry of the selected subject and to obtain a rendered neural texture thereby.

[0167] The rendering neural network may obtain an image of the subject based on the rendered neural texture.

[0168] The trainable parameters of the image of the subject can be optimizable vectors of training frames, wherein each training frame corresponds to its own optimizable vector of the training frame, wherein each optimizable vector of the training frame encodes information about the warping that needs to be fed to the warping neural network to obtain the warping of the spherical neural texture for this training frame. Moreover, the spherical neural texture can be determined by the function of calculating the values of the neural texture at any point on the canonical sphere and the texture correspondence that maps the points of the 3D geometry of the subject to points on the surface of the canonical sphere using the 3D mesh of the spherical neural texture.

[0169] Moreover, the 3D mesh of the spherical neural texture can be represented by polygons (in the prior art, triangles are usually used, which will be considered further), each pair of which has one common edge and two common vertices, wherein each of the vertices of the 3D mesh of the spherical neural texture lies on the surface of the canonical sphere of the neural texture and uniquely corresponds to one vertex of the 3D geometry of the subject, wherein the vertices of the 3D mesh of the spherical neural texture are connected in the same way as the vertices of the 3D geometry of the subject. Moreover, the prediction of the warping of the spherical neural texture is a prediction of the coordinates of points on the deformed 3D mesh of the spherical neural texture, depending on which training frame image parameters are fed to the input of the warping neural network. Moreover, the pre-processing module is additionally configured for: image segmentation and calculation of the foreground mask for the image in the frame from the selected frames. The warping neural network may be represented by two deep neural networks interacting through adaptive normalization parameters.

[0170] A method for training the mentioned system for creating an image of a subject in a desired pose is also proposed, comprising the steps in which, using the pre-processing module, retrieving a spherical neural texture from the memory of an electronic device, selecting training frames in an arbitrary way from a sequence of frames with an image of the subject in different poses for further processing. wherein for each frame from the selected training frames obtaining trainable parameters of the image of the subject, obtaining the 3D geometry of the subject, which is a polygonal 3D mesh corresponding to the pose and shape of the subject, while the number of vertices and faces of the 3D geometry of the subject is the same for the image of the subject in all selected frames, while the vertices of the 3D geometry of the subject are connected in the same way as the vertices of the 3D mesh of the spherical neural texture, for each frame from the selected training frames: using the warping prediction module:

[0171] - feeding to the input of the warping neural network:

[0172] - the trainable parameters of the image of the subject,

[0173] - the 3D geometry of the subject,

[0174] - the spherical neural texture, obtaining, at the output of the warping neural network, a prediction of the warping of the spherical neural texture, feeding the spherical neural texture and the obtained warping prediction to the input of the warping application module, obtaining a deformed spherical neural texture at the output, rendering the deformed neural texture and the 3D geometry of the subject, using the neural texture rendering module, obtaining a rendered neural texture, feeding the rendered neural texture to the input of the rendering neural network, at the output of which an image of the subject is obtained, using the optimization module: comparing the obtained image of the subject with the image of the subject from the mentioned training frame from the selected training frames, determining the values of the loss function, based on which the error is calculated, performing backpropagation of the error to update the parameters of the entire system for creating an image of the subject, wherein the following are updated:

[0175] - the parameters of the spherical neural texture,

[0176] - the trainable parameters of the warping prediction module, including the trainable parameters of the image of the subject,

[0177] - the weights of the rendering neural network, wherein all steps of training the system for creating an image of the subject are repeated for an arbitrary subset of the selected training frames, wherein the training of the system is repeated until the average difference between the obtained images of the subject and the corresponding images of the subject in the training frames ceases to decrease or a fixed number of iterations.

[0178] The trainable parameters of the image of the subject can be optimizable vectors of training frames, when each training frame corresponds to its own optimizable vector of the training frame, wherein each optimizable vector of the training frame encodes information about the warping that needs to be fed to the warping neural network to obtain the warping of the spherical neural texture for this training frame. The mentioned comparison can be additionally performed using one of the discriminative network and adversarial loss function, pixel-by-pixel difference calculation, perceptual loss function calculation, segmentation loss function, regularization loss function applied to the warping prediction.

[0179] A method of operating the mentioned system for creating an image of a subject in a desired pose, trained by means of the mentioned method, is proposed, comprising the steps in which: obtaining, using the pre-processing module, the 3D geometry of an arbitrary subject in the desired pose, which is in the pose selected by the user, wherein the number of vertices and faces of the 3D geometry of the arbitrary subject in the desired pose is equal to the number of vertices and faces of the mentioned 3D geometry of the subject used during training, calculating, using the warping prediction module, a weighted sum of the trained image parameters of the subject obtained as a result of training the system, feeding to the input of the warping neural network of the warping prediction module:

[0180] - the weighted sum of the trained image parameters of the subject,

[0181] - the 3D geometry of the selected subject,

[0182] - trained spherical neural texture,

[0183] - obtaining a prediction of the warping of the spherical neural texture at the output of the warping neural network, feeding the obtained warping prediction and the trained spherical neural texture to the input of the warping application module, obtaining a deformed spherical neural texture at the output, rendering, using the neural texture rendering module, the deformed spherical neural texture and the 3D geometry of the selected subject, obtaining a rendered neural texture, feeding the rendered neural texture to the input of the trained rendering neural network with updated weights, at the output of which an image of the subj ect in the desired pose selected by the user is obtained.

[0184] The trained parameters of the image of the subject can be optimized vectors of training frames obtained during the update of the trainable parameters of the image of the subject, which are optimizable vectors of training frames after training the mentioned system. The step of obtaining the 3D geometry of an arbitrary subject in the desired pose may consist in extracting the 3D geometry of an arbitrary subject in the desired pose from the memory of an electronic device using the pre-processing module.

[0185] At least one of the plurality of modules may be implemented via an Al (artificial intelligence) model. The Al-related function may be performed via a non-volatile memory, a volatile memory, and a processor. The processor may include one or more processors. At this time, one or more processors may be a general-purpose processor such as a central processing unit (CPU), an application processor (AP), or the like, a graphics processor such as a graphics processor (GPU), a visual processor (VP), and / or a specialized Al processor such as a neural processor (NP). The one or more processors control the processing of input data in accordance with a predetermined operating rule or Al model stored in the non-volatile memory and the volatile memory. The predetermined operating rule or artificial intelligence model is provided via learning or training. Here, providing via learning means that a predetermined operating rule or Al model of a desired characteristic is created by applying a learning algorithm to a plurality of training data. The training may be performed in the device itself, in which the Al according to the embodiment is executed, and / or may be implemented through a separate server / system. The Al model may consist of a plurality of neural network layers. Each layer has a plurality of weight values and performs the layer operation by computing the previous layer and the operation of the plurality of weights. Examples of neural networks include, but are not limited to, a convolutional neural network (CNN), a deep neural network (DNN), a recurrent neural network (RNN), a restricted Boltzmann machine (RBM), a deep belief network (DBN), a bidirectional recurrent deep neural network (BRDNN), generative adversarial networks (GANs), and deep Q-networks. The training algorithm is a method of training a predetermined target device (e.g., a robot) using a plurality of training data to force, allow, or control the target device to make a decision or make a prediction. Examples of learning algorithms include, but are not limited to, supervised learning, unsupervised learning, semi-supervised learning, or reinforcement learning. The method described in this application, performed by an electronic device, may be performed using an artificial intelligence model.

[0186] Subject recognition methods can obtain image or person recognition outputs on the image using the image data as input to an artificial intelligence model. The artificial intelligence model can be obtained by training. Here, "obtained by training" means that a predetermined rule of operation or artificial intelligence model tuned to perform the desired function (or goal) is obtained by training a basic artificial intelligence model with several pieces of training data using a training algorithm. The artificial intelligence model may include multiple neural network layers. Each of the multiple neural network layers includes a plurality of weights and performs a neural network calculation by calculating between the result of the calculation by the previous layer and the plurality of weights. Visual understanding is a method of image recognition and processing, like human vision, and includes, for example, object recognition, object tracking, image search, subject recognition, scene recognition, 3D reconstruction / localization, or image enhancement.

[0187] FIG. 8 schematically illustrates controlling method of an electronic apparatus as an embodiment.

[0188] The electronic apparatus (100) is for training an AI( Artificial Intelligence) model to generate an image of an object based on an input pose.

[0189] The electronic apparatus (100) may include at least one processor including processing circuitry, and memory storing instructions.

[0190] The at least one processor may obtain a plurality of images for training the Al model, obtain training images including an object from the plurality of images, obtain at least one parameter corresponding to each of the training images, obtain 3D geometry information of the object based on the at least one parameter, obtain a spherical neural texture mapped with the 3D geometry information, obtain warping prediction information for the spherical neural texture, update the spherical neural texture based on the warping prediction information, and train the Al model based on the updated spherical neural texture.

[0191] The training images may describes as sample images, selected images or extracted images.

[0192] The object may describes as a target object, a movable object. For example, the object is human object.

[0193] The 3D geometry information may describes as 3D structural information, 3D shape data or 3D model information.

[0194] The 3D geometry information may refer to data representing the three-dimensional structure of an object. The 3D geometry information may include at least one of spatial coordinates of points, surface shapes or topological relationships among geometric elements.

[0195] The spherical neural texture may refer to a neural representation mapped onto a canonical sphere. Each point on the sphere may correspond features of the object’s 3D geometry.

[0196] The spherical neural texture may be described as a spherical feature map, a neural surface representation, a spherical texture model, a neural sphere mapping data, or a 3D feature sphere.

[0197] The warping prediction information may refer to data indicating deformation of the spherical neural texture. The warping prediction information may be described as deformation data, transformation parameters, displacement information, warping field data, or geometry adjustment information.

[0198] Updating the spherical neural texture based on the warping prediction information may refer to modifying the texture according to predicted deformations.

[0199] The at least one processor may obtain first points indicating the object based on the 3D geometry information, obtain second points on a surface of a canonical sphere, and obtain the spherical neural texture by mapping the points indicating the object to the second points.

[0200] The at least one processor may identify the object capable of changing a pose from a plurality of objects included in the sequence of images.

[0201] The training images may be a sequence of images.

[0202] The parameters may be trainable parameters. The trainable parameters may be described as learnable parameters, adjustable parameters, optimizable parameters, adaptive parameters, or model parameters.

[0203] The warping prediction information may include information for deforming the spherical neural texture.

[0204] The at least one processor may obtain an output image based on the updated spherical neural texture, obtain a loss value by comparing the output image with the training images, and train the Al model based on the loss value.

[0205] The at least one processor may obtain the loss value by comparing the output image with the training images based on a loss function for the Al model, obtain error information for backpropagation to the Al model, and update weight information of the Al model based on the error information.

[0206] The weight information may correspond to the at least one parameter used to obtain the 3D geometry information of the object.

[0207] The at least one processor may obtain optimizable vector corresponding to each of the training images based on the at least one parameter, and obtain the 3D geometry information of the object based on the optimizable vector.

[0208] The at least one processor may obtain a canonical sphere, obtain correspondence information between points on a surface of the canonical sphere and the 3D geometry information of the object, and obtain the spherical neural texture mapped with the 3D geometry information based on the correspondence information.

[0209] The at least one processor may obtain a 3D mesh of the object based on the correspondence information, and obtain the spherical neural texture based on the 3D mesh.

[0210] The 3D mesh may include adjacent triangles sharing a common edge and two common vertices, and the two common vertices may correspond to points on the surface of the canonical sphere.

[0211] The at least one processor may obtain the warping prediction information for the spherical neural texture by predicting coordinates of points on the 3D mesh.

[0212] The warping prediction information may include predicted displacements or transformations of points on the 3D mesh.

[0213] The update adjusts feature values or spatial mappings on the sphere to reflect the predicted changes in the 3D mesh.

[0214] The 3D mesh may refer to a data structure representing the shape of a three-dimensional object. The 3D mesh consists of vertices, edges, and polygonal faces, typically triangles, forming a connected network that defines the object’s surface in 3D space.

[0215] In an embodiment, A controlling method of an electronic apparatus (100), the controlling method comprising obtaining a plurality of images for training the Al model (S810), obtaining training images including an object from the plurality of images (S820), obtaining at least one parameter corresponding to each of the training images (S830), obtaining 3D geometry information of the object based on the at least one parameter (S840), obtaining a spherical neural texture mapped with the 3D geometry information (S850), obtaining warping prediction information for the spherical neural texture (S860), updating the spherical neural texture based on the warping prediction information (S870), and training the Al model based on the updated spherical neural texture (S880).

[0216] Although the invention has been described in connection with certain illustrative embodiments, it should be understood that the essence of the invention is not limited to these particular embodiments. On the contrary, it is intended that the essence of the invention includes all alternatives, modifications and equivalents that may be included in the essence and scope of the claims.

[0217] Moreover, the invention retains all equivalents of the claimed invention, even if the clauses of the claims are changed during the review process.

[0218] List of references

[0219] [1] Thies, Justus, Michael Zollhbfer, and Matthias NieBner. "Deferred neural rendering: Image synthesis using neural textures." Acm Transactions on Graphics (TOG) 38.4 (2019): 1- 12.

[0220] [2] Georgios Pavlakos, Vasileios Choutas, Nima Ghorbani, Timo Bolkart, Ahmed A. A. Osman, Dimitrios Tzionas and Michael J. Black. Expressive body capture: 3d hands, face, and body from a single image. Published in Proceedings IEEE Conf, on Computer Vision and Pattern Recognition (CVPR), 2019.

[0221] [3] Vaswani, A. "Attention is all you need. " Advances in Neural Information Processing Systems (2017).

[0222] [4] Mildenhall, Ben, et al. "Nerf: Representing scenes as neural radiance fields for view synthesis." Communications of the ACM 65.1 (2021): 99-106.

[0223] [5] Muller, Thomas, et al. "Instant neural graphics primitives with a multiresolution hash encoding." ACM transactions on graphics (TOG) 41.4 (2022): 1-15.

[0224] [6] Huang, Xun, and Serge Belongie. "Arbitrary style transfer in real-time with adaptive instance normalization." Proceedings of the IEEE international conference on computer vision. 2017.

[0225] [7] Weng, Chung- Yi, et al. "Humannerf: Free-viewpoint rendering of moving people from monocular video." Proceedings of the IEEE / CVF conference on computer vision and pattern Recognition. 2022.

[0226] [8] Goodfellow, Ian, et al. "Generative adversarial networks." Communications of the ACM 63.11 (2020): 139-144.

[0227] [9] Isola P. et al. Image-to-image translation with conditional adversarial networks / / Proceedings of the IEEE conference on computer vision and pattern recognition. - 2017. - p. 1125-1134.

[0228]

[0010] Dosovitskiy, Alexey, and Thomas Brox. "Generating images with perceptual similarity metrics based on deep networks." Advances in neural information processing systems 29 (2016).

[0229]

[0011] Justin Johnson, Alexandre Alahi and Li Fei-Fei. Perceptual losses for real-time style transfer and super-resolution. Published in Bastian Leibe, Jiri Matas, Nicu Sebe, and Max Welling, editors, Proc. ECCV, 2016.

[0230]

[0012] Milletari, Fausto, Nassir Navab, and Seyed- Ahmad Ahmadi. "V-net: Fully convolutional neural networks for volumetric medical image segmentation." 2016 fourth international conference on 3D vision (3DV). leee, 2016.

[0231]

[0013] Li, Ruilong, et al. "Tava: Template-free animatable volumetric actors." European Conference on Computer Vision. Cham: Springer Nature Switzerland, 2022.

[0014] Li, Zhe, et al. "Avatarcap: Animatable avatar conditioned monocular human volumetric capture." European Conference on Computer Vision. Cham: Springer Nature Switzerland, 2022.

[0232]

[0015] Liu, Lingjie, et al. "Neural actor: Neural free-view synthesis of human actors with pose control." ACM transactions on graphics (TOG) 40.6 (2021): 1-16.

[0233]

[0016] Tiwari, Garvita, et al. "Neural-gif: Neural generalized implicit functions for animating people in clothing." Proceedings of the IEEE / CVF International Conference on Computer Vision. 2021.

[0234]

[0017] Weng, Chung- Yi, et al. "Humannerf: Free-viewpoint rendering of moving people from monocular video." Proceedings of the IEEE / CVF conference on computer vision and pattern Recognition. 2022.

[0235]

[0018] Gong K. et al. Graphonomy: Universal human parsing via graph transfer learning / / Proceedings of the IEEE / CVF Conference on Computer Vision and Pattern Recognition. - 2019. - p. 7450-7459.

Claims

[CLAIMS]

1. An electronic apparatus for training an AI( Artificial Intelligence) model to generate an image of an object based on an input pose comprises: at least one processor including processing circuitry, and memory storing instructions, wherein the instructions, when executed by the at least one processor individually or collectively, cause the electronic apparatus to: obtain a plurality of images for training the Al model, obtain training images including an object from the plurality of images, obtain at least one parameter corresponding to each of the training images, obtain 3D geometry information of the object based on the at least one parameter, obtain a spherical neural texture mapped with the 3D geometry information, obtain warping prediction information for the spherical neural texture, update the spherical neural texture based on the warping prediction information, and train the Al model based on the updated spherical neural texture.

2. The electronic apparatus of claim 1, wherein the instructions, when executed by the at least one processor individually or collectively, cause the electronic apparatus to: obtain first points indicating the object based on the 3D geometry information, obtain second points on a surface of a canonical sphere, and obtain the spherical neural texture by mapping the points indicating the object to the second points.

3. The electronic apparatus of claim 1, wherein the instructions, when executed by the at least one processor individually or collectively, cause the electronic apparatus to: identify the object capable of changing a pose from a plurality of objects included in the sequence of images.

4. The electronic apparatus of claim 1, wherein the training images are a sequence of images.

5. The electronic apparatus of claim 1, wherein the parameters are trainable parameters.

6. The electronic apparatus of claim 1, wherein the warping prediction information includes information for deforming the spherical neural texture.

7. The electronic apparatus of claim 1, wherein the instructions, when executed by the at least one processor individually or collectively, cause the electronic apparatus to: obtain an output image based on the updated spherical neural texture, obtain a loss value by comparing the output image with the training images, and train the Al model based on the loss value.

8. The electronic apparatus of claim 7, wherein the instructions, when executed by the at least one processor individually or collectively, cause the electronic apparatus to: obtain the loss value by comparing the output image with the training images based on a loss function for the Al model, obtain error information for backpropagation to the Al model, and update weight information of the Al model based on the error information.

9. The electronic apparatus of claim 8, wherein the weight information corresponds to the at least one parameter used to obtain the 3D geometry information of the object.

10. The electronic apparatus of claim 1, wherein the instructions, when executed by the at least one processor individually or collectively, cause the electronic apparatus to: obtain optimizable vector corresponding to each of the training images based on the at least one parameter, and obtain the 3D geometry information of the object based on the optimizable vector.

11. The electronic apparatus of claim 1, wherein the instructions, when executed by the at least one processor individually or collectively, cause the electronic apparatus to: obtain a canonical sphere, obtain correspondence information between points on a surface of the canonical sphere and the 3D geometry information of the object, and obtain the spherical neural texture mapped with the 3D geometry information based on the correspondence information.

12. The electronic apparatus of claim 11, wherein the instructions, when executed by the at least one processor individually or collectively, cause the electronic apparatus to: obtain a 3D mesh of the object based on the correspondence information, and obtain the spherical neural texture based on the 3D mesh.

13. The electronic apparatus of claim 12, wherein the 3D mesh includes adjacent triangles sharing a common edge and two common vertices, and wherein the two common vertices correspond to points on the surface of the canonical sphere.

14. The electronic apparatus of claim 13, wherein the instructions, when executed by the at least one processor individually or collectively, cause the electronic apparatus to: obtain the warping prediction information for the spherical neural texture by predicting coordinates of points on the 3D mesh.

15. A controlling method of an electronic apparatus, the controlling method comprising: obtaining a plurality of images for training the Al model, obtaining training images including an object from the plurality of images, obtaining at least one parameter corresponding to each of the training images, obtaining 3D geometry information of the object based on the at least one parameter, obtaining a spherical neural texture mapped with the 3D geometry information, obtaining warping prediction information for the spherical neural texture, updating the spherical neural texture based on the warping prediction information, and training the Al model based on the updated spherical neural texture.

Images