Apparatus for generating image of object based on three-dimensional data and operating method thereof
The method generates a training data set from three-dimensional object data to address the limitations of two-dimensional image generation, ensuring accurate reflection of three-dimensional structures and maintaining consistency through optimal training and evaluation, enhancing processing speed and memory efficiency.
Patent Information
- Application Number
- PCT/KR2026/005938
- Authority / Receiving Office
- WO · WO
- Patent Type
- Applications
- Current Assignee / Owner
- Priority Date
- 2025-11-13
- Filing Date
- 2026-05-08
- Publication Date
- 2026-11-12
AI Technical Summary
Conventional image generation technologies using two-dimensional data fail to accurately reflect three-dimensional spatial information, leading to shape distortion and visual incongruity when camera angles change or complex poses are involved, and lack the ability to select optimal training methods based on object attributes.
A method and apparatus that generate a training data set from three-dimensional object data, incorporating images and metadata from various viewpoints, analyze attribute information to determine optimal training methods, and perform composite training to maintain consistent style and shape, with evaluation and retraining based on quality assessment.
Accurately reflects three-dimensional structures, maintains consistency under environmental changes, reduces errors in image generation, and enhances processing speed and memory efficiency by direct 3D extraction, ensuring high-quality image generation.
Smart Images

Figure KR2026005938_12112026_PF_FP_ABST
Abstract
Description
APPARATUS FOR GENERATING IMAGE OF OBJECT BASED ON THREE-DIMENSIONAL DATA AND OPERATING METHOD THEREOF
[0001] The present disclosure relates to a device for generating an image and a method of operating the device.
[0002] Recently, as generative artificial intelligence (Generative AI) technology has advanced, technology for automatically generating a new image based on text or an example image input by a user has been utilized in various industrial fields. In particular, diffusion models, generative adversarial networks (GANs), and Transformer-based models have attracted significant attention in the fields of game, animation, and metaverse content production.
[0003] However, since conventional technologies mostly use only two-dimensional image-based training data, there is a problem in that three-dimensional spatial information, such as depth information of an object, a surface direction, and a camera viewpoint, cannot be accurately reflected. Due to this limitation, when a camera angle is changed or a complex pose is included, shape distortion or visual incongruity occurs in a generated image.
[0004] In addition, existing fine-tuning methods often retrain an entire model or apply only a single LoRA method, and thus lack a function of automatically selecting or compositely applying an optimized training method according to attributes such as the number of polygons, the number of joints, a style type, and a data source of an object. Accordingly, there is a need for technology for generating a high-quality image in which consistency is maintained based on three-dimensional data.
[0005] A technical problem to be solved by the present disclosure is to generate an accurate training data set based on three-dimensional object data and to include images, metadata, and camera metadata corresponding to various viewpoints, thereby accurately reflecting a three-dimensional structure of an object.
[0006] Another technical problem to be solved by the present disclosure is to analyze attribute information, such as polygon complexity, the number of joints, a style type, and a data source of an object to be learned, thereby determining an optimal training method from among a plurality of training methods.
[0007] Another technical problem to be solved by the present disclosure is to provide a structure capable of performing composite training by combining different training modules as well as a single training method, thereby providing an image in which detail is reflected while a consistent style and shape are maintained.
[0008] Another technical problem to be solved by the present disclosure is to evaluate quality of an image generated after completion of training and to perform retraining based on an evaluation result, thereby effectively updating an image generation model.
[0009] According to one aspect, there is provided a method of operating a server, the method comprising: obtaining three-dimensional data for an object included in an image; obtaining, by rendering the three-dimensional data according to a plurality of viewpoints, a training data set including, for each of the plurality of viewpoints, a two-dimensional image and metadata having three-dimensional attribute information; performing training using the training data set based on a training method determined from attribute information of the three-dimensional data; and obtaining an image generation model for the object based on a result of the training.
[0010] According to an embodiment, the metadata may comprise at least one of depth information, surface normal information, motion information, and occlusion information for each pixel.
[0011] According to an embodiment, the method of operating the server may further comprise: obtaining, based on the three-dimensional data, the attribute information including at least one of a type of the three-dimensional data, a type of the object, a number of polygons for the object, complexity of joints, a resolution of a texture, and a style; and determining, from among a plurality of training methods, a training method corresponding to the attribute information.
[0012] According to an embodiment, the plurality of training methods may be classified based on at least one of a modification range of model parameters, an amount of training data used during training, a style of the object, and a type of the object.
[0013] According to an embodiment, the determining of the training method corresponding to the attribute information may comprise determining, for each criterion for classifying the plurality of training methods, a training method corresponding to the attribute information, and the training of the training data set may comprise training the training data set by combining the training methods determined for each criterion.
[0014] According to an embodiment, the obtaining of the training data set may further comprise obtaining, as part of the training data set, camera metadata related to a setting of a camera corresponding to the plurality of viewpoints.
[0015] According to an embodiment, the camera metadata may comprise position and posture information of the camera in a three-dimensional space, and may comprise at least one photographing control parameter including a focal length, an aperture value, a shutter speed, a sensor size, ISO sensitivity, and white balance.
[0016] According to an embodiment, the method may further comprise: obtaining quality information on an image generated through the image generation model; and updating the image generation model based on the quality information.
[0017] According to an embodiment, the quality information on the image may comprise information on at least one of a matching degree between text input for generating the image and the image, a proportion and a color reflecting consistency of the object, a three-dimensional structure matching degree, realism of the image, and a matching degree between the image and a camera parameter.
[0018] According to an embodiment, the updating of the image generation model based on the quality information may comprise adjusting, based on the quality information, a weight of a parameter associated with a feature having an image quality evaluation score lower than a preset reference value or a weight of a loss function of the image generation model.
[0019] According to an embodiment, the obtaining of the three-dimensional data may comprise obtaining the three-dimensional data from a two-dimensional image obtained by photographing a real object corresponding to the object or from a three-dimensional model.
[0020] According to an embodiment, the method of operating the server may further comprise obtaining a target image based on setting information for a three-dimensional target image and the image generation model.
[0021] According to an embodiment, the obtaining of the target image may comprise: obtaining the setting information in which at least one of a pose of the object, an expression of the object, a composition of a camera, and an arrangement of lighting is set; obtaining, by rendering the three-dimensional data based on the setting information, a control map representing a visual structural feature corresponding to the setting information; and obtaining the target image based on the control map and the image generation model.
[0022] According to an embodiment, the three-dimensional data may be three-dimensional scene data including pose and spatial arrangement information of the object corresponding to the target image.
[0023] According to an embodiment, the control map may comprise a depth map generated by rendering the three-dimensional data on a virtual three-dimensional viewport and extracting a value from a Z-buffer of the viewport.
[0024] According to an embodiment, the control map may comprise at least one of a depth map, a surface normal map, a pose map, an expression map, a contour map, a camera map, and a lighting map.
[0025] According to an embodiment, the obtaining of the target image based on the control map and the image generation model may comprise obtaining, through the image generation model, the target image in which detail of the object is reflected, based on the visual structural feature set in the control map.
[0026] According to an embodiment, the method of operating the server may further comprise: obtaining quality information on the target image; and correcting the target image based on the quality information.
[0027] According to another aspect, there is provided a server comprising: a communication device configured to communicate with an external device; a processor; and a memory storing instructions executable by the processor, wherein the processor, by executing the instructions, is configured to: obtain three-dimensional data for an object included in an image; obtain, by rendering the three-dimensional data according to a plurality of viewpoints, a training data set including, for each of the plurality of viewpoints, a two-dimensional image and metadata having three-dimensional attribute information; perform training using the training data set based on a training method determined from attribute information of the three-dimensional data; and obtain an image generation model for the object based on a result of the training.
[0028] According to still another aspect, there is provided a non-transitory computer-readable storage medium storing instructions that, when executed by a processor, cause the processor to perform operations comprising : obtaining three-dimensional data for an object included in an image; obtaining, by rendering the three-dimensional data according to a plurality of viewpoints, a training data set including, for each of the plurality of viewpoints, a two-dimensional image and metadata having three-dimensional attribute information; performing training using training data set based on a training method determined from attribute information of the three-dimensional data; and obtaining an image generation model for the object based on a result of the training.
[0029] One advantageous effect of the present disclosure is that an accurate training data set may be generated based on three-dimensional object data, and images, metadata, and camera metadata corresponding to various viewpoints may be included, thereby accurately reflecting a three-dimensional structure of an object. In addition, attribute information, such as polygon complexity, the number of joints, a style type, and a data source of an object for training, may be analyzed, thereby determining an optimal training method from among a plurality of training methods.
[0030] Another advantageous effect of the present disclosure is that, by providing a structure capable of performing composite training by combining different training modules as well as a single training method, an image in which detail is reflected while a consistent style and shape are maintained may be provided.
[0031] Another advantageous effect of the present disclosure is that, by generating an image through direct extraction of data from 3D rendering of an object, an image in which consistency of an appearance feature and a texture of the object is maintained even under changes in environment, such as a background, an angle, lighting, and a viewpoint, may be provided.
[0032] Another advantageous effect of the present disclosure is that, in particular, unlike some approaches relying on 2D image-based estimation, such as depth estimation or normal estimation, control signals are directly extracted from a Z-buffer, a normal buffer, a pose rig, or the like in a 3D rendering stage. Accordingly, an original three-dimensional structure may be reflected as it is without loss of pixel-level precision, an actual shape, a distance, a lighting response, and a surface texture of an object may be maintained with high precision, and spatial recognition may be achieved during model training without distortion or omission.
[0033] Another advantageous effect of the present disclosure is that the control signals provide structural information in a three-dimensional space, for example, depth, a surface normal, a pose, lighting, and a camera position, as condition inputs of an image generation model, thereby supporting the image generation model to learn not only a level of "visual similarity" but also a physical relationship of an object and a scene composition principle. As a result, even when the same object is reproduced under various viewpoints or environmental conditions, both shape consistency and visual coherence may be secured.
[0034] Another advantageous effect of the present disclosure is that a 3D direct extraction approach may substantially reduce an error accumulation problem, such as depth distortion, edge loss, or occlusion misjudgment, which may occur in 2D estimation-based ControlNet, thereby enabling stable structure recognition and image generation even in a complex background or a multi-object scene. Accordingly, by directly reflecting realistic structural information of three-dimensional data, a technical effect of improving accuracy, reproducibility, and consistency of an AI image generation process may be provided. Furthermore, this direct 3D extraction approach improves the functioning of the computer itself by avoiding computationally intensive and error-prone 2D-to-3D estimation processes, thereby enhancing the processing speed and memory efficiency of the image generation system.
[0035] Another advantageous effect of the present disclosure is that quality of an image generated after completion of training may be evaluated, and retraining may be performed based on an evaluation result, thereby effectively updating an image generation model.
[0036] The present disclosure may be readily understood by a combination of the following detailed description and the accompanying drawings, and reference numerals indicate structural elements.
[0037] FIG. 1 is a flowchart illustrating a method of operating a server for obtaining an image generation model based on three-dimensional data according to an embodiment.
[0038] FIG. 2 is a flowchart illustrating a method of operating a server for obtaining a training data set for an object according to an embodiment.
[0039] FIG. 3A is a diagram for describing a process of obtaining a training data set according to an embodiment.
[0040] FIG. 3B illustrates an example of source code for describing a process of generating a training data set from a three-dimensional model according to an embodiment.
[0041] FIG. 4 is a flowchart illustrating a method of operating a server for determining a training method based on attribute information of three-dimensional data according to an embodiment.
[0042] FIG. 5A is a diagram for describing a process of selecting a candidate method for each item based on attribute information of three-dimensional data according to an embodiment.
[0043] FIG. 5B is a flowchart illustrating a method of operating a server for generating one image generation model by integrating composite candidate training methods based on attribute information of three-dimensional data according to an embodiment.
[0044] FIG. 6 is a diagram for describing an operation of an image generation model for performing training using a training data set according to an embodiment.
[0045] FIG. 7 is a flowchart illustrating a method of operating a server for obtaining a target image using an image generation model according to an embodiment.
[0046] FIG. 8 is a flowchart illustrating a method of operating a server for setting three-dimensional scene data according to an embodiment.
[0047] FIG. 9 is a flowchart illustrating a method of operating a server for generating a target image based on a control map according to an embodiment.
[0048] FIG. 10 is a flowchart illustrating a method of operating a server for correcting a target image based on quality information on the target image according to an embodiment.
[0049] FIG. 11 is a block diagram illustrating a configuration of a server according to an embodiment.
[0050] According to one aspect, there is provided a method of operating a server, the method comprising: obtaining three-dimensional data for an object included in an image; obtaining, by rendering the three-dimensional data according to a plurality of viewpoints, a training data set including, for each of the plurality of viewpoints, a two-dimensional image and metadata including three-dimensional attribute information; performing training using the training data set based on a training method determined from attribute information of the three-dimensional data; and obtaining an image generation model for the object based on a result of the training.
[0051] Hereinafter, various embodiments will be described in detail with reference to the drawings. The embodiments described below may be implemented by being modified into various different forms. In order to more clearly describe features of the embodiments, detailed descriptions of matters widely known to those of ordinary skill in the art to which the following embodiments pertain will be omitted.
[0052] Meanwhile, in the present specification, when a component is referred to as being "connected" to another component, this includes not only a case in which the component is "directly connected" to the other component, but also a case in which the component is connected to the other component with still another component interposed therebetween. In addition, when a component is referred to as "including" another component, this means that the component may further include another component, rather than excluding the other component, unless specifically stated otherwise.
[0053] In addition, terms including ordinal numbers, such as "first" or "second," used in the present specification may be used to describe various components, but the components should not be limited by the terms. The terms are used only for the purpose of distinguishing one component from another component.
[0054] In the present specification, "three-dimensional data" may include structural information and spatial information of an object. "Metadata" may mean data including spatial, optical, and physical attribute information calculated in a process of rendering three-dimensional data from a plurality of viewpoints. In the present specification, "three-dimensional attribute information" may refer to specific data included in the "metadata." Specifically, the three-dimensional attribute information may refer to 3D structure data, which is rendered and extracted pixel by pixel from a 3D model so that an AI model directly learns a 3D structure, for example, a depth map, a surface normal map, motion information, and occlusion information, and may be used as part of the "metadata" for allowing the AI model to "directly learn" the 3D structure. "Attribute information of three-dimensional data" may be information representing structural complexity and visual expression characteristics of the three-dimensional data. For example, the "attribute information of three-dimensional data" may mean intrinsic characteristics of the 3D model itself, for example, a number of polygons, joint complexity, and a style type, which are used as criteria for determining a "training method" of the AI model. A "control map" may be data including a spatial structure or geometric information obtained from three-dimensional scene data.
[0055] In the present specification, "training data" may refer to data used for training an image generation model or a training module coupled to the image generation model. For example, the training data may include a two-dimensional image, metadata, three-dimensional attribute information, camera metadata, a control map, a prompt, a reference image, a quality evaluation result, or a combination thereof.
[0056] In the present specification, a "training data set" may refer to a data set in which one or more pieces of training data are configured as a structured unit. For example, the training data set may include a data structure in which two-dimensional images for a plurality of viewpoints, metadata corresponding to each two-dimensional image, camera metadata, and a prompt are matched to one another on a one-to-one basis.
[0057] In the present specification, the expression "performing training using the training data set" may be understood not as limiting the training data set itself as a training target, but as including an operation of learning, adjusting, or updating parameters of an image generation model, a training module coupled to the image generation model, a LoRA module, an adapter module, or a combination thereof by using the training data set.
[0058] In the present specification, a "training method" may refer to a method, an algorithm, a model structure, a parameter adjustment method, or a combination thereof, which is applied to train an image generation model or a training module by using training data or a training data set. For example, the training method may include a full fine-tuning method, a low-rank-based parameter addition method, a LoRA method, a DreamBooth method, a method of injecting a specific object or style, a method of inserting a modular adapter, or a composite training method in which a plurality of training modules are applied in combination.
[0059] In the present specification, an "image generation model" may mean a neural network model that includes a pre-trained diffusion-based neural network or generative adversarial network, receives condition information such as a text prompt, a control map, and a camera parameter, and outputs a two-dimensional image or a plurality of images satisfying the condition information. The image generation model may be trained using an additional modules or training technique, such as LoRA, Adapter, Fine-Tuning, or ControlNet, and may be optimized for a specific object or a specific style.
[0060] In the present specification, "quality information" may mean a set of indicators quantitatively representing how accurately an image generated by the image generation model reproduces an appearance, a structure, a motion, a style, and viewpoint consistency of an object.
[0061] FIG. 1 is a flowchart illustrating a method of operating a server 10 for obtaining an image generation model based on three-dimensional data according to an embodiment.
[0062] Referring to FIG. 1, in operation S110, the server 10 may obtain three-dimensional data for an object constituting an image.
[0063] For example, the three-dimensional data may include structural information and spatial information of the object. Specifically, the three-dimensional data may include at least one of geometry of the object, vertex coordinates, a polygon structure, that is, a mesh structure, surface normal information, that is Normal Vector, texture information, color information, depth information, material information, and joint and skeleton structure information, for example, skeleton and rigging information, and may mean data capable of reconstructing the object in a three-dimensional space.
[0064] In addition, the three-dimensional data may be obtained as a model file in a predetermined three-dimensional format. For example, the three-dimensional data may be obtained in the form of a digital model file, for example, .fbx, .obj, .gltf, or the like, a point cloud, or volumetric data.
[0065] In addition, the three-dimensional data may not only be directly uploaded by a user, but may also be automatically generated from a two-dimensional image or a real-world image through an AI-based 3D reconstruction technology such as Photogrammetry, Neural Radiance Field (NeRF), or Gaussian Splatting. Even in this case, the result may be regarded as the three-dimensional data including geometric information, depth information, surface normal information, color information, and the like of the object.
[0066] Preferably, the AI-based 3D reconstruction technology may include a "hybrid 3D data preparation" workflow combined with a manual supplementation operation by an artist.
[0067] In an embodiment, the server 10 first automatically reconstructs an "initial 3D model" from a two-dimensional image through NeRF or Gaussian Splatting technology. However, an AI reconstruction result may often include geometry errors, for example, a mesh having a hole, or texture omission.
[0068] In this case, the artist may load the "initial 3D model" into a 3D DCC tool, for example, Blender or Maya, and perform a "3D data correction / supplementation" operation of "modifying" a broken mesh or "supplementing" an omitted part. The server 10 finally obtains "final 3D data," for which the supplementation operation by the artist has been completed, as "ground truth" for a training operation, that is, rendering.
[0069] In addition, after obtaining the three-dimensional data, the server 10 may perform an "automatic refinement and optimization" operation, that is, pre-flighting, before a rendering operation. In an embodiment, the server 10 may automatically check whether geometry of the obtained three-dimensional data is a closed structure, for example, a watertight structure, whether a surface normal direction is not reversed, whether UV coordinates are correctly set, and the like. In addition, when the obtained number of polygons is too low compared to a training target, for example, in the case of a low-poly model, the server 10 may automatically apply tessellation or subdivision to subdivide a mesh. Conversely, when the obtained three-dimensional data is excessively complex, for example, in the case of original 3D scan data, the server 10 may automatically simplify the mesh through decimation or retopology to prepare a 3D asset optimized for training.
[0070] The server 10 may use an AI-based three-dimensional reconstruction technology (3D Reconstruction Technology) in order to process a target for which a three-dimensional model does not previously exist, such as a real person or an actual background. The server 10 may perform a series of processes for reconstructing a three-dimensional shape from two-dimensional images obtained by photographing a real-life target from a plurality of viewpoints.
[0071] As a specific example, the server 10 may apply NeRF (Neural Radiance Fields) technology and use two-dimensional image data obtained by photographing a subject from 20 or more different viewpoints as an input, thereby generating a three-dimensional representation in a continuous volumetric form.
[0072] NeRF may train a neural network-based continuous radiance field (Neural Radiance Field) and precisely reconstruct a surface and an internal structure of an actual object by predicting a color and density of a ray according to a change in a viewpoint. The reconstruction process may provide a high-quality result, but may require large-scale computational resources, and training may take about 24 hours.
[0073] As another example, the server 10 may apply a Gaussian Splatting method to perform rapid three-dimensional reconstruction based on a point cloud.
[0074] Gaussian Splatting is a method of approximating density and color information in space by using a plurality of Gaussian points. Although reconstruction accuracy may vary depending on implementation conditions, a 3D model may be generated in about 2 hours, and Gaussian Splatting may be suitable when real-time rendering or rapid visual feedback is required.
[0075] For example, in relation to a 3D reconstruction process, the server 10 may perform: 1) an operation of collecting a plurality of viewpoint images, that is, multi-view images, of a subject; 2) an operation of performing camera calibration for correcting a camera position and posture of each image by using a structure reconstruction tool such as COLMAP; 3) an operation of performing three-dimensional reconstruction for reconstructing a point cloud in a three-dimensional space based on corrected viewpoint information; 4) an operation of extracting a polygon mesh through a Marching Cubes algorithm or the like by using reconstructed point cloud data; and 5) an operation of performing UV mapping and texture baking for generating texture coordinates on the extracted mesh and applying color information thereto. The server 10 may perform the 3D reconstruction process by omitting some of the operations.
[0076] The reconstructed three-dimensional data may include information on at least one of a shape, a position, a size, and a surface of the object, and may be used in a subsequent operation as a condition input of control map generation, training data configuration, or the image generation model.
[0077] That is, the server 10 may secure a high-quality three-dimensional data set even for a real person or an actual environment, thereby generating a target image having high structural consistency and visual consistency.
[0078] In operation S120, the server 10 may obtain, by rendering the three-dimensional data according to a plurality of viewpoints, a training data set including, for each of the plurality of viewpoints, a two-dimensional image and metadata having three-dimensional attribute information.
[0079] The "operation of rendering three-dimensional data according to a plurality of viewpoints" may mean a process in which the server 10 places a virtual camera, lighting, and an object in a three-dimensional space, and generates two-dimensional image data as if the object were photographed from the plurality of viewpoints while changing at least one of a position and an orientation of the camera, for example, roll, pitch, and yaw, and a camera parameter, for example, a focal length, an aperture effect, or the like. In this process, three-dimensional attributes such as depth information of the object, surface normal information, material attributes, contour information, for example, edge information, pose information, and a skeleton structure may also be generated or stored together.
[0080] For example, the server 10 may automatically generate a training data set in which a two-dimensional image corresponding to each viewpoint and metadata including three-dimensional attribute information are combined, by rendering the three-dimensional data according to the plurality of viewpoints.
[0081] The operation of rendering the three-dimensional data according to the plurality of viewpoints by the server 10 is intended to prevent an overfitting problem that may occur when a model is trained on a specific object only with respect to a predetermined angle or lighting environment, and to allow an intrinsic shape, texture, and structural consistency of the object to be recognized even under various conditions.
[0082] Specifically, the server 10 may perform an automated multi-angle dataset rendering process in relation to a rendering operation. The server 10 may place the three-dimensional data for the object in a virtual space of a rendering engine, for example, Blender, Unreal Engine, or the like, and set a plurality of physical variables such as a camera position, a lighting condition, and a focal length to be controllable. In addition, the server 10 may set a virtual spherical shell centered on a three-dimensional object through a Python-based automation module, and may photograph the object while sequentially moving a position of a camera at predetermined intervals, for example, at intervals of 5 degrees, in elevation and azimuth directions along a surface of the sphere. Through the photographing operation, the server 10 may obtain hundreds to thousands of two-dimensional images from various viewpoints.
[0083] In addition, the server 10 may apply a randomization technique for each frame of each rendering. For example, the server 10 may apply different HDRI (High Dynamic Range Imaging) maps to respective viewpoints to change a lighting direction, a color temperature, brightness, and the like, and may randomly adjust a focal length and a depth of field of the camera, thereby obtaining training data under various optical conditions and environmental conditions even for the same object. The training data obtained in response to the various conditions may maximize diversity of data and allow the image generation model to learn visual generalization ability robust to environmental changes.
[0084] In addition, at each rendering viewpoint, the server 10 may generate a metadata packet (Mdata) defining the corresponding viewpoint together with an image, and may store the metadata packet and the image as a pair. For example, the metadata packet for each viewpoint may include the following items.
[0085] Mdata = { Iimage, Tprompt, Pcam, Rcam, Kintrinsics }
[0086] Here, Iimage may mean the rendered two-dimensional image, Tprompt may mean text information associated with the rendered two-dimensional image, Pcam may mean position information of the camera, Rcam may mean rotation value information of the camera, and Kintrinsics may mean camera intrinsic parameter information such as a focal length, a sensor size, and a distortion coefficient.
[0087] The structured metadata packet described above may be used as key information for determining quality of a fine-tuning process performed in a subsequent operation. Preferably, the "training data set" is composed of a pair of a rendered "two-dimensional image" and "metadata," and in this case, the "metadata" is stored as a "structured metadata packet," for example, in a JSON format. This packet may include a data structure that binds at least the following information on a one-to-one (1:1) basis. For example, the packet may include (1) a path of a rendered 2D image, (2) a path of three-dimensional attribute metadata, for example, a depth map and a normal map, (3) camera metadata, for example, a 3D position, a focal length, and an aperture value, (4) attribute information of original 3D data, for example, a number of polygons and a number of joints, and (5) an automatically generated text prompt describing the corresponding image.
[0088] The structured packet serves as a basis that enables the server 10 to precisely query and use data of a specific condition during training and quality evaluation.
[0089] In addition, the rendering environment may be provided in a controlled environment to secure accurate metadata, and, as an example, rendering may be performed under conditions including a rendering engine of Blender Cycles, for example, physically based rendering, an Academy Color Encoding System (ACES) standard color space of Linear, for example, 16-bit float and HDR, a resolution of 2048Х2048, and sampling of 256 samples per pixel.
[0090] According to the multi-angle rendering operation of the server 10, a high-quality and diverse training data set may be generated in large quantities without user intervention, and the image generation model may consistently recognize structural features and textural attributes of the object under various environmental conditions and generate an image.
[0091] When rendering is completed, metadata corresponding to the two-dimensional image generated for each viewpoint may be obtained. The metadata may mean data including spatial, optical, and physical attribute information calculated in the process of rendering the three-dimensional data from the plurality of viewpoints. The metadata may correspond to the two-dimensional image of each viewpoint and may include three-dimensional structure information such as depth information, surface normal information, occlusion information, and motion information. In addition, the metadata may include camera parameter information at the time of rendering, for example, a position, a focal length, and an aperture value, and lighting condition information, for example, a light source position, intensity, and a color temperature.
[0092] The server 10 may obtain the metadata by directly extracting accurate three-dimensional attribute information from a Z-buffer, that is, a depth buffer, of the rendering engine. In this case, the Z-buffer is a buffer that records, for each pixel of a screen, a depth value and a position in space of a corresponding point, and may more accurately restore three-dimensional information lost in a two-dimensional image.
[0093] The metadata includes spatial information, optical information, and physical attribute information generated in the process of rendering the three-dimensional data from the plurality of viewpoints, and may be stored by being matched one-to-one with the two-dimensional image of each viewpoint. In particular, because the server 10 directly extracts data from the Z-buffer, unlike an existing method of estimating depth information from a two-dimensional image, the server 10 may provide metadata including measured-level three-dimensional attribute information with reduced quantitative error.
[0094] Accordingly, accuracy of the metadata included in the training data set may be improved, and the image generation model may accurately recognize and learn the shape, depth, spatial relationship, and the like of the object.
[0095] The metadata may be stored by being matched one-to-one with the two-dimensional image for each viewpoint, and may be used as an element constituting the training data set. That is, the training data set may be composed of a two-dimensional image derived from the three-dimensional data and metadata including three-dimensional attribute information.
[0096] The metadata may be used to generate a control map or may be used as an input of the image generation model in a fine-tuning stage, thereby supporting the image generation model to learn an accurate three-dimensional-based visual condition.
[0097] That is, the server 10 may overcome the limitations of a two-dimensional estimation method by directly extracting data based on the Z-buffer of a three-dimensional rendering engine, and may automatically configure a training data set for which accuracy and consistency are ensured. In addition, this direct 3D extraction approach improves the functioning of the computer itself by avoiding computationally intensive and error-prone 2D-to-3D estimation processes, thereby enhancing the processing speed and memory efficiency of the image generation system.
[0098] In addition, the server 10 may obtain, as the training data set, camera metadata related to a setting of a camera corresponding to the plurality of viewpoints. Here, the camera metadata may include position and posture information of the camera in a three-dimensional space, and may include at least one photographing control parameter including a focal length, an aperture value, a shutter speed, a sensor size, ISO sensitivity, and white balance.
[0099] For example, camera metadata at each rendering viewpoint may be recorded with the following precision. Position information may be defined by X, Y, and Z coordinate values and may have precision in millimeters (mm). Rotation information may represent rotation angles of Pitch, Yaw, and Roll, and may be expressed in units of 0.1°. Lens information may include focal length range information from a wide angle to a telephoto angle, for example, a focal length from 24 mm wide angle to 135 mm telephoto angle, and may express various perspective and background compression effects. Aperture information may include parameter information on an aperture, for example, a value in a range of f / 1.4 to f / 16, and may be used as an element for adjusting depth of field.
[0100] Because the camera metadata is stored together with the training data set, the image generation model may recognize and learn not only the shape or color of the object but also a photographing viewpoint and an optical property. For example, the trained image generation model may accurately interpret and reproduce camera-related prompts such as "a portrait photographed with a telephoto lens" and "a background blur effect having a shallow depth of field."
[0101] In an embodiment, the training data set obtained in operation S120 may include metadata composed of object information, data obtained by extracting a Depth Map, a Normal Map, an Edge Map, a Motion Vector, and Ambient Occlusion of each pixel from a Z-buffer after 3D scene rendering of the object, camera metadata, and a prompt automatically generated based on the data, for example, "photographed with a telephoto lens," "studio lighting," and "sunset lighting."
[0102] By including actual physical parameters of the camera in the training data set together with the metadata, the trained image generation model may naturally simulate various photographing compositions and optical effects, which may consequently provide a technical effect of improving viewpoint consistency and photographic realism.
[0103] In operation S130, the server 10 may perform training on the training data set based on a training method determined from attribute information of the three-dimensional data.
[0104] For example, the server 10 may determine, from among a plurality of training methods, a training method corresponding to the attribute information of the three-dimensional data, and may perform training on the training data set based on the training method. The plurality of training methods may be classified based on at least one of a modification range of model parameters, an amount of training data used during training, a style of the object, and a type of the object.
[0105] For example, the attribute information of the three-dimensional data may be information representing structural complexity and visual expression characteristics of the three-dimensional data. The attribute information of the three-dimensional data may include information on at least one of a type of the three-dimensional data, a type of the object, a number of polygons for the object, a number of joints, a resolution of a texture, and a style. For example, the type of the object may include at least one of a human, an animal, and a background. In addition, the style may indicate whether the three-dimensional data is based on a real-life image or based on animation.
[0106] The server 10 may independently determine or combine and apply the training method based on at least one of the modification range of model parameters, a scale of data, a type of a style, and the type of the object. In addition, the server 10 may combine and compositely apply different training methods for respective detailed items of the attribute information of the three-dimensional data.
[0107] In operation S140, the server 10 may obtain an image generation model for the object based on a result of performing training on the training data set.
[0108] In addition, the server 10 may obtain quality information on an image generated through the image generation model. The server 10 may update the image generation model based on the quality information. Here, the quality information on the image may include information on at least one of a matching degree between text input for generating the image and the image, a ratio and a color reflecting consistency of the object, a three-dimensional structure matching degree, realism of the image, and a matching degree between the image and the camera parameter. Here, the proportion reflecting consistency of the object may include at least one of an appearance proportion, a body proportion, a relative size between partial components, a contour proportion, and an arrangement proportion between main feature portions of the object. The color reflecting consistency of the object may include at least one of a representative color, a color distribution, a color contrast, a texture color, and a style color of the object.
[0109] FIG. 2 is a flowchart illustrating a method of operating the server 10 for obtaining a training data set for an object according to an embodiment.
[0110] Referring to FIG. 2, in operation S210, the server 10 may obtain a two-dimensional image by rendering three-dimensional data for the object. Specifically, the server 10 may set a plurality of viewpoints by changing a position and direction of a camera and a lighting environment in a three-dimensional space, and may obtain the two-dimensional image by rendering the three-dimensional data according to the plurality of viewpoints.
[0111] For example, the server 10 may set a virtual camera and a lighting environment in the three-dimensional space, and may generate the two-dimensional image by projecting the three-dimensional data from various viewpoints using a rendering engine, for example, Blender Cycles, Unreal Engine, Unity, or the like.
[0112] For example, the server 10 may arrange the camera based on a virtual spherical path centered on the object, and may implement multi-angle and multi-illuminance photographing conditions by changing at least one of a position, an orientation, a focal length, an aperture value, and a lighting direction of the camera.
[0113] The position of the camera may be defined by X-axis, Y-axis, and Z-axis coordinate values, and an angle, for example, azimuth or elevation, may be set to automatically traverse the plurality of viewpoints while rotating at predetermined intervals, for example, intervals of 5° to 15°
[0114] The server 10 may perform rendering such that a shape, a texture, and a material of the object are visually expressed at each viewpoint, and may obtain a color image generated as a result thereof as two-dimensional image data.
[0115] In this case, a rendering result may include visual elements such as pixel-level color information, for example, RGB values, specular reflection, a shadow, and a surface highlight, and the corresponding image may subsequently be stored as part of the training data set.
[0116] For example, during a rendering process, the server 10 may generate data while periodically changing camera movement for each angle, lighting environment randomization, for example, Random HDRI Sampling, focal length and depth of field variation, exposure, color temperature adjustment, and the like. Through the rendering process, hundreds to thousands of multi-angle and multi-illuminance images may be automatically secured, thereby improving generalization performance for a shape, a material, and lighting changes of an object in a training process of an image generation model.
[0117] In operation S220, the server 10 may obtain metadata corresponding to the two-dimensional image. For example, the metadata may include at least one of depth information, surface normal information, motion information, occlusion information, color information, pose information, and Canny edge information for each pixel. The metadata may be directly extracted from a Z-buffer of the rendering engine or 3D geometry data, that is, mesh data, and may be used as a high-precision control signal reflecting accurate attributes of a three-dimensional structure. Meanwhile, a "surface normal map" may refer to a map representing, on a pixel-by-pixel basis, a direction vector of a surface of an object, and may also be referred to as a "normal map" depending on context. In this case, the surface normal map and the normal map are not intended to mean different information, but may be understood as the same or corresponding data representing surface direction information corresponding to each pixel.
[0118] Depth information may be information indicating a distance of each pixel from the camera, and may be expressed as a Z-axis coordinate or a Z-buffer value in the three-dimensional space. For example, a depth value for each pixel, which is directly extracted from a Z-buffer of a viewport in which a virtual camera is set, may be used. In this case, the depth value may be stored with 32-bit floating-point precision, and may represent, with high precision, a relative distance from the camera or a position in a Z-axis direction for each pixel on a screen. The depth information may be used to restore a distance from the camera, a distance relationship between objects, and a spatial stereoscopic effect, and may enable quantitative reproduction of perspective of a scene when an image is generated.
[0119] Surface normal information may be information indicating a direction vector in which a surface of each pixel faces, and may be defined by a surface gradient and a direction of the pixel. The surface normal information may reflect fine irregularities or a gradient of the surface without loss, and consequently may contribute to maintaining texture fidelity, a specular reflection effect, and three-dimensional structure matching degree.
[0120] Motion information may be information indicating a temporal movement vector of a pixel between consecutive frames, and may represent in which direction and at what speed each pixel has moved. The motion information may be used for learning a change between viewpoints or a motion of an animation-type object. The motion information may contribute to securing temporal consistency between frames.
[0121] Occlusion information may be information indicating whether each pixel is occluded by another object or a background, and may be expressed in a form of a binary mask that distinguishes a visible region from an occluded region. The occlusion information may be used for boundary recognition of the object, pixel-level segmentation, and distortion correction during training for multiple objects.
[0122] Pose information may be generated based on 3D rig and joint data of the object, and may be configured by projecting a position and a rotation angle of each joint. The pose information may precisely express movement, a posture, a direction, and the like of a three-dimensional character, and in particular, may substantially reduce a flip ambiguity or an occlusion problem that may occur in an existing 2D pose estimation method.
[0123] Canny edge information may be data obtained by extracting a clear edge of the object by applying a Canny Edge Detector algorithm to the rendered two-dimensional image. Since edge detection is performed on a noise-free 3D rendering result, highly refined and stable silhouette information may be secured, and the Canny edge information may be used as a control map or an image condition, thereby enabling the image generation model to clearly recognize a boundary of the object.
[0124] In addition, ambient occlusion (AO) information is data representing a soft contact shadow generated according to spatial proximity between adjacent object surfaces. In the present embodiment, an AO map may be used as a control signal that enables the image generation model to learn spatial interaction of a 3D scene, and thereby may greatly improve depth perception and photorealism of a resulting image.
[0125] In operation S230, the server 10 may obtain the training data set including the two-dimensional image and the metadata.
[0126] Meanwhile, in a process of generating the two-dimensional image based on the three-dimensional data, a shape, perspective, and an occlusion relationship of the object may vary depending on a position of the camera and an angle at which photographing is performed. Therefore, even the same object may be expressed as a completely different image when a viewpoint is different, and when viewpoint information is not learned, a problem may occur in that a training model fails to maintain viewpoint consistency or fails to infer a three-dimensional spatial structure. Accordingly, by storing camera metadata together with the image in a process of configuring a training data set, image generation ability robust to viewpoint changes may be secured in a model training process.
[0127] In addition, the server 10 may obtain, as the training data set, camera metadata including position and posture information of the camera in the three-dimensional space. In addition, the camera metadata may include at least one photographing control parameter including a focal length, an aperture value, a shutter speed, a sensor size, ISO sensitivity, and white balance.
[0128] For example, information on camera parameters, such as a position of the camera, for example, X, Y, and Z, a rotation, for example, Pitch, Yaw, and Roll, a focal length, for example, 24 mm to 135 mm, an aperture, for example, an f-number, a sensor size, and an exposure time, may be included in the camera metadata.
[0129] The camera metadata may be used as viewpoint information when a control map is generated, and may be used as key data that enables the training model to maintain a positional relationship of the object and viewpoint consistency.
[0130] FIG. 3A is a diagram for describing a process of obtaining a training data set according to an embodiment.
[0131] Referring to FIG. 3A, three-dimensional data 310 for an object may be a three-dimensional model of an object for training, for example, a character, a background prop, a vehicle, or the like. The server 10 may obtain a set 320 of two-dimensional images for respective viewpoints by arranging the three-dimensional data 310 for the object in a three-dimensional space, moving a virtual camera to various positions and postures, and rendering the three-dimensional data 310 from a plurality of viewpoints. In addition, the server 10 may maximize diversity of generated data by applying different High Dynamic Range Imaging (HDRI) maps at respective photographing viewpoints to change a lighting environment, and by randomly adjusting a focal length and a depth of field setting of the camera.
[0132] The server 10 may extract metadata having three-dimensional attribute information and camera metadata at each viewpoint. Specifically, the server 10 may extract, from the set 320 of two-dimensional images for respective viewpoints, metadata including at least one of depth information, surface normal information, motion information, color information, and occlusion information for each pixel, and may extract camera metadata including position and posture information of the camera in the three-dimensional space.
[0133] The server 10 may obtain, as a training data set 330, the two-dimensional image, the metadata having the three-dimensional attribute information, and the camera metadata. By obtaining the training data set 330, the server 10 may simultaneously secure geometric matching with respect to viewpoint changes, physical consistency such as pose, depth, and normal, and photographing condition correspondence with camera parameters, compared to a data set based on a single 2D image. Accordingly, in a training stage, the server 10 may precisely learn a structure, motion, and optical response of the object. The image generation model may stably produce a high-quality image even at various angles and under various lighting conditions.
[0134] In addition, the training data set 330 may further include the image, the metadata having the three-dimensional attribute information, the camera metadata, and a prompt automatically generated based on the data, for example, "photographed with a telephoto lens," "studio lighting," "sunset lighting," or the like.
[0135] FIG. 3B illustrates an example of source code for describing a process of generating a training data set from a three-dimensional model according to an embodiment.
[0136] Referring to source code 351 of FIG. 3B, the server 10 may receive a three-dimensional object model as an input, and may perform multi-angle 3D rendering while combinatorially changing a camera angle, a camera height, and a lens focal length.
[0137] For example, the server 10 may sequentially apply 24 viewpoints in which the camera angle is set from 0° to 360° at intervals of 15°, three heights in a unit range of 0.5 to 1.5, and various lens conditions such as 24 mm, 50 mm, 85 mm, and 135 mm as the focal length.
[0138] Referring to source code 352 of FIG. 3B, the server 10 may directly refer to a Z-buffer from a rendering result generated at each viewpoint, and may extract a depth map and a surface normal map.
[0139] Since the Z-buffer holds distance and surface direction information for each pixel of a 3D scene, accurate three-dimensional structure information may be maintained even in a process of being converted into a two-dimensional image.
[0140] In this case, an extract_from_zbuffer() function may directly sample a corresponding attribute channel from an internal buffer of a rendering engine, and may convert the corresponding attribute channel into a metadata form.
[0141] Referring to source code 353 of FIG. 3B, the server 10 may generate a metadata structure by integrating a rendered image and extracted attribute maps.
[0142] metadata = {
[0143] 'image': render_output['color'],
[0144] 'depth': depth_map,
[0145] 'normal': normal_map,
[0146] 'camera': {'position': [camera_angle, camera_height], 'focal_length': focal_length, 'lens_type': get_lens_type(focal_length)},
[0147] 'prompt': generate_description(render_output)
[0148] }
[0149] Here, the "image" item may mean a two-dimensional color image, for example, RGB values, obtained by rendering the 3D object, and may be used as target output data to be learned by an AI model.
[0150] In addition, the "depth" and "normal" items may represent depth information and surface direction information directly extracted from the Z-buffer.
[0151] In addition, the "camera" item may mean camera metadata including viewpoint information such as a focal length of a lens, a camera position, and a lens type.
[0152] In addition, the "prompt" item may be automatically generated based on the image and the camera metadata.
[0153] A prompt generation function may generate a situation-based text prompt by referring to environmental information such as a lens focal length, an optical condition (for example, lens type), lighting, weather, and a color tone. For example, situation-based text prompts such as "Character Bigi photographed with a telephoto lens" and " Character Bigi under lighting on a rainy day" may be automatically generated.
[0154] The automatically generated text prompt is not a simple name tag, but is a conditioned natural language description, that is, a conditioned prompt, matching an actual visual result, and may allow the image generation model to learn an accurate association between a linguistic condition, for example, "telephoto lens" or "cloudy weather," and a visual result, for example, background compression or a color tone change.
[0155] The metadata may be stored in a form in which a 2D image, structural attributes, for example, depth and normal, and a text condition, for example, a prompt, are connected to one another on a one-to-one basis.
[0156] For example, the server 10 may generate a LoRA weight model for 2D image generation, which has learned a shape, a texture, and viewpoint consistency of the three-dimensional object, by performing LoRA-based fine-tuning using the training data set.
[0157] The trained LoRA weight model does not directly include 3D geometry data, for example, .fbx, .obj, or the like, but may integrally reflect structural consistency of the three-dimensional object and physical conditions such as a camera, a lens, and lighting. Accordingly, the image generation model may provide an effect of having "3D spatial understanding" in a 2D image generation process.
[0158] FIG. 4 is a flowchart illustrating a method of operating the server 10 for determining a training method based on attribute information of three-dimensional data according to an embodiment.
[0159] Referring to FIG. 4, in operation S410, the server 10 may obtain attribute information of three-dimensional data.
[0160] The server 10 may obtain, based on the three-dimensional data, the attribute information including at least one of a type of the three-dimensional data, a type of an object, a number of polygons for the object, joint complexity, texture resolution, and a style. Here, the attribute information may be information for representing structural complexity, a degree of detail, or visual expression characteristics of the object.
[0161] For example, the type of the three-dimensional data may be information indicating whether the three-dimensional data is realistic data obtained by scanning an actual target or synthetic data generated through virtual modeling. In addition, the type of the object may be classification information such as a humanoid type, an object type, and a background type.
[0162] For example, the number of polygons may mean the number of vertices constituting a mesh of a three-dimensional model, and may indicate a degree of detail of the model. For example, when the number of polygons is 50,000 or more, the object may be classified as a high-precision model, and when the number of polygons is 10,000 or less, the object may be classified as a simple model.
[0163] In addition, the server 10 may determine fundamentally different training methods according to not only the "attribute information of the three-dimensional data" but also an "importance level of IP identity." In an embodiment, when generating an "IP-specific model," the server 10 selectively applies a "parameter-efficient fine-tuning (PEFT)" method and a "full model fine-tuning" method. For example, lightweight style and object adaptation, for example, a LoRA method, may be applied. When an object for training is a specific art style, various costumes of a character, or a prop that can be flexibly replaced, the server 10 selects a "Low-Rank Adaptation (LoRA)" method. In this method, weights of a base model are frozen, and only a small-scale "adapter" matrix is added and trained. Therefore, the method is advantageous for storage and sharing because it is generated as a small file of tens to hundreds of MB, and is advantageous for real-time combination, for example, "applying costume B LoRA to character A." Alternatively, precise IP identity injection, for example, a DreamBooth method, may be applied. When the object for training is a "face of a main character" or a "unique appearance," which is a "core identity" that must not be changed under any circumstances, the server 10 may select a "DreamBooth" method or a full fine-tuning method similar thereto. This method finely readjusts a token embeddings of a unique identifier, that is, a token, such as "a photo of sks-character," and entire U-Net weights of the base model, thereby deeply injecting "digital DNA" of the corresponding IP into the base model itself.
[0164] As described above, the automatic selection or combined application, by the server 10, of an optimal training method such as LoRA or DreamBooth according to attributes and purposes of IP is may be an important technique for achieving "consistency" of the present disclosure.
[0165] Preferably, the operation of determining the training method may be performed based on a "meta-learning" model. In an embodiment, the server 10 may previously construct, as a "meta data set," a plurality of 3D model fine-tuning histories performed in the past, in which an input is 3D attribute information and an output is a quality score of a result. When new three-dimensional data is input, the server 10 may automatically determine, through the meta-learning model, by "inference," an optimal hyperparameter combination, for example, a LoRA rank, a learning rate, and an optimal LoRA module combination, which has shown the highest quality score in the past for given 3D attributes.
[0166] For example, joint complexity may be defined based on the number of joints included in the object and a connection structure. For example, when the object includes 50 or more joints and has a multi-stage joint connection structure, as in a humanoid character, the object may be classified as a high-degree-of-freedom object having high complexity.
[0167] For example, texture resolution may indicate resolution information of a texture map applied to a surface of the three-dimensional object. As the texture resolution increases, detailed texture expression becomes richer, and thus the object may be classified as an object requiring high-precision training.
[0168] For example, the style may be information regarding a visual expression method of the object, and may include at least one of a realistic type and an animation type. The style information may be used as a criterion for enabling the model to reproduce a specific rendering tone or texture expression while maintaining visual consistency.
[0169] In operation S420, the server 10 may determine a training method based on the attribute information of the three-dimensional data. The training method may be classified based on at least one of a modification range of model parameters, an amount of training data used during training, complexity of the object, and a style. The server 10 may determine, for each criterion for classifying a plurality of training methods, a training method corresponding to the attribute information.
[0170] Training methods may be classified as techniques for imparting new knowledge, a style, or a function to a pre-trained model. For example, the training methods may be classified into a full fine-tuning method for retraining an entire model, a low-rank-based parameter addition method, a method of injecting a specific object or style, and a method of inserting a modular adapter.
[0171] The full fine-tuning method may be a training technique for reconstructing the entire internal representation structure of the model by performing retraining on new data with respect to all parameters of the pre-trained model. The full fine-tuning method may provide a high degree of freedom and adaptability, and may precisely reflect unique shape information and detailed style information by retraining all parameters. However, since all parameters are modified, large-scale computational resources are required, and a problem may occur in that previously learned knowledge is lost.
[0172] The low-rank-based parameter addition method may be referred to as a Low-Rank Adaptation (LoRA) method. A LoRA training method is a method of learning an output difference of the model by inserting a low-dimensional matrix, that is, a low-rank matrix, into a specific operation layer, for example, a self-attention layer or a fully connected layer, without changing all parameters of an existing model. Through this, the model may additionally express a new pattern or attribute information while maintaining existing knowledge. LoRA uses less memory, may be quickly trained with a small amount of data and a small amount of computation, and each LoRA module may be independently stored and replaced. Therefore, a plurality of LoRA modules may be combined for respective purposes. For example, in a character model for generating an image of a character, LoRA for representing a pose, LoRA for representing a texture, LoRA for representing lighting responses, and the like may be combined for respective purposes.
[0173] The method of injecting a specific object or style may be a technique for learning an identity of a specific person, object, or style by inputting a small amount of data, for example, 3 to 20 images, to a pre-trained model. The method of injecting a specific object or style may be adjusted to remember unique visual features, for example, a shape, a color, a style, or the like, of a specific object while maintaining a detailed generation ability of an existing model.
[0174] The method of inserting a modular adapter may be a technique for selectively learning only a specific function without changing an overall structure of a model by inserting a small neural network module between major layers of a base model. The adapter may mainly be configured as a low-dimensional bottleneck structure, and may be added or replaced on an individual function basis, such as a lighting effect, depth information recognition, and language conversion.
[0175] The full fine-tuning method for retraining the entire model, the low-rank-based parameter addition method, the method of injecting a specific object or style, and the method of inserting a modular adapter are not mutually exclusive structures, and may be used in combination according to an implementation purpose. For example, a basic style of the model may be learned by the full fine-tuning method, a specific character may be trained by the method of injecting a specific object or style, and a pose, lighting, or texture may be learned by the low-rank-based parameter addition method or the modular adapter method.
[0176] The server 10 may improve quality and consistency of an image by determining at least one training method from among the plurality of training methods based on the attribute information of the three-dimensional data, and thereby generating the same person or object in a consistent form even in various backgrounds, poses, and lighting environments.
[0177] Meanwhile, the training method may be determined according to a scale of training data. For example, when a large-scale data set including various viewpoints, poses, and optical conditions is secured, the full fine-tuning method may be applied. When only a limited number of pieces of data, for example, 3 to 20 images, exists, such as for a specific character or a specific style, the low-rank-based parameter addition method or the method of injecting a specific object or style may be applied.
[0178] In addition, the training method may be determined according to a style or expression method of a target object. For example, for a realistic object, a method capable of accurately learning physical-based optical characteristics such as lighting reflection, a shadow, depth, and a material may be applied to maximize realism, and as an example, Photorealistic LoRA may be applied as a model. Specifically, training may be performed in consideration of material characteristics, for example, skin, hair, and clothing, studio and natural lighting effects, lens depth of field and bokeh effects, high-resolution output, and the like, and photographic-level visual quality may additionally be secured while maintaining a general style generation ability of an existing base model. Accordingly, by applying the Photorealistic LoRA to content types such as a photorealistic human, a product advertisement, fashion photography, and a high-quality composite image, a technical effect that a generated image "looks as if it were photographed with a camera" may be achieved.
[0179] For an animation-type character, a style-specific fine-tuning method reflecting an edge, flat shading, an emphasized texture pattern, and the like may be applied.
[0180] In addition, the training method may vary according to a type and characteristics of the object. For example, a humanoid character includes a joint structure, a pose change, and an expression change, and motion diversity implementation is an important part thereof. Accordingly, motion-based fine-tuning, for example, motion LoRA, a pose adapter, or an expression control adapter, may be applied. In contrast, for a static object such as a building, furniture, or a background, maintaining spatial consistency is an important part thereof. Accordingly, a Spatial Consistency LoRA, a texture LoRA, or an environment adapter method that intensively learns positions, sizes, materials, textures, and perspective between objects may be applied.
[0181] The server 10 may independently determine or combine and apply the training method based on at least one of the modification range of model parameters, the scale of data, a type of the style, and the type of the object. For example, the server 10 may construct an image generation model reflecting all of joint movement, surface texture, and lighting responses by simultaneously applying motion LoRA, texture LoRA, and a lighting adapter to a humanoid character. Through a combination of composite training methods, the server 10 may generate the same character or object in a consistent form even in various backgrounds, poses, and lighting environments, and may improve quality and visual consistency of an image.
[0182] FIG. 5A is a diagram for describing a process of selecting a candidate method for each item based on attribute information of three-dimensional data according to an embodiment.
[0183] The server 10 may automatically classify and select an optimized candidate training method for each item according to a type, a style, structural complexity, and a use of input three-dimensional data.
[0184] Referring to FIG. 5A, polygon complexity 501 may be an indicator representing shape complexity of an object. According to the shape complexity of the object, the number of parameters that a training model needs to secure and a rank of LoRA may vary.
[0185] As a specific example, when the number of polygons is less than 10,000, the structure is relatively simple, and the number of faces and edges is small, so that detailed texture expression is limited. Therefore, lightweight LoRA rank 16 having a small number of parameters may be applied to enable rapid training. For example, LoRA rank 16 may be applied to a simple robot arm, a low-resolution game character, a casual-style character having no muscle structure, or the like.
[0186] As another example, when the number of polygons is 10,000 or more and less than 50,000, the appearance of a character is somewhat complex and detailed elements such as a face, clothing, and equipment are present, and thus standard LoRA rank 32 may be applied. For example, LoRA rank 32 may be applied to an animation character, a game character, or the like.
[0187] As still another example, when the number of polygons is 50,000 or more, because the number of polygons is large and muscles, clothing wrinkles, accessories, reflective surfaces, and the like require detailed expression, high-dimensional LoRA rank 64 or higher or full fine-tuning may be applied. For example, full fine-tuning or LoRA rank 64 or higher may be applied to a realistic human character for an AAA game, a CG model for a movie, a humanoid object in a live-action movie, or the like.
[0188] The number of joints and pose diversity 502 may be considered to determine whether the object is static or dynamic. When the number of joints is 40 or more or 20 or more pieces of various pose information are present, motion LoRA may be applied. For example, motion LoRA may be applied to an object including complex movements of arms, fingers, a spine, and the like.
[0189] Style characteristics and rendering characteristics 503 may be considered to determine a type of the object and characteristics of rendering. In the case of a cartoon, animation, or non-photorealistic rendering (NPR) style, style LoRA may be applied. In addition, when the object includes reflective characteristics or refractive characteristics such as metal, glass, or a wet surface, a lighting adapter may be applied.
[0190] Standard LoRA or Continuous LoRA may be applied according to a data source 504. As an example, when the data source is an original 3D model, that is, a mesh, standard LoRA may be applied by prioritizing shape accuracy. In the case of NeRF reconstruction data, Continuous LoRA may be applied because viewpoint and expression continuity is important. In the case of three-dimensional data represented by a Gaussian Splatting method, Fast LoRA may be applied by prioritizing processing speed and rendering training efficiency.
[0191] In addition to the items illustrated in FIG. 5A, the server 10 may analyze a content type of the object, and may apply one or more of the following candidate strategies according to a result of the analysis.
[0192] For example, when the content type of the object is a character type, the server 10 may select a Dynamic Motion LoRA method in consideration of characteristics in which motion and pose changes of the object frequently occur.
[0193] This method is designed to maintain motion consistency between frames around joint and rig information of a 3D character, and may enable training without structural distortion even in various poses and animation states.
[0194] In addition, a motion adapter for each viewpoint change may be applied together to reflect detailed changes such as an expression and a hand motion of the character.
[0195] For example, when the content type of the object is a background type or an environment type, the server 10 may select a Spatial Consistency LoRA method in consideration of characteristics in which lighting changes, a sense of distance, perspective distortion, and the like frequently occur in the same space.
[0196] This method may learn relative positional relationships of background structures observed from different viewpoints, for example, a wall, a road, a building, a natural terrain, or the like, to be matched, thereby maintaining the same spatial configuration even at a plurality of camera angles. Accordingly, the method may implement realistic 3D spatial expression by minimizing spatial discontinuity between images generated from multiple angles.
[0197] For example, when the content type of the object is a photorealistic human type, the server 10 may select a Photorealistic LoRA method to reflect optical characteristics based on actual photography, such as detailed texture of a human body, skin reflected light, and hair material.
[0198] This method may precisely reproduce a stereoscopic effect, reflected light, and color contrast of a subject by using together lighting data based on high dynamic range (HDR) and metadata corresponding to actual camera parameters, for example, a lens focal length, an aperture value, ISO, and the like. The method may improve a synthesized result of 3D rendering data while maintaining photorealism at a level similar to a live-action photograph.
[0199] The server 10 may apply one of the candidate methods described above alone, or may apply Dynamic Motion LoRA and Spatial Consistency LoRA in combination when a plurality of content types are mixed.
[0200] For example, when a background scene including a person is generated, naturalness of motion and spatial matching may be secured simultaneously by applying dynamic motion LoRA to a character and applying spatial consistency LoRA to a surrounding environment in parallel.
[0201] The server 10 may support the image generation model to process various types of content, such as a character, a background, and a photorealistic human, according to respective characteristics by subdividing the attribute information of the three-dimensional data and selecting different LoRA training modules for respective types and characteristics of objects.
[0202] Therefore, by allowing the image generation model to adaptively select an optimized training path according to data attributes without relying on a training pattern of a single structure, an effect of improving realism, consistency, and expression diversity of a target image may be provided.
[0203] FIG. 5B is a flowchart illustrating a method of operating the server 10 for generating one image generation model by integrating composite candidate training methods based on attribute information of three-dimensional data according to an embodiment.
[0204] Referring to FIG. 5B, in operation S510, the server 10 may obtain attribute information of three-dimensional data. For example, the server 10 may analyze an input three-dimensional character model, and may obtain attribute information indicating that the number of polygons is about 12,000, the number of joints is 52, and a rendering style is a 3D cartoon, that is, a non-photorealistic rendering (NPR) format.
[0205] In operation S520, the server 10 may determine a plurality of candidate training methods based on the attribute information of the three-dimensional data. For example, the server 10 may determine a candidate training method mapped to an item corresponding to the attribute information. Referring to the description of FIG. 5A, the server 10 may select standard-capacity LoRA (rank 32) as a basic candidate appearance training method because the number of polygons corresponds to an intermediate level, may select motion LoRA as candidate training because the number of joints is greater than 40, and may select style LoRA as candidate training because the style is cartoon.
[0206] In operation S530, the server 10 may prepare training according to the plurality of candidate training methods.
[0207] The server 10 may prepare training by using a composite training method rather than a single training method. The server 10 may secure visual identity, motion diversity, and style consistency of the object simultaneously by applying a composite fine-tuning architecture rather than a single training module.
[0208] For example, the server 10 may set Base LoRA (rank 32) as a base model for maintaining a basic shape, proportion, and identity of the character, may set motion LoRA (rank 48) in consideration of the number of joints, 52, of the character, and may add style LoRA as an independent training module to maintain style consistency. The server 10 may complete a composite fine-tuning architecture in which appearance, motion, and style characteristics are learned simultaneously by applying a plurality of LoRA modules to one object in parallel or sequentially
[0209] In this case, the server 10 may apply regularization based on weight orthogonalization in a training process to minimize interference between modules and preserve unique characteristics of each module. For example, by designing a loss function such that a weight vector of a LoRA module for learning "motion" maintains orthogonality with a weight vector of a LoRA module for learning "style," an influence of a motion change on a style may be minimized. Alternatively, at a time of inference, a desired combination result may be generated by dynamically adjusting and merging weights of each LoRA module.
[0210] Before performing training, the server 10 may prepare a data configuration and an execution procedure necessary for training. For example, the server 10 may be set to secure at least 30 pieces of various pose data for motion training, and to prepare at least 60 reference images or frames for learning style characteristics. As a training execution method, Base LoRA, motion LoRA, and style LoRA may be sequentially trained in this order. Alternatively, the three modules may be trained in parallel and then integrated by weight merging or adapter stacking.
[0211] In addition, LoRA training may be implemented by a parameter-efficient fine-tuning (PEFT) method. LoRA may be characterized by a lightweight structure in which a small-scale matrix-type adapter is added next to an existing network and trained without retraining entire weights of an existing model.
[0212] As a specific example, based on hundreds of automatically generated two-dimensional rendered images and corresponding detailed metadata, that is, Mdata, the server 10 may automatically generate refined captions, for example, prompts, reflecting content, lighting, a style, a viewpoint, and the like of each image, and may train a LoRA adapter based thereon.
[0213] The trained LoRA result may generally be generated as a lightweight model file having a size of tens to hundreds of megabytes (MB), may be easily stored and shared, and may be combined in multiple ways during execution.
[0214] For example, by combining one character LoRA with another style LoRA, a user may easily implement a composite production such as "applying a Ghibli-style background to a specific character or a version in which the specific character wears another costume." Accordingly, flexible object adaptation and style transfer through a combination of multiple LoRAs may be supported.
[0215] In addition, when precise IP identity injection, that is, identity-preserving fine-tuning, is required, namely, for an object for which visual changes are not allowed, such as a main character or a brand mascot, the server 10 may perform a DreamBooth-based automated pipeline.
[0216] The DreamBooth method may automatically select dozens of images having the most ideal composition and lighting conditions from a 3D asset, designate the images as "instance images," and automatically generate, for each image, a text prompt including a unique identifier, that is, an instance token, such as "a photo of sks-character."
[0217] In addition, the server 10 may perform stable training by using "class-preservation images" together so as not to lose a general concept of a person or character previously trained. In this process, the server 10 may cross-train instance images and class-preservation images, and may fine-adjust some of entire U-Net weights of the model, thereby optimizing unique visual features of a specific person or IP object to be directly connected to a linguistic token.
[0218] In addition, through the process, a unique name "sks-character" may be learned as a digital identity corresponding one-to-one to visual attributes of the corresponding IP, such as a shape, a proportion, a color, and a texture.
[0219] The server 10 applies a composite fine-tuning architecture rather than a single training module, and provides an integrated training framework that supports both lightweight training based on parameter-efficient LoRA and precise identity training based on DreamBooth. Accordingly, the server 10 may automatically select and combine the most efficient and stable training methods according to characteristics of an object, and may implement a high-quality generative AI model that can flexibly respond to various styles and versions while preserving visual identity of unique IP.
[0220] In operation S540, the server 10 may execute the plurality of candidate training methods and integrate modules through verification.
[0221] The server 10 may quantitatively evaluate a training result of each module. For example, the server 10 may determine quality based on semantic alignment between a text prompt and a generated image, an error between a predicted pose and actual skeleton joint coordinates, similarity of color and texture distributions between styles before and after training, structural consistency with metadata-based depth and normal information, and the like. When a specific item is equal to or less than a preset criterion, the server 10 may selectively retrain only a corresponding module or may improve performance by increasing a LoRA rank.
[0222] The server 10 may integrate Base LoRA, motion LoRA, and style LoRA modules into one image generation model. When a plurality of modules are trained in parallel, integration may be performed using weight merging or a stacked structure. Together with the completed model, a basis for selecting the training method, for example, attribute values such as the number of polygons, the number of joints, and a style type, an applied rule ID, a final LoRA rank value, and the like may be recorded and stored in a meta log format. The stored information may be used as basis data when a model is updated, retrained, or provided as a user-customized model.
[0223] FIG. 6 is a diagram for describing an operation of an image generation model for performing training using a training data set according to an embodiment.
[0224] Referring to FIG. 6, the image generation model may receive the training data set 330 as an input, update parameters based on a predetermined training method (S610), and update the model using quality information on a generated image after completion of training (S620). Here, the training data set 330 may include a two-dimensional image rendered at each viewpoint, metadata having three-dimensional attribute information on the two-dimensional image, and camera metadata.
[0225] Preferably, the "image generation model," for example, a Diffusion U-Net, may operate based on a "multi-conditional architecture." This architecture is designed such that a noise predictor, that is, a U-Net, of a diffusion model simultaneously receives conditioning from a plurality of sources. For example, a text prompt may be injected into a cross-attention layer through a text encoder, for example, CLIP, a "3D control map," for example, a depth map, a pose map, or the like, may be injected into each decoder block of the U-Net through a lightweight encoder such as ControlNet, an "IP-specific model," that is, LoRA, may be directly merged into weights of the U-Net, and a "reference image," for example, for a style, may be injected into the cross-attention layer through a separate image encoder.
[0226] In operation S610, the image generation model may perform training using training data based on a training method determined from attribute information of three-dimensional data. For example, the training method may include a full fine-tuning method for retraining an entire model, a low-rank-based parameter addition method, a method of injecting a specific object or style, and a method of inserting a modular adapter.
[0227] The image generation model may independently determine or combine and apply the training method based on at least one of a modification range of model parameters, a scale of data, a type of a style, and a type of an object. For example, the image generation model may construct an image generation model reflecting all of joint movement, surface texture, and lighting responses by simultaneously applying Base LoRA, motion LoRA, texture LoRA, and a lighting adapter as a base model for maintaining a basic shape, proportion, and identity of a humanoid character.
[0228] Meanwhile, the image generation model may sequentially perform training on Base LoRA, motion LoRA, texture LoRA, and the lighting adapter. Specifically, the image generation model may first learn an appearance, a proportion, and an identity using Base LoRA, next learn joint rotation and motion expression using motion LoRA, and then learna style or optical response using texture LoRA or the lighting adapter.
[0229] In addition, the image generation model may perform training on Base LoRA, motion LoRA, texture LoRA, and the lighting adapter in parallel. After independently training each of the plurality of modules, the image generation model may integrate the plurality of modules by weight merging or adapter stacking.
[0230] In operation S620, the image generation model may update the image generation model based on quality information on the generated image.
[0231] Specifically, the update process may be implemented as an "automated quality evaluation and feedback loop."
[0232] In an embodiment, the server 10 first defines a unique identity of IP, for example, a specific color value, a body proportion, or a unique feature, as a "machine-readable IP guideline file," for example, JSON. During a training process, the server 10 periodically generates a test image, automatically compares the test image with the "guideline file," and calculates an "IP consistency score." If the score is less than a preset reference value, the server 10 performs an "automatic retraining loop" for increasing a weight of related data or adjusting a training parameter so as to satisfy the corresponding guideline, for example, "proportion" or "color."
[0233] In particular, the quality information may include an indicator for quantitatively measuring "geometric 3D consistency."
[0234] In an embodiment, during the training process, the server 10 generates two test images by requesting that the same object be rendered from two different virtual camera viewpoints, that is, View 1 and View 2.
[0235] Thereafter, the server 10 analyzes "epipolar geometry" between the two images, and calculates an "epipolar error" by determining whether a feature point of one image is accurately positioned on an epipolar line of the other image.
[0236] When the error value is less than a preset reference value, this proves that the two generated images are mathematically different viewpoints of the same 3D object. When the error value is large, the server 10 may automatically retrain the model in a direction of increasing a three-dimensional structure matching degree.
[0237] In addition, the update operation may include an "active learning" feedback loop. For example, when the server 10 identifies a "weakness" in which a primarily trained model generates a particularly low-quality image under a specific condition, for example, "a 45-degree angle looking up from below," the server 10 may go beyond simply adjusting parameters and may provide feedback to an operation of obtaining a training data set, thereby instructing additional automatic generation of a rendered image and metadata of the "corresponding weak 45-degree angle" from an original 3D model. The server 10 performs secondary training, that is, retraining, using this reinforced data set, thereby compensating for the weakness of the model and updating quality.
[0238] Quality information may mean a set of indicators quantitatively representing how accurately an image generated by the image generation model reproduces an appearance, a structure, a motion, a style, and viewpoint consistency of an object. The quality information may be used as a criterion for determining whether to terminate training, whether partial retraining is necessary, or whether parameter adjustment is necessary.
[0239] As a specific example, the quality information may include at least one of text-image matching degree, pose and motion accuracy, three-dimensional structure matching degree, style consistency or preservation degree, and optical characteristic reproduction degree.
[0240] The text-image matching degree may mean an indicator for evaluating how semantically consistent an input text prompt or condition statement and a generated image are. For example, the text-image matching degree may be calculated by a multimodal embedding-based similarity measure such as CLIP Score.
[0241] The pose and motion accuracy may be an indicator representing how accurately a joint position or posture of an object in the generated image matches a target pose. For example, the pose and motion accuracy may be evaluated by a mean squared error (MSE) of skeleton keypoints or a Percentage of Correct Keypoints (PCK) value.
[0242] The three-dimensional structure matching degree may be an indicator for evaluating whether rendered depth information, normal information, or silhouette information is reproduced identically to original three-dimensional data or metadata. For example, the three-dimensional structure matching degree may be measured by an SSIM (Structural Similarity Index) of a depth map and a normal map or an IoU (Intersection over Union) value of a silhouette.
[0243] Style consistency or preservation degree may indicate a measure of whether visual style elements, such as a color distribution, a texture pattern, and contour characteristics, match training data or a target style. For example, the style consistency or preservation degree may be measured using Kullback-Leibler (KL) divergence between color histograms, a Gram Matrix-based texture loss, Learned Perceptual Image Patch Similarity (LPIPS), or the like.
[0244] The optical characteristic reproduction degree may be an indicator representing whether a generated image accurately reflects a lighting direction, reflected light, or a highlight element in an actual or virtual environment. For example, the optical characteristic reproduction degree may be evaluated based on a BRDF (bidirectional reflectance distribution function) error or representation accuracy of specular reflection.
[0245] The image generation model may update the image generation model by adjusting, based on the quality information, a weight of a parameter associated with a feature having an image quality evaluation score lower than a preset reference value or a weight of a loss function of the image generation model.
[0246] In addition, when the quality information on the generated image does not satisfy a preset reference value, the image generation model may update the image generation model by selectively performing retraining or adjusting a parameter with respect to a training module corresponding to each quality item.
[0247] For example, when pose accuracy is equal to or less than a reference value, the image generation model may retrain only a motion LoRA or pose adapter module. When style consistency is insufficient, the image generation model may perform rank expansion or additional training with respect to style LoRA.
[0248] In addition, when structural matching with a depth map or a normal map is evaluated to be low, the image generation model may increase a geometry consistency loss using Z-buffer-based metadata. In addition, when lighting reproducibility is insufficient, the image generation model may perform correction training for each module by adjusting a lighting adapter or an ambient light parameter.
[0249] For example, quality evaluation may be periodically performed. For example, quality of an image generated every 100 steps may be quantitatively measured, and when a measurement result does not satisfy a preset threshold, the server 10 may perform a correction procedure.
[0250] As a specific example, quality evaluation may be performed using multiple criteria such as CLIP Score, FID Score (Frechet Inception Distance), an Identity Preservation (IP) consistency index for evaluating identity consistency, and camera parameter accuracy. Each indicator may be defined as follows.
[0251] Contrastive Language-Image Pre-training (CLIP) Score is an indicator representing semantic alignment between a generated image and a text prompt, and may evaluate how much the model has generated an image matching content of the prompt.
[0252] Frechet Inception Distance (FID) Score is an indicator quantifying a difference between a distribution of a generated image and a distribution of an actual data set, and a lower value indicates more realistic image quality.
[0253] Identity Preservation (IP) consistency is a value representing a degree to which a visual identity of the same object, for example, a specific character or person, is maintained, and may measure similarity of major visual features such as a proportion, a color, a face shape, and a hairline of the object.
[0254] Camera parameter accuracy is an indicator representing how much a composition and perspective of a generated image match three-dimensional camera metadata, and may include restoration accuracy for a lens focal length, a viewpoint angle, an aperture value, and the like.
[0255] In addition, in the present specification, camera accuracy or camera parameter matching degree may be understood as a quality evaluation indicator that is the same as or corresponds to the camera parameter accuracy. For example, the camera accuracy or the camera parameter matching degree may indicate a degree to which a viewpoint, perspective, composition, lens effect, or depth of field of the generated image matches camera metadata included in the training data set or camera setting information input when generating a target image. In addition, the camera parameter accuracy may further include a degree to which a background blur degree, lens distortion, focal length effect, or aperture effect of the generated image matches the camera metadata or the camera setting information.
[0256] The server 10 may update the model in real time by applying the following automatic adjustment rules based on a result of evaluating the quality indicators.
[0257] For example, when CLIP Score < 0.85, the server 10 may determine that the image generation model has not sufficiently learned a visual interpretation of a text instruction, that is, a prompt, and may decrease a learning rate by 10% and increase a ratio of "hard samples" to perform retraining. Through this, the text-image matching degree may be gradually improved.
[0258] As another example, when FID Score > 15, the server 10 may determine that a visual distribution of a generated image excessively deviates from an actual image, and may adjust the model to maintain a training balance by increasing normalization strength or a stabilization coefficient of a loss function.
[0259] As another example, when IP consistency < 0.9, the server 10 may determine that the image generation model has failed to consistently maintain a visual identity of the same object, and may improve identity preservation degree by strengthening a weight for a feature embedding of the corresponding object and performing retraining based on an instance image.
[0260] As another example, when camera parameter accuracy < 0.9, the server 10 may determine that the generated image has not sufficiently reflected an optical characteristic corresponding to camera metadata, for example, a lens, a focal length, an aperture, or the like, and may improve viewpoint and composition matching by increasing a camera conditioning weight.
[0261] The automatic adjustment procedure described above may be performed fully automatically without user intervention during the training process, and an evaluation result of each quality indicator may be logged in real time and reflected in an update cycle of the model. That is, the server 10 may maintain an optimal training state by executing a quality evaluation module at every predetermined step and dynamically readjusting a learning rate, normalization strength, feature weight, and the like according to a condition for each indicator.
[0262] Through the training and retraining described above, the server 10 may generate an image generation model in which consistency of an object is maintained even according to changes in an environment and a viewpoint. As a specific example, the server 10 may generate an image generation model capable for generating an image in which a ratio between a body and a head is constantly maintained at 1:1.2 at all angles, RGB values of a brown portion are consistently maintained at (101, 67, 33) even in consideration of lighting changes, and detailed features such as six whiskers, round eyes, and triangular ears geometrically match accurately.
[0263] FIG. 7 is a flowchart illustrating a method of operating the server 10 for obtaining a target image using an image generation model according to an embodiment.
[0264] Referring to FIG. 7, in operation S710, the server 10 may obtain user input information including a text prompt. The text prompt may mean condition information of a target image or scene to be generated. The text prompt may be information representing, in natural language, content, an appearing object, an action, a background, lighting, a style, an atmosphere, and the like of the target image or scene to be generated. For example, the server 10 may obtain a text prompt such as "a scene in which a woman wearing a black dress dances under a spotlight at the center of a stage."
[0265] In addition, the server 10 may further obtain a reference image. The reference image may be used as condition information for image generation together with the text prompt.
[0266] In operation S720, the server 10 may arrange a position and a composition of an object in a three-dimensional space based on the user input information. The server 10 may determine whether three-dimensional scene data is required to be configured based on the user input information. The three-dimensional scene data may include pose and spatial arrangement information of the object. As a specific example, when a scene in which "two people are standing back to back" is input as the text prompt, it may be required to configure the three-dimensional scene data because it is difficult to generate an accurate image without camera matching and spatial configuration of the objects based on the three-dimensional space. In this case, when a three-dimensional image having metadata having three-dimensional attribute information is input as the reference image, it may be unnecessary to configure the three-dimensional scene data.
[0267] The server 10 may obtain setting information in which at least one of character setting, camera composition setting, and lighting arrangement is set in the three-dimensional space. The server 10 may configure an overall shape of the scene by arranging the position and the composition of the object in the three-dimensional space based on the setting information. Configuring the overall shape of the scene may be referred to as three-dimensional blocking. In this case, the server 10 may use previously generated three-dimensional scene data or newly perform a three-dimensional blocking operation based on the user input information.
[0268] On the other hand, when configuring the three-dimensional scene data is not required, the server 10 may generate the target image using the text prompt and the reference image. As a specific example, when a text prompt saying "make the background blurry" is input, it may be unnecessary to configure the three-dimensional scene data.
[0269] In operation S730, the server 10 may extract a control map based on the three-dimensional scene data. The control map may mean data including a spatial structure or geometric information on two-dimensional data generated by rendering the three-dimensional scene data. For example, the control map may include one or more of: a depth map including distance information from a camera; a surface normal map including a direction vector in which a surface of a character or object faces; a pose map in which a skeleton or a joint position of a human body is projected and displayed on a two-dimensional plane; an edge map emphasizing a contour or outer shape of the object; an expression map including emotion expression data obtained by quantifying movements of eyes, a mouth, eyebrows, and the like based on facial rigging or blendshape data of a 3D character; a camera map including a camera position, orientation, lens focal length, aperture value, field of view, and the like in the three-dimensional space; a lighting map including information on a position, color temperature, intensity, shadow direction, and specular highlight of a light source in the three-dimensional scene; a motion map, as a control map representing a pixel movement vector between consecutive frames, which represents in which direction and at what speed the object moves; and an ambient map representing a soft contact shadow generated when objects in the 3D scene come close to one another.
[0270] The control maps may be used in a single form, but according to embodiments, a plurality of control maps may be combined in parallel or hierarchically to form a more precise structural condition.
[0271] For example, a shape of a space may be defined by combining the depth map and the surface normal map, and optical responses and viewpoint information may be simultaneously reflected by additionally applying the lighting map and the camera map thereto. When multiple control maps are combined and trained in this manner, the image generation model may generate a high-precision image that understands both 3D spatial information and visual interaction beyond a simple 2D image level.
[0272] In operation S740, the server 10 may obtain the target image based on the control map and the image generation model.
[0273] The server 10 may obtain, through the image generation model, the target image in which detail of the object is reflected, based on a visual structural feature set in the control map. Here, the control map is data including a spatial structure or geometric information obtained from the three-dimensional scene data, and may serve as a guide that enables the image generation model to generate an image while maintaining an accurate shape, composition, and viewpoint, rather than merely performing text-based image generation.
[0274] Specifically, the server 10 may convert the text prompt into an embedding vector using a text encoder, and may input the embedding vector to the image generation model as condition information.
[0275] In this case, the condition information input to the image generation model may be configured according to various embodiments depending on a purpose of a user and characteristics of the three-dimensional scene data.
[0276] (First embodiment: generation based on a 3D structure map) As a first embodiment, the server 10 may use three-dimensional structure information extracted from the three-dimensional scene data, for example, a depth map, a pose map, a surface normal map, or the like, as the "control map." This method focuses on generating, from noise, an image that accurately follows a geometric structure of the three-dimensional scene, and may strongly control a shape, a composition, and perspective of the object.
[0277] (Second embodiment: rough render-based conversion) As a second embodiment, the server 10 may generate a "rough render image" in which some details are omitted from the three-dimensional scene, and may use the rough render image as an input image of the image generation model. In this case, the image generation model may perform img2img conversion or inpainting based on the rough render image, and may correct or convert detail, texture, or a style of the image according to an instruction of the text prompt.
[0278] (Third embodiment: composite control method) As the most preferable third embodiment, the server 10 may perform composite control by combining the two methods. That is, the server 10 may enforce structural constraints, such as pose and depth, through a three-dimensional structure map, that is, the control map of the first embodiment, and may simultaneously control a "style and visual feature," for example, color, lighting, and a unique appearance of a character, by receiving together the text prompt and / or a "color reference image," for example, a rough render image or a separate reference image.
[0279] As a preferable embodiment, the "user input information" may be obtained through a dedicated Digital Content Creation (DCC) tool plugin, for example, a plugin for Blender or Maya, operating on a client device of an artist.
[0280] The server 10 may input the control map through a control-dedicated path or a cross-attention structure so that visual structural information on a position, a shape, a posture, a viewpoint, a direction, and the like of the object is maintained in the image generation model. For example, when a direction of an arm or a head of a character is set in the control map, the image generation model may generate the target image by supplementing detail and texture while the direction of the arm or the head of the character set in the control map is maintained.
[0281] The target image generated through the image generation model may not merely reflect a text description corresponding to the text prompt, but may be a result reflecting metadata and camera metadata based on the three-dimensional scene data. For example, when a text prompt saying "a knight holding a sword is standing while looking to the left" is input, and pose information and viewpoint information of the knight are given in the control map, detailed elements such as an angle of an arm, a position of a weapon, a direction of a face, and a rotation angle of a body may be accurately reflected in the target image.
[0282] In addition, in an image generation process, the server 10 may reduce a problem in which a boundary is unclear or a shape is distorted by using the depth map and the surface normal map. In addition, in the image generation process, the server 10 may generate an image in which rotation and movement of joints are accurately expressed without damage(or deformation, artifacts) to a skeleton structure by using the pose map. Here, the generated image may simultaneously satisfy a style requested by the user, a consistent structure, a natural pose, and an accurate viewpoint.
[0283] Unlike a method of operating the image generation model based only on the text prompt, the server 10 may generate a more accurate and consistent target image by combining the control map and the trained image generation model.
[0284] FIG. 8 is a flowchart illustrating a method of operating the server 10 for setting three-dimensional scene data according to an embodiment.
[0285] Referring to FIG. 8, in operation S810, the server 10 may set a pose of an object based on user input information.
[0286] In this case, a method by which the server 10 obtains setting information of a three-dimensional scene, for example, a pose of the object, an expression, a camera composition, or the like, may have various embodiments.
[0287] (First embodiment: direct control by an artist) As a preferable embodiment, the "user input information" may be "3D blocking scene data" itself, which is directly set by an artist, that is, a user, in an external three-dimensional Digital Content Creation (DCC) tool, for example, Blender, Maya, or the like. In this case, the server 10 receives, from a client of the artist, specific "setting information" such as a joint rotation value, a camera position, and an expression blendshape value of the object included in the 3D blocking scene, and obtains the setting information.
[0288] (Second embodiment: semi-automatic or automatic generation by the server) As another embodiment, the server 10 may semi-automatically or automatically generate the setting information of the three-dimensional scene. For example, when the user provides a simplified text prompt or a reference image, such as "a cat walking," as the "user input information," the server 10 may automatically set or arrange a pose and a camera composition corresponding to a "walking motion" based on a 3D model and a motion library held by the server 10. This method enables even a user unfamiliar with 3D work to obtain a result.
[0289] (Third embodiment: mixed control method) As still another embodiment, a "mixed control method" combining the first embodiment and the second embodiment is also possible. In this method, the server 10 first automatically sets and arranges an "initial composition" of the three-dimensional scene based on a simplified input of the user, for example, text, as in the second embodiment. Thereafter, the artist may load the initial composition into a DCC tool, that is, a client, and may additionally and precisely control a camera angle, a pose, an expression, and the like as in the first embodiment. The server 10 receives, from the client, "final setting information" for which the precise control by the artist has been completed, and finally obtains the final setting information. This mixed method may maximize work efficiency by combining a rapid automatic arrangement function of AI and precise control ability of the artist.
[0290] When a three-dimensional production program is executed, a three-dimensional model corresponding to the object may be loaded in the three-dimensional space. The three-dimensional model has a previously prepared skeleton structure, and rotation values of major joints, such as a shoulder, an elbow, a knee, and a hip joint, for example, a rotation angle or a position coordinate based on an FK or IK method, may be adjusted based on the user input information.
[0291] Setting the pose of the object using the three-dimensional model in the three-dimensional space may serve as a basis for a motion direction, a center of gravity, and emotional expression of the object in a target image, and may also be associated with a camera viewpoint.
[0292] For example, setting the pose of the object may be performed based on information extracted from a reference image or a text prompt. As a specific example, when a scene in which "a knight holding a sword looks to the left and takes an attacking pose" is input as the text prompt, the server 10 may automatically or manually adjust a head direction, a neck rotation, an angle of an arm, a position of a hand, a degree to which legs are spread, and the like, thereby generating a rough pose.
[0293] For example, a result of setting the pose of the object may be composed of skeleton data including rotation and position information of each joint of the object, a pose image indicating a current pose state, and a pose map representing the pose on a pixel basis.
[0294] In operation S820, the server 10 may set an expression of the object. The expression of the object may be set using facial rigging technology that simulates muscle deformation of a facial region. For example, movements of eyebrows, eyelids, mouth corners, a jaw, cheeks, and the like may be adjusted through a blendshape.
[0295] For example, in order for a character to have an angry expression, values such as "lowering eyebrows (brow_down = 0.5), tightly closing a mouth (mouth_close = 0.7), and moving eye gaze to the left (eye_left = 0.6)" may be adjusted. When setting values for the expression are converted into a control map, the image generation model may also learn an emotion-based facial shape change and reflect the emotion-based facial shape change during image generation.
[0296] In addition, when setting the expression, the server 10 may prevent unnatural distortion by adjusting a gaze direction and a head tilt together based on the user input information so as to be consistent with a head direction, a neck rotation, and the like set in the pose operation.
[0297] In operation S830, the server 10 may set a camera composition. The camera composition may be an element that determines a viewpoint and a composition of a scene shown in the target image. For example, the server 10 may set a position and a rotation of a virtual camera in the three-dimensional space based on the user input information, and may select a focal length, a sensor size, and an aspect ratio.
[0298] Information on the set camera composition may be stored as camera metadata. The camera metadata may serve as a reference for reproducing the same view during multi-view rendering, and may be used as a condition signal for generating, in the image generation model, an image matching a camera position and a viewpoint.
[0299] In operation S840, the server 10 may set an arrangement of lighting. An atmosphere of a scene and texture expression of the object may be set through lighting. For example, the arrangement of lighting may be set with at least one of a position, a direction, brightness, a color temperature, and shadow intensity of each light. Preferably, information on 3D "virtual lighting" arranged by the artist in the operation may be extracted in a form of a "lighting map." The "lighting map" does not merely mean rendered lighting information, but may operate as a "control signal" that conveys a "style and atmosphere of lighting," for example, dramatic Rembrandt lighting or strong rim light, intended by the artist, to the AI model.
[0300] In this case, the AI model may combine an ability previously learned in various lighting environments, that is, a lighting separation ability, with the input "lighting map," and may realistically generate and reflect, that is, perform generative relighting of, a lighting effect intended by the artist in a final image.
[0301] FIG. 9 is a flowchart illustrating a method of operating the server 10 for generating a target image based on a control map according to an embodiment.
[0302] Referring to FIG. 9, in operation S910, the server 10 may input a control map to an image generation model as condition information. For example, the server 10 may input a control map extracted from three-dimensional scene data to the image generation model as condition information. The condition information may be used as information for structure preservation.
[0303] For example, the control map may include a depth map including distance information from a camera, a surface normal map including a direction vector in which a surface of a character or object faces, a pose map in which a skeleton or a joint position of a human body is projected and displayed on a two-dimensional plane, and an edge map emphasizing a contour or outer shape of the object.
[0304] For example, when a text prompt and a reference image are provided together, the server 10 may convert the text prompt and the reference image into embedding vectors through a text encoder and an image encoder, and may combine the embedding vectors with a control map embedding to configure a model input packet.
[0305] In operation S920, the server 10 may generate an intermediate image corresponding to the condition information based on the image generation model. For example, the server 10 may provide condition input information to the image generation model and perform noise removal iterations, thereby generating an intermediate image reflecting a structure of the control map. The intermediate image may not have complete color, texture, gloss, and the like, but may satisfy morphological constraints such as a pose and joint arrangement, a camera composition and perspective, shading, and an object boundary. In addition, the server 10 may sequentially inject multiple control maps to gradually fix the structure, or may inject the multiple control maps in parallel to adjust a weight for each control map. In addition, the server 10 may maintain viewpoint consistency by using camera metadata.
[0306] In operation S930, the server 10 may generate a target image by reflecting detail generation and style information with respect to the intermediate image.
[0307] For example, the server 10 may receive the intermediate image as an input and reinforce detailed texture and optical responses through a detail generation module. Specifically, detailed details may include skin pores, wrinkles and stitches of clothing, strand representation of hair, specular highlights of metal and glass, a contribution of global illumination, a contact shadow, and the like.
[0308] The server 10 may generate the target image by reflecting a style desired by a user or a predefined style condition, that is, a style prior, in the intermediate image. Here, the "style condition" may represent information specifying in what atmosphere or texture an image should be expressed. For example, the style condition may include a realistic style, that is, Photoreal, an animation or cartoon style, a painterly style, and the like.
[0309] In particular, when the style condition is designated as non-photorealistic rendering (NPR), such as an "animation or cartoon style," the server 10 may perform a preprocessing operation of intentionally modulating or simplifying an extracted "original control map" without using the original control map as it is.
[0310] For example, for "cartoon rendering," the server 10 may posterize a "depth map" to simplify depth stages, or for "2D animation," may ignore a "depth map" and a "normal map" and use only a "pose map" and a "Canny edge map" to enforce a clear line art structure. The server 10 injects this "modulated control map" into a model in which the corresponding style has been learned, thereby generating a result that accurately follows a 3D structure while perfectly expressing the style in a non-photorealistic manner.
[0311] For example, when the style condition is set to a realistic style, the server 10 may preferentially apply lighting responses and material expressions based on a rendering principle so that surfaces of metal, cloth, glass, leather, and the like reflect or absorb light like actual materials, and thus physical characteristics of the materials may be accurately expressed as if the materials were photographed with an actual camera.
[0312] For example, when the style condition is set to an animation or cartoon style, the server 10 may emphasize simplified light and shade and line expression unique to animation rather than complex light responses.
[0313] For example, when the style condition is set to a painterly style, the server 10 may add brush strokes, color bleeding, and texture expression so that the target image looks like an actual painting.
[0314] In a style application process, the server 10 may use a specific module so that the trained image generation model can change only a style without losing an original shape or structure of the object. For example, the specific module may be a style-preservation loss or a style-dedicated adapter module, for example, Style LoRA or Style Adapter. The image generation model may operate in a manner of "changing only the style while maintaining the object as it is."
[0315] In addition, the present disclosure may be applied not only to still image generation but also to temporally consistent video generation. In an embodiment, an artist may set, as "3D blocking," not a single pose but a "3D animation," for example, a walking motion, having a specific time, for example, 2 seconds or 48 frames, in a three-dimensional space.
[0316] In this case, the server 10 extracts the control map not as a single frame but as a time series, for example, a sequence of 48 consecutive pose maps. The image generation model may receive the entire "time-series control map" as a condition, and may generate a high-quality video in which the 3D structure of each frame perfectly matches an original 3D animation and an appearance, clothing, and texture of a character are temporally consistent between frames.
[0317] FIG. 10 is a flowchart illustrating a method of operating the server 10 for correcting a target image based on quality information on the target image according to an embodiment.
[0318] Referring to FIG. 10, in operation S1010, the server 10 may obtain quality information on a target image generated through an image generation model. Here, the quality information may include information on at least one of a matching degree between text input for image generation and an image, a proportion and a color reflecting consistency of an object, a three-dimensional composition matching degree, realism of the image, and a matching degree between the image and a camera parameter. That is, the quality information may mean a result of evaluating whether the image maintains an accurate structure, whether noise or distortion has occurred, whether a style is consistently maintained, whether resolution and detail are sufficient, and the like.
[0319] Specifically, structural accuracy may be evaluated by an indicator such as Structural Similarity Index (SSIM) or LPIPS, based on how much depth, a surface normal, or the like of the generated image matches an original control map.
[0320] Whether noise distortion occurs may detect whether an abnormal phenomenon such as blur, a glitch, damage(or deformation, artifacts) to a finger or an eye, or the like exists in the image.
[0321] Style consistency may be evaluated through color histogram comparison, a style loss, or the like, based on whether the generated image accurately follows a designated style, for example, live-action, cartoon, animation, or the like.
[0322] Detail and sharpness may be checked by edge information analysis or a texture similarity indicator, based on whether skin texture, a fabric pattern, a contour, or the like is clearly expressed.
[0323] The server 10 may analyze the quality information, and may correct the target image or perform regeneration, if necessary, in a subsequent operation.
[0324] In operation S1020, the server 10 may correct the target image based on the quality information. For example, the server 10 may adjust, based on the quality information, a weight of a parameter associated with a feature having an image quality evaluation score lower than a preset reference value or a weight of a loss function of the image generation model.
[0325] Preferably, the correction operation may include a "structure-aware iteration" loop including an "interactive correction" input of an artist, that is, a user.
[0326] In an embodiment, when the artist is not satisfied with a specific region, for example, a "shoulder part of a character," of a final image generated by AI, the artist may designate the corresponding region as a mask on a DCC tool plugin and request "correction."
[0327] In this case, unlike general 2D inpainting, the server 10 loads together a "control map of an original 3D blocking scene," for example, a depth map or a pose map, corresponding to the corresponding mask region. The server 10 regenerates the mask region based on a correction prompt of the artist, for example, "make the shoulder armor more rounded," and performs inpainting by forcing a "3D control map structure of the corresponding region," for example, a 3D shape and depth of the shoulder, not to be violated, that is, by constraining the 3D control map structure. This is a core workflow that enables correction while maintaining 3D consistency.
[0328] In addition, when an automatic correction attempt fails more than a preset number of times, for example, three times, the server 10 may perform an "automatic fallback" procedure. This is to stably provide even a next-best result instead of a fatal error by automatically switching a currently used image generation model, for example, a "high-resolution realistic model," to a "fallback algorithm," for example, a "base model" having high stability although its speed is high, and performing regeneration.
[0329] For example, when a structure is distorted, for example, pose distortion or abnormal body representation, the server 10 may regenerate an image by inputting the original control map back into the model, or may regenerate the image with the same structure by strengthening a structure-preservation loss.
[0330] For example, when noise or an artifact is present, the server 10 may locally correct only the corresponding region through diffusion model-based noise removal, inpainting, or latent resampling, or may fine-adjust some of LoRA / adapter weights of the model so that the error is not repeated.
[0331] For example, when the style is not properly reflected, the server 10 may additionally apply Style LoRA or Style Adapter, or may regenerate an image while minimizing a style loss value using reference image-based style mapping. When a color error is present, the server 10 may correct the color error based on a brand palette.
[0332] For example, when detail or resolution is insufficient, the server 10 may upscale an image to a 2Х to 4Х high-resolution image using a super-resolution model, or may apply a detail correction module that additionally supplements skin texture, clothing texture, and the like, for example, by adjusting a Detail LoRA weight.
[0333] Through this process, the target image may be iteratively improved until user requirements, structural accuracy, and visual completeness are satisfied. In addition, when completeness is still insufficient even after automatic correction is performed a preset number of times, for example, three times, an output of the image generation model may be corrected. Here, correction may be performed not by retraining of the model, but by adjusting an inference parameter at a time of image generation.
[0334] The server 10 may periodically evaluate quality during an image generation process. For example, automatic correction may be repeatedly performed up to three times for the same input condition. When the quality information does not satisfy a reference value in each correction loop, the server 10 may improve output quality by automatically adjusting generation condition parameters described in the following operation. This iterative correction process may be automatically performed without user intervention.
[0335] The server 10 may adjust one or more inference parameters based on the quality information, for example, CLIP Score, FID Score, IP consistency, camera parameter accuracy, or the like. The inference parameters may include a guidance scale, a control map weight, a denoising strength, and a number of sampling steps.
[0336] The guidance scale is a value for adjusting fidelity of the model to the text prompt. As the value of the guidance scale increases, a generated image becomes more closely matched to the prompt, and semantic consistency is strengthened. As the value decreases, a more creative and free image may be generated. For example, when CLIP Score is calculated to be low, the server 10 may gradually increase the guidance scale, thereby improving a text-image matching degree.
[0337] The control map weight is a value for adjusting a reflection ratio of a control map such as a depth map, a pose map, and a surface normal map. When FID Score or structural consistency is equal to or less than a reference value, the server 10 may increase a ControlNet weight, thereby strengthening matching based on a 3D structure.
[0338] The denoising strength indicates an intensity of a noise removal operation during image generation. As this value increases, new detailed descriptions are strongly reflected, and as this value decreases, an original form is maintained. For example, when intellectual property (IP) consistency is evaluated to be low, the server 10 may decrease the denoising strength, thereby performing correction in a direction of maintaining a form of an existing object.
[0339] The number of sampling steps is a value indicating the number of denoising iterations in a diffusion process. As the number of sampling steps increases, detailed description is possible but a generation time increases. As the number decreases, speed is high but quality may be degraded. For example, when quality is equal to or less than a reference value, the server 10 may increase the number of sampling steps, thereby improving detail quality.
[0340] That is, when the reference value is not satisfied even after three automatic corrections, the server 10 may apply a fallback algorithm. The fallback algorithm may reset a combination of parameters and retry image generation in a direction of prioritizing structural matching over semantic restriction of the prompt by, for example, relaxing the guidance scale and increasing the ControlNet weight.
[0341] The quality-based automatic correction and parameter adjustment procedure described above has an advantage of immediately improving generation quality without retraining the model. That is, the server 10 may adapt to various quality requirements only by adjusting parameters in an inference stage while maintaining the same trained model. Through this, the server 10 may generate, in real time, a target image satisfying an optimized quality criterion.
[0342] The server 10 may form an automatic feedback loop of quality evaluation, correction, and regeneration, thereby implementing a self-adaptive quality improvement system capable of immediately responding to quality degradation without user intervention. As a result, visual quality, a matching degree, and intellectual property (IP) consistency of a target image generation result may be stably maintained.
[0343] An image finally secured in quality through the above-described operations may be provided as a result.
[0344] In addition, the present disclosure may include a "generative inverse texturing" workflow that uses a generated high-quality image to improve an original 3D asset.
[0345] In an embodiment, the server 10 may generate, through the technology of the present disclosure, an image sequence "ensured to be consistent from multiple angles," for example, a front image, a side image, a 45-degree image, and a rear image, for a specific object. The server 10 may refer to "camera metadata" stored together with these images, and may accurately perform "inverse projection" to determine which part, that is, which UV coordinate, of an original 3D model pixel information of each image corresponds to.
[0346] Through this process, high-quality detail generated by AI, for example, realistic fur texture, may be inversely applied as a "new texture" of the original 3D model, thereby automatically upgrading a low-resolution 3D asset to a high-resolution next-generation asset.
[0347] FIG. 11 is a block diagram illustrating a configuration of the server 10 according to an embodiment.
[0348] Referring to FIG. 11, the server 10 may include a communication device 1110, a memory 1120, and a processor 1130. However, all of the illustrated components are not essential components. The server 10 may be implemented by more components than the illustrated components, or may be implemented by fewer components than the illustrated components. Hereinafter, the components will be described. The server 10 illustrated in FIG. 11 may identically correspond to the server 10 described with reference to FIGS. 1 to 10.
[0349] The communication device 1110 may communicate with an external device. For example, the communication device 1110 may be connected to a network in a wired or wireless manner and may communicate with the external device. Here, the external device may be an electronic device.
[0350] The communication device 1110 may include a communication module supporting one of various wired and wireless communication methods. The communication module may be a short-range communication module or a wired communication module.
[0351] The memory 1120 may include at least one type of storage medium among a flash memory type, a hard disk type, a multimedia card micro type, a card-type memory, for example, an SD memory, an XD memory, or the like, RAM (Random Access Memory), SRAM (Static Random Access Memory), ROM (Read-Only Memory), EEPROM (Electrically Erasable Programmable Read-Only Memory), PROM (Programmable Read-Only Memory), magnetic memory, a magnetic disk, and an optical disk.
[0352] The memory 1120 may store at least one program for causing the server 10 to execute a method of operating the server 10, the method including obtaining three-dimensional data for an object, perform training using a training data set obtained from the three-dimensional data according to a training method determined from characteristics of the three-dimensional data, and obtaining an image generation model. The at least one program stored in the memory 1120 may be classified into a plurality of modules according to functions.
[0353] The processor 1130 controls an overall operation of the server 10, and may include at least one processor such as a CPU. The processor 1130 may include at least one specialized processor corresponding to each function, or may be a processor integrated into one.
[0354] The processor 1130 may execute a program stored in the memory 1120, may read data or a file stored in the memory 1120, or may store a new file in the memory 1120. In addition, the processor 1130 may execute instructions stored in the memory 1120.
[0355] The processor 1130 may obtain three-dimensional data for an object constituting an image. For example, the three-dimensional data may include structural information and spatial information of the object.
[0356] The processor 1130 may obtain, by rendering the three-dimensional data according to a plurality of viewpoints, a training data set including, for each of the plurality of viewpoints, a two-dimensional image and metadata having three-dimensional attribute information.
[0357] When rendering is completed, the processor 1130 may obtain metadata corresponding to the two-dimensional image generated for each viewpoint. The metadata may mean data including spatial, optical, and physical attribute information calculated in a process of rendering the three-dimensional data from the plurality of viewpoints. The metadata may correspond to the two-dimensional image of each viewpoint, and may include three-dimensional structure information such as depth information, surface normal information, occlusion information, and motion information. In addition, the metadata may include camera parameter information at the time of rendering, for example, a position, a focal length, an aperture value, or the like, and lighting condition information, for example, a light source position, intensity, a color temperature, or the like.
[0358] The metadata may be stored by being matched one-to-one with the two-dimensional image for each viewpoint, and may be used as an element constituting the training data set. That is, the training data set may be composed of a two-dimensional image derived from the three-dimensional data and metadata including three-dimensional attribute information.
[0359] In addition, the processor 1130 may obtain, as part of the training data set, camera metadata related to a setting of a camera corresponding to the plurality of viewpoints. Here, the camera metadata may include position and posture information of the camera in a three-dimensional space, and may include at least one photographing control parameter including a focal length, an aperture value, a shutter speed, a sensor size, ISO sensitivity, and white balance.
[0360] The processor 1130 may perform training using the training data set based on a training method determined from attribute information of the three-dimensional data.
[0361] For example, the processor 1130 may determine, from among a plurality of training methods, a training method corresponding to the attribute information of the three-dimensional data, and may perform training using the training data set based on the training method. The plurality of training methods may be classified based on at least one of a modification range of model parameters, an amount of training data used during training, a style of the object, and a type of the object.
[0362] For example, the attribute information of the three-dimensional data may be information representing structural complexity and visual expression characteristics of the three-dimensional data. The attribute information of the three-dimensional data may include information on at least one of a type of the three-dimensional data, a type of the object, a number of polygons for the object, a number of joints, a resolution of a texture, and a style. For example, the type of the object may include at least one of a person, an animal, and a background. In addition, the style may indicate whether the three-dimensional data is based on a live-action image or based on animation.
[0363] The processor 1130 may independently determine or combine and apply the training method based on at least one of the modification range of model parameters, a scale of data, a type of the style, and the type of the object. In addition, the processor 1130 may combine and compositely apply different training methods for respective detailed items of the attribute information of the three-dimensional data.
[0364] The processor 1130 may obtain an image generation model for the object based on a result of training.
[0365] In addition, the processor 1130 may obtain quality information on an image generated through the image generation model. The processor 1130 may update the image generation model based on the quality information. Here, the quality information on the image may include information on at least one of a matching degree between text input for image generation and the image, a proportion and a color reflecting consistency of the object, a three-dimensional structure matching degree, realism of the image, and a matching degree between the image and the camera parameter.
[0366] The processor 1130 may obtain user input information including a text prompt. The text prompt may mean condition information of a target image or scene to be generated.
[0367] In addition, the processor 1130 may further obtain a reference image. The reference image may be used as condition information for image generation together with the text prompt.
[0368] The processor 1130 may arrange a position and a composition of an object in a three-dimensional space based on the user input information. The processor 1130 may determine whether three-dimensional scene data is required to be configured based on the user input information. In this case, the server 10 may include a "mapping engine" that automatically converts camera-related natural language included in the text prompt into actual 3D camera parameters.
[0369] This engine uses various pieces of camera metadata knowledge learned in an "offline training" stage. For example, when text such as "a telephoto lens feeling" is input, the mapping engine may automatically set a "focal length (focal_length)" of a 3D scene to a range of [100 mm, 200 mm], and when text such as "a blurry background feeling" is input, may automatically set an "aperture value (f_stop)" to a low range of [1.4, 2.8]. This may be used to allow AI to accurately reproduce learned 3D camera characteristics using only text when configuring a 3D scene is not required, for example, in the case of a simple camera effect.
[0370] The three-dimensional scene data may include pose and spatial arrangement information of the object. As a specific example, when a scene in which "two people are standing back to back" is input as the text prompt, it may be required to configure the three-dimensional scene data because it is difficult to generate an accurate image without camera matching and spatial configuration of the objects based on the three-dimensional space. In this case, when a three-dimensional image having metadata having three-dimensional attribute information is input as the reference image, it may be unnecessary to configure the three-dimensional scene data.
[0371] The processor 1130 may obtain setting information in which at least one of character setting, camera composition setting, and lighting arrangement is set in the three-dimensional space. The processor 1130 may configure an overall shape of a scene by arranging the position and the composition of the object in the three-dimensional space based on the setting information. Configuring the overall shape of the scene may be referred to as three-dimensional blocking.
[0372] On the other hand, when configuring the three-dimensional scene data is not required, the processor 1130 may generate a target image using the text prompt and the reference image.
[0373] The processor 1130 may extract a control map based on the three-dimensional scene data. The control map may mean data including a spatial structure or geometric information on two-dimensional data generated by rendering the three-dimensional scene data. Preferably, the control map may include not only a "depth map" indicating distance and a "surface normal map" indicating a surface direction, but also an "ambient occlusion (AO) map" representing a "contact shadow" and "spatial interaction" generated when objects come close to each other, for example, when an arm contacts a torso, or in a gap.
[0374] The AI model may receive the AO map as a condition, and may generate a final image in which stereoscopic effect and realism of a region to which lighting is not directly applied are greatly improved. The AO map operates as a core control signal representing a "realistic 3D depth perception" that is difficult to implement only with a depth map.
[0375] In addition, in connection with video generation, that is, time-series control, the control map may include a "motion vector map" indicating a movement direction and a speed of pixels between frames. This is used for AI to realistically generate a "motion blur" effect and maximize temporal stability.
[0376] Furthermore, the control map may include an "occlusion map" specifying an occlusion relationship between objects in a 3D scene. This helps AI consistently process, without errors, an object that disappears behind another object, that is, is occluded, or appears again, that is, is disoccluded.
[0377] The processor 1130 may obtain the target image based on the control map and the image generation model. The processor 1130 may obtain, through the image generation model, the target image in which detail of the object is reflected, based on a visual structural feature set in the control map. Here, the control map is data including a spatial structure or geometric information obtained from the three-dimensional scene data, and may serve as a guide that enables the image generation model to generate an image while maintaining an accurate shape, composition, and viewpoint, rather than merely performing text-based image generation.
[0378] Specifically, the processor 1130 may convert the text prompt into an embedding vector using a text encoder, and may input the embedding vector to the image generation model as condition information.
[0379] In addition, the server 10 may obtain and process a "region-based hybrid control" signal by which an artist, that is, a user, differently designates a control method for each region of the three-dimensional scene. In an embodiment, after configuring a three-dimensional blocking scene in a Digital Content Creation (DCC) tool, the artist may designate different control methods for specific regions of an image to be generated.
[0380] For example, a "character" region may be designated to follow an accurate three-dimensional pose by designating a "pose map" as a control map, a "background" region may be designated to maintain a three-dimensional spatial sense by designating a "depth map" as a control map, and a "specific prop" region, for example, a magic sword in concept art, for which 3D modeling is difficult, may be designated to follow a corresponding style and shape by designating a "color reference image."
[0381] The server 10 may receive a "hybrid control map" in which different types of control signals are compositely configured for respective regions as described above, and may generate a final image by simultaneously applying a control method suitable for each region.
[0382] In addition, as an alternative embodiment of the present disclosure, instead of extracting a "control map" from the three-dimensional scene data, the server 10 may first generate a "rough render image" including a basic shape and color of the object. The "rough render image" may be used as an "input image" of an "img2img" or "inpainting" process of the image generation model. In this case, the server 10 may load an "IP-specific model," for example, LoRA, together to enforce a unique identity of the object, and may generate, that is, convert, a final target image in which detail and texture are reinforced while a 3D structure and IP identity of the rough render image are maintained according to a text prompt of the artist, for example, "more realistic fur texture, detail correction."
[0383] The processor 1130 may input the control map through a control-dedicated path or a cross-attention structure so that visual structural information on a position, a shape, a posture, a viewpoint, a direction, and the like of the object is maintained in the image generation model. For example, when a direction of an arm or a head of a character is set in the control map, the image generation model may generate the target image by supplementing detail and texture while the direction of the arm or the head of the character set in the control map is maintained.
[0384] The target image generated through the image generation model may not merely reflect a text description corresponding to the text prompt, but may be a result reflecting metadata and camera metadata based on the three-dimensional scene data. For example, when a text prompt saying "a knight holding a sword is standing while looking to the left" is input, and pose information and viewpoint information of the knight are given in the control map, detailed elements such as an angle of an arm, a position of a weapon, a direction of a face, and a rotation angle of a body may be accurately reflected in the target image.
[0385] In addition, in an image generation process, the processor 1130 may reduce a problem in which a boundary is unclear or a shape is distorted by using the depth map and the surface normal map. In addition, in the image generation process, the processor 1130 may generate an image in which rotation and movement of joints are accurately expressed without damage to a skeleton structure by using the pose map. Here, the generated image may simultaneously satisfy a style requested by the user, a consistent structure, a natural pose, and an accurate viewpoint.
[0386] Unlike a method of operating the image generation model based only on the text prompt, the processor 1130 may generate a more accurate and consistent target image by combining the control map and the trained image generation model.
[0387] The server 10 described in the present disclosure may be implemented by hardware components, software components, and / or a combination of hardware components and software components. In addition, the present disclosure may be provided in the form of a computer program stored in a computer-readable storage medium so as to perform a method of operating the server 10. In addition, the present disclosure may be written as a program executable on a computer, and may be implemented in a general-purpose digital computer that operates such a program using a computer-readable storage medium.
[0388] The computer-readable storage medium may be read-only memory (ROM), random-access memory (RAM), flash memory, CD-ROMs, CD-Rs, CD+Rs, CD-RWs, CD+RWs, DVD-ROMs, DVD-Rs, DVD+Rs, DVD-RWs, DVD+RWs, DVD-RAMs, BD-ROMs, BD-Rs, BD-R LTHs, BD-REs, magnetic tapes, floppy disks, magneto-optical data storage devices, optical data storage devices, hard disks, and solid-state disks (SSDs), and may be any device capable of storing instructions or software, related data, data files, and data structures, and capable of providing the instructions or software, the related data, the data files, and the data structures to a processor or computer so that the processor or computer can execute the instructions.
[0389] Although the embodiments have been described in detail above, the scope of the present disclosure is not limited thereto, and various modifications and improvements made by those of ordinary skill in the art using the basic concept of the present disclosure defined in the following claims also fall within the scope of the present disclosure.
Claims
1.A method of operating a server, the method comprising:obtaining three-dimensional data for an object included in an image;obtaining, by rendering the three-dimensional data according to a plurality of viewpoints, a training data set including, for each of the plurality of viewpoints, a two-dimensional image and metadata including three-dimensional attribute information;performing training using the training data set based on a training method determined from attribute information of the three-dimensional data; andobtaining an image generation model for the object based on a result of the training.2.The method of claim 1, wherein the metadata comprises at least one of depth information, surface normal information, motion information, and occlusion information for each pixel.3.The method of claim 1, further comprising:obtaining, based on the three-dimensional data, the attribute information including at least one of a type of the three-dimensional data, a type of the object, a number of polygons for the object, complexity of joints , a resolution of a texture, and a style; anddetermining, from among a plurality of training methods, the training method corresponding to the attribute information.4.The method of claim 3, wherein the plurality of training methods are classified based on at least one of a modification range of model parameters, an amount of training data used during the training, the style of the object, and the type of the object.5.The method of claim 3,wherein the determining of the training method corresponding to the attribute information comprisesdetermining, for each criterion for classifying the plurality of training methods, a training method corresponding to the attribute information, andthe training of the training data set comprisestraining the training data set by combining the training methods determined for each criterion.6.The method of claim 1, wherein the obtaining of the training data set further comprisesobtaining, as part of the training data set, camera metadata related to a setting of a camera corresponding to the plurality of viewpoints.7.The method of claim 6, wherein the camera metadata comprisesposition and posture information of the camera in a three-dimensional space, andat least one photographing control parameter including a focal length, an aperture value, a shutter speed, a sensor size, ISO sensitivity, and white balance.8.The method of claim 1, further comprising:obtaining quality information on an image generated through the image generation model; andupdating the image generation model based on the quality information.9.The method of claim 8, wherein the quality information on the image comprisesinformation on at least one of a matching degree between the image and text input for generating the image, a ratio and a color reflecting consistency of the object, a three-dimensional structure matching degree, realism of the image, and a matching degree between the image and a camera parameter.10.The method of claim 8, wherein the updating of the image generation model based on the quality information comprisesadjusting, based on the quality information, a weight of a parameter associated with a feature having an image quality evaluation score lower than a preset reference value or a weight of a loss function of the image generation model.11.The method of claim 1, wherein the obtaining of the three-dimensional data comprisesobtaining the three-dimensional data from a two-dimensional image obtained by photographing a real object corresponding to the object or from a three-dimensional model.12.The method of claim 1, further comprising:obtaining a target image based on setting information for a three-dimensional target image and the image generation model.13.The method of claim 12, wherein the obtaining of the target image comprises:obtaining the setting information in which at least one of a pose of the object, an expression of the object, a composition of a camera, and an arrangement of lighting is set;obtaining, by rendering the three-dimensional data based on the setting information, a control map representing a visual structural feature corresponding to the setting information; andobtaining the target image based on the control map and the image generation model.14.The method of claim 13, wherein the three-dimensional data is three-dimensional scene data including pose and spatial arrangement information of the object corresponding to the target image.15.The method of claim 13, wherein the control map comprisesa depth map generated by rendering the three-dimensional data on a virtual three-dimensional viewport and extracting a value from a Z-buffer of the viewport.16.The method of claim 13, wherein the control map comprisesat least one of a depth map, a surface normal map, a pose map, an expression map, a contour map, a camera map, and a lighting map.17.The method of claim 13, wherein the obtaining of the target image based on the control map and the image generation model comprisesobtaining, through the image generation model, the target image in which a detail of the object is reflected, based on the visual structural feature set in the control map.18.The method of claim 12, further comprising:obtaining quality information on the target image; andcorrecting the target image based on the quality information.19.A server comprising:a communication device configured to communicate with an external device;a processor; anda memory storing instructions executable by the processor,wherein the processor is configured to execute the instructions to:obtain three-dimensional data for an object included in an image;obtain, by rendering the three-dimensional data according to a plurality of viewpoints, a training data set including, for each of the plurality of viewpoints, a two-dimensional image and metadata having three-dimensional attribute information;perform training using the training data set based on a training method determined from attribute information of the three-dimensional data; andobtain an image generation model for the object based on a result of the training.20.A non-transitory computer-readable storage medium storing instructions that, when executed by a processor, cause the processor to perform operations comprising: obtaining three-dimensional data for an object included in an image;obtaining, by rendering the three-dimensional data according to a plurality of viewpoints, a training data set including, for each of the plurality of viewpoints, a two-dimensional image and metadata having three-dimensional attribute information;performing training using the training data set based on a training method determined from attribute information of the three-dimensional data; andobtaining an image generation model for the object based on a result of the training.
