Hierarchical artificial intelligence-based stepwise content generation method, device and computer program

Abstract

Disclosed is a hierarchical artificial intelligence-based stepwise content generation method performed by at least one processor, the method comprising the steps of: generating a content generation prompt on the basis of target subject content including a target subject and first condition information corresponding to an initial constraint condition of to-be-generated content; generating first content as initial content on the basis of the generated content generation prompt and second condition information corresponding to a component constraint condition of the to-be generated content; and generating second content as the to-be-generated content by synthesizing the generated first content and the target subject content.

Description

Hierarchical AI-based content stepwise generation method, device, and computer program

[0001] The present disclosure relates to a hierarchical artificial intelligence-based method, apparatus, and computer program for sequentially generating content. Specifically, it relates to a method, apparatus, and computer program for sequentially generating content according to conditions set for each stage.

[0002] Content generation technology utilizing generative artificial intelligence (AI) has been developing rapidly recently, but there are several limitations that need to be resolved before this technology can reach a level where it generates and edits content according to the user's intent.

[0003] Conventional generative AI models generate outputs by transforming data within the neural network's latent space, a process that generally exhibits black-box characteristics. In other words, since the data processing within the latent space is not clearly understood, users must manually adjust input values ​​and repeat tests multiple times to obtain the desired results. This results in the consumption of significant computation time and computing resources.

[0004] For example, image generation models such as Stable Diffusion generate the final image by transforming input data into a latent space, expanding the data by adding noise, and then undergoing a diffusion process that gradually removes the noise.

[0005] Looking specifically at the image generation process based on Stable Diffusion, the first step involves preparing an input dataset and training a Variational Autoencoder (VAE) model capable of regenerating the data using an auto-encoding method. Subsequently, the VAE encoder generates latent spatial vectors for the input dataset. These latent spatial vectors are transformed into data with added noise according to specific rules and processed iteratively step-by-step until the noise is removed. This process is referred to as the diffusion (noise removal) process. However, there is a limitation in that it is difficult to maintain the consistency of the output due to data loss occurring during noise removal.

[0006] Furthermore, since it is difficult to reuse or modify latent space vectors, the process of adding and removing noise must be repeated from scratch to reflect new conditions. During this process, users conduct multiple tests and go through trial and error until appropriate input values ​​are derived. In particular, satisfying complex conditions or detailed requirements requires more time and resources. This repetitive and inefficient process leads to significant waste of computation time, energy, and computing resources, as well as increased user fatigue.

[0007] This process is called prompt engineering, but ultimately, from the user's perspective, it requires a significant amount of time and effort to obtain the result, leading to reduced efficiency.

[0008] In conclusion, conventional generative AI technologies have limitations in precisely generating desired content due to the fixed nature of latent space vectors and the incompleteness of noise removal. To address these issues, it is necessary to introduce technologies that enable detailed control during the generation process and the efficient reuse of existing latent vectors.

[0009] The aforementioned background technology is one that the inventor possessed or acquired in the process of deriving the content of the disclosure of the present application, and it cannot be considered as prior art disclosed to the general public prior to the filing of this application.

[0010] The present disclosure provides a hierarchical artificial intelligence-based content stepwise generation method, apparatus (system), and computer program for solving the above-mentioned problems.

[0011] The present disclosure may be implemented in various ways, including a method, a system (device), or a computer program stored on a readable storage medium.

[0012] A method for generating content in a hierarchical structure based on artificial intelligence, performed by at least one processor according to one embodiment of the present disclosure, comprising: generating a content generation prompt based on first condition information corresponding to initial constraints of target content including a target object and content to be generated; generating a first content as initial content based on second condition information corresponding to component constraints of the generated content generation prompt and content to be generated; and generating a second content as content to be generated by synthesizing the generated first content and the target content.

[0013] In one embodiment of the present disclosure, the first condition information includes at least one of the context in which the content to be generated is used, the size of the content to be generated, and visual characteristic information of the content to be generated.

[0014] In one embodiment of the present disclosure, the second condition information includes at least one of condition information regarding a first component to be included in the foreground of the content to be generated, condition information regarding a second component to be included in the background of the content to be generated, condition information regarding position and posture information of the first component and the second component, and condition information regarding external characteristics of the first component and the second component.

[0015] In one embodiment of the present disclosure, the step of generating a content generation prompt includes the step of generating a content generation prompt by analyzing target content and first condition information based on a previously trained language model, and the generated content generation prompt includes one or more feature information about a target extracted from target content, first condition information, and a command that directs content generation based on one or more feature information and first condition information.

[0016] In one embodiment of the present disclosure, the step of generating a first content includes the step of generating a first image corresponding to text including a generated content generation prompt and second condition information using a text-to-image conversion model, wherein the generated first image includes a user corresponding to the generated content generation prompt and second condition information, and is an image including a body part among the user's body parts to which a target object is applied.

[0017] In one embodiment of the present disclosure, the step of generating second content includes the step of generating a second image by synthesizing a first image generated using an inpainting-based image synthesis model with a target content, and the generated second image is an image including a user to whom a target is applied by filling a target target into an area corresponding to a body part included in the first image generated based on the target content.

[0018] In one embodiment of the present disclosure, the method further includes the step of individually correcting at least one of first condition information and second condition information based on the attributes of the acquired feedback when feedback is obtained for the generated second content.

[0019] In one embodiment of the present disclosure, the step of correcting includes correcting first condition information based on the acquired feedback when the acquired feedback relates to an initial constraint condition for content to be generated, and the step of generating a content generation prompt includes regenerating a content generation prompt based on the target content and the corrected first condition information.

[0020] In one embodiment of the present disclosure, the step of correcting includes correcting second condition information based on the acquired feedback when the acquired feedback relates to a component constraint of the content to be generated, and the step of generating the first content includes regenerating the first content based on the generated content generation prompt and the corrected second condition information.

[0021] A computer program stored on a computer-readable recording medium is provided to execute a method for generating a hierarchical artificial intelligence-based content stepwise according to one embodiment of the present disclosure on a computer.

[0022] An apparatus according to one embodiment of the present disclosure comprises a communication module, a display, a memory, and at least one processor connected to the memory and configured to execute at least one computer-readable program included in the memory, wherein the at least one program comprises instructions for generating a content creation prompt based on first condition information corresponding to initial constraints for a product image including a target product and content to be created, obtaining second condition information corresponding to constraints for a component to be included in the content to be created, generating an initial product content based on the generated content creation prompt and the obtained second condition information, and generating a final product content corresponding to the content to be created by reflecting the product image in the generated initial product content.

[0023] In various embodiments of the present disclosure, content is generated step by step based on an artificial intelligence model with a hierarchical structure, and condition information corresponding to different constraints is input at each step, and by generating content based thereon, content optimized for the user's needs can be generated.

[0024] In various embodiments of the present disclosure, in the process of regenerating content as feedback is obtained from a user, content can be generated more quickly and efficiently by selectively re-performing only some necessary steps instead of re-performing all steps performed for content generation.

[0025] In various embodiments of the present disclosure, different types of artificial intelligence models are used hierarchically at each step of generating content, and by individually designing the performance of each artificial intelligence model, the quality of the content or the computational speed can be dynamically adjusted according to the situation or conditions.

[0026] The effects of the present disclosure are not limited to those mentioned above, and other unmentioned effects will be clearly understood by a person skilled in the art to which the present disclosure pertains (referred to as "person skilled in the art") from the description in the claims.

[0027] Embodiments of the present disclosure will be described with reference to the accompanying drawings described below, wherein similar reference numerals indicate similar elements, but are not limited thereto.

[0028] FIG. 1 is a diagram illustrating an example of a hierarchical artificial intelligence-based content stepwise generation interface according to one embodiment of the present disclosure.

[0029] FIG. 2 is a schematic diagram showing a configuration in which an information processing system is connected to communicate with a plurality of user terminals in order to provide a hierarchical artificial intelligence-based content step-by-step generation service according to one embodiment of the present disclosure.

[0030] FIG. 3 is a block diagram showing the internal configuration of a user terminal and an information processing system according to one embodiment of the present disclosure.

[0031] FIG. 4 is a block diagram showing the internal configuration of a processor according to one embodiment of the present disclosure.

[0032] FIG. 5 is a flowchart of a step-by-step method for generating AI-based content of a hierarchical structure according to one embodiment of the present disclosure.

[0033] FIG. 6 is a flowchart illustrating the step-by-step creation process of AI-based content of a hierarchical structure according to one embodiment of the disclosure.

[0034] FIG. 7 is a drawing showing an example of target content according to one embodiment of the present disclosure.

[0035] FIG. 8 is a drawing showing an example of a first content according to one embodiment of the present disclosure.

[0036] FIG. 9 is a drawing showing an example of a second content according to one embodiment of the present disclosure.

[0037] Hereinafter, specific details for implementing the present disclosure will be described in detail with reference to the attached drawings. However, in the following description, specific descriptions regarding widely known functions or configurations will be omitted if there is a risk that the gist of the present disclosure may be unnecessarily obscured.

[0038] In the attached drawings, identical or corresponding components are assigned the same reference numerals. Additionally, in the description of the following embodiments, the description of identical or corresponding components may be omitted. However, even if a description of a component is omitted, it is not intended that such component is not included in any embodiment.

[0039] The advantages and features of the disclosed embodiments and the methods for achieving them will become clear by referring to the embodiments described below in conjunction with the accompanying drawings. However, the present disclosure is not limited to the embodiments disclosed below but may be implemented in various different forms, and the embodiments provided are merely to make the present disclosure complete and to fully inform those skilled in the art of the scope of the invention.

[0040] The terms used in this specification will be briefly explained, and the disclosed embodiments will be described in detail. The terms used in this specification have been selected to be as generally used as possible, taking into account their functions in this disclosure; however, these terms may vary depending on the intent of those skilled in the art, case law, the emergence of new technologies, etc. Additionally, in specific cases, terms may be arbitrarily selected by the applicant, and in such cases, their meanings will be described in detail in the relevant description of the invention. Therefore, the terms used in this disclosure should be defined not merely by their names, but based on their meanings and the content throughout this disclosure.

[0041] In this specification, singular expressions include plural expressions unless the context clearly specifies them as singular. Additionally, plural expressions include singular expressions unless the context clearly specifies them as plural. Throughout the specification, when a part is described as including a certain component, this means that, unless specifically stated otherwise, it does not exclude other components but may include additional components.

[0042] Additionally, the terms 'module' or 'part' as used in the specification refer to software or hardware components, and the 'module' or 'part' performs certain roles. However, the meaning of 'module' or 'part' is not limited to software or hardware. The 'module' or 'part' may be configured to reside in an addressable storage medium or configured to run on one or more processors. Thus, as an example, the 'module' or 'part' may include components such as software components, object-oriented software components, class components, and task components, and at least one of processes, functions, attributes, procedures, subroutines, segments of program code, drivers, firmware, microcode, circuits, data, databases, data structures, tables, arrays, or variables. The components and the functions provided within the 'module' or 'part' may be combined into a smaller number of components and 'modules' or 'parts', or further separated into additional components and 'modules' or 'parts'.

[0043] According to one embodiment of the present disclosure, a ‘module’ or ‘part’ may be implemented as a processor and memory. The term ‘processor’ should be broadly interpreted to include a general-purpose processor, a central processing unit (CPU), a microprocessor, a digital signal processor (DSP), a controller, a microcontroller, a state machine, etc. In some environments, the term ‘processor’ may refer to an application-specific integrated circuit (ASIC), a programmable logic device (PLD), a field programmable gate array (FPGA), etc. The term ‘processor’ may also refer to a combination of processing devices, such as, for example, a combination of a DSP and a microprocessor, a combination of multiple microprocessors, a combination of one or more microprocessors combined with a DSP core, or any other combination of such configurations. Additionally, the term ‘memory’ should be broadly interpreted to include any electronic component capable of storing electronic information. 'Memory' may refer to various types of processor-readable media, such as Random Access Memory (RAM), Read-Only Memory (ROM), Non-Volatile Random Access Memory (NVRAM), Programmable Read-Only Memory (PROM), Erasable-Programmable Read-Only Memory (EPROM), Electrically Erasable PROM (EEPROM), Flash Memory, Magnetic or Marked Data Storage Devices, Registers, etc. If a processor can read information from memory and / or write information to memory, the memory is said to be in an electronic communication state with the processor. Memory integrated into a processor is in an electronic communication state with the processor.

[0044] In addition, terms such as first, second, A, B, (a), (b), etc. used in the following embodiments are used merely to distinguish one component from another, and the essence, order, or sequence of the said component is not limited by such terms.

[0045] Additionally, in the following embodiments, where it is stated that one component is 'connected', 'coupled', or 'joined' to another component, it should be understood that the component may be directly connected or joined to the other component, but that another component may also be 'connected', 'coupled', or 'joined' between each component.

[0046] Additionally, as used in the following embodiments, 'comprises' and / or 'comprising' do not exclude the presence or addition of one or more other components, steps, actions, and / or elements to the mentioned components, steps, actions, and / or elements.

[0047] Hereinafter, various embodiments of the present disclosure will be described in detail with reference to the attached drawings.

[0048] FIG. 1 is a diagram illustrating an example of a hierarchical AI-based content stepwise generation interface according to one embodiment of the present disclosure. As illustrated, the hierarchical AI-based content stepwise generation interface (100) may include a target content upload area (110) and a condition information input area (120).

[0049] In one embodiment, target content can be uploaded through the target content upload area (110). Here, the target may be clothing products, accessory products, etc., and the target content may be an image including clothing products, but is not limited thereto. In the target content upload area (110), several targets that the user can select may be displayed, and the user may select one of these several targets as the target.

[0050] In one embodiment, the uploaded target content (111) (or content including a target selected by the user as a target among several targets) may be displayed in the target target content upload area (110).

[0051] In one embodiment, the condition information input area (120) may include an area for inputting condition information for the content to be generated. For example, the condition information input area (120) may include a first condition information input area for inputting first condition information corresponding to the initial constraint of the content to be generated, and a second condition information input area for inputting second condition information corresponding to the component constraint of the content to be generated.

[0052] Here, the first condition information includes the context in which the content to be generated is used, the size of the content to be generated, and information on the visual characteristics of the content to be generated, and the first condition information input area may include a field for entering such information. Additionally, the second condition information may include condition information regarding a first component to be included in the foreground of the content to be generated, condition information regarding a second component to be included in the background of the content to be generated, condition information regarding the position and pose information of the first and second components, and condition information regarding the external characteristics of the first and second components, and the second condition information input area may include a field for entering such information.

[0053] The user can freely set various constraints for the content they wish to create (e.g., gender, age group, race, body type, posture, location, etc. of the person to be included in the image they wish to create) through the condition information input area (120). Additionally, the user can input feedback such as modifying specific condition information or setting additional condition information based on the second content, which is the final content.

[0054] In one embodiment, the condition information input area (120) may further include an email address input area for receiving an email address, and the second content (final content) generated based on the second condition information obtained from the user and the target content may be transmitted to the email address entered through the email address input area.

[0055] Each component within the AI-based content step-by-step generation interface (100) of the hierarchical structure illustrated in FIG. 1 is exemplary and is not limited thereto. Areas included in the AI-based content step-by-step generation interface (100) of the hierarchical structure may be configured differently in location or shape from that illustrated in FIG. 1, additional components (e.g., a feedback input area for modifying condition information) may be included, or some components may be omitted.

[0056] FIG. 2 is a schematic diagram showing a configuration in which an information processing system is connected to communicate with a plurality of user terminals to provide a hierarchical AI-based content step-by-step generation service according to one embodiment of the present disclosure. The information processing system (230) may include system(s) capable of providing a hierarchical AI-based content step-by-step generation service. In one embodiment, the information processing system (230) may include one or more server devices and / or databases capable of storing, providing, and executing computer-executable programs (e.g., downloadable applications) and data related to a hierarchical AI-based content step-by-step generation service, or one or more distributed computing devices and / or distributed databases based on cloud computing services.

[0057] The hierarchical artificial intelligence-based content step-by-step generation service provided by the information processing system (230) can be provided to the user through applications installed on each of the multiple user terminals (210_1, 210_2, 210_3).

[0058] Multiple user terminals (210_1, 210_2, 210_3) can communicate with an information processing system (230) through a network (220). The network (220) can be configured to enable communication between the multiple user terminals (210_1, 210_2, 210_3) and the information processing system (230). Depending on the installation environment, the network (220) may be configured as a wired network such as Ethernet, Power Line Communication, telephone line communication devices and RS-serial communication, a mobile communication network, a Wireless LAN (WLAN), Wi-Fi, Bluetooth and ZigBee, or a combination thereof. The communication method is not limited and may include not only communication methods utilizing communication networks that the network (220) may include (e.g., mobile communication network, wired internet, wireless internet, broadcasting network, satellite network, etc.) but also short-range wireless communication between user terminals (210_1, 210_2, 210_3).

[0059] For example, a plurality of user terminals (210_1, 210_2, 210_3) can transmit a request to an information processing system (230) via a network (220), and the information processing system (230) can receive this and then transmit a response corresponding to the request to the plurality of user terminals (210_1, 210_2, 210_3). For instance, if a user terminal (210_1) transmits a target content, a first condition information, a second condition information, and a request for content creation / output to the information processing system (230) (request), the information processing system (230) can create a second content as the target content for creation based on the received information and request and transmit it to the user terminal (210_1) (response).

[0060] In FIG. 2, a mobile phone terminal (210_1), a tablet terminal (210_2), and a PC terminal (210_3) are illustrated as examples of user terminals, but are not limited thereto. The user terminals (210_1, 210_2, 210_3) may be any computing device capable of wired and / or wireless communication and capable of installing and running a hierarchical AI-based content step-by-step generation / output application. For example, user terminals may include medical devices, smartphones, mobile phones, navigation systems, computers, laptops, digital broadcasting terminals, PDAs (Personal Digital Assistants), PMPs (Portable Multimedia Players), tablet PCs, game consoles, wearable devices, IoT (Internet of Things) devices, VR (Virtual Reality) devices, AR (Augmented Reality) devices, etc. Additionally, FIG. 2 illustrates three user terminals (210_1, 210_2, 210_3) communicating with an information processing system (230) through a network (220), but is not limited thereto, and may be configured so that a different number of user terminals communicate with an information processing system (230) through a network (220).

[0061] In FIG. 2, it is described that an information processing system generates content and provides it to a user terminal, but it is not limited thereto. For example, the user terminal may generate content directly without communicating with the information processing system.

[0062] FIG. 3 is a block diagram showing the internal configuration of a user terminal and an information processing system according to an embodiment of the present disclosure. The user terminal (210) may refer to any computing device capable of executing content step-by-step generation / output applications, etc., and capable of wired / wireless communication, and may include, for example, the mobile phone terminal (210_1), tablet terminal (210_2), PC terminal (210_3) of FIG. 2. As illustrated, the user terminal (210) may include a memory (312), a processor (314), a communication module (316), and an input / output interface (318). Similarly, the information processing system (230) may include a memory (332), a processor (334), a communication module (336), and an input / output interface (338). As illustrated in FIG. 3, the user terminal (210) and the information processing system (230) may be configured to communicate information and / or data through the network (220) using their respective communication modules (316, 336). Additionally, the input / output device (320) may be configured to input information and / or data to the user terminal (210) or output information and / or data generated from the user terminal (210) through the input / output interface (318).

[0063] The memory (312, 332) may include any non-transient computer-readable recording medium. According to one embodiment, the memory (312, 332) may include a non-perishable permanent mass storage device such as ROM (read-only memory), a disk drive, an SSD (solid-state drive), or a flash memory. As another example, a non-perishable permanent mass storage device such as ROM, an SSD, a flash memory, or a disk drive may be included in the user terminal (210) or the information processing system (230) as a separate permanent storage device distinct from the memory. Additionally, the memory (312, 332) may store an operating system and at least one program code (e.g., code for a content-stepping / output application).

[0064] These software components may be loaded from a computer-readable recording medium separate from memory (312, 332). This separate computer-readable recording medium may include a recording medium that can be directly connected to the user terminal (210) and the information processing system (230), for example, a computer-readable recording medium such as a floppy drive, disk, tape, DVD / CD-ROM drive, or memory card. As another example, the software components may be loaded into memory (312, 332) via a communication module (316, 336) rather than a computer-readable recording medium. For example, at least one program may be loaded into memory (312, 332) based on a computer program (e.g., a content-staged creation / output application, etc.) that is installed by files provided through a network (220) by developers or a file distribution system that distributes installation files for the application.

[0065] The communication module (316, 336) may provide a configuration or function for the user terminal (210) and the information processing system (230) to communicate with each other via the network (220), and may provide a configuration or function for the user terminal (210) and / or the information processing system (230) to communicate with another user terminal or another system (e.g., a separate cloud system). For example, a request or data generated by the processor (314) of the user terminal (210) according to program code stored in a recording device such as memory (312) may be transmitted to the information processing system (230) via the network (220) under the control of the communication module (316). Conversely, a control signal or command provided under the control of the processor (334) of the information processing system (230) may be received by the user terminal (210) through the communication module (316) of the user terminal (210) via the communication module (336) and the network (220).

[0066] The input / output interface (318) may be a means for interfacing with an input / output device (320). As an example, the input device may include a device such as a camera including an audio sensor and / or an image sensor, a keyboard, a microphone, or a mouse, and the output device may include a device such as a display, a speaker, or a haptic feedback device. As another example, the input / output interface (318) may be a means for interfacing with a device in which the configuration or function for performing input and output is integrated into one, such as a touchscreen. Although the input / output device (320) is depicted in FIG. 3 as not being included in the user terminal (210), it is not limited thereto and may be configured as a single device with the user terminal (210). Additionally, the input / output interface (338) of the information processing system (230) may be a means for interfacing with a device (not shown) for input or output that is connected to the information processing system (230) or that the information processing system (230) may include. In FIG. 3, the input / output interface (318, 338) is shown as an element configured separately from the processor (314, 334), but is not limited thereto, and the input / output interface (318, 338) may be configured to be included in the processor (314, 334).

[0067] The user terminal (210) and the information processing system (230) may include more components than those of FIG. 3. However, it is not necessary to clearly illustrate most of the conventional technical components. In one embodiment, the user terminal (210) may be implemented to include at least some of the input / output devices (320) described above. Additionally, the user terminal (210) may further include other components such as a transceiver, a GPS (Global Positioning System) module, a camera, various sensors, a database, etc. For example, if the user terminal (210) is a smartphone, it may include components that are generally included in a smartphone, and may be implemented to include various components such as an accelerometer, a gyroscope, a microphone module, a camera module, various physical buttons, buttons using a touch panel, input / output ports, and a vibrator for vibration.

[0068] According to one embodiment, the processor (314) of the user terminal (210) may be configured to operate an application or web browser application that provides a synthetic image generation / output service. At this time, program code associated with the application may be loaded into the memory (312) of the user terminal (210). While the application is running, the processor (314) of the user terminal (210) may receive information and / or data provided from an input / output device (320) through an input / output interface (318) or receive information and / or data from an information processing system (230) through a communication module (316), and may process the received information and / or data and store it in the memory (312). Additionally, such information and / or data may be provided to the information processing system (230) through the communication module (316).

[0069] While the application is running, the processor (314) can receive voice data, text, images, videos, etc. that are input or selected through an input device such as a touch screen, keyboard, audio sensor and / or image sensor, camera, microphone, etc. connected to the input / output interface (318), and can store the received voice data, text, images and / or videos, etc. in memory (312) or provide them to an information processing system (230) through a communication module (316) and a network (220).

[0070] The processor (314) of the user terminal (210) can transmit information and / or data to an input / output device (320) through an input / output interface (318) and output it. For example, the processor (314) of the user terminal (210) can output the processed information and / or data through an output device (320), such as a display output device (e.g., touch screen, display, etc.) or a voice output device (e.g., speaker).

[0071] The processor (334) of the information processing system (230) may be configured to manage, process, and / or store information and / or data received from a plurality of user terminals (210) and / or a plurality of external systems. The information and / or data processed by the processor (334) may be provided to the user terminals (210) through a communication module (336) and a network (220).

[0072] FIG. 4 is a block diagram showing the internal configuration of a processor according to one embodiment of the present disclosure. As illustrated, the processor (400) (e.g., the processor (314) of the user terminal of FIG. 3 or the processor (334) of the information processing system) may include a content creation prompt generation unit (410), a first content creation unit (420), and a second content creation unit (430), etc.

[0073] The content creation prompt generation unit (410) can generate a content creation prompt based on target target content including a target target and first condition information.

[0074] In one embodiment, the content creation prompt generation unit (410) can generate a content creation prompt by analyzing target content and first condition information based on a pre-trained language model.

[0075] Here, the target content is content containing the target, and may be, for example, a product image containing only clothing or accessory products to be virtually fitted. Additionally, here, the first condition information is condition information corresponding to the initial constraints of the content to be generated, and may include, for example, the context in which the content to be generated is used, the size of the content to be generated (size, time, format, etc.), and visual characteristic information of the content to be generated (e.g., level of detail depiction of the result, dynamic range, exposure, color tone, and information related to the level of overall quality such as the style (look-and-feel) of the overall content (e.g., cartoon, oil painting, pen drawing, photograph, illustration, etc.). The content generation prompt generated based on such target content and the first condition information may include one or more feature information about the target extracted from the target content, the first condition information, and a command instructing content generation based on the one or more feature information and the first condition information.

[0076] The first content generation unit (420) can generate the first content based on the content generation prompt generated through the content generation prompt generation unit (410) and the first condition information. Here, the first content may refer to the initial content generated based on the content generation prompt and the first condition information.

[0077] In one embodiment, the first content generation unit (420) can generate a first image as first content corresponding to text including a content generation prompt and second condition information using a text-to-image conversion model.

[0078] Here, the first condition information is condition information corresponding to the component constraints of the content to be generated, and may include at least one of condition information regarding a first component to be included in the foreground of the content to be generated, condition information regarding a second component to be included in the background of the content to be generated, condition information regarding the position and posture information of the first component and the second component, and condition information regarding the external characteristics of the first component and the second component. The first image generated based on such second condition information and the content generation prompt may be an image that includes a user corresponding to the content generation prompt and the second condition information, and may include a body part among the user's body parts to which the target object is to be applied.

[0079] The second content generation unit (430) can generate second content by synthesizing the first content generated through the first content generation unit (420) and the target content. Here, the second content may refer to the final content, i.e., the final result, generated based on the target content and the first content.

[0080] In one embodiment, the second content generation unit (430) can generate a second image as the second content by synthesizing the first image (first content) and the target content using an inpainting-based image synthesis model.

[0081] Here, the target content is an image of a clothing product that includes only the clothing product to be virtually fitted, and the first image is an image that includes a user to wear the clothing product to be virtually fitted and a body part to wear the clothing product. The second image generated by synthesizing the target content and the first image may mean an image that includes a user to whom the target is applied, that is, an image that includes a user wearing the clothing product to be virtually fitted on a body part, by filling the target with the target in the area corresponding to the body part included in the first image based on the target content.

[0082] The internal configuration of the processor (400) illustrated in FIG. 4 is merely an example, and in some embodiments, additional configurations other than the illustrated internal configuration may be included, and some configurations may be omitted. For example, if the processor (400) is the processor (334) of the information processing system of FIG. 3 and some of the above internal configurations are omitted, the processor (314) of the user terminal may be configured to perform the functions of the omitted internal configurations. Furthermore, although the internal configuration of the processor (400) in FIG. 4 is described by separating it by function, this does not necessarily mean that they are physically separated. The content creation prompt generation unit (410), the first content creation unit (420), and the second content creation unit (430) have been described separately, but this is for the purpose of aiding understanding of the invention and is not limited thereto.

[0083] FIG. 5 is a flowchart of a method for generating AI-based content in stages with a hierarchical structure according to one embodiment of the present disclosure, and FIG. 6 is a flowchart illustrating a process for generating AI-based content in stages with a hierarchical structure according to one embodiment of the disclosure. The method (500) may be performed by at least one processor (e.g., a processor of a user terminal, a processor of an information processing system, or a processor of a content stage generation device).

[0084] In the method (500), the processor can obtain a target target content (Packshot) containing a target target and a first condition information (System Prompt) corresponding to the initial constraints of the target content to be created (S510). In one embodiment, the processor can provide a hierarchical AI-based content stepwise creation interface (e.g., 100 in FIG. 1) to a user terminal (210_1, 210_2, 210_3), and can receive uploaded target target content containing a target target through the hierarchical AI-based content stepwise creation interface. Here, the processor can also obtain the first condition information through the hierarchical AI-based content stepwise creation interface, but is not limited thereto, and the first condition information may be information that is set in advance. Here, the target target content is content containing a target target, and may be, for example, a clothing product image (700) containing only the clothing product to be virtually fitted, as shown in FIG. 7. Additionally, the first condition information may include, but is not limited to, at least one of the context in which the content to be generated is used, the size of the content to be generated, and the visual characteristic information of the content to be generated.

[0085] Subsequently, the processor can generate a content generation prompt based on the target content and the first condition information (S520). In one embodiment, the processor can generate a content generation prompt (Model Prompt) as output data by inputting the target content and the first condition information into a pre-trained language model.

[0086] Here, the pre-trained language model is a generative AI model, which may be a Transformer-based Large Language Model (LLM) (e.g., a GPT-family model such as GPT-4o). The Transformer model is a structure that encodes an input sequence and generates an output using only an attention mechanism, and it is a model that overcomes the limitations of existing Recurrent Neural Networks (RNNs) and LSTMs. The Transformer is designed to efficiently learn the relationships between each word in an input sequence through self-attention and multi-head attention. The generative AI-based language model takes the task of Next Token Prediction as its learning objective. The model performs self-supervised learning by predicting the probability of the next word in an input word sequence based on large-scale text data; through this, the model understands context and can perform various natural language processing (NLP) tasks such as generating natural sentences, generating responses to questions, summarizing, and translation.

[0087] Subsequently, the processor may obtain second condition information corresponding to the component constraints of the content to be created (S530). In one embodiment, the processor may provide a hierarchical AI-based content stepwise creation interface (e.g., 100 of FIG. 1) to a user terminal (210_1, 210_2, 210_3), and through the hierarchical AI-based content stepwise creation interface, obtain second condition information corresponding to the component constraints of the content to be created. Here, the second condition information may include, but is not limited to, at least one of condition information regarding a first component to be included in the foreground of the content to be created, condition information regarding a second component to be included in the background of the content to be created, condition information regarding the position and posture information of the first component and the second component, and condition information regarding the external features of the first component and the second component.

[0088] Subsequently, the processor may generate a first content as initial content based on a content creation prompt and second condition information (Add text) (S540). In one embodiment, the processor may generate a first image (Model Image) corresponding to text containing the content creation prompt and the second condition information using a text-to-image conversion model (e.g., FLUX T2I). Here, the first image may be an image (800) that includes a user corresponding to the content creation prompt and the second condition information, for example, as shown in FIG. 8, and includes a body part among the user's body parts to which a target object is applied.

[0089] Subsequently, the processor can generate a second content (Output Image) as the content to be generated by synthesizing the first content and the target content (S550). In one embodiment, the processor can generate a second image by synthesizing the first image and the target content generated using an inpainting-based image synthesis model (e.g., VTON(KOLOR)). Here, the second image may be an image (900) containing a user to whom the target is applied, by filling the target target into an area corresponding to a body part included in the first image based on the target content, as shown in FIG. 9.

[0090] In one embodiment, when feedback is obtained regarding the second content, the processor may select and individually correct at least one of the first condition information and the second condition information based on the attributes of the feedback.

[0091] For example, if the feedback is related to an initial constraint on the content to be generated, the processor can correct the first condition information based on the feedback, and accordingly, can regenerate a content generation prompt based on the target content and the corrected first condition information, can regenerate a first image based on the regenerated content generation prompt and second condition information, and can regenerate a second image by synthesizing the regenerated first image and the target content.

[0092] As another example, if the feedback is related to the component constraints of the content to be generated, the processor can correct the second condition information based on the feedback, regenerate the first image based on the content generation prompt and the corrected second condition information, and regenerate the second image by synthesizing the regenerated first image and the target content.

[0093]

[0094] In other words, the processor has the advantage of enabling faster and more efficient content generation by selectively re-executing only some necessary steps during the process of regenerating content based on feedback from the user, rather than re-executing all steps required for content creation.

[0095] In addition, by hierarchically using different types of AI models at each stage of content generation and designing the performance of each AI model individually, there is an advantage in being able to dynamically adjust the quality of the content or the speed of computation depending on the situation or conditions.

[0096] The method described above may be provided as a computer program stored on a computer-readable recording medium for execution on a computer. The medium may continuously store a computer-executable program, or temporarily store it for execution or download. Additionally, the medium may be various recording or storage means in the form of a single or multiple hardware components, and may not be limited to a medium directly connected to a computer system but may exist distributed over a network. Examples of media may include magnetic media such as hard disks, floppy disks, and magnetic tapes; optical recording media such as CD-ROMs and DVDs; magneto-optical media such as floptical disks; and media configured to store program instructions, including ROM, RAM, and flash memory. Furthermore, other examples of media may include recording or storage media managed by sites, servers, etc., that supply or distribute various software, such as app stores that distribute applications.

[0097] The methods, operations, or techniques of the present disclosure may be implemented by various means. For example, these techniques may be implemented in hardware, firmware, software, or a combination thereof. Those skilled in the art will understand that the various exemplary logical blocks, modules, circuits, and algorithmic steps described in connection with the disclosure herein may be implemented in electronic hardware, computer software, or a combination of both. To clearly illustrate such interchangeability between hardware and software, various exemplary components, blocks, modules, circuits, and steps have been generally described above in terms of their functional aspects. Whether such functions are implemented in hardware or in software depends on the design requirements imposed on the specific application and the overall system. Those skilled in the art may implement the functions described in various ways for each specific application, but such implementations should not be construed as departing from the scope of the present disclosure.

[0098] In a hardware implementation, the processing units used to perform the techniques may be implemented in one or more ASICs, DSPs, digital signal processing devices (DSPDs), programmable logic devices (PLDs), field programmable gate arrays (FPGAs), processors, controllers, microcontrollers, microprocessors, electronic devices, other electronic units designed to perform the functions described in this disclosure, computers, or a combination thereof.

[0099] Accordingly, the various exemplary logic blocks, modules, and circuits described in connection with the present disclosure may be implemented or performed by any combination of general-purpose processors, DSPs, ASICs, FPGAs or other programmable logic devices, discrete gate or transistor logic, discrete hardware components, or those designed to perform the functions described herein. A general-purpose processor may be a microprocessor, but alternatively, the processor may be any conventional processor, controller, microcontroller, or state machine. The processor may also be implemented as a combination of computing devices, for example, a DSP and a microprocessor, a plurality of microprocessors, one or more microprocessors coupled with a DSP core, or any other combination of configurations.

[0100] In firmware and / or software implementations, techniques may be implemented as instructions stored on a computer-readable medium such as random access memory (RAM), read-only memory (ROM), non-volatile random access memory (NVRAM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable PROM (EEPROM), flash memory, compact disc (CD), magnetic or optical data storage devices, etc. The instructions may be executable by one or more processors, and may cause the processor(s) to perform specific aspects of the functions described in this disclosure.

[0101] Although the embodiments described above have been described as utilizing aspects of the subject matter disclosed herein in one or more standalone computer systems, the present disclosure is not limited thereto and may be implemented in conjunction with any computing environment, such as a network or a distributed computing environment. Furthermore, aspects of the subject matter in the present disclosure may be implemented in a plurality of processing chips or devices, and storage may be similarly affected across a plurality of devices. Such devices may include PCs, network servers, and portable devices.

[0102] Although the present disclosure has been described in relation to some embodiments, various modifications and changes may be made without departing from the scope of the present disclosure as understood by a person skilled in the art to which the invention of the present disclosure pertains. Furthermore, such modifications and changes should be considered to fall within the scope of the claims appended to this specification.

Claims

1. A hierarchical artificial intelligence-based content stepwise generation method performed by at least one processor, wherein A step of generating a content creation prompt based on first condition information corresponding to initial constraints of target content including a target and creation target content; A step of generating a first content as initial content based on second condition information corresponding to the generated content generation prompt and the component constraints of the content to be generated; and A step of generating a second content as the target content by synthesizing the first content generated above and the target content above. A hierarchical AI-based content stepwise generation method including 2. In Paragraph 1, The above first condition information is, A hierarchical AI-based content stepwise generation method comprising at least one of the context in which the content to be generated is used, the size of the content to be generated, and the visual characteristic information of the content to be generated.

3. In Paragraph 1, The above second condition information is, A hierarchical AI-based content stepwise generation method comprising at least one of condition information regarding a first component to be included in the foreground of the content to be generated, condition information regarding a second component to be included in the background of the content to be generated, condition information regarding position and pose information of the first component and the second component, and condition information regarding external characteristics of the first component and the second component.

4. In Paragraph 1, The step of generating the above content creation prompt is, A step of generating a content creation prompt by analyzing the target content and the first condition information based on a previously trained language model. Includes, The content creation prompt generated above is, A hierarchical AI-based content stepwise generation method comprising one or more feature information about the target extracted from the target content, the first condition information, and a command instructing content generation based on the one or more feature information and the first condition information.

5. In Paragraph 1, The step of generating the first content above is, A step of generating a first image corresponding to text including the generated content generation prompt and the second condition information using a text-to-image conversion model. Includes, The first image generated above is, A hierarchical AI-based content stepwise generation method comprising a user corresponding to the generated content generation prompt and the second condition information, wherein the image includes the body part to which the target target is applied among the body parts of the user.

6. In Paragraph 5, The step of generating the second content above is, A step of generating a second image by synthesizing the first image generated above and the target content using an inpainting-based image synthesis model. Includes, The second image generated above is, A hierarchical AI-based content stepwise generation method, wherein the target target is filled into an area corresponding to a body part included in the first image generated based on the target target content, thereby including an image containing a user to whom the target target is applied.

7. In Paragraph 1, When feedback is obtained regarding the second content generated above, the step of individually correcting at least one of the first condition information and the second condition information by selecting it based on the attributes of the obtained feedback. A hierarchical AI-based content stepwise generation method including further 8. In Paragraph 7, The above correction step is, If the above-mentioned acquired feedback relates to initial constraints on the content to be generated, the step of correcting the first condition information based on the above-mentioned acquired feedback Includes, The step of generating the above content creation prompt is, A step of regenerating a content creation prompt based on the above target content and the above corrected first condition information. A hierarchical AI-based content stepwise generation method including 9. In Paragraph 7, The above correction step is, If the above-mentioned acquired feedback relates to the component constraints of the content to be generated, the step of correcting the second condition information based on the above-mentioned acquired feedback Includes, The step of generating the first content above is, A step of regenerating the first content based on the generated content creation prompt and the corrected second condition information. A hierarchical AI-based content stepwise generation method including 10. In Paragraph 7, The above correction step is, If the above-mentioned acquired feedback relates to the component constraints of the content to be generated, the step of correcting the second condition information based on the above-mentioned acquired feedback Includes, The step of generating the first content above is, A step of regenerating the first content based on the generated content creation prompt and the corrected second condition information. A hierarchical AI-based content stepwise generation method including 11. In the device, Communication module; display; Memory; and At least one processor connected to the memory and configured to execute at least one computer-readable program contained in the memory. Includes, The above at least one program is, A content creation prompt is generated based on first condition information corresponding to initial constraints on a product image including a target product and content to be created, and Obtaining second condition information corresponding to the constraints of the components to be included in the above-mentioned target content for generation, and Initial product content is generated based on the above-generated content generation prompt and the above-acquired second condition information, and A device comprising a command for generating final product content corresponding to the target content to be generated by reflecting the product image in the initial product content generated above.

Images