Webtoon generation method using webtoon generation ai

Abstract

The present invention relates to a webtoon generation method using webtoon generation AI that automatically converts a conte (sketch) of a webtoon creator into line art. The present invention comprises the steps of: (A) training a webtoon generation AI algorithm; (B) receiving conte data input from a user; and (C) generating a line art image from the conte data input through the webtoon generation AI. According to the present invention, a conversion time from a conte to line art is significantly reduced (by 90% or more), thereby enabling a webtoon creator to focus creative energy more on story planning and conte work, and thus significantly improving the efficiency of a webtoon production process.

Description

Webtoon generation method using webtoon generation AI

[0001] The present invention relates to the field of webtoon production, and more specifically, to a method for generating webtoons using a webtoon generation AI that automatically converts a webtoon artist's storyboard (sketch) into line art.

[0002]

[0003] In the webtoon production process, the transition from storyboards to line art is a task that requires significant time and effort. Traditional methods involved a manual process where artists directly drew line art based on storyboards, which has been a major factor in reducing work efficiency. Particularly for serialized webtoons, where new episodes must be produced weekly or bi-weekly, this repetitive and time-consuming work has placed a heavy burden on artists.

[0004] Although there have been remarkable achievements in the field of image processing and generation recently due to advancements in artificial intelligence and deep learning technologies, technology capable of automatically generating consistent character expressions while accurately reflecting the individual and unique style of webtoon artists has not yet been sufficiently developed. In particular, maintaining character consistency, preserving the artist's unique style, and ensuring a natural transition from storyboards to line art remained major challenges to be addressed.

[0005]

[0006] The present invention was devised to solve the aforementioned conventional problems, and the present invention aims to provide a webtoon generation method with significantly improved efficiency by converting a webtoon artist's storyboard into high-quality line art according to the characteristics of each artist.

[0007] Furthermore, the present invention aims to provide a webtoon generation method that accurately analyzes the artist's unique style and applies it to the line art generation process, thereby enabling the line art automatically generated through artificial intelligence to maintain the artist's original style.

[0008] In addition, the present invention aims to provide a method for generating a webtoon that recognizes character information indicated in a storyboard and, based thereon, generates new line art while maintaining the characteristics of each character consistently.

[0009] And the present invention aims to provide a method for generating webtoons that applies deepfake technology to precisely generate a character's face and facial expressions, thereby enriching the character's emotional expressions.

[0010] Furthermore, the present invention aims to provide a webtoon generation method that minimizes the artist's intervention while allowing the artist to intervene at the line art stage if necessary, thereby minimizing work errors and modifications.

[0011]

[0012] According to the features of the present invention for achieving the above-mentioned purpose, the present invention comprises: (A) a step of learning a webtoon generation AI algorithm; (B) a step of receiving storyboard data from a user; and (C) a step of generating a line art image from the storyboard data input through the webtoon generation AI.

[0013] At this time, the above step (A) may be performed by a webtoon generation algorithm learning unit training a webtoon generation AI algorithm using the webtoon content of a designated author as training data.

[0014] And the above AI algorithm is composed of a ResNet (Residual Network) architecture, and can also map by assigning the line thickness element of the image to the feature vector of the ResNet architecture.

[0015] Additionally, the above step (A) may be performed by a webtoon generation algorithm learning unit comprising: (A1) a step of classifying the work data of a designated author chronologically to analyze the line art stage; (A2) a step of classifying the work data by line art stage to generate learning data for each line art stage; and (A3) a step of learning the webtoon generation algorithm for each line art stage through the learning data for each line art stage.

[0016] And in step (C) above, line art images for each line art stage may be generated through the webtoon generation algorithm learned from the storyboard data in step (A3) above.

[0017] In addition, the line art images of step (C) above may be generated for each object by dividing the storyboard data by object.

[0018] And the above object may be composed of multiple characters, backgrounds, and text.

[0019] In addition, the above step (C) may provide the line art image of the above line art stage to the user and receive the user's approval command to generate the line art image of the next stage.

[0020] And the above step (C) may provide the line art image of the above line art stage to the user, and if the user inputs image modification data, may generate the line art image of the next stage for the line art image reflecting the input image modification data.

[0021] In addition, when user image modification data is input by step (C), the above step (A) may generate line art step-by-step learning data for the generated line art image to learn the webtoon generation algorithm.

[0022] And the above step (A) may be repeated so that the FID (Frechet Inception Distance) score between the line art image generated without reflecting the user image modification data and the webtoon content of the specified author becomes less than or equal to a preset value.

[0023] In addition, the above FID score may be calculated for each identical object by extracting objects included in the above line art image and webtoon content.

[0024] And the identification and recognition of the above objects may also be performed by a YOLO model having a backbone structure of CSPDarknet53.

[0025] In addition, the present invention may further include the steps of: (D) generating an ignition image from a line drawing image generated through the webtoon generation AI; and (E) generating webtoon content by adding a background image and a text image to the ignition image through the webtoon generation AI.

[0026] Meanwhile, the learning of the webtoon generation AI algorithm in step (A) of the present invention is performed by including an image style analysis process for the learning data, wherein the image style analysis process may be performed by using a convolutional neural network (CNN) to extract feature vectors of the author's style; fine-tuning a pre-trained model for each author through transfer learning; and determining whether the learning is complete by evaluating a style consistency index.

[0027] And the style consistency index evaluation of step (A) above may be calculated by defining a Gram Matrix-based style loss function between the generated line art image and the training data; measuring the similarity between the generated image and the original style by calculating the Frechet Inception Distance (FID); and evaluating the overall degree of structural preservation using the Structural Similarity Index (SSIM).

[0028] Additionally, the above step (A) may be performed by including a preprocessing process for the stored training data, and the preprocessing process may be performed through an image normalization process that converts the stored training data to a standard resolution and color space; a noise removal process that removes noise generated during the scanning process by combining an intermediate filter and a Gaussian filter; and an image enhancement process that diversifies the target image through rotation, inversion, and scaling of the image.

[0029]

[0030] The following effects can be expected from the webtoon generation method according to the present invention as described above.

[0031] In other words, the present invention significantly reduces the time required to convert from storyboards to line art (by more than 90%), allowing the author to focus their creative energy more on story conception and storyboarding, thereby having the effect of greatly improving the efficiency of the webtoon production process.

[0032] In addition, the present invention has the effect of accurately reproducing the author's unique line texture, shading processing, etc. through deep learning-based style analysis, thereby enabling the creation of webtoon content that maintains consistency in the author's style.

[0033] In addition, the present invention provides character descriptions that are consistent throughout the episode by classifying and learning features by character, and has the effect of increasing the richness of the character's facial expressions and emotional expressions through the use of deepfake technology.

[0034] Furthermore, the present invention significantly reduces the time required for line art work, thereby shortening the production period of Webkoon content and consequently alleviating the economic burden on the production company.

[0035] In addition, the present invention allows for freeing from repetitive tasks and enabling more time to be invested in developing new stories and characters, which facilitates experimentation with various styles and results in an improvement in the quality of the work.

[0036]

[0037] FIG. 1 is a block diagram illustrating the configuration of a webtoon generation system using a webtoon generation AI according to a specific embodiment of the present invention.

[0038] FIG. 2 is a flowchart illustrating a method for generating a webtoon using a webtoon generation AI according to a specific embodiment of the present invention.

[0039] FIG. 3 is a flowchart illustrating an example of the learning process of a webtoon generation AI according to the present invention.

[0040] FIG. 4 is a flowchart illustrating a method for generating a webtoon using a webtoon generation AI according to a specific embodiment of the present invention.

[0041] FIGS. 5 to 7 are exemplary diagrams illustrating an example in which webtoon content is generated from storyboard data according to the present invention.

[0042] FIGS. 8 to 12 are exemplary diagrams illustrating other examples in which webtoon content is generated from storyboard data according to the present invention.

[0043]

[0044] A method for generating a webtoon using a webtoon-generating AI according to the present invention as described above comprises: (A) a step of learning a webtoon-generating AI algorithm; (B) a step of receiving storyboard data from a user; (C) a step of generating a line art image from the input storyboard data through the webtoon-generating AI; (D) a step of generating an illustration image from the line art image generated through the webtoon-generating AI; and (E) a step of generating webtoon content by adding a background image and a text image to the illustration image through the webtoon-generating AI; wherein step (A) comprises: (A1) a step of classifying the work data of a designated artist chronologically to analyze the line art stage; (A2) a step of classifying the work data by line art stage to generate learning data for each line art stage; (A3) A step of learning the webtoon generation algorithm for each line drawing step through the learning data for each line drawing step; wherein the step (C) preferably generates a line drawing image for each line drawing step through the webtoon generation algorithm learned for each line drawing step by the storyboard data in step (A3).

[0045]

[0046] Hereinafter, we will examine a method for generating a webtoon using a webtoon generation AI according to a specific embodiment of the present invention with reference to the attached drawings.

[0047] Before proceeding with the explanation, the effects, features, and methods for achieving the present invention will become clear from the embodiments described below in detail together with the accompanying drawings. However, the present invention is not limited to the embodiments disclosed below but can be implemented in various different forms. These embodiments are provided merely to ensure that the disclosure of the present invention is complete and to fully inform those skilled in the art of the scope of the invention, and the present invention is defined only by the scope of the claims.

[0048] In describing the embodiments of the present invention, if it is determined that a detailed description of known functions or configurations may unnecessarily obscure the essence of the invention, such detailed description will be omitted. Furthermore, the terms described below are defined considering the functions in the embodiments of the present invention, and these may vary depending on the intentions or conventions of the user or operator. Therefore, such definitions should be based on the content throughout this specification.

[0049] Combinations of each block of the attached block diagram and each step of the flowchart may be executed by computer program instructions (execution engine), and since these computer program instructions may be loaded into the processor of a general-purpose computer, a special-purpose computer, or other programmable data processing equipment, the instructions executed through the processor of the computer or other programmable data processing equipment create a means to perform the functions described in each block of the block diagram or each step of the flowchart.

[0050] Since these computer program instructions may be stored in computer-available or computer-readable memory that can be directed toward a computer or other programmable data processing equipment to implement a function in a specific way, the instructions stored in said computer-available or computer-readable memory may also be used to produce a manufactured item containing instruction means that perform the function described in each block of a block diagram or each step of a flowchart.

[0051] And, since computer program instructions can be loaded onto a computer or other programmable data processing equipment, instructions that perform a series of operation steps on a computer or other programmable data processing equipment to create a process executed by a computer and that execute the computer or other programmable data processing equipment may also provide steps for executing the functions described in each block of the block diagram and each step of the flowchart.

[0052] Additionally, each block or each step may represent a module, segment, or part of code containing one or more executable instructions for executing specific logical functions, and in some alternative embodiments, the functions mentioned in the blocks or steps may occur out of order.

[0053] In other words, the two blocks or steps described can actually be performed substantially simultaneously, and can also be performed in the reverse order of their corresponding functions as needed.

[0054]

[0055] Hereinafter, a method for generating a webtoon using the webtoon generation AI according to the present invention as described above will be explained in detail with reference to the attached drawings.

[0056] FIG. 1 is a block diagram illustrating the configuration of a webtoon generation system using a webtoon generation AI according to a specific embodiment of the present invention, FIG. 2 is a flowchart illustrating a method for generating a webtoon using a webtoon generation AI according to a specific embodiment of the present invention, FIG. 3 is a flowchart illustrating an example of a learning process of a webtoon generation AI according to the present invention, FIG. 4 is a flowchart illustrating a method for generating a webtoon using a webtoon generation AI according to a specific embodiment of the present invention, FIG. 5 is an illustrative diagram illustrating an example of generating webtoon content from storyboard data according to the present invention, and FIG. 6 is an illustrative diagram illustrating another example of generating webtoon content from storyboard data according to the present invention.

[0057] First, as illustrated in FIG. 1, a webtoon generation system implementing the present invention is configured to include a webtoon generation algorithm learning unit (100), a data input unit (200), a webtoon content generation unit (300), and a storage unit (400).

[0058] The above webtoon generation algorithm learning unit (100) is a part that learns an artificial intelligence algorithm for generating webtoon content. The above webtoon generation algorithm learning unit (100) learns a webtoon generation AI algorithm for each author, but since the drawing style of the character may change for each work even for the same author, it is more preferable that the webtoon generation AI algorithm is learned for each work.

[0059] In addition, the webtoon generation algorithm learning unit (100) can learn the line art generation process by dividing it into line art stages according to each artist's line art style, and for this purpose, the work process of each artist can be stored as time-series data in the storage unit.

[0060] And the above data input unit (200) is a part that receives storyboard (sketch) data (image) from a user, and various input means such as an image generation tablet, a scanner, etc. can be applied.

[0061] At this time, the storyboard data refers to various types of images that visually organize the overall flow of the webtoon in advance, such as images showing character types, placement, features, and dialogue along with a schematic composition, as illustrated in FIG. 5a or FIG. 6a.

[0062] In addition, the webtoon content generation unit (300) is a part that generates webtoon content using the webtoon generation AI algorithm based on the input storyboard data.

[0063] At this time, the webtoon content generation unit (300) classifies the line art stages by reflecting the line art style of each artist, generates the line art images for each line art stage, and then generates the final webtoon content, which will be explained in detail later.

[0064] And the storage unit (400) is a part where the learning data to be learned by the webtoon generation algorithm learning unit (100) and the webtoon content generated by the webtoon content generation unit (300) are stored.

[0065] In this case, while the training data primarily stores webtoon images generated for each author and work, it is desirable to store the entire work data of the corresponding work for learning at the line art stage.

[0066] That is, the above work data refers not only to completed webtoon images but also to data that stores the process of the artist drawing the webtoon images. When the artist works using electronic devices, such work data can be obtained from work equipment such as a tablet.

[0067] And the webtoon content generated by the webtoon content generation unit (300) is stored in the storage unit (400), and the webtoon content generated by the webtoon content generation unit (300) can also be used as training data.

[0068] Meanwhile, the above webtoon content generation unit (300) is configured to include an object recognition module (310), an image generation module (320), a consistency maintenance module (330), and an output module (340) in order to generate webtoon content.

[0069] The object recognition module (310) is for recognizing, classifying, and identifying objects included in the storyboard data and training data, and can be implemented with various image identification algorithms, but preferably can be implemented by a YOLO model having a backbone structure of CSPDarknet53.

[0070] And the image generation module (320) is a part that generates webtoon content, and as shown in FIG. 1, it may be configured to include a line drawing image generation unit (321), a drawing image generation unit (323), a background image generation unit (325), and a text image generation unit (327).

[0071] The above line art image generation unit (321) is a part that generates line art images using storyboard data, and various image generation algorithms may be applied, but preferably it is implemented based on a Conditional Generative Adversarial Network (Conditional GAN).

[0072] The configuration and function of the above-mentioned line drawing image generation unit (321) will be explained in detail again.

[0073] And the above-mentioned ignition image generating unit (323) is a part that generates a colored ignition image by adding color to the line drawing image generated in the above-mentioned line drawing image generating unit (321).

[0074] Additionally, the background image generation unit (325) is a part that adds a background image to the ignition image generated by the ignition image generation unit (323).

[0075] And the text image generation unit (327) is a part that completes webtoon content by adding speech bubbles and other text to an image with a background image added.

[0076] The main objective of the present invention is to maximize the efficiency of webtoon production by automating line art work, which requires the most human resources from the author during webtoon production and cannot be replaced by other personnel. Therefore, while it is essential to generate line art images from storyboard data through the line art image generation unit (321), it is also possible to optionally apply the line art images, background images, and text through the line art image generation unit (323), background image generation unit (325), and text image generation unit (327).

[0077] Meanwhile, the consistency maintenance module (330) is a deepfake technology applied to maintain consistency of the same object (specific character), and 1) generates various angles and expressions of the character using 3D face modeling technology, 2) predicts character movement in consecutive frames using a sequence-to-sequence model, and 3) evaluates and corrects the consistency maintenance results.

[0078] And the output module (340) is intended to provide the generated webtoon content to the user, and various output means may be applied, but preferably, an image work tablet is applied as an output means so that the user can input modification data during the line art image generation stage.

[0079]

[0080] The detailed functions of the main components constituting the webtoon generation system as described above will be explained below together with the webtoon generation method according to the present invention.

[0081] As illustrated in FIG. 2, the webtoon generation method according to the present invention begins with a webtoon generation algorithm learning unit (100) learning a webtoon generation AI algorithm (S100).

[0082] At this time, the learning of the webtoon generation AI algorithm is performed by the webtoon generation algorithm learning unit (100) using the webtoon content of a designated author as learning data, and preferably, if there are multiple works by the author, the webtoon generation AI algorithm is trained for each work.

[0083] In addition, the above AI algorithm can be composed of a ResNet (Residual Network) architecture, and by assigning and mapping the line thickness element of the image to the feature vector of the ResNet architecture, the AI ​​algorithm can be trained by accurately reflecting the line art characteristics of each artist according to line thickness.

[0084] The characteristics of line art images by artist are basically determined by the drawing style, but even in cases of similar drawing styles, the characteristics of line art images are distinguished by the thickness of the lines used.

[0085] Therefore, if the above webtoon generation AI algorithm is configured as a ResNet architecture and trained by assigning image line thickness elements as feature vectors, it becomes possible to learn detailed features specific to each author.

[0086] Meanwhile, the sequence of line art work for webtoon images varies depending on the line art style of each artist, and the aforementioned webtoon generation AI algorithm learns the line art work process for each artist (work) in order to synchronize the line art image generation process with the artist's work sequence.

[0087] For example, depending on the artist's style, some artists may create line art for each figure according to the importance of each figure, others may create line art by dividing the entire image according to the thickness of the lines, and still others may draw specific body parts (eyes, nose, mouth, etc.) of each figure first and then draw other body parts based on them.

[0088] In the present invention, the reason the webtoon generation AI algorithm learns the line art image generation process is to facilitate modification of the generated line art image by the corresponding author. If modification is required at each stage of the line art, the user (author) inputs the modification details so that the subsequent image generation process can be carried out based on the modified image, thereby minimizing the user's modification process.

[0089] A specific embodiment of the learning process of the webtoon generation AI algorithm of the above 100th step, as shown in FIG. 3, begins with classifying the line drawing stage of the training data (S110).

[0090] That is, the above webtoon generation algorithm learning unit (100) classifies the work data of a designated author in a time series and analyzes the line art stage.

[0091] At this time, the above work data is data containing the work process of the artist (or work), and can be collected by storing the process of the artist performing the work through electronic input equipment.

[0092] Accordingly, the above line drawing stage may be distinguished differently depending on the characteristics of the artist's work; for example, depending on the artist's style, it may be set by character, type of line (thickness), or specific body parts of the character (eyes, nose, mouth, etc.).

[0093] Next, the webtoon generation algorithm learning unit (100) classifies the work data by line art stage and generates line art stage learning data (S120).

[0094] The aforementioned line art stage training data refers to the process of classifying the line art work into stages and organizing the intermediate results of each stage in a chronological order.

[0095] Afterwards, the webtoon generation algorithm learning unit (100) trains the webtoon generation AI algorithm through learning the training data for the generated line art stage.

[0096] Accordingly, the above webtoon generation AI algorithm learns the line art stage results of the corresponding author and becomes capable of generating line art images for each stage.

[0097] Through this process, after the webtoon generation AI algorithm is trained, the webtoon content generation unit (300) receives storyboard data from the user and generates webtoon content.

[0098] Looking at the process for this, as illustrated in FIG. 2, the webtoon content generation unit (300) receives storyboard data from the user (S200).

[0099] And the webtoon content generation unit (300) generates a line art image from the input storyboard data through the learned webtoon generation AI (S300).

[0100] At this time, when the webtoon generation AI is trained at each line art stage, the webtoon content generation unit (300) generates line art images at each line art stage and obtains approval from the user at each stage to generate line art images.

[0101] Specifically, as illustrated in FIG. 4, when storyboard data is input from a user, the object recognition module (310) of the webtoon content generation unit (300) identifies objects included in the storyboard data (S310).

[0102] In this way, the line art images may be generated for each object by dividing the storyboard data by object, but it is also possible to omit the above 310 step and generate line art images for the entire storyboard as a whole, or to group specific objects and generate line art images sequentially.

[0103] Here, the object may be multiple characters, backgrounds, or text, etc.

[0104] Meanwhile, the identification and recognition of the above object can be performed by a YOLO model having a backbone structure of CSPDarknet53.

[0105] Afterwards, the image generation module (320) of the above webtoon content generation unit (300) determines the line art stage set for the user (or work) and generates a line art image corresponding to the first stage (S320, S330).

[0106] Specifically, the line art image is generated by the line art image generation unit (321) of the image generation module (320).

[0107] Next, the line art image generation unit (321) receives confirmation from the user regarding approval of the generated line art image (S340).

[0108] At this time, if approval is confirmed by the user as a result of the verification in step 340 above, it is determined whether the line drawing stage in which the line drawing image was generated is the final stage, and if it is not the final line drawing stage, the line drawing image of the next line drawing stage is generated (S370).

[0109] Meanwhile, if, as a result of the verification in step 340 above, approval from the user is not confirmed and a modification data is received for the generated line art image (S350), the line art image generation unit (321) modifies the line art image by reflecting the modification data (S360).

[0110] Here, the aforementioned modified data refers to the user (artist) modifying the generated line art image.

[0111] For example, when the line art image generation unit (321) provides the generated line art image to the user, the user may approve the generated line art image to generate the next stage line art image, or if it is determined that there is a part to modify, the user may modify the provided line art image and input it.

[0112] Of course, in this case, modifications to the line art image can be performed via the user's tablet.

[0113] At this time, the user modifies the line art image and inputs the modified data, and the line art image generation unit (321) generates the next step of the line art image based on the modified line art image.

[0114] In this way, by generating the above line art images by classifying them into line art stages by artist (or work) and verifying approval and revision history, compared to generating line art images all at once, it is possible to minimize the artist's revision process and prevent repetitive revision work, and there is an effect of obtaining a line art image satisfactory to the artist with minimal error processes.

[0115] The above-mentioned line drawing image generation unit (321) repeats steps 310 through 370 up to the final line drawing stage to generate a final line drawing image (S380).

[0116] Figures 5 and 6 illustrate an example in which webtoon content is generated from storyboard data (images) in this manner.

[0117] As illustrated herein, an example is shown in which a line art image (Figs. 5b, Fig. 6b) is generated from storyboard data (image) (Figs. 5a, Fig. 6a) containing the approximate shape (proportion, direction, etc.) of a character and text, and a webtoon content is generated by adding a drawing image, a background image, and text.

[0118] As described above, after the line drawing image is generated, the ignition image generation unit (323) of the image generation module (320) colors the line drawing image to generate an ignition image (S400).

[0119] Of course, if the webtoon itself is uncolored, the coloring process can be omitted.

[0120] Next, the background image generation unit (325) of the image generation module (320) generates a background image (S500), and the text image generation unit (327) of the image generation module (320) generates a speech bubble (S600), thereby finally generating webtoon content (S700).

[0121] At this time, it is possible to recognize the background as a single object so that the creation of the background image is performed together with the creation of the line art image, and it is also possible to recognize the text including the speech bubble as a single object so that it is created together with the line art image.

[0122] Meanwhile, the learning process of the webtoon generation AI algorithm in steps 110 to 130 above can be continuously performed using the webtoon content generated through steps 300 to 700 above as learning data, and can be maintained and learned so that the learning result of the webtoon generation AI algorithm is above a certain level.

[0123] At this time, the learning level of the webtoon generation AI algorithm can be determined by the FID (Frechet Inception Distance) score between the final line art image generated by the webtoon generation AI algorithm without performing steps 350 and 360 and the webtoon content of the corresponding author stored as training data. That is, as described above, training can be repeated so that the FID score becomes less than or equal to a preset value.

[0124] Meanwhile, the above FID score may also be calculated for each identical object by extracting objects included in the above line art image and webtoon content.

[0125]

[0126] Below, we will examine a specific system configuration for generating webtoon content using the webtoon generation AI according to the present invention.

[0127] First, looking at the CPU (Central Processing Unit) configuration, the CPU utilizes high-performance multi-core processors such as the Intel Xeon or AMD EPYC series, enabling the execution of complex artificial intelligence algorithms through massive parallel processing capabilities.

[0128] Next, looking at the configuration of the GPU, the GPU uses a high-performance GPU such as the NVIDIA Tesla V100 or A100 series to accelerate the training and inference processes of the deep learning model.

[0129] And if we look at the configuration of the memory, the memory uses high-speed DDR4 RAM with a processing speed of at least 128GB to secure sufficient memory space for rapid processing of large datasets and intermediate results.

[0130] Next, for storage devices, a combination of a high-speed NVMe SSD of at least 1TB and a large-capacity HDD of at least 10TB is used to simultaneously support fast data access and storage of large-capacity training data.

[0131] In addition, the network interface is configured to support high-speed Ethernet of 10Gbps or higher, providing a high-bandwidth network for efficient data transmission in a distributed processing environment.

[0132] Meanwhile, the input / output interface utilizes high-speed data transfer ports such as USB 3.1 and Thunderbolt 3 to ensure compatibility with various input devices (tablets, scanners, etc.).

[0133]

[0134] Next, the key technologies applied in the data processing and image processing of the present invention will be described.

[0135]

[0136] 1. Input Data Preprocessing

[0137] A preprocessing process to ensure that the identification and recognition of the conti data received in the above 200th step is clear is performed through 1) image normalization, 2) noise removal, and 3) image augmentation. Of course, the preprocessing process can also be applied to the stored training data.

[0138]

[0139] First, 1) the above image normalization process converts the input storyboard data into a standard resolution and color space, specifically, through a process of ① resizing the image to be processed to a size of 512x512 pixels, ② normalizing each channel value of the resized image to a range of 0-1 using the RGB color space, and ③ improving image sharpness by performing contrast-limited adaptive histogram equalization (CLAHE).

[0140]

[0141] Next, 2) the noise removal process involves removing noise generated during the scanning process by combining a median filter and a Gaussian filter. Specifically, ① salt-and-pepper noise is removed by applying a median filter with a kernel size of 3x3, ② overall noise is reduced using a Gaussian filter with a standard deviation of 0.5, and ③ noise is removed while preserving edges using an anisotropic diffusion filter.

[0142]

[0143] And, 3) the image augmentation process can be performed by diversifying the training data through rotation / inversion / scale change, specifically by ① randomly rotating within a range of ±15 degrees, ② randomly inverting in horizontal and vertical directions, ③ randomly adjusting brightness and contrast within a range of ±10%, ④ randomly zooming in / out within a range of up to 20%, or ⑤ applying a cutout technique to improve the robustness of the model by randomly masking a part of the image.

[0144]

[0145] 2. Image Style Analysis

[0146] The style analysis of the image is basically applied to train the webtoon generation AI algorithm in the above 100th step, and is performed by 1) extracting feature vectors of the author's style using a convolutional neural network (CNN), 2) fine-tuning the pre-trained model for each author through transfer learning, and 3) applying an indicator for evaluating style consistency.

[0147]

[0148] First, 1) feature vector extraction based on a Convolutional Neural Network (CNN) involves ① using a pre-trained model based on the ResNet-152 architecture, ② removing the last fully connected layer and adding a global average pooling layer, and ③ using the extracted 2048-dimensional feature vector for the author style representation.

[0149]

[0150] Next, 2) fine-tuning the model for each author through transfer learning is performed by: ① using a dataset consisting of 100-500 existing works by each author; ② training only the last 3 convolutional layers and the newly added fully connected layer; ③ setting the learning rate to 0.0001, the Adam optimizer, and the batch size to 32; and ④ applying an Early Stopping technique where training is stopped if the validation loss does not improve for 10 epochs.

[0151]

[0152] And the style consistency evaluation metric defines ① a Gram Matrix-based style loss function, ② measures the similarity between the generated image and the original style by calculating the Frechet Inception Distance (FID), and ③ evaluates the overall degree of structural preservation using the Structural Similarity Index (SSIM).

[0153]

[0154] 3. Character Recognition and Classification

[0155] Character recognition and classification involves classifying and recognizing characters among objects within the training data during the learning process of the webtoon generation AI algorithm. This is achieved by 1) identifying the location of characters within the storyboard using a multi-scale object detection algorithm, 2) extracting character names using OCR combined with a recurrent neural network (RNN) and an attention mechanism, and 3) learning unique features for each character and building a database through a face recognition algorithm.

[0156]

[0157] First, 1) the multi-scale object detection process is as follows: ① CSPDarknet53 is adopted as the backbone network using the YOLOv5 architecture; ② the anchor box sizes are set to [(10,13), (16,30), (33,23), (30,61), (62,45), (59,119), (116,90), (156,198), (373,326)]; ③ predictions are performed at three scales (13x13, 26x26, 52x52); and ④ duplicate detections are removed by applying a non-maximum suppression (NMS) threshold of 0.45.

[0158]

[0159] Next, 2) character name extraction through OCR uses ① a CRNN (Convolutional Recurrent Neural Network) model, ② a convolutional layer with a 3x3 kernel, 32-64-128-256 channel structure, ③ a bidirectional LSTM 2-layer (256 hidden units) structure, ④ a CTC (Connectionist Temporal Classification) loss function, and ⑤ data augmentation with random gradient and elastic distortion.

[0160]

[0161] And 3) the face recognition algorithm is based on the FaceNet architecture, uses the Inception-ResNet-v1 backbone, is trained with the Triplet Loss function, sets the margin value to 0.2, applies the Online Hard Negative Mining technique, generates a 128-dimensional face embedding vector, and identifies characters by measuring cosine similarity, determining them to be the same character when the threshold is 0.7 or higher.

[0162]

[0163] Such character identification and classification methods are optimized for webtoon images; however, if the method of identifying people through real photographs is applied, there is a problem in that recognition errors increase, where the same characters are perceived as different depending on factors such as exaggerated facial expressions, magnification of specific body parts, and the character's orientation.

[0164]

[0165] Accordingly, through the method described above, it becomes possible to provide an identification / recognition algorithm with optimized identification power for webtoon images.

[0166]

[0167] 4. Line Art Conversion Algorithm

[0168] The line art conversion algorithm is an algorithm that generates line art images from storyboard images, and refers to the webtoon generation AI algorithm of the present invention, which 1) converts the storyboard into line art based on a Conditional Generative Adversarial Network (Conditional GAN), 2) maintains a balance between the overall structure and detailed features by introducing a multi-discriminator structure, and 3) performs differential processing based on the importance of characters and backgrounds by utilizing an attention mechanism.

[0169]

[0170] Specifically, 1) the Conditional Generative Adversarial Network (Conditional GAN) applies ① a U-Net structure to the generator, with 8 encoder-decoder blocks each, ② a PatchGAN structure to the discriminator, which determines authenticity in 70x70 patch units, ③ an adversarial loss + L1 loss (weights 100:1) to the loss function, ④ batch normalization and the ReLU activation function, and ⑤ a Dropout layer (rate=0.5) to prevent overfitting.

[0171]

[0172] Next, 2) the multiple discriminator structure is used by combining ① a global discriminator that evaluates the consistency of the entire image, ② a local discriminator that evaluates the detail quality for a 64x64 patch, and ③ the loss of each discriminator in a 1:1 ratio.

[0173]

[0174] And 3) the attention mechanism applies ① a spatial attention module consisting of a 3x3 convolutional layer and a sigmoid activation function, ② a channel attention module using a Squeeze-and-Excitation block, ③ inserts the attention module into the intermediate layer of the generator, and ④ additionally provides an attention map visualization function as a basis for the model's decision.

[0175]

[0176] 5. Consistency Maintenance Techniques

[0177] The consistency maintenance technology is a consistency maintenance module (300) that applies deepfake technology to maintain consistency of the same object (specific character), and 1) generates various angles and expressions of the character using 3D face modeling technology, 2) predicts character movement in consecutive frames using a sequence-to-sequence model, and 3) evaluates and corrects the consistency maintenance results.

[0178]

[0179] Specifically, 1) 3D face modeling technology ① reconstructs the face based on 3DMM (3D Morphable Model), ② extracts 68 face landmark points, ③ controls geometric deformation and expression changes with separate parameters, and ④ applies a Phong shading model during the rendering process.

[0180]

[0181] And 2) the sequence-to-sequence model uses ① an LSTM network with an encoder-decoder structure, ② a 2-layer bidirectional LSTM (512 hidden units) encoder, ③ a 2-layer unidirectional LSTM (512 hidden units) decoder, and ④ an attention mechanism using Bahdanau attention.

[0182] In addition, ⑤ the length of the input sequence is set to 10 frames and the length of the output sequence is set to 5 frames, and ⑥ the Teacher Forcing technique is applied so that the actual previous frame is used with an 80% probability during training.

[0183]

[0184] In addition, 3) consistency evaluation and correction involves ① checking the consistency of motion based on optical flow, ② estimating motion between consecutive frames using the FlowNet 2.0 architecture, ③ applying a smoothing algorithm when abrupt changes in motion vectors are detected, and ④ introducing Cycle Consistency Loss for long-term consistency.

[0185]

[0186] 6. Post-processing and quality improvement

[0187] Post-processing and quality enhancement technologies can be applied optionally to improve the quality of generated webtoon content through 1) super-resolution processing, 2) noise removal and line refinement, and 3) style transfer fine-tuning.

[0188]

[0189] Specifically, 1) super-resolution processing is performed using ESRGAN (Enhanced Super-Resolution Generative Adversarial Networks), ① applying a 4x upscaling model (512x512 → 2048x2048), ② adopting a Residual-in-Residual Dense Block (RRDB) structure, and ③ processing combined with Perceptual Loss and Texture Loss.

[0190]

[0191] And 2) noise removal and line refinement are performed by ① removing fine noise using the BM3D (Block-Matching and 3D filtering) algorithm, ② refining lines through morphological operations, and ③ sequentially applying opening operations followed by closing operations (kernel size: 3x3).

[0192]

[0193] In addition, 3) fine-tuning of style transfer involves ① performing style transfer based on AdaIN (Adaptive Instance Normalization), ② extracting features from the 'relu1_1', 'relu2_1', 'relu3_1', and 'relu4_1' layers of the VGG-19 network, and ③ setting the weight ratio of content loss to style loss to 1:10.

[0194]

[0195] The rights of the present invention are not limited to the embodiments described above but are defined by the claims, and it is obvious that a person skilled in the art may make various modifications and adaptations within the scope of the rights described in the claims.

[0196]

[0197] The present invention relates to a method for generating webtoons using a webtoon generation AI that automatically converts a webtoon artist's storyboard (sketch) into line art. According to the present invention, the time required to convert from storyboard to line art is significantly reduced (by more than 90%), allowing the artist to focus their creative energy more on story conception and storyboard work, thereby significantly improving the efficiency of the webtoon production process.

Claims

1. (A) A step for training a webtoon generation AI algorithm and; (B) A step of receiving storyboard data from the user; (C) A method for generating a webtoon using a webtoon-generating AI, characterized by including the step of generating a line art image from the storyboard data input through the webtoon-generating AI.

2. In Paragraph 1, The above (A) step is, A method for generating a webtoon using a webtoon-generating AI, characterized in that a webtoon generation algorithm learning unit is performed by training a webtoon generation AI algorithm using webtoon content of a designated author as training data.

3. In Paragraph 2, The above webtoon generation AI algorithm is, It is composed of a ResNet (Residual Network) architecture, but, A method for generating webtoons using a webtoon generative AI characterized by mapping image line thickness elements by assigning them to feature vectors of a ResNet architecture.

4. In Paragraph 2, The line art image of step (C) above is, A method for generating a webtoon using a webtoon-generating AI, characterized by dividing the above-mentioned storyboard data by object and generating each object.

5. In Paragraph 4, The above object is, A method for generating a webtoon using a webtoon-generating AI characterized by being composed of multiple characters, backgrounds, and text.

6. In Paragraph 5, The identification and recognition of the above object is, A method for generating webtoons using a webtoon generative AI characterized by being performed by a YOLO model having a backbone structure of CSPDarknet53.

7. In Paragraph 5, The learning of the webtoon generation AI algorithm in step (A) above is, It is performed including an image style analysis process for the training data, but, The above image style analysis process is, Extract feature vectors of the author's style using a Convolutional Neural Network (CNN); Fine-tuning a pre-trained model for each author through transfer learning; A method for generating webtoons using a webtoon-generating AI characterized by being performed by evaluating style consistency indicators to determine whether learning is complete.

8. In Paragraph 7, The evaluation of style consistency indicators in Step (A) above is, Define a Gram Matrix-based style loss function between the generated line art images and the training data; Calculates the Frechet Inception Distance (FID) to measure the similarity between the generated image and the original style; A method for generating webtoons using a webtoon-generating AI characterized by evaluating the degree of overall structural preservation using the Structural Similarity Index (SSIM).

9. In Paragraph 8, The above FID score is, A method for generating a webtoon using a webtoon-generating AI characterized by extracting objects included in the above-mentioned line art image and webtoon content, and producing output for each identical object.

10. In any one of paragraphs 1 through 9, The above (A) step is, It is performed including a preprocessing process for the stored training data, and The above preprocessing process is, An image normalization process that converts stored training data into a standard resolution and color space; A noise removal process that removes noise generated during the scanning process by combining an intermediate filter and a Gaussian filter; and A method for generating a webtoon using a webtoon-generating AI, characterized by being performed through an image augmentation process that diversifies the target image through rotation, inversion, and scaling changes of the image.

11. In Paragraph 10, (D) A step of generating an ignition image from a line art image generated through the above webtoon generation AI; (E) A method for generating a webtoon using a webtoon-generating AI, further comprising the step of generating webtoon content by adding a background image and a text image to the ignition image through the webtoon-generating AI.

Images