Ai-based text-to-comic device and method

Abstract

The present application discloses an AI-based text-to-comic device and method. A script parsing module extracts image parameters, and in combination with LoRA model training and a Stable Diffusion framework, a Regional Prompter plug-in, a ControlNet plug-in, and an Adetailer plug-in are used to achieve multi-dimensional control of character positioning, actions, and scene style, so as to generate highly consistent keyframe images; and a local repainting module repairs details by means of segmentation and inpainting technology, and finally a comic or a video is synthesized. The present invention solves the problem of inconsistent character appearances and incoherent scenes in images generated by existing AI tools, significantly improves the controllability and efficiency of generated content, reduces manual adjustment costs, and is applicable to automated generation of comic creation and storyline explanation videos.

Description

An AI-based graphic novel device and method Technical Field

[0001] This invention relates to the fields of artificial intelligence and image generation technology, specifically to an AI-based comic book creation device and method, which uses controllable image generation technology to achieve consistency of characters, actions, and scenes in comic book or video production. Background Technology

[0002] With the development of artificial intelligence technology, AI is being applied more and more widely in the field of image generation. While existing AI-generated comics tools can produce comic images with a certain artistic effect, they suffer from shortcomings in the controllability of the generated images. Specifically:

[0003] Inconsistent character and scene appearance: Due to the randomness of the generated model, it is difficult to maintain consistency between character appearance and scene in different images. For example, the facial features, clothing details, or body shape of the same character may differ in different images, and background elements (such as architectural style and lighting effects) cannot remain consistent in consecutive frames. This results in poor continuity in comics or videos.

[0004] Difficulty in controlling movement and facial expressions: Existing AI tools struggle to accurately control the movements and expressions of each character when generating multi-person scenes. For example, when generating interactive scenes with multiple people, there are significant discrepancies between character positioning, body language, and the descriptions in the prompts. This results in generated images that do not meet expectations.

[0005] Insufficient detail restoration capabilities: The generated images are prone to distortion in details, such as the face and limbs of people, which affects the viewing experience.

[0006] In existing technologies, some solutions improve the generation effect by fine-tuning pre-trained models, but these suffer from high training costs and poor flexibility; other solutions rely on manual post-processing corrections, which are inefficient. Therefore, there is an urgent need for an AI-generated comics technology that can achieve high controllability and ensure the coherence and consistency of comics or videos. Summary of the Invention

[0007] This invention aims to provide an AI-based comic strip production device and method. By training a dedicated LoRA model and combining Stable Diffusion and its functional plugins, it solves the problems of poor controllability and insufficient consistency in the generated images in the prior art, and significantly improves the efficiency and quality of comic or video production.

[0008] To achieve the above objectives, the present invention adopts the following technical solution:

[0009] In the first aspect, this application provides an AI-based graphic novel device, comprising:

[0010] The script parsing module is used to parse the input comic script and extract the character images, actions, and scene parameters required for each image;

[0011] The model training module is used to train LoRA models of characters and scenes based on the output of the script parsing module, and store the trained models in the model library.

[0012] The image generation module integrates the Stable Diffusion framework and is equipped with the Regional Prompter plugin, the Adetailer plugin, and at least one ControlNet plugin. It is used to control the character's position, actions, expressions, and the scene in which they are located, and to generate keyframe images that meet the requirements of the script.

[0013] The local repainting module segments the image region using the Segment Anything plugin and calls the Stable Diffusion Inpainting function to repair the specified region.

[0014] The output module is used to synthesize the processed image sequence into comics or videos, and supports adding text, subtitles, and exporting in multiple formats.

[0015] In a preferred embodiment, the model training module specifically includes:

[0016] Character Model Training Unit: Used to train the LoRA model of each character in order to control the character's appearance;

[0017] Scene Model Training Unit: Used to train LoRA models for each scene in order to control scene style.

[0018] In a preferred embodiment, the image generation module supports direct invocation of a pre-trained LoRA model.

[0019] In a preferred embodiment, the image generation module specifically includes:

[0020] Prompt word control unit: Used to introduce the LoRA model into the prompt words and set weight coefficients to control the character image and scene style;

[0021] Regional control unit: The screen is divided into multiple regions by the Regional Prompter plugin, and each region is bound to the position coordinates and action prompts of a specific character;

[0022] Detail Restoration Unit: Automatically detects and restores facial and limb details using the Adetailer plugin;

[0023] Style control unit: Add reference images and adjust the Style Fidelity parameter through the ControlNet Reference Only plugin to control the consistency of scene style;

[0024] Motion control unit: Captures the skeletal motion of the person in the reference image through the ControlNet OpenPose plugin, and controls the posture and position of the person in the generated image.

[0025] In a preferred embodiment, the local redrawing module specifically includes:

[0026] Segmentation Unit: Use the Segment Anything plugin to segment the image region to be adjusted and generate a mask;

[0027] Repaint Unit: Calls the Stable Diffusion Inpainting function to perform local repainting of the masked area.

[0028] In a preferred embodiment, the output module supports OCR text recognition, automatically matching script dialogue to the image timeline and generating subtitle files.

[0029] In a preferred embodiment, the output module supports exporting to MP4, GIF, or PDF formats, with an adjustable resolution range from 720p to 4K.

[0030] Secondly, this application also provides an AI-based method for creating textual comics, comprising the following steps:

[0031] Step S1: Analyze the comic script and plan the character design, actions, and scene parameters for each image;

[0032] Step S2: Train the LoRA model for the character and scene based on the parameters, or call the pre-trained LoRA model library;

[0033] Step S3: Use Stable Diffusion to generate keyframe images, use the Regional Prompter plugin to divide the screen area and bind the character's position, and use the ControlNet plugin to control the character's movements and scene style;

[0034] Step S4: Segment and redraw inconsistent local areas in the generated image to ensure the continuity of characters and scene details in the previous and subsequent images;

[0035] Step S5: Combine the corrected images into a comic or video according to the script order, and perform post-production.

[0036] In a preferred embodiment, step S3 specifically includes the following steps:

[0037] When generating multi-person scenes, the Regional Prompter plugin is used to divide the image into multiple regions for partition control, controlling the position and interaction of characters in each region.

[0038] When generating full-body and wide-angle photos, the Adetailer plugin automatically detects people and repairs their expressions and limbs.

[0039] The ControlNet Reference Only plugin controls the consistency of image scene style. A reference image is added, and the Style Fidelity parameter is adjusted to control the similarity between the generated image style and the reference image.

[0040] Control the actions and positions of people in an image using the ControlNet OpenPose plugin.

[0041] In a preferred embodiment, after generating the keyframe image in step S3, the method further includes: automatically filtering images through consistency scoring rules, wherein the consistency scoring rules include the matching degree of character clothing, the integrity of background elements, and the accuracy of actions.

[0042] In a preferred embodiment, step S4 specifically includes the following steps:

[0043] Use the Segment Anything plugin to segment out the parts of the image that need fine-tuning and generate a mask;

[0044] The Stable Diffusion Inpainting feature allows for localized repainting of areas inside or outside the mask until the generated image meets expectations.

[0045] Compared with the prior art, the technical solution of the present invention has the following beneficial effects:

[0046] This invention provides an AI-based comic book creation device and method. It extracts image parameters through a script analysis module, combines LoRA model training with the Stable Diffusion framework, and utilizes Regional Prompter, ControlNet, and Adetailer plugins to achieve multi-dimensional control over character positioning, actions, and scene style, generating highly consistent keyframe images. A local repainting module repairs details through segmentation and inpainting techniques, ultimately synthesizing comics or videos. This invention solves the problems of inconsistent characters and disjointed scenes in images generated by existing AI tools, significantly improving the controllability and efficiency of generated content, reducing manual adjustment costs, and is suitable for the automated generation of comic book creation and narrative videos. Attached Figure Description

[0047] To more clearly illustrate the specific embodiments of the present invention or the technical solutions in the prior art, the drawings used in the description of the specific embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0048] Figure 1 is a structural block diagram of the AI-based text-based comic device in an embodiment of the present invention;

[0049] Figure 2 is a flowchart illustrating the AI-based text-based comic strip method in an embodiment of the present invention. Detailed Implementation

[0050] To make the above and other features and advantages of the present invention clearer, the invention will be further described below with reference to the accompanying drawings. It should be understood that the specific embodiments given herein are for the purpose of explanation to those skilled in the art and are exemplary only, not restrictive.

[0051] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this invention are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be used interchangeably where appropriate. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion. For example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to these processes, methods, products, or apparatuses. Example

[0052] As shown in Figure 1, this embodiment provides an AI-based text-based comic device, including: a script parsing module, a model training module, an image generation module, a local redrawing module, and an output module. Each module is described in detail below.

[0053] 1.1 Script Analysis Module

[0054] The script parsing module is used to parse the input comic script and extract the character images, actions, and scene parameters required for each image.

[0055] Character design: Based on the analysis of the comic script, define the visual characteristics of each character, such as hairstyle, clothing, and facial features.

[0056] Scene setting: Define the background features of each scene, such as location, time, and weather.

[0057] Action design: Design the actions and expressions of each character in each scene to ensure that the actions and expressions meet the requirements of the script.

[0058] 1.2 Model Training Module

[0059] The model training module is used to train LoRA models of characters and scenes based on the output of the script parsing module, and store the trained models in the model library.

[0060] The purpose of training a LoRA model is to more accurately control image generation. Relying solely on prompts to shape characters and scenes results in highly random images, making it difficult to guarantee consistency in character and scene appearance. Of course, some readily available LoRA models can also be used directly online.

[0061] Specifically, the model training module includes:

[0062] The character model training unit is used to train the LoRA model of each character in order to control the character's appearance;

[0063] Scene model training unit, used to train LoRA models for each scene in order to control scene style.

[0064] 1.3 Image Generation Module

[0065] The image generation module integrates the Stable Diffusion framework and is configured with the Regional Prompter plugin, the Adetailer plugin, and at least one ControlNet plugin. It collaboratively controls the generation of keyframe images through the following units:

[0066] (1) A prompt word control unit is used to introduce the LoRA model into the prompt words and set weight coefficients to control the character image and scene style. For example, the prompt words may include "Male character A is wearing red clothes and standing in the park".

[0067] (2) Area control unit: The screen is divided into multiple areas by plugins such as Regional Prompter, and each area is bound to the position and action prompts of a specific character.

[0068] When rendering multi-character scenes, the accuracy of the generated images is relatively low. For example, when generating an image of a man and a woman talking face-to-face with prompts, the number of characters, their genders, and their positions are likely to be incorrect. Plugins such as Regional Prompter can be used to divide the image into multiple regions for zone control, allowing control over the characters' positions and interactions in each region, thus improving the success rate of the generated images. For example, "Left zone: Character A raises their hand; Right zone: Character B runs."

[0069] (3) Detail repair unit: Automatically detects and repairs facial and limb details of the character through the Adetailer plugin.

[0070] For example, when generating full-body or wide-angle photos, the face and limbs of the person may appear distorted. Adetailer can be used to automatically detect the person and repair their expression and limbs.

[0071] (4) Style control unit: Add reference images and adjust the Style Fidelity parameter through the ControlNet Reference Only plugin to control the consistency of scene style. The higher the coefficient, the closer the style of the generated image is to the reference image.

[0072] (5) Motion control unit: captures the motion of the human skeleton in the reference image through the ControlNet OpenPose plugin, and controls the posture and position of the human in the generated image.

[0073] For example, if an image is obtained during the image generation process where the character's movements meet the requirements but other aspects do not, this image can be used as a reference image. OpenPose can be used to capture the character's position in the frame, and the character's posture and stance can be restored in the generated image.

[0074] 1.4 Local Redraw Module

[0075] The Local Repaint module further adjusts images with some flaws. Specifically, it segments the image region using the Segment Anything plugin and calls the Stable Diffusion Inpainting function to repair the specified region.

[0076] The local redraw module specifically includes:

[0077] The segmentation unit uses the Segment Anything plugin to segment the image region that needs fine-tuning and generate a mask. For example, it can separate the tie part of a person.

[0078] The repaint unit calls the Stable Diffusion Inpainting function to locally repaint the area inside or outside the mask until the generated image meets expectations. For example, only the tie part can be repainted until the generated image meets expectations.

[0079] 1.5 Output Module

[0080] The output module is used to process and edit the processed image sequence, combine it into a comic or video, and add text or subtitles to create a comic or story narration video.

[0081] Specifically, it includes:

[0082] Image processing: Process the generated keyframe images, including color adjustment, sharpening, blurring, and other operations to improve image quality.

[0083] Editing: The processed keyframe images are edited according to the script order to form a continuous sequence of images.

[0084] Add text or subtitles: Add text or subtitles in appropriate places according to the script requirements to enhance the expressiveness of the visuals.

[0085] Video compositing: Combine edited image sequences into a video, and add background music, sound effects, and other elements to create a complete comic or story narration video. Example

[0086] As shown in Figure 2, this embodiment provides an AI-based text-based comic strip method, including the following steps:

[0087] Step S1: Analyze the comic script and plan the character design, actions, and scene parameters for each image.

[0088] Step S2: Train the LoRA model for the character and scene based on the parameters, or call the pre-trained LoRA model library.

[0089] Step S3: Use Stable Diffusion to control the character's position, actions, expressions, and the scene, and generate images to select keyframe images that best meet expectations.

[0090] Step S3 specifically includes the following steps:

[0091] Step S3.1: Introduce the LoRA model into the prompt words and set weight coefficients to control the character image and scene style.

[0092] Step S3.2 uses some functional plugins of Stable Diffusion to further improve the controllability of the output graph, including:

[0093] When drawing scenes with multiple characters, the accuracy of the generated images is relatively low. For example, when generating an image of a man and a woman talking face to face with prompts, the number of characters, their genders, and their positions are likely to be incorrect. Plugins such as Regional Prompter can be used to divide the image into multiple regions for zone control, thereby controlling the position and interaction of characters in each region and improving the success rate of the generated images.

[0094] When generating full-body or wide-angle photos, the face and limbs of the subject are prone to distortion. Adetailer can be used to automatically detect the subject and repair their expression and limbs.

[0095] The ControlNet Reference Only plugin controls the consistency of image scene style. Add a reference image and set the style Fidelity parameter. The higher the coefficient, the closer the style of the generated image is to the reference image.

[0096] The ControlNet OpenPose plugin allows you to control the movements and positions of people in images. For example, if you get an image where the person's movements meet your requirements but other aspects don't, you can use this image as a reference. OpenPose can then capture the person's position within the frame, allowing you to recreate their pose and position in the generated image.

[0097] Step S3.3: Following steps S3.1 and S3.2, the controllability and efficiency of the output image are greatly improved. Stable Diffusion is used to output the image and select keyframe images that meet the expectations.

[0098] In a preferred embodiment, after generating keyframe images, the method may further include: automatically filtering images using consistency scoring rules, wherein the consistency scoring rules include the matching degree of character clothing, the integrity of background elements, and the accuracy of actions.

[0099] Step S4: Segment and redraw inconsistent local areas in the generated image to ensure the continuity of characters and scene details in the previous and next images.

[0100] Further adjustments were made to some images with minor flaws, including:

[0101] Step S4.1: Use plugins such as Segment Anything to segment out the parts of the image that need fine-tuning and generate a mask.

[0102] Step S4.2 involves using the Stable Diffusion Inpainting function to locally redraw parts inside or outside the mask until the generated image meets expectations. For example, if the tie color of a person is inconsistent, the image area containing the tie is separated, and only the tie is redrawn while the rest of the image remains unchanged. This method can also be used to replace the image background or the person's character.

[0103] Step S5: Combine the corrected images into a comic or video according to the script order, and perform post-production.

[0104] After generating keyframe images that match the script, the images are processed, edited, and have text or subtitles added to create comics or plot explanation videos for subsequent promotion and operation.

[0105] To better understand the technical solution of the present invention, the following detailed explanation is provided through specific examples. Case Example

[0106] Imagine a comic strip script depicting a story of two friends taking a walk in a park. The script is divided into three scenes:

[0107] Scene 1: Two friends meet at the park entrance.

[0108] Scene 2: Two friends are taking a walk on a path in the park.

[0109] Scene 3: Two friends are resting on a park bench.

[0110] The implementation process is described in detail below based on the technical solution of this invention.

[0111] Step 1: Script planning and parameter extraction.

[0112] Input script: The script text is parsed using an NLP module to extract key elements for each scene.

[0113] 1) Scenario 1:

[0114] Characters: Character A (male, blue hoodie, short hair), Character B (female, red dress, ponytail);

[0115] Action: The two stand at the park entrance, smiling and looking at each other;

[0116] Background: The park entrance features an arched iron gate and green flower beds.

[0117] 2) Scenario Two:

[0118] Action: Character A is in front, and character B is behind. They walk along the path and talk.

[0119] Background: A stone-paved path lined with cherry blossom trees on both sides.

[0120] 3) Scenario 3:

[0121] Action: Character A sits on the left side of the bench, and Character B sits on the right side of the bench, both gazing at the lake.

[0122] Background: A wooden bench, a lake in the distance, and the setting sun.

[0123] Output parameters:

[0124] Character's clothing, hairstyle, and position coordinates (e.g., character A is at the left 30% of the screen).

[0125] Background style keywords (such as "cherry blossom tree - pink petals" and "sunset - warm tones").

[0126] Step 2: Model training.

[0127] A large amount of image data related to characters A and B was collected and preprocessed. Then, a LoRA model was trained using a deep learning framework. During training, the weight coefficients were adjusted to enable the model to more accurately control the visual features of characters A and B.

[0128] Step 3: Image generation.

[0129] Images for each scene are generated using the Stable Diffusion algorithm and various plugins:

[0130] 1) Scene 1 generation (park entrance):

[0131] The prompt reads: "At the park entrance, there are two friends. Character A is wearing a blue hoodie and smiling, and Character B is wearing a red dress and smiling. There is an iron archway, flower beds, and bright sunshine."

[0132] Use the Regional Prompter to divide the image into two regions, controlling the position and movement of character A and character B respectively.

[0133] For example: Divide the area into left (30%), right (30%), and background (40%); bind the left area to "Character A is standing on the left, wearing a blue hoodie"; bind the right area to "Character B is standing on the right, wearing a red dress".

[0134] Use Adetailer to repair the faces and limbs of characters A and B.

[0135] Use the ControlNet Reference Only plugin to control the background style to match the park entrance.

[0136] Use the ControlNet OpenPose plugin to control the actions and positions of characters A and B. For example, import a "standing face-to-face" skeleton diagram.

[0137] 2) Scene 2 generation (Cherry Blossom Path):

[0138] The prompt reads: "Two friends are walking on a path, with character A in front and character B behind. Cherry blossom trees line both sides, and pink petals are falling in the wind."

[0139] Use the Regional Prompter to divide the image into two regions, controlling the position and movement of character A and character B respectively.

[0140] For example: Divide the area into front (40%), back (30%), and background (30%); bind the front area to "Character A in front"; bind the back area to "Character B in back".

[0141] Use Adetailer to repair the faces and limbs of characters A and B.

[0142] Use the ControlNet Reference Only plugin to reference the background of Scene 1 and maintain a consistent cherry blossom color tone.

[0143] Use the ControlNet OpenPose plugin to control the actions and positions of character A and character B. For example, bind a "walking pose" skeleton map to character A and character B respectively.

[0144] 3) Scene 3 Generation (Rest on a Bench):

[0145] The prompt reads: "Two friends are sitting on a wooden bench, with character A on the left and character B on the right. They are looking at the lake as the sun sets."

[0146] Use the Regional Prompter to divide the image into two regions, controlling the position and movement of character A and character B respectively.

[0147] For example: Divide the area into left (40%), right (40%), and background (20%); bind the left area to "Character A is sitting on the left, looking at the lake"; bind the right area to "Character B is sitting on the right, looking at the lake".

[0148] Use Adetailer to repair the hand details of characters A and B (to avoid finger distortion).

[0149] Use the ControlNet OpenPose plugin to control the actions and positions of characters A and B. For example, import a "seated gazing" skeleton image and adjust the character's head angle.

[0150] Step 4: Local redraw.

[0151] Perform partial redrawing on the generated image:

[0152] 1) Scene 1 Correction:

[0153] Problem: Character A's hoodie is too gray (it needs to be adjusted to a bright blue).

[0154] Operation: Use Segment Anything to select the hoodie area and generate a mask; Inpainting prompt: "Bright blue hoodie with a delicate zipper"; Noise reduction intensity 0.3, enable "Color Transfer" to preserve the original image shadows.

[0155] 2) Scenario 2 Correction:

[0156] Problem: Character B's arm position is unnatural.

[0157] Procedure: Use Segment Anything to select the arm area and generate a mask; Inpainting prompt: "Natural arm posture, slightly bent"; After repainting, use OpenPose to verify the continuity of the motion.

[0158] 3) Scenario 3 Correction:

[0159] Problem: The lake surface reflects too much light, obstructing the distant view.

[0160] Procedure: Use Segment Anything to select the lake area and generate a mask; Inpainting prompt: "Calm lake surface, faint sunset reflection"; Adjust the noise reduction intensity to 0.5 to preserve the outline of the distant mountains.

[0161] Step 5: Output and Composition.

[0162] Image sequence processing: Arrange the corrected keyframe images in the order of the script, and use image processing software to crop, resize, and perform other operations on the images to meet the layout requirements of the comic. Add transition effects between images to enhance the continuity of the comic; for example, use "fade in / fade out" (0.5 seconds duration) between scenes, and use "panning camera" within scenes to simulate dynamic perspectives.

[0163] Subtitle addition: The script dialogue text is recognized by OCR and automatically matched with the timeline to generate subtitle files.

[0164] In summary, this invention provides an AI-based comic creation device and method. It extracts image parameters through a script analysis module, combines LoRA model training with the Stable Diffusion framework, and utilizes Regional Prompter, ControlNet, and Adetailer plugins to achieve multi-dimensional control over character positioning, actions, and scene style, generating highly consistent keyframe images. A local repainting module repairs details through segmentation and inpainting techniques, ultimately synthesizing comics or videos. This invention solves the problems of inconsistent characters and disjointed scenes in images generated by existing AI tools, significantly improving the controllability and efficiency of generated content, reducing manual adjustment costs, and is suitable for the automated production of comic creation and narrative videos.

[0165] The specific embodiments of the present invention have been described in detail above, but they are merely examples, and the present invention is not limited to the specific embodiments described above. For those skilled in the art, any equivalent modifications and substitutions to the present invention are also within the scope of the present invention. Therefore, all equivalent transformations and modifications made without departing from the spirit and scope of the present invention should be covered within the scope of the present invention.

Claims

1. An AI-based text-to-manga device, characterized by, Comprise: a script analysis module for analyzing an input comic script and extracting character image, action and scene parameters required for each picture; a model training module for training LoRA models of characters and scenes based on the output of the script analysis module, and storing the trained models in a model library; an image generation module integrated with the Stable Diffusion framework, configured with Regional Prompter plug-in, Adetailer plug-in and at least one ControlNet plug-in, for controlling character positions, actions, expressions and scenes to generate key frame pictures meeting the requirements of the script; a local redrawing module for segmenting image regions by Segment Anything plug-in and calling Stable Diffusion Inpainting function to repair specified regions; an output module for combining processed picture sequences into comics or videos, and supporting adding text, subtitles and multi-format export. 2.The AI-based text-to-comic device of claim 1, wherein, The model training module specifically comprises: a character model training unit for training LoRA models of each character to control the character image; a scene model training unit for training LoRA models of each scene to control the scene style. 3.The AI-based text-to-comic device of claim 1, wherein, The image generation module specifically comprises: a prompt word control unit for introducing LoRA models into prompt words and setting weight coefficients to control character image and scene style; a region control unit for dividing the picture into multiple regions by Regional Prompter plug-in, and binding the position coordinates and action prompt words of specific characters in each region; a detail repair unit for automatically detecting characters and repairing facial and body details by Adetailer plug-in; a style control unit for adding reference pictures and adjusting Style Fidelity parameters by ControlNet Reference Only plug-in to control scene style consistency; an action control unit for capturing character skeleton actions in reference pictures by ControlNet OpenPose plug-in to control the posture and position of characters in generated pictures. 4.The AI-based text-to-comic device of claim 1, wherein, The local redrawing module specifically comprises: a segmentation unit for segmenting the image area to be adjusted by Segment Anything plug-in to generate a mask; a redrawing unit for calling Stable Diffusion Inpainting function to redraw the mask area locally. 5.The AI-based text-to-comic device of claim 1, wherein, The output module supports OCR text recognition function, automatically matches script dialogues to picture timeline and generates subtitle files. 6.The AI-based text-to-comic device of claim 1, wherein, The output module supports exporting in MP4, GIF or PDF format, and the resolution can be adjusted in the range of 720p to 4K.

7. An AI-based text-to-manga method, characterized by, Comprise the following steps: Step S1: analyze the comic script and plan the character image, action and scene parameters of each picture; Step S2: train LoRA models of characters and scenes based on the parameters, or call the pre-trained LoRA model library; Step S3: Generate key frame pictures using Stable Diffusion, divide the picture area and bind the character position through the Regional Prompter plugin, and control the action of the character and the scene style through the ControlNet plugin; Step S4: Segment and redraw the inconsistent local area in the generated picture to ensure the coherence of the characters and scene details in the front and back pictures; Step S5: Combine the corrected pictures into a comic or video according to the script order and perform post-production.

8. The AI-based text-to-comic method of claim 7, wherein, The step S3 specifically includes the following steps: When generating a multi-person scene, divide the picture into multiple areas for partition control through the Regional Prompter plugin, control the character position and interaction action in each area; When generating full-length photos and long-range photos, automatically detect the character and repair its expression and body through the Adetailer plugin; Control the consistency of the picture scene style through the ControlNet Reference Only plugin, add a reference picture, and adjust the Style Fidelity parameter to control the similarity of the generated picture style to the reference picture; Control the action and position of the character in the picture through the ControlNet OpenPose plugin. 9.The AI-based text-to-comic method of claim 7, wherein, In the step S3, after generating the key frame picture, the method further includes: Automatically filter the picture through the consistency scoring rule, which includes the matching degree of character clothing, the completeness of background elements, and the accuracy of action.

10. The AI-based text-to-comic method of claim 7, wherein, The step S4 specifically includes the following steps: Segment the part of the picture that needs to be fine-tuned to generate a mask using the Segment Anything plugin; Redraw the part inside or outside the mask through the Stable Diffusion Inpainting function until the generated picture effect meets the expectation.

Images