Display device

Abstract

The present disclosure provides a display device. The display device comprises a display and at least one processor and is configured to: display a function page of a picture book application on the display, the function page displaying a prompt for generating a picture book by means of voice interaction; in response to a voice interaction instruction, display a picture book generation waiting page on the display, the picture book generation waiting page displaying the progress of picture book generation; upon generating the picture book, display a picture book playback page on the display, the picture book playback page comprising an image display area and a text display area, the generated picture book comprising at least one piece of story content in sequence and an image corresponding to each piece of story content; and sequentially display each piece of story content in the text display area, voice-broadcast the currently displayed story content, and display, in the image display area, the image corresponding to the story content. In this way, a picture book conforming to user requirements is automatically generated on the basis of the voice interaction instruction, and is displayed and played.

Description

A display device

[0001] Cross-references to related applications

[0002] This disclosure claims priority to Chinese applications filed on December 31, 2024, No. 202411982475.5; No. 202411982780.4; and No. 202411980675.7, the entire contents of which are incorporated herein by reference. Technical Field

[0003] This disclosure relates to the technical field of display devices, and more particularly to a display device. Background Technology

[0004] With the rapid development of display devices, the functions they can provide to users are becoming increasingly diverse. Currently, display devices include smart TVs, smart set-top boxes, smart set-top boxes, and products with smart display screens. Taking smart TVs as an example, smart TVs enable more and more scenarios, no longer just for watching TV programs in the home, but also for playing games, playing electronic photo albums, displaying information, and more.

[0005] Users can view preset images or videos through display devices. For example, users can view preset picture books through display devices, but cannot view picture books that are not preset. Therefore, traditional display devices have the problem of not being able to display picture books that meet the personalized needs of users. Summary of the Invention

[0006] In a first aspect, this disclosure provides a display device, comprising: a display configured to display content from a broadcasting system or network and / or a user interface; and at least one processor connected to the display, configured to: in response to an instruction to open a picture book application, display a function page of the picture book application on the display, the function page displaying a prompt instructing the generation of a picture book through voice interaction; in response to a voice interaction instruction, display a picture book generation waiting page on the display, the picture book generation waiting page displaying the progress of picture book generation; after the picture book is generated, display a picture book playback page on the display, the picture book playback page including an image display area and a text display area, the generated picture book including at least one story content and an image corresponding to each story content; each story content having a corresponding order; according to the order, sequentially displaying each story content in the at least one story content as target story content in the text display area, and performing voice playback on the target story content; and displaying the image corresponding to the target story content in the image display area.

[0007] Secondly, this disclosure provides a display device, comprising: a display configured to display content from a broadcast system or network and / or a user interface; at least one processor connected to the display and configured to: display a function page of the picture book application on the display in response to an instruction to open a picture book application; generate a target picture book in response to a picture book generation instruction triggered on the function page, the target picture book comprising at least one picture book page having a page order, each picture book page comprising picture book text and picture book image; display a picture book playback page on the display, the picture book playback page comprising a picture book display area and a picture book editing control; the picture book display area being used to display the at least one picture book page according to the page order; in response to a triggering operation of the picture book editing control, acquiring newly added text content input by a user, generating a target image corresponding to the newly added text content based on the newly added text content and a picture book style corresponding to the target picture book; combining the newly added text content and the target image to form a new picture book page; inserting the new picture book page into the target picture book to obtain an updated picture book, and displaying the updated picture book in the picture book display area. Attached Figure Description

[0008] Figure 1 is a flowchart of a picture book processing method according to some embodiments;

[0009] Figure 2 is a schematic diagram of a picture book application according to some embodiments;

[0010] Figure 3 is a schematic diagram of a functional page according to some embodiments;

[0011] Figure 4 is a schematic diagram of a picture book generating a waiting page according to some embodiments;

[0012] Figure 5 is a schematic diagram of a picture book playback page according to some embodiments;

[0013] Figure 6 is a schematic diagram of the composition of a text image model according to some embodiments;

[0014] Figure 7 is a flowchart of yet another picture book processing method according to some embodiments;

[0015] Figure 8 is a schematic diagram of yet another functional page according to some embodiments;

[0016] Figure 9 is a schematic diagram of yet another picture book playback page according to some embodiments;

[0017] Figure 10 is a data processing flowchart for visually generating large models according to some embodiments;

[0018] Figure 11 is a flowchart illustrating another picture book generation method according to some embodiments;

[0019] Figure 12 is a schematic diagram of the content of a picture book according to some embodiments;

[0020] Figure 13 is a schematic diagram of a picture book according to some embodiments;

[0021] Figure 14 is an internal structure diagram of a computer device according to some embodiments. Detailed Implementation

[0022] In this embodiment of the disclosure, display device 200 generally refers to a device with screen display and data processing capabilities. For example, display device 200 includes, but is not limited to, smart TVs, mobile terminals, computers, monitors, advertising screens, wearable devices, virtual reality devices, augmented reality devices, etc.

[0023] Users can operate the display device 200 via touch operation, mobile terminal 300, and control device 100. The control device 100 receives user input commands and converts them into control commands that the display device 200 can recognize and respond to. For example, the control device 100 can be a remote control, stylus, gamepad, etc.

[0024] In some embodiments, the control device 100 may be a remote control, and the communication between the remote control and the display device includes at least one of infrared protocol communication, Bluetooth protocol communication, and other short-range communication methods, controlling the display device 200 wirelessly or via a wired connection. Users can control the display device 200 by inputting user commands through buttons on the remote control, voice input, control panel input, etc.

[0025] The mobile terminal 300 can serve as a control device for performing human-computer interaction between the user and the display device 200. The mobile terminal 300 can also serve as a communication device for establishing a communication connection with the display device 200 and exchanging data.

[0026] Display device 200 may include at least one of the following: tuner / demodulator 210, communication device 220, detector 230, device interface 240, controller 250, display 260, audio output device 270, memory, power supply, and user input interface.

[0027] In some embodiments, the detector 230 is used to collect signals from the external environment or to interact with the outside world.

[0028] In some embodiments, device interface 240 may include, but is not limited to, a high-definition multimedia interface (HDMI), an analog or data high-definition component input interface (component), a composite video input interface (CVBS), a USB input interface (USB), an RGB port, etc. Device interface 240 may also be a composite input / output interface formed by multiple of the above interfaces.

[0029] Display device 200 can connect to external device 500 via device interface 240. External device 500 can be a set-top box, game console, PC, or other similar device. When display device 200 connects to external device 500 via device interface 240, a corresponding signal source channel can be established, and input signals can be received through this signal source channel. For example, display device 200 can connect to a game device via an HDMI interface to establish an HDMI channel, and receive audio and video signals sent by the game device based on the HDMI channel.

[0030] In some embodiments, the communication device 220 is a component used to communicate with external devices or the server 400 according to various communication protocol types. The display device 200 may have multiple communication devices 220 depending on the supported communication methods. For example, when the display device 200 supports wireless network communication, it may have a communication device 220 with WiFi functionality. When the display device 200 supports Bluetooth connectivity, it needs to have a communication device 220 with Bluetooth functionality.

[0031] The communication device 220 enables the display device 200 to communicate with external devices or the server 400 via wireless or wired connections. Wired connections utilize data cables, interfaces, or other components to connect the display device 200 to external devices. Wireless connections utilize wireless signals or wireless networks. The display device 200 can directly establish a connection with external devices or indirectly through gateways, routers, or other connection devices.

[0032] In some embodiments, the controller 250 may include at least one of a central processing unit, a video processor, an audio processor, a graphics processor, and a power processor, and a first to an nth interface for input / output. The controller 250 controls the operation of the display device and responds to user operations through various software control programs stored in memory. The controller 250 controls the overall operation of the display device 200.

[0033] In some embodiments, the controller 250 and the tuner 210 may be located in different separate devices, that is, the tuner 210 may also be located in an external device of the main device where the controller 250 is located, such as an external set-top box.

[0034] In some embodiments, a user can input user commands through a graphical user interface (GUI) displayed on a display 260, and the user input interface receives user input commands through the graphical user interface (GUI).

[0035] In some embodiments, the audio output device 270 can be a built-in speaker of the display device 200 or an external audio output device connected to the display device 200. For the external audio output device connected to the display device 200, the display device 200 may also be provided with an external audio output terminal, through which the audio output device can be connected to the display device 200 to output sound from the display device 200.

[0036] In some embodiments, the user input interface 280 can be used to receive instructions from user input.

[0037] In this embodiment, the application framework layer includes a view system, managers, and content providers. The view system designs and implements the application's interface and interactions, and includes lists, grids, textboxes, and buttons. The managers include at least one of the following modules: an activity manager for interacting with all running activities in the system; a location manager for providing system services or applications with access to system location services; a package manager for retrieving various information related to application packages currently installed on the device; a notification manager for controlling the display and clearing of notification messages; and a window manager for managing icons, windows, toolbars, wallpapers, and desktop widgets on the user interface.

[0038] Traditional picture books are primarily illustrated books with a small amount of text. With the rapid development of display devices, the functions they offer users have become increasingly diverse. Users can view preset images or videos on display devices, such as picture book content. Display devices can have preset picture books, each containing multiple images or videos. These images or videos can be displayed sequentially, allowing users to view the preset picture books. However, if a user requests a picture book on a specific theme, but that theme is not preset on the display device, the user cannot view it. Therefore, traditional display devices have the problem of not being able to display picture books that meet users' personalized needs.

[0039] To address the problem that traditional display devices cannot display picture books that meet users' personalized needs, this disclosure provides a picture book processing method. Figure 1 is a flowchart of a picture book processing method according to some embodiments. The method includes:

[0040] S100. In response to the instruction to open the picture book application, the function page of the picture book application is displayed on the display, and the function page displays a prompt indicating that a picture book can be generated through voice interaction.

[0041] The display device has multiple pre-installed applications, with the picture book application being a pre-installed application used to generate and display picture books. Figure 2 is a schematic diagram of a picture book application according to some embodiments. The application named "AI Speedy Picture Book" in the figure is the icon for the picture book application. After the user triggers the icon of the picture book application, the processor of the display device responds to the trigger operation of the icon and generates an instruction to open the picture book application.

[0042] The function page refers to the page used to showcase the functions of the picture book application. The function page displays a prompt indicating that picture books can be generated through voice interaction. This prompt suggests that users can automatically generate picture books by inputting voice commands. Figure 3 is a schematic diagram of a function page according to some embodiments. The prompt on the function page could be: "Just one sentence is all it takes to create a colorful children's picture book for you," for example, "Tell me a little kitten's adventure story."

[0043] In some embodiments, after displaying the function page of the picture book application, the voice recording function is enabled to receive voice interaction commands input by the user. For example, the voice interaction command could be: "The child doesn't want to take medicine, please tell him a story to teach him to take medicine." In some embodiments, the text corresponding to the voice interaction command can be displayed on the function page.

[0044] In some embodiments, the function page also displays at least one example text for voice interaction. The example text for voice interaction is used to provide the user with an example of voice interaction. Referring to Figure 3, the example text for voice interaction could be: Try saying: Tell me a story about a kitten's adventure.

[0045] In some embodiments, the function page also includes a picture book generation start control, as shown in Figure 3, with the button named "AI Picture Book Generation". In response to the triggering of the picture book generation start button, the voice recording function is activated to receive voice interaction commands input by the user.

[0046] S200, in response to voice interaction commands, displays a picture book generation waiting page on the screen, which shows the progress of picture book generation.

[0047] Voice interaction commands refer to instructions obtained after processing user-inputted voice. The picture book generation waiting page refers to the page displayed on the screen during the process of the processor generating the picture book. In some embodiments, the picture book generation waiting page displays a picture book generation waiting prompt to remind the user to wait for the picture book generation to complete. For example, the picture book generation waiting prompt could be: "The picture book is being drawn."

[0048] The picture book generation is performed by the processor's picture book generation module. The module sends progress information to the processor, which then displays the progress on a picture book generation waiting page. For example, the progress information can be a percentage of the current picture book generation duration relative to a preset duration. Figure 4 is a schematic diagram of a picture book generation waiting page according to some embodiments. The picture book generation waiting page can display the progress of picture book generation in the form of a progress bar.

[0049] In some embodiments, if the waiting time for picture book generation exceeds a preset time, that is, if the picture book generation is not completed after the preset time has been exceeded, a failure message is returned to indicate that the current picture book generation has failed.

[0050] S300. After generating the picture book, a picture book playback page is displayed on the monitor. The picture book playback page includes an image display area and a text display area. The generated picture book includes at least one story segment and an image corresponding to each story segment. Each story segment has a corresponding order.

[0051] Upon receiving progress information indicating that picture book generation is complete, the picture book is generated. The generated picture book typically contains at least one sequential story segment and corresponding images for each segment. The sequence of the story segments refers to their logical order within the context of the story. These segments, when combined in order, form a complete picture book story. The images corresponding to each story segment display the main information of that segment.

[0052] A picture book playback page refers to the page on which the generated picture book is played. Since the generated picture book includes story content and images, the picture book playback page can include an image display area and a text display area, thus dividing the story content and images for separate display. Figure 5 (left side) is a schematic diagram of a picture book playback page according to some embodiments. The picture book playback page includes an image display area and a text display area. The text display area displays the first paragraph of the story content, and the image display area displays the images corresponding to the first paragraph of the story content.

[0053] S400. In the order described, each segment of the at least one story content is selected as the target story content to be displayed in the text display area, and the target story content is read aloud. Additionally, the image corresponding to the target story content is displayed in the image display area.

[0054] Referring to the left side of Figure 5, the text display area of ​​the picture book playback page is used to display at least one segment of the story content, with one segment displayed at a time. The target story content refers to the story content currently displayed in the text display area. When the target story content is displayed in the text display area, it can be read aloud via voice. For example, a preset voice tone can be used to read the target story content aloud through the voice playback module. Alternatively, the user's voice tone can be determined by voice interaction commands, and then used to read the target story content aloud via the voice playback module. The voice tone used for voice playback can be determined based on the user's selection.

[0055] In addition, while displaying the target story content in the text display area, the corresponding image can also be displayed in the image display area. This allows users to view the picture book through the story content, images, and audio playback displayed on the device, enhancing the user experience. For example, following the sequence of at least one story segment, the i-th segment of the story content is displayed in the text display area, narrated audio, and the corresponding image is displayed in the image display area. After the audio playback of the i-th segment is completed, the display and audio playback of the (i+1)-th segment of the story content, image, and the last segment of the story content are continued, where i is a positive integer.

[0056] As shown on the right side of Figure 5, after the last paragraph (the 9th paragraph) of the story content is played, the text display area shows the text indicating that the picture book playback has ended, the audio playback stops, and the image display area shows the picture book playback completion screen.

[0057] In the above embodiments, the display device has a pre-installed picture book application. In response to the instruction to open the picture book application, the function page of the picture book application is displayed. The function page displays a prompt to generate a picture book through voice interaction, instructing the user to input a voice interaction command to generate the picture book. In response to the voice interaction command, a picture book generation waiting page is displayed, showing the progress of picture book generation so that the user can keep track of the progress. After the picture book is generated, a picture book playback page is displayed, where at least one segment of story content is displayed sequentially in the text display area, and the target story content being displayed is narrated by voice. At the same time, the image display area of ​​the picture book playback page displays the image corresponding to the target story content. In this way, a picture book can be automatically generated using the input voice interaction command, and at least one segment of story content and the image corresponding to each segment of story content can be displayed and played sequentially on the display. The display device can display picture books that meet the user's personalized needs, improving the user experience.

[0058] In some embodiments, the function page also displays at least one historically generated picture book, and the picture book processing method further includes: in response to a trigger operation on a first picture book among the at least one picture book, displaying a picture book playback page on the display and playing the first picture book on the picture book playback page; in response to a deletion operation on a second picture book among the at least one picture book, deleting the second picture book from the at least one picture book.

[0059] Each time a picture book is generated, the processor can save it to the picture book library, which stores all previously generated picture books. The function page can also display at least one previously generated picture book. Referring to Figure 3, at least one previously generated picture book is displayed on the function page as a triggerable control.

[0060] The first picture book refers to the picture book triggered by the user in at least one of the displayed picture books. In response to the triggering operation of the first picture book, the processor displays a picture book playback page on the display and plays the first picture book on the picture book playback page.

[0061] The second picture book refers to the picture book selected by the user from at least one of the displayed picture books. When the focus box of the remote control moves to the second picture book, in response to the trigger operation of the menu button on the remote control, the second picture book is deleted, and the function page will no longer display the second picture book.

[0062] In some embodiments, when the focus frame of the remote control moves to the second picture book, a play option and a delete option are displayed. In response to the triggering operation of the play option, a picture book playback page is displayed on the screen, and the second picture book is played on the picture book playback page. In response to the triggering operation of the delete option, the second picture book is deleted from at least one picture book, and the function page no longer displays the second picture book.

[0063] In some embodiments, the function page can display M historically generated picture books at a time. If the remote control's focus box moves to the j-th picture book among the M picture books, the focus box continues to move to the M-th picture book. If the focus box continues to move, the (M+1)-th picture book is displayed on the function page, and the first picture book is removed from the function page. This process continues, and other historically generated picture books can be displayed. j and M are both positive integers, and j is less than or equal to M.

[0064] This embodiment of the disclosure can display at least one historically generated picture book on the function page. Users can trigger any of the at least one picture book to play that picture book on the picture book playback page. In this way, users can repeatedly watch historically generated picture books, avoiding the resource and time consumption caused by repeatedly generating picture books with the same needs, which is conducive to improving the user experience. In addition, users can also delete any picture book in the at least one picture book to remove picture books that they are not satisfied with, which is also conducive to improving the user experience.

[0065] In some embodiments, after the target story text is generated based on voice interaction commands, the target story text is displayed on the picture book generation waiting page. The target story text refers to all the story content in the generated picture book. The picture book generation process involves first generating the target story text based on voice interaction commands, and then generating an image corresponding to each segment of story content based on at least one segment of story content in the target story text; that is, the generation of the target story text precedes the generation of the images.

[0066] After the target story text is generated, it can be displayed on the picture book generation waiting page until all the images corresponding to the story content have been generated, or until the progress information indicates that the picture book generation is complete. For example, referring to Figure 4, when the progress information is 51%, since the target story text has been generated, it can be displayed on the picture book generation waiting page, but the image generation is not yet complete, and the picture book generation will continue.

[0067] In some embodiments, the picture book generation waiting page also includes a prompt to return to the homepage, indicating that the user can stop picture book generation and return to the homepage by pressing the back button on the remote control. Users can stop picture book generation by pressing the back button on the remote control to return to the homepage, thereby improving the user experience.

[0068] In this embodiment of the disclosure, after generating the target story text based on voice interaction commands, the target story text is displayed on the picture book generation waiting page so that users can understand the story content of the generated picture book in time before the picture book images are generated, thereby improving the user experience.

[0069] In some embodiments, a static virtual image is displayed on the picture book generation waiting page; during the process of narrating the target story content being displayed in the text display area, the virtual image is controlled to perform corresponding actions in response to the narration. Here, the virtual image refers to a digital image displayed on the monitor, which can be displayed in two-dimensional or three-dimensional form. The type of virtual image can be a virtual character, a virtual animal, etc. Referring to Figure 4, the virtual image can be displayed statically on the picture book generation waiting page. Referring to the left side of Figure 5, the virtual image can be displayed dynamically on the picture book playback page; for example, during the process of the processor narrating the target story content being displayed in the text display area, the virtual image can be controlled to perform corresponding actions in response to the narration.

[0070] This embodiment of the disclosure can display a static virtual image on the picture book generation waiting page and a dynamic virtual image on the picture book playback page. The virtual image performs corresponding actions in response to the voice broadcast, which helps to improve the richness and vividness of the displayed content and enhance the user experience.

[0071] In some embodiments, interactive voice input from a user is received, and voice interaction instructions are generated based on the interactive voice; in response to the voice interaction instructions, the interactive voice is converted into target text; based on the target text and a pre-set first prompt word, a first input text is assembled, the first prompt word being used to guide the language model to generate character information and story theme that meet the first target requirements; the first input text is input into the language model to obtain target character information and target story theme; based on the target character information, target story theme, and a pre-set second prompt word, a second input text is assembled, the second prompt word being used to guide the language model to generate a story title, story synopsis, and story content that meet the second target requirements; the second input text is input into... The language model generates the target story text, which includes the target story title, synopsis, and content. Based on the target character information, target story theme, target story text, and pre-set third prompt words, a third input text is generated. This third prompt word guides the language model to generate a visual description text that meets the third target requirement. The third input text is then input into the language model to obtain the target description text. Based on the keywords in the target text, the picture book style to be generated is determined. The style prompt words corresponding to the picture book style to be generated are retrieved from the preset style library. The target description text and the style prompt words are then concatenated to obtain the concatenated text. The concatenated text is then input into the text-to-image model to generate the picture book.

[0072] The processor receives user-input interactive voice commands either through an internally configured voice command acquisition module or through an externally configured voice control device. It then converts the interactive commands into voice interaction commands, and in response, performs speech recognition to obtain the corresponding target text. For example, the target text could be: "A Little Adventure in the Four Seasons," or "The Friendship Between the Snowman and the Little Black Cat."

[0073] The first objective requirement indicates the requirements that the language model must follow when outputting target character information and target story theme. The first cue word refers to a pre-set cue word in the processor, used to guide the language model in generating character information and story theme that meet the first objective requirement. The processor can assemble the target text and the first cue word into the first input text, and input the first input text into the language model to generate the target character information and target story theme.

[0074] The first prompt can contain an extraction requirement text and a requirement description text. The extraction requirement text indicates the object to be extracted and the content to be extracted from the object. The requirement description text indicates the format of the output result and examples of formats. For example, the extraction requirement text for the first prompt could be: "Please extract the roles and themes contained in qurey based on the qurey I entered." Here, the object to be extracted is qurey, and the content to be extracted from the object is roles and themes.

[0075] The first prompt text can be: Do not output my input content, output the result directly, including the character information role_name and the story theme story_info; the output format is as a JSON object containing two keys, role_name and story_info; an example of the format is: {"role_name":[],"story_info":""}.

[0076] In some embodiments, the target text can be used as the object to be extracted, and then assembled with the first prompt word to obtain the first input text. The first target requirement indicates the requirements that the language big model outputs target character information and target story theme must follow, including the format of the output results, key information to complete the character image, separation of character name and character body, storage format of multiple character images, character information extraction requirements, and story theme extraction requirements, etc.

[0077] For example, the first objective requirement may include the following:

[0078] The output must strictly adhere to the following requirements:

[0079] 1. Directly output standard JSON format string data, disabling other content;

[0080] 2. Provide a suitable description for the extracted character image, in the format of [Character Name] + description. You can use your imagination, but it must be reasonable. For example, for people, you can define skin color, hairstyle, clothing, decorations, etc., and for animals, you can define color, decorations, etc.

[0081] 3. Each [character name] should contain only one character. If a word contains two or more characters, they should be defined separately. For example, if the character name contains "grandpa" or "grandma", "grandpa" and "grandma" should be defined separately.

[0082] 4. Store the values ​​in role_name using a Python list format, with the number of roles greater than 0 and less than 3;

[0083] 5. If no character can be found in the query, provide a suitable character based on the context information;

[0084] 6. If a character with the same name appears in Qurrey, give the character a unique name;

[0085] 6. Keep the story theme as concise as possible, but retain the key content from Qurrey.

[0086] The following is an example of generating target character information and target story theme based on the target text and the first cue word using a large language model:

[0087] Example 1: The target text is: "The story of a puppy venturing alone into the forest." The output of the language model could be:

[0088] {"role_name":["[Puppy]A yellow Shiba Inu puppy with a red collar around its neck"],

[0089] "story_info":"The story of a puppy embarking on a forest adventure out of curiosity and ultimately gaining courage."}

[0090] Example 2, the target text is: Xiaohong and Xiaoming went to school; the output of the language model can be:

[0091] {"role_name":["[Xiaohong] A Chinese girl with short black hair, wearing a white T-shirt, a black school skirt, yellow sneakers, and a red scarf","[Xiaoming] A Chinese boy with a buzz cut, wearing a blue T-shirt, blue jeans, white sneakers, and a red scarf"],"story_info":"The story of Xiaohong and Xiaoming helping each other at school and eventually gaining friendship"}.

[0092] Example 3, the target text is: Two children go to school; the output of the language model can be:

[0093] {"role_name":["[Xiaohong] A Chinese little girl with short black hair, wearing a white T-shirt, a black school skirt, yellow sneakers, and a red scarf","[Xiaoming] A Chinese little boy with a black buzz cut, wearing a blue T-shirt, blue jeans, white sneakers, and a red scarf"],"story_info":"The story of Xiaohong and Xiaoming helping each other and ultimately gaining friendship"}.

[0094] After generating target character information and target story theme based on target text and pre-set first cue words, the target story text can be further generated using a large language model.

[0095] The second objective requirement indicates the requirements that the language model's output target story text must meet. The second cue word is a pre-set cue word in the processor, used to guide the language model in generating a story title, story synopsis, and story content that conforms to the second objective requirement. The processor can assemble target character information, target story theme, and the second cue word into a second input text, and input this second input text into the language model to generate the target story text, which includes the target story title, target story synopsis, and target story content.

[0096] The second prompt can include story generation requirement text and requirement specification text. The story generation requirement text indicates the basis for generating the story text, the type of story text, and the required text paragraphs. The requirement specification text indicates the format of the output result and examples of formats.

[0097] For example, the story generation requirement text for the second prompt word could be: "Please create a children's story based on the story character_name and story summary I input, requiring clear paragraphs." Here, the story text is generated based on the story character_name and story summary_summary, the story text type is "children's story," and the paragraph requirement is clear paragraphs.

[0098] The second prompt's requirement description text can be: Do not output my input content, directly output the result, including the title, summary, and content; the output format should be a JSON object containing three keys: title, summary, and content; an example of the format is as follows:

[0099] {"title":"","summary":"","content":""}.

[0100] The processor uses the target role information as the story role_name in the second prompt and the target story theme as the story summary_summary in the second prompt, and assembles them together to obtain the second input text.

[0101] The second objective requirement specifies the requirements that the output story text of the large language model must follow, including the format of the output results, the format requirements for multiple story segments, the requirements for story segmentation, the story genre, the requirements for character naming, the requirements for sensitive words, and the requirements for character handling.

[0102] For example, the second objective requirement may include the following:

[0103] The output must strictly adhere to the following requirements:

[0104] 1. Directly output standard JSON format string data, disabling other content;

[0105] 2. The content in the output JSON data needs to be enclosed in double quotes, while single quotes can only be used inside the content if necessary.

[0106] 3. The story needs to have multiple parts, and each part **MUST** begin with **PAGE:** and cannot contain special characters such as (“\"\\| / ).

[0107] 4. Directly output the story synopsis; there is no need to provide unnecessary information such as page numbers and page breaks.

[0108] 5. The story should have 6-10 segments, with a minimum of 6 and a maximum of 10. Do not use segmented prompts.

[0109] 6. The story is primarily narrative and does not require character dialogue;

[0110] 7. Do not give characters random names;

[0111] 8. Sensitive words are strictly prohibited in the content;

[0112] 9. Double quotation marks are not allowed in the text content;

[0113] 10. Each story segment should not exceed two sentences;

[0114] 11. Only characters listed in Role_name may appear in the story; other characters or interactive figures are not allowed.

[0115] The following is an example of generating target story text using a large language model based on target character information, target story theme, and second cue words:

[0116] Example 1, Target Role Information Role_name: ["[Paul] A Shiba Inu with a red collar and brown eyes."], Target Story Theme Story_summary: ["Paul explores the city and ultimately finds happiness"].

[0117] Output target story text:

[0118] {"title":"Paul's Amazing Urban Adventures",

[0119] "summary": "Paul is a brave little Shiba Inu who wears a red collar and explores every corner of the city with his curious brown eyes."

[0120] "content":"PAGE: On a warm spring afternoon, Paul embarked on a journey to explore the city. His red collar gleamed in the sunlight. PAGE: Paul sniffed out various aromas in the market. PAGE: Then, Paul discovered a small park, and he happily rolled around on the grass. PAGE: As dusk fell, Paul discovered a tall tower, and he decided to climb it to overlook the entire city. PAGE: After descending from the tower, Paul encountered a flock of pigeons, which surrounded him, making him feel very warm. PAGE: Night fell, and Paul found his way home under the moonlight. PAGE: Paul rested at home for the night, preparing for new adventures. PAGE: The next day, Paul discovered a small river, and he played by its banks. PAGE: Paul found some beautiful stones by the river, and he took them home. PAGE: Paul's adventures made him very happy, and he looked forward to more explorations."}

[0121] After generating the target story text based on the target character information, the target story theme, and pre-set second cue words, the target description text can be further generated using a large language model.

[0122] The third objective requirement is used to instruct the language model on the requirements that the output target description text must meet. The third cue word is a pre-set cue word in the processor, used to guide the language model to generate image description text that conforms to the third objective requirement. The processor can assemble the target character information, target story theme, target story text, and third cue word into a third input text, which is then input into the language model to generate the target description text. Since the target story text includes multiple story segments, the target description text can include multiple description text segments, each describing the information of the corresponding image.

[0123] The third prompt can include the basis for generating the description text and the requirements for generating the description text. The basis for generating the description text indicates the basis for generating the description text. For example, the third prompt could be: "Based on the provided Title, Summary, Content, Role_list, and Role_info, generate the screen description text corresponding to each PAGE, with a strict one-to-one correspondence between PAGE and description text!" Here, the basis for generating the description text is Title, Summary, Content, Role_list, and Role_info, and the requirement is to generate the screen description text corresponding to each PAGE, with a strict one-to-one correspondence between PAGE and description text.

[0124] In some embodiments, the target story title, target story summary, and target story content can be used as Title, Summary, and Content in the third prompt, respectively. The target character information includes the name and basic information of at least one character. The name of at least one character is used as Role_list in the third prompt, and the basic information of each character is used as Role_info. Together with the third prompt, they are assembled to obtain the third input text.

[0125] The third objective requirement indicates the requirements that the output target description text of the large model should follow, including the output format, text opening description requirements, action description requirements, the correspondence between each element in the description text and the character, vocabulary and grammar requirements, the correspondence between picture book pages and description text, the correspondence between picture book pages and scene, the number of picture book pages required, and the character name output requirements, etc.

[0126] For example, the third objective requirement may include the following:

[0127] The output must strictly adhere to the following requirements:

[0128] 1. The screen contains a Role_name, which is a character description. The description text format A is: [Role_name] Action / Posture / Position / Background, with [Role_name] at the beginning of the sentence.

[0129] 2. The image does not contain a Role_name; it is a description of the scenery or background. The descriptive text format is B: [NC] Scenery description, indicated by [NC] at the beginning of the sentence. For example, "A vast grassland, full of life."

[0130] 3. For the actions of Role_name, use simple action words, such as "walk," "sit," "stand," "touch," etc., to describe them. Increase the diversity of foreground and background views;

[0131] 4. If there is no obvious action for Role_name, expand the posture according to the scenario. For example, sleeping can be expanded to actions such as "lying flat" or "closing eyes".

[0132] 5. Each PAGE description text should include the positional relationship between each element and the Role_name, such as "There is a green tree next to Paul" or "There is a sun above Jack's head";

[0133] 6. Each page description text should not contain any psychological or emotional actions. Delete psychological descriptions such as "very happy", "feel very fulfilled", and "dreaming". Only keep descriptions of body movements and element positions.

[0134] 7. Please ensure that the wording is simple, including simple grammar, simple vocabulary, and no modifiers or adjectives, so that the image generation model can understand it;

[0135] 8. Output simple Chinese syntax combinations, with each description text not exceeding 25 words in length;

[0136] 9. Avoid outputting irrelevant or analytical information; try to retain only one role in each output statement.

[0137] 10. [NC] Only describes landscape images;

[0138] 11. Each page should only output one text description;

[0139] 12. First, output the number of pages and the number of frames to be generated, strictly ensuring that the two numbers are consistent. Note! The minimum number of frames is 6.

[0140] 13. Determine the number of pages based on the content, and do not output any other content exceeding this number;

[0141] 14. Do not output notes! Do not output content that does not match the page in the content;

[0142] 15. [Role_name] The character name appears in the output. When the name is output, square brackets are added on both sides, such as [name]. The same name can only appear once in each segment.

[0143] The following is an example of generating target description text using a large language model based on target character information, target story text, and pre-set third cue words:

[0144] Example 1, Title: Jack's Beach Camping Adventure.

[0145] Summary: During his summer vacation, Jack decides to go to the beach alone for a camping adventure.

[0146] Content: PAGE: Jack shouldered his small backpack and set off for the beach. He planned to spend an unforgettable night on the sand. PAGE: As he arrived at the beach, the sun was slowly setting. Jack found the perfect campsite. PAGE: He skillfully pitched his tent, preparing for the night. PAGE: As night fell and the stars twinkled in the sky, Jack lit a campfire. PAGE: He roasted marshmallows, enjoying the tranquility brought by the sound of the waves. PAGE: Suddenly, a sea breeze blew by, and Jack felt a slight chill, but he wasn't afraid. PAGE: He took out a flashlight and strolled along the beach, exploring the mysteries of the night. PAGE: Under the moonlight, Jack discovered a group of small crabs busily working on the beach. PAGE: The next morning, as the first rays of sunlight shone on the tent, Jack was awakened by the warm light. PAGE: He packed up the tent, filled with memories, and set off on his way home.

[0147] Role_list:[Jack].

[0148] Role_info: [Jack, a black kitten].

[0149] The output target description text can be: Since there are 10 picture book pages, output the image description text for each of the 10 picture book pages, and output 10 scenes.

[0150] 1. [Jack] walks along the road in front of the beach with a small backpack on his back, against a backdrop of blue sky and white clouds.

[0151] 2. [NC] At dusk, the red sun hangs in the sky, and the sea is reflected in the sun, turning it red. The background is the sunset on the beach.

[0152] 3. [Jack] stood by a tent on the beach, with the red sunset over the sea and the sunset over the beach in the background.

[0153] 4. [Jack] sat by a campfire on the beach, with stars in the sky and the beach at night in the background.

[0154] 5. [Jack] sat by the campfire, holding a cotton candy in his hand, with the sea in the distance, the moon hanging in the sky, and the beach at night in the background.

[0155] 6. [NC] Waves on the sea, a tent on the beach, the moon hanging in the sky, the background is the beach at night.

[0156] 7. [Jack] was walking on the beach, the moon shining silvery light in the sky.

[0157] 8. [Jack] stood on the beach with a group of small crabs beside him and the moon hanging in the sky.

[0158] 9. [Jack] lay in a tent by the beach, the sun hanging in the sky, the background being morning.

[0159] 10. [Jack] walked home in the morning sun, carrying his duffel bag.

[0160] A text-to-image (TPE) model refers to a large-scale deep learning-based model that generates an image corresponding to the input text description by parsing and understanding the text. In some embodiments, the target descriptive text is input into the TPE model, generating at least one image corresponding to the target descriptive text. Each segment of the target descriptive text corresponds to one image, and multiple segments of descriptive text and their corresponding images constitute a picture book.

[0161] In other embodiments, the processor extracts keywords from the target text to obtain the keywords in the target text, and determines the picture book style of the picture book to be generated based on style-related keywords among the keywords. The picture book style of the picture book to be generated includes, but is not limited to, comic style, cartoon style, illustration style, etc. If the keywords do not contain style-related keywords, the default style is used as the picture book style of the picture book to be generated. For example, the default style is cartoon style.

[0162] A preset style library stores the correspondence between picture book styles and style hints. Querying the preset style library determines the style hints corresponding to the style of the picture book to be generated. The processor concatenates the target description text with the style hints to obtain concatenated text. This concatenated text is then input into the text-to-image model to generate a picture book with the style of the picture book to be generated.

[0163] In this embodiment, the target text in the user-input interactive voice is assembled with a first prompt word to form a first input text, thereby guiding the language model to generate target character information and target story theme. Then, the target character information, target story theme, and second prompt word are assembled into a second input text, thereby guiding the language model to generate target story text. Next, the target character information, target story text, and a third prompt word are assembled into a third input text, thereby guiding the language model to generate target description text. The target description text is a series of continuous story content segments. Based on the target description text, the text-to-image model generates images corresponding to each of the multiple continuous story content segments, resulting in a picture book. This achieves the automatic generation of picture books that meet user needs using the input interactive voice. Furthermore, users can indicate the picture book style through interactive voice, thereby automatically generating picture books that meet the user's style requirements, simplifying user operations and improving user experience.

[0164] In some embodiments, a training set can be constructed based on the style of the picture book to be adjusted; wherein, the style of the picture book to be adjusted represents the style of the picture book with unsatisfactory generation effect; the training set includes multiple training samples, each training sample including a sample image under the style of the picture book to be adjusted, and the content description text corresponding to the sample image;

[0165] For each sample image in the training set, the content description text corresponding to the current sample image is input into the text-generated graph network in the text-generated graph model; a target sample image is randomly selected from other sample images in the training set besides the current sample image; the target sample image is input into the adapter layer in the text-generated graph model, and the adapter layer extracts features from the target sample image to obtain the feature representation of the target sample image; the feature representation is output to the text-generated graph network; the text-generated graph network processes the received content description text and feature representation, and outputs a predicted image; the model loss is calculated based on the current sample image and the predicted image, the parameters of the adapter layer are adjusted based on the model loss, and training continues until the model loss meets the preset conditions.

[0166] Figure 6 is a schematic diagram of the composition of a text-based graph model according to some embodiments. The text-based graph model includes a text-based graph network and an adapter layer. When adjusting the parameters of the text-based graph model using the training set, the parameters of the text-based graph network can be frozen, and only the parameters of the adapter layer can be fine-tuned. This can significantly reduce the computational complexity and memory usage during model training, while also having advantages such as knowledge preservation, reduced resource overhead, improved generalization ability, and flexibility.

[0167] Specifically, for each sample image in the training set, the content description text corresponding to the current sample image is input into the text-based graph network in the text-based graph model. The target sample image from other sample images is input into the adapter layer. The adapter layer is used to extract features from the target feature image to obtain the feature representation of the target sample image. The feature representation is input into the text-based graph model. In this way, the text-based graph model processes the content description text and the feature representation to obtain the predicted image. The processor calculates the model loss based on the current sample image, the predicted image, and a preset loss function, and adjusts the parameters of the adapter layer based on the model loss. Training continues until the model loss meets the preset conditions, resulting in the parameter-adjusted text-based graph model. For example, LoRA (Low-Rank Adaptation) can be used to fine-tune the parameters of the text-based graph model.

[0168] In this embodiment of the disclosure, for the picture book style to be adjusted that is not ideal in terms of the generated effect of manual evaluation, the generation effect of the picture book style to be adjusted can be improved by fine-tuning the parameters of the adapter layer of the text-to-image model. At the same time, the parameter fine-tuning method can significantly reduce the computational complexity and memory usage during model training, and has the advantages of preserving knowledge, reducing resource consumption, improving generalization ability and flexibility.

[0169] Figure 7 is a flowchart of yet another picture book processing method according to some embodiments. Applied to a display device, it includes:

[0170] S1001. In response to the instruction to open the picture book application, the function page of the picture book application is displayed on the screen.

[0171] The display device includes a monitor and at least one processor. The monitor is configured to display content from a broadcast system or network and / or a user interface. The at least one processor is connected to the monitor and configured to execute picture book processing methods. Multiple applications are installed on the display device, and the picture book application is an application installed on the display device for generating, editing, and displaying picture books. A function page refers to a page used to demonstrate the functions of the picture book application. For example, the function page displays prompts for generating and editing picture books according to user needs.

[0172] In some embodiments, the display shows application icons for multiple applications. After the user triggers the icon of the picture book application, the processor responds to the triggering operation of the picture book application by generating an instruction to open the picture book application. In response to the instruction to open the picture book application, the processor opens the picture book application and displays the function page of the picture book application on the display.

[0173] In some embodiments, the function page further includes a picture book portfolio control and a featured portfolio control. In response to a trigger operation of the picture book portfolio control on the function page, a picture book portfolio page is displayed on the screen, the picture book portfolio page including at least one picture book; in response to a trigger operation of any picture book on the picture book portfolio page, the corresponding picture book content is played on the screen. Similarly, in response to a trigger operation of the featured portfolio control on the function page, a featured portfolio page is displayed on the screen, the picture book portfolio page including at least one featured picture book, which is a picture book selected from the picture book portfolio according to preset rules; in response to a trigger operation of any picture book on the featured picture book portfolio page, the corresponding picture book content is played on the screen.

[0174] S1002, in response to a picture book generation instruction triggered on a function page, generate a target picture book, the target picture book containing at least one picture book page with page order, each picture book page containing picture book text and picture book image; display a picture book playback page on the display, the picture book playback page containing a picture book display area and a picture book editing control; the picture book display area is used to display at least one picture book page according to page order.

[0175] The functional page includes a picture book generation control, such as a button. In response to a trigger operation of the picture book generation control, a picture book generation instruction is generated, and in response to the picture book generation instruction, a target picture book is generated. In some embodiments, the functional page may also include at least one of a story character input control, a story theme input control, and a style selection control. Figure 8 shows a schematic diagram of the functional page on the left. In response to a trigger operation of the story character input control, the user-inputted story character is obtained; in response to a trigger operation of the story theme input control, the user-inputted story theme is obtained. The functional page may also include a story character selection control. In response to a selection operation of the story character selection control, the target character selected by the user from multiple preset characters is obtained and used as the story character. Additionally, the user can input the story character, story theme, and picture book style via voice or text. The processor can, in response to a triggered picture book generation instruction, obtain the user-inputted story character and story theme, and generate the target picture book using a large model based on the story character and story theme. The processor can also, in response to a selection operation of the style selection control, obtain the target style selected by the user from multiple preset styles and use it as the picture book style.

[0176] In other embodiments, the processor may respond to a triggered picture book generation instruction, acquire interactive voice input by the user, identify the interactive text corresponding to the interactive voice, and generate a target picture book based on the interactive text using a large model.

[0177] The target picture book contains at least one page with a page number. Each page contains picture book text and picture book illustrations. The picture book texts from multiple pages can be combined in page order to form a picture book story with a logical contextual order. The picture book illustrations can display the story characters, story themes, and story content indicated by the picture book text.

[0178] The picture book playback page refers to the page where the picture book is played. The picture book playback page can be displayed immediately after the picture book generation command is triggered, or, after the picture book generation command is triggered, a picture book waiting page can be displayed first to instruct the user to wait for the picture book generation to complete; the picture book playback page can then be displayed after the picture book generation is complete. Figure 8 on the right shows a schematic diagram of a picture book waiting page according to some embodiments. The picture book waiting page displays prompt text, which can be: "Picture book creation in progress." For example, after the picture book text is generated, the picture book text can be displayed on the picture book waiting page until all picture book images are generated, and then the user exits the picture book waiting page.

[0179] The picture book playback page includes a picture book display area and picture book editing controls. The picture book display area is used to display at least one page of the picture book in page order. The picture book display area can display one page of the picture book at a time, or it can display multiple pages of the picture book at a time.

[0180] Picture book editing controls are used in response to user triggers, instructing the processor to edit the target picture book. There can be one or more picture book editing controls. For example, a picture book playback page may have one picture book editing control to instruct the processor to edit all the text in the target picture book. Alternatively, a picture book playback page may have multiple picture book editing controls, each instructing the processor to edit a corresponding page of the picture book.

[0181] S1003. In response to the trigger operation of the picture book editing control, obtain the new text content input by the user, generate the target image corresponding to the new text content based on the new text content and the picture book style corresponding to the target picture book; combine the new text content and the target image to form a new picture book page; insert the new picture book page into the target picture book to obtain the updated picture book, and display the updated picture book in the picture book display area.

[0182] The newly added text content is text input by the user, used to rewrite or continue the target picture book. In some embodiments, in response to a trigger operation on the picture book editing control, the processor displays a text input control on the picture book playback page, and the user receives the newly added text content input by the user.

[0183] The picture book style refers to the style type corresponding to the picture book images of the target picture book, such as cartoon style, comic style, illustration style, etc. In some embodiments, referring to the left side of Figure 8, the user can actively select the desired picture book style before the target picture book is generated. In other embodiments, the user may not specify the picture book style, and the large model will automatically select the appropriate picture book style based on the picture book text.

[0184] The target image is generated using a large model based on the newly added text content and the style of the target picture book. The target image can display the story characters, story themes, and story content indicated by the newly added text content, and the style of the target image is consistent with the style of the picture book corresponding to the target picture book.

[0185] In this embodiment, newly added text content is used as picture book text, and the target image is used as picture book image, thus forming a new picture book page. The new picture book page, as an independent page, can be inserted into the target picture book. For example, the insertion position can be the position indicated by the user, or the position indicated by the large model based on contextual semantic logic. After the new picture book page is inserted, an updated picture book is obtained, realizing picture book rewriting. Alternatively, the insertion position can be at the end of the picture book; after the new picture book page is inserted, an updated picture book is obtained, realizing picture book continuation.

[0186] The picture book display area showcases updated picture books, ensuring that the picture books displayed on the device meet the user's personalized needs.

[0187] In some embodiments, the picture book display area includes a text display area and an image display area. The text display area is used to display the picture book text in the picture book page, and the image display area is used to display the picture book image in the picture book page. In response to a trigger operation on the picture book editing control, the method of obtaining newly added text content input by the user includes: in response to a trigger operation on the picture book editing control, clearing the target picture book text currently displayed in the text display area, receiving the text content input by the user in the text display area, and using the text content as newly added text content.

[0188] Referring to Figure 9, the picture book display area includes a text display area and an image display area. The text display area is used to display the picture book text from at least one page of the picture book in page order, and the image display area is used to display the picture book images from at least one page of the picture book in page order. It is understood that picture book text and images belonging to the same page order are displayed in the same picture book display area.

[0189] In some embodiments, a text input control can be displayed on the picture book playback page to receive text content entered by the user. Then, the currently displayed target picture book text in the text display area is cleared, and the newly added text content is displayed in the text display area. The text content entered by the user can be determined by displaying a virtual keyboard on the screen, which displays multiple characters. In response to the movement operation of the remote control, the multiple characters selected by the user are determined, and the text content is determined by the multiple characters.

[0190] In this embodiment of the disclosure, the text display area of ​​the picture book display area displays the picture book text, and the image display area displays the picture book images. Users can input new text content through text input. The new text content can be displayed in the text display area and can be used as story content for rewriting or continuing the picture book. This is beneficial for rewriting or continuing the target picture book and thus generating a picture book that meets the user's personalized needs.

[0191] In some embodiments, in response to a triggering operation on the picture book editing control, obtaining newly added text content input by the user includes: in response to a triggering operation on the picture book editing control, displaying a voice input prompt on the picture book playback page; receiving interactive voice input by the user, converting the interactive voice into text, and using the text as newly added text content.

[0192] In some embodiments, new text content can also be obtained through voice input. Specifically, in response to a trigger operation on the picture book editing control, the processor displays a voice input prompt on the picture book playback page. The voice input prompt is used to instruct the user to input the picture book content to be edited in the form of voice.

[0193] In some embodiments, a voice module can also be activated to receive interactive voice input from the user, perform speech recognition on the interactive voice to obtain the corresponding text, and use this text as new text content. The voice module can be a module configured inside the processor or an external voice control device.

[0194] The new text content is usually a rewrite or continuation of the target picture book. For example, the theme of the target picture book is: Bai Bai is a little white cat with blue eyes. It explores the changing seasons and experiences the joy of growing up. The target picture book has 7 pages. The new text content could be: This year, Bai Bai made many friends in each season. It hopes to bring this joy of growth to more friends next year, so that everyone can grow together.

[0195] For example, the processor can display newly added text content on the currently displayed picture book page, and can also display picture book pages without picture book text and picture book images on the picture book playback page before the currently displayed picture book page, after the currently displayed picture book page, before the last picture book page, or after the last picture book page, and display the newly added text content in the text display area of ​​the picture book page.

[0196] In this embodiment of the disclosure, users can input new text content through voice input. The new text content can be used as story content for rewriting or continuing the picture book, which is beneficial for rewriting or continuing the target picture book and thus generating a picture book that meets the user's personalized needs.

[0197] In some embodiments, in response to a trigger operation on the forward expansion control, the page number of the target picture book currently displayed in the picture book display area can be used as the page number of the newly added picture book page, and the page number of the target picture book page and all subsequent picture book pages can be incremented by one. Here, the forward expansion control refers to a triggerable control on the picture book playback page. For example, the forward expansion control can be a button; clicking the button triggers a trigger operation on the forward expansion control. As another example, referring to Figure 9, the forward expansion control can be a triggerable option in a drop-down list; in response to a trigger operation on the drop-down list, the forward expansion option is displayed; clicking the forward expansion option triggers a trigger operation on the forward expansion control.

[0198] In some embodiments, the page number of the currently displayed target picture book page can be used as the page number of the newly added picture book page, and the page number of the target picture book page and all subsequent picture book pages can be incremented by one, that is, the newly added picture book page can be inserted before the target picture book page. For example, if the target picture book page is page 7, after the newly added picture book page is constructed, the newly added picture book page can be used as the new page 7, and the original page 7, as well as pages 8, 9, etc., are all incremented by one, becoming the new page 8, page 9, page 10, etc.

[0199] In this embodiment of the disclosure, a triggerable forward expansion control is pre-set in the picture book playback page. Based on the user's trigger operation on the forward expansion control, the newly added picture book page can be inserted before the currently displayed target picture book page, which is beneficial to meeting the user's rewriting needs of the picture book.

[0200] In some embodiments, the picture book playback page further includes a backward expansion control; inserting a new picture book page into the target picture book includes: in response to a trigger operation on the backward expansion control, using the page number of the next picture book page of the target picture book currently displayed in the picture book display area as the page number of the new picture book page, and incrementing the page number of the next picture book page and all subsequent picture book pages by one.

[0201] In this context, "expand backward control" refers to a triggerable control on the picture book playback page. For example, an expand backward control can be a button; clicking the button triggers a response from the processor to the expand backward control. Alternatively, referring to Figure 9, an expand backward control can be a triggerable option in a drop-down list; responding to a drop-down list's trigger action, the expand backward option is displayed; clicking this option triggers a response from the processor to the expand backward control.

[0202] In some embodiments, the page number of the next page of the target picture book currently displayed can be used as the page number of the newly added picture book page. The page number of the next page and all subsequent pages is incremented by one, meaning the newly added picture book page is inserted after the target page. For example, if the target page is page 7, after the new page is created, it can be used as the new page 8, and the original page 8, page 9, page 10, etc., are all incremented by one, becoming the new page 9, page 10, page 11, etc.

[0203] In this embodiment of the disclosure, a triggerable backward expansion control is pre-set in the picture book playback page. Based on the user's trigger operation on the backward expansion control, the newly added picture book page can be inserted after the currently displayed target picture book page, which is beneficial to meet the user's rewriting needs of the picture book.

[0204] In some embodiments, generating a target image corresponding to the newly added text content based on the picture book style corresponding to the target picture book includes: constructing a picture book editing instruction based on the newly added text content and the picture book style corresponding to the target picture book; the picture book editing instruction carries the newly added text content, the picture book style corresponding to the target picture book, and the identification data of the target picture book; sending the picture book editing instruction to the server to instruct the server to generate a target image corresponding to the newly added text content based on the picture book editing instruction; and receiving the target image fed back by the server.

[0205] In some embodiments, the processor stores at least one picture book, and the identification data is an identifier that distinguishes each picture book. Each picture book has unique identification data, such as a number. In some embodiments, the display device has a device identifier, the user has a user identifier, the picture book has a story identifier, and the picture book page has a page identifier. According to actual needs, the device identifier, user identifier, and story identifier can be combined to form the identification data of the picture book, and the device identifier, user identifier, story identifier, and page identifier can be combined to form the identification data of the picture book page.

[0206] Because the picture book editing instructions carry the new text content, the picture book style corresponding to the target picture book, and the target picture book's identifier data, the instructions can direct the server to generate the target image corresponding to the new text. The target image can display the scene indicated by the new text content and maintain consistency with the picture book style of the target picture book. This enables the function of rewriting or continuing picture books based on the user's editing needs, improving the flexibility of picture book generation.

[0207] Figure 10 is a flowchart of the data processing method for a large-scale visual generation model according to some embodiments. The picture book style is pre-encoded using a pre-trained encoder, converting it into a style feature vector, which can be a set of numerical feature vectors. The newly added picture book text is encoded into cue word vectors based on a pre-trained language model. Subsequently, the style feature vectors and cue word vectors are integrated to obtain a conditional vector. Specifically, this integration can be achieved through concatenation or weighted summation. Next, the conditional vector is formatted into a format recognizable and processable by the large-scale visual generation model. This conditional vector is then passed to the large-scale visual generation model via API or direct call, guiding the model to generate the target image. Specifically, the large-scale visual generation model always uses the conditional vector as guidance, considering visual elements such as character features and style type during the target image generation process, while also incorporating the content of the newly added picture book text, ensuring that the generated target image both conforms to the user-inputted text content and maintains the original picture book style.

[0208] The large-scale visual generation model can be a diffusion model based on a cross-attention mechanism. A diffusion model is a generative model that generates high-quality data samples by progressively adding noise and then learning to denoise it. Cross-attention is an attention mechanism that allows the model to focus on information from different modalities during the generation process. In this embodiment, the cross-attention mechanism has been adjusted, specifically for the query variable (Q variable), key variable (K variable), and value variable (V variable). The Q variable represents the content features of the current image in the current generation process, while the K and V variables capture the degree of correlation between the context and different parts of the current image. Specifically, the K variable is a pre-calculated image feature related to the target picture book, providing a reference for the currently generated image. The model evaluates the similarity between the Q and K variables to find which queried image features best match the needs of the currently generated image content. The V variable provides detailed information after a successful match; once it is determined which image features best suit the needs of the currently generated image, the V variable provides specific visual details or style guidance to enrich the quality of the generated image.

[0209] Specifically, unlike traditional attention mechanisms where Q, K, and V variables are all based on the feature vector of the currently generated image, this embodiment uses the image feature data of the currently generated image as the query variable (Q variable), while the K and V variables use the style feature vector in the conditional vector.

[0210] The server integrates style feature vectors and cue word vectors to obtain conditional vectors. These conditional vectors are then input into a diffusion model based on a cross-attention mechanism. Starting with a noisy image, the diffusion model iteratively reduces noise over time steps until a clear image is generated. During each iteration of denoising, the cross-attention mechanism allows the model to adjust its generation strategy based on information from the conditional vectors. Specifically, the image feature data of the currently generated image is used as the query variable (Q variable), and the visual feature data from the conditional vectors are used as the K and V variables. The model calculates the similarity between the Q and K variables and weights and aggregates the V variable based on this similarity, thus guiding the update of the current image features. The diffusion model learns how to recover meaningful image features from noise, gradually removing noise until a final clear image is generated. Furthermore, the server can perform post-processing on the generated picture book images, such as color correction and resolution adjustment, to improve the image quality.

[0211] If a user wants to adjust the style of a picture book, such as changing it from "comic" to "traditional Chinese style", the server can encode the adjusted picture book style to obtain the adjusted style feature vector, thereby updating the condition vector and guiding the visual generation model to generate a target image that matches the adjusted picture book style.

[0212] In this embodiment of the disclosure, by integrating style feature vectors and prompt word vectors into a conditional vector, and using this conditional vector to guide the visual generation model, the target image generated by the visual generation model can maintain consistency with the style of the target picture book, and the content of the target image matches the newly added text content input by the user, which is beneficial to improving the user experience.

[0213] In some embodiments, in response to a user's trigger command to the picture book editing control, the newly added text content input by the user can be obtained, and the identification data is used to obtain the picture book text of the target picture book from the character feature library. The newly added picture book content, the picture book text, and the first prompt word are assembled into a first input text, which serves as a prompt word for the first language big model to guide the first language big model to generate story text; based on the story text and the second prompt word, a second input text is assembled into a second input text, which serves as a prompt word for the second language big model to guide the second language big model to generate newly added picture book text; based on the newly added picture book text and picture book style, a target image is generated through a visual generation big model; the newly added text content and the target image form a new picture book page, which is then inserted into the target picture book for display.

[0214] In the above embodiments, the display device has a pre-installed picture book application. In response to an instruction to open the picture book application, the application can be opened and its function page displayed. In response to a picture book generation instruction triggered on the function page, a target picture book can be automatically generated, and a picture book playback page can be displayed. The picture book playback page can display at least one page of the target picture book with a page order, thus realizing the automatic generation and display of picture books according to the user's picture book generation needs. In response to the triggering operation of the picture book editing control on the picture book playback page, the newly added text content input by the user is obtained. Based on the newly added text content and the picture book style corresponding to the target picture book, a target image corresponding to the newly added text content is generated. The newly added text content and the target image can form a new picture book page. The new picture book page is inserted into the target picture book and displayed. Since the new picture book page has text content that meets the user's needs, an image that matches the text content, and a picture book style consistent with the target picture book, it realizes the ability to rewrite or continue the generated target picture book according to the user's picture book editing needs, and display a picture book that meets the user's personalized needs on the display device. The server can receive picture book editing instructions sent by the display device. These instructions carry new text content, the picture book style corresponding to the target picture book, and the target picture book's identifier data. Based on the new text content and identifier data, the server uses the text-to-text capability of the language model to generate new text content, and then uses the text-to-image capability of the visual generation model to generate a target image that matches the new text content and picture book style. Finally, the server sends the target image to the display device. This method, based on the user's picture book editing needs and combined with the text-to-text and text-to-image capabilities of the large model, can generate new text content and a target image with a consistent style that meet the user's picture book editing requirements. This facilitates the function of rewriting or continuing the target picture book, thereby displaying picture books that meet the user's personalized needs on the display device.

[0215] Figure 11 is an overall flowchart of another picture book processing method according to some embodiments, which can be applied to a display device 200, including:

[0216] Step S1401: When the picture book content is displayed on the user interface, receive the rewriting request information for the picture book content, and use the current picture book identifier as the picture book identifier corresponding to the rewriting request information.

[0217] Step S1402: Identify the text information corresponding to the rewrite requirement information, and retrieve the picture book information corresponding to the picture book identifier from the database based on the picture book identifier corresponding to the rewrite requirement information; the picture book information represents the multi-dimensional information of the picture book corresponding to the picture book identifier.

[0218] Step S1403: Input the text information and the information representing the characters in the picture book information into the text processing model to obtain the rewritten text content; the rewritten text content represents the text content composed of the text information and the information representing the characters; the text processing model is used to generate picture book text content based on the input text information and the information representing the characters, and output the rewritten text content.

[0219] Step S1404: Input the rewritten text content, the style information in the picture book information, and the character feature information in the picture book information into the image processing model to obtain rewritten image data; the rewritten image data represents image data that matches the rewritten text content, style information, and character feature information; the image processing model is used to generate picture book image data based on the input rewritten text content, style information, and character feature information, and output the rewritten image data.

[0220] Step S1405: Based on the information representing the timbre of the rewritten text content and the picture book information, perform audio data synthesis processing to obtain the audio data corresponding to the rewritten image data; and based on the information representing the theme and style in the picture book information, perform background audio data generation processing to obtain the background audio data corresponding to the rewritten image data.

[0221] Step S1406: Replace the picture book image data in the picture book content corresponding to the picture book identifier with rewritten image data, and replace the broadcast audio data and background audio data corresponding to the picture book image data with the broadcast audio data and background audio data corresponding to the rewritten image data to obtain the rewritten picture book; the number of pages in the rewritten picture book remains unchanged from the number of pages in the picture book corresponding to the picture book identifier.

[0222] The user interface can be an interactive interface used to display picture book content and receive user actions, such as the picture book's display interface. Picture book content can be a combination of images and text, such as illustrated content containing a storyline. Rewrite request information can be a user's modification request for the picture book content, such as modifications input by the user via voice, like "Rewrite this story as: The child cries loudly and knocks over the medicine bottle." Picture book identifiers can be unique identification information for a picture book, such as a unique identifier generated for each story. The database can be a data storage system for storing picture book-related information, such as a database system for storing character information, style information, story theme, story content, narrator voice ID, character reference characteristics, etc. Text information can be specific text content identified from the rewrite request information, such as text content converted from user voice input. Picture book information can be all information related to the picture book stored in the database, such as character information, style information, story theme, story content, narrator voice ID, character reference characteristics, etc.

[0223] Multi-dimensional information can be a collection of information describing various aspects of a picture book, such as information including characters, style, theme, content, and tone. A text processing model can be a model used to handle text generation tasks, such as a large language model. Rewritten text content can be new text content generated by the model based on user input and character information; for example, it can be new story content that meets the user's rewriting needs while maintaining character consistency. Information representing characters can be information describing the characteristics of the story's characters, such as character names and attributes.

[0224] Image processing models can be models used to generate images, such as a text-based image model. Rewritten image data can be new images generated based on rewritten text content, such as images generated by a text-based image model that match the new story content. Information representing style can describe the style of the picture book, such as cartoon, realistic, hand-drawn, or watercolor styles. Information representing character characteristics can be feature information used to maintain character consistency, such as character attribute-related features extracted from the text-based image model. Information representing narration timbre can be timbre information used for speech synthesis, such as system-integrated timbre for little girls, little boys, mature men, and mature women. Narration audio data can be speech data converted from text content, such as story narration audio generated by a speech synthesis model. Information representing the theme can describe the story's theme, such as the overall theme and style information of the story. Background audio data can be background music data that complements the story, such as light music generated based on the story's theme and style.

[0225] When displaying picture book content on the user interface, the system receives rewriting request information from the user via a speech recognition module and generates a unique picture book identifier for the current picture book. The rewriting request information is then input into a large language model for speech-to-text processing to obtain text information. Simultaneously, picture book information corresponding to the identifier is retrieved from the database. This picture book information includes multi-dimensional information such as character information, style information, theme information, and narration tone information. The text information and the character representation information from the picture book information are input into the large language model to generate rewritten text content that conforms to the character settings. The rewritten text content, along with the style representation information and character characteristic information from the picture book information, are input into a text-to-image large model. A consistency attention mechanism is used to maintain character image consistency, generating rewritten image data. Based on the rewritten text content and the narration tone information, a multimodal generation large model synthesizes narration audio data, and background audio data is generated based on the theme representation information and style representation information. Finally, the images, narration audio, and background audio in the original picture book content are replaced with the newly generated content to obtain the rewritten picture book, while maintaining the same number of pages.

[0226] In this embodiment, by associating picture book identifiers with multi-dimensional information and storing them in a database, and uniformly obtaining information such as roles, styles, and themes based on the picture book identifiers during rewriting, redundant operations such as repeated acquisition and manual input of information are avoided. By inputting the acquired information into text processing models, image processing models, and audio processing models for automated processing, and accurately replacing pages while keeping the page count unchanged, the consistency between the rewritten content and the original picture book in terms of roles, styles, and themes is maintained. This facilitates automated rewriting of picture books, reduces manual operation steps, and improves the efficiency of picture book rewriting.

[0227] In some embodiments, when picture book content is displayed on the user interface, an input selection operation for picture book content can be received. Then, if the selected picture book content is identified based on the selection operation, rewriting request information for the picture book content can be received.

[0228] Among them, the picture book content selection operation can be an operation in which the user selects the picture book content through an input device, such as the operation in which the user focuses on a picture book image through a remote control.

[0229] For example, referring to Figure 12, the display 260 shows the picture book image and text content of the picture book story page in the user interface of the picture book application. The picture book image may refer to the child holding medicine in Figure 12, and the picture book text content may refer to the text part in Figure 12, such as "The child smiled, he reached out, gently picked up the medicine, and prepared to eat it." The picture book story page in Figure 12 may be the 8th page of the picture book story with the theme "The Adventure of Medicine". The user can also return to the home page by clicking the back button.

[0230] Specifically, when the user interface displays picture book content, it receives the picture book content selection operation input by the user via remote control. When the user moves the focus to a certain page of the picture book content, it recognizes the specific picture book content selected by the user and activates the microphone of the voice receiving module. After recognizing the specific picture book content selected by the user based on the picture book content selection operation, it receives the rewriting request information of the selected picture book content input by the user through the voice receiving module.

[0231] In this embodiment of the disclosure, by first receiving the user's selection operation on the displayed picture book content, then identifying the specific picture book content based on the selection operation, and only then receiving the rewriting request information for the selected content, it is beneficial to accurately locate the specific content that the user wants to rewrite, avoiding the problem of unclear rewriting scope; thereby, it is beneficial to ensure that there is a clear correspondence between the rewriting request information and the picture book content selected by the user, and improve the accuracy of picture book rewriting.

[0232] In some embodiments, as shown in Figure 5, after receiving the input of a picture book content selection operation, the method further includes:

[0233] Step S801: Respond to the picture book content selection operation and identify the picture book content corresponding to the current focus;

[0234] Step S802: Identify the picture book content corresponding to the current focus as the selected picture book content.

[0235] The current focus can be the location currently selected by the remote control on the user interface.

[0236] The picture book content corresponding to the current focus can be the picture book image displayed at the current focus position of the remote control. For example, it can be the picture book image displayed on page 8 of the picture book story page when the user places the focus on the picture book image displayed on page 8 of the picture book story page using the remote control.

[0237] Specifically, the application control module receives the user's selection of picture book content via remote control and monitors the focus position of the remote control on the user interface of the picture book application in real time. When a change in the focus position of the remote control is detected, the coordinate information of the current focus position is obtained, and the corresponding picture book content is located in the user interface of the picture book application based on the coordinate information. After locating the picture book content corresponding to the current focus, the picture book content is used as the picture book content selected by the user, so as to receive subsequent rewriting request information for the picture book content.

[0238] In this embodiment of the disclosure, by responding to the picture book content selection operation and identifying the picture book content corresponding to the current focus, it is beneficial to accurately locate the specific content position that the user wants to rewrite and avoid misidentifying the user's rewriting intention; by identifying the picture book content corresponding to the current focus as the selected picture book content, it is beneficial to establish an accurate correspondence between the focus position and the specific content; thereby, it is beneficial to accurately understand the user's rewriting needs and improve the accuracy of picture book rewriting.

[0239] In some embodiments, before retrieving the picture book information corresponding to the picture book identifier from the database based on the picture book identifier corresponding to the rewritten requirement information, the method further includes: performing intent recognition processing on the text information corresponding to the rewritten requirement information to obtain an intent recognition result corresponding to the rewritten requirement information; the intent recognition result is used to characterize whether data is retrieved from the database; based on the intent recognition result, it is determined whether data is retrieved from the database; retrieving the picture book information corresponding to the picture book identifier from the database based on the picture book identifier corresponding to the rewritten requirement information includes: if it is determined that data is retrieved from the database, retrieving the picture book information corresponding to the picture book identifier from the database based on the picture book identifier corresponding to the rewritten requirement information.

[0240] Intent recognition processing can be a process of analyzing and understanding the rewriting requirements of user input.

[0241] The intent recognition result can be a classification result obtained after processing the user's input rewriting request information. For example, it can be the result of determining that the user wants to rewrite the content of the picture book and needs to obtain relevant data from the database.

[0242] Specifically, upon receiving the rewrite request information, the text information corresponding to the rewrite request information is sent to the domain intent recognition module for intent recognition processing; the intent recognition result returned by the domain intent recognition module is received, which is used to indicate whether data needs to be retrieved from the database; based on the intent recognition result, it is determined whether data needs to be retrieved from the database. When it is determined that data needs to be retrieved from the database, the picture book information corresponding to the picture book identifier corresponding to the rewrite request information, such as character information, style information, story theme, story content, narration voice ID, and character reference features, is retrieved from the database.

[0243] In this embodiment of the disclosure, by performing intent recognition processing on the text information corresponding to the rewriting requirement information, and determining whether it is necessary to obtain data from the database based on the intent recognition result, it is beneficial to avoid unnecessary database access operations; by obtaining the picture book information from the database only when it is indeed necessary to obtain data, it is beneficial to realize the on-demand execution of data acquisition operations; thereby, it is beneficial to reduce resource consumption and improve overall operating efficiency while ensuring the normal implementation of the rewriting function.

[0244] In some embodiments, the information representing the character characteristics is also obtained by: selecting picture book image data containing information representing the character from the picture book corresponding to the picture book identifier, as reference picture book image data; and extracting information representing the character characteristics from the reference picture book image data.

[0245] Among them, the picture book image data can be generated image frames that contain specific characters in the picture book, such as the image data corresponding to a page of the picture book that contains the character "Yaoyao".

[0246] The reference picture book image data can be image frames selected from the picture book to extract character features. For example, it can be an image selected from multiple image frames containing the character "Yaoyao" to maintain the consistency of character features in the subsequent generation process.

[0247] Specifically, from the picture books corresponding to the picture book identifier, a frame of picture book image data containing the character is selected as reference picture book image data through a random sampling function; the reference picture book image data is input into the U-Net (U network) diffusion model, and tokens features are extracted through the attention layer; the extracted tokens features are used as information representing the character features and stored in the database to maintain the consistency of character features in subsequent generation processes.

[0248] In this embodiment of the disclosure, by selecting picture book image data containing character information from the picture book as a reference, it is beneficial to obtain standardized visual features of the character; by extracting information representing character features from the reference picture book image data, it is beneficial to establish benchmark data for character features; thereby, it is beneficial to maintain the consistency of character image in the subsequent picture book generation process and improve the quality of picture book generation.

[0249] In some embodiments, text information is input into a text processing model to obtain key text information; key text information represents key information in the text information; the text processing model is used to output the key text information in the text information based on the input text information; key text information and information representing characters are input into the text processing model to obtain rewritten text content; rewritten text content represents text content composed of key text information and information representing characters; the text processing model is used to perform picture book text content generation processing based on the input key text information and information representing characters, and output the rewritten text content. Here, key text information can be the core information content in the user input text obtained after processing by the text processing model, such as key information like "the child is crying" and "the medicine bottle is overturned" extracted from the content "rewrite this story as, the child cries loudly and knocks over the medicine bottle on the ground".

[0250] Specifically, the user-inputted text information is fed into a large language model for processing. The core content of the text information is extracted through a key information extraction module to obtain key text information. The extracted key text information, together with the information representing roles obtained from the database, is input into the large language model. The large language model generates new picture book text content based on this input information and converts the generated picture book text content into content suitable for image generation, thereby obtaining rewritten text content.

[0251] In this embodiment of the disclosure, by first inputting text information into a text processing model to extract key text information, it is beneficial to accurately grasp the core content of the user's rewriting requirements; by inputting the extracted key text information together with the information representing the character into the text processing model, it is beneficial to generate new text content while maintaining the character characteristics; thus, it is beneficial to ensure that the rewritten content meets the user's needs while maintaining the consistency of the picture book character image and improving the quality of the generated rewritten text content.

[0252] In some embodiments, rewritten text content, style information from picture book information, and character feature information from picture book information are input into an image processing model to obtain rewritten image data. This includes: inputting the rewritten text content into a text processing model to obtain image generation prompt information corresponding to the rewritten text content; the image generation prompt information is used to represent text content adapted to the picture book image data generation process; the text processing model is used to adjust the input rewritten text content and output the adjusted text content; the adjusted text content is the image generation prompt information; inputting the image generation prompt information, style information, and character feature information into the image processing model to obtain rewritten image data; the rewritten image data represents image data matching the image generation prompt information, style information, and character feature information; and the image processing model is used to perform picture book image data generation processing based on the input image generation prompt information, style information, and character feature information, and output the rewritten image data.

[0253] The image generation prompts can be text descriptions that have been adjusted and optimized to be more suitable for image generation. For example, "The child is very happy" can be rewritten as "The child raises his hands and his face is full of smiles" to be more specific and detailed descriptive text.

[0254] Specifically, the rewritten text content is input into a large language model for processing, transforming simple descriptions like "the children are very happy" into more detailed prompts suitable for image generation. Information on representational style and character features is retrieved from the database, and the transformed image generation prompts, representational style information, and character feature information are input together into the text-to-image large model. The input information is processed through the Consistency Attention Module (CAB) in the U-Net diffusion model, and the consistency of character features is maintained by using a cross-attention mechanism in the downsampling and upsampling layers, ultimately outputting the rewritten image data.

[0255] In this embodiment of the disclosure, by converting the rewritten text content into prompts that are more suitable for image generation, the accuracy of text-to-image conversion is improved; by inputting the image generation prompts together with information representing style and information representing character characteristics into the image processing model, the consistency of style and character characteristics is maintained during the image generation process; thus, it is beneficial to generate high-quality image data that meets the user's rewriting needs while maintaining the original picture book style and character characteristics.

[0256] In some embodiments, inputting image generation prompts, style representation information, and character feature representation information into an image processing model to obtain rewritten image data includes: inputting image generation prompts and character feature representation information into an image processing model to obtain initial rewritten image data; the initial rewritten image data represents image data that matches the image generation prompts and character feature representation information; the image processing model is used to perform picture book image data generation processing based on the input image generation prompts and character feature representation information, and output the initial rewritten image data; inputting the initial rewritten image data and style representation information into the image processing model to obtain rewritten image data; the style information of the rewritten image data is the same as the style representation information, and the image content of the rewritten image data is the same as the image content of the initial rewritten image data; the image processing model is used to perform style representation information rendering processing on the input initial rewritten image data, and output the rewritten image data.

[0257] The initial rewritten image data can be the raw image data generated by the image processing model based on image generation prompts and character feature information, before style rendering. Specifically, the image generation prompts and character feature information are first input into the Wenshengtu large model, and processed by the Consistent Attention Module (CAB) in the downsampling and upsampling layers of the U-Net diffusion model. Reference image features containing the same character are selected from the database using the RandSample function, and the cross-attention between the currently generated image features and the reference image features is calculated in the U-Net diffusion model to generate the initial rewritten image data. The generated initial rewritten image data and style information are then input into the Wenshengtu large model for style rendering to obtain rewritten image data with the specified style.

[0258] In this embodiment of the disclosure, the process of generating rewritten image data is divided into two steps. First, initial rewritten image data is generated based on image generation prompts and information representing character characteristics. This helps to ensure the accuracy of the generated image content and the consistency of character characteristics. Then, the initial rewritten image data is subjected to style rendering processing to obtain rewritten image data. This helps to achieve style uniformity while keeping the image content unchanged. This improves the generation quality of rewritten image data, so that the generated image can accurately express the content and maintain style consistency.

[0259] In some embodiments, based on the information representing the timbre of the rewritten text content and the picture book information, playback audio data synthesis processing is performed to obtain playback audio data corresponding to the rewritten image data; and based on the information representing the theme and style in the picture book information, background audio data generation processing is performed to obtain background audio data corresponding to the rewritten image data, including: inputting the rewritten text content and the information representing the timbre of the playback audio into the playback audio data processing model to obtain playback audio data corresponding to the rewritten image data; the timbre information of the playback audio data is the same as the information representing the timbre of the playback audio, and the text content corresponding to the playback audio data is the same as the rewritten text content; the playback audio data processing model is used to... The input rewritten text content is converted into corresponding initial broadcast audio data. The initial broadcast audio data is adjusted to reflect the timbre of the broadcast, and the broadcast audio data corresponding to the rewritten image data is output. The information representing the theme and style is input into the background audio data processing model to obtain the background audio data corresponding to the rewritten image data. The background audio data is labeled with the background audio labels corresponding to the information representing the theme and style. The background audio data processing model is used to generate background audio data based on the background audio labels corresponding to the input information representing the theme and style, and outputs the background audio data corresponding to the rewritten image data.

[0260] The audio data processing model can be an AI model for converting text content into speech, such as a multimodal generation model. The initial playback audio data can be the speech data after the audio data processing model has converted the text into speech, but before any timbre adjustment; for example, it can be speech data converted from text content to a default timbre. The background audio data processing model can be an AI model for generating background music, such as a multimodal generation model. Background audio tags can be identifiers specifying the type of background music, such as tags for upbeat or relaxing music styles.

[0261] Specifically, the rewritten text content and information representing the timbre of the broadcast are input into the multimodal generation model. First, the rewritten text content is converted into initial broadcast audio data. Then, based on the information representing the timbre of the broadcast, the initial broadcast audio data is processed to adjust its timbre, generating broadcast audio data corresponding to the rewritten image data. Simultaneously, information representing the theme and style is input into the multimodal generation model. Based on the story theme and style characteristics, corresponding background audio labels are determined. Background audio data matching the rewritten image data is generated through a background audio data processing model. Finally, the broadcast audio data and background audio data are mixed to form a complete audio output.

[0262] For example, as shown in Figure 13, the display 260 shows multiple picture book stories on the user interface of the picture book application. These picture book stories may include "The Wonderful Journey of Pill," "The Wonderful Life of Kitten," "The Cute World of So-and-So," and so on. On this interface, users can see a brief introduction to the picture book story, click on AI picture book generation to automatically generate a picture book, and also delete a customized picture book by pressing the menu button.

[0263] In this embodiment of the disclosure, by acquiring multi-dimensional information of the original picture book and using it as a constraint for continuation writing, it is beneficial to ensure that the continuation content remains consistent with the original picture book in terms of character image, artistic style, and story theme. By collaboratively generating text content, image data, narration audio, and background audio, and determining the appropriate continuation position based on semantic recognition, it is beneficial to achieve a natural connection of multimodal content, maintaining the coherence and integrity of the story while expanding the picture book content. This facilitates the automated continuation of picture books, reduces manual operation steps, and thus improves the efficiency of picture book continuation writing.

[0264] In some embodiments, when a picture book is displayed on the user interface, an input picture book selection operation is received; if the selected picture book is identified based on the picture book selection operation, continuation request information for the picture book is received. The picture book selection operation can be an operation where the user selects a specific picture book through an input device, for example, an operation where the user controls the focus to fall on a picture book using a remote control.

[0265] Specifically, the user interface displays a list of picture books. When the user selects a picture book via remote control, the focus is shifted to the selected picture book, and the voice input function is activated. The voice receiving module is then invoked to turn on the microphone and receive information about the need to continue writing the picture book.

[0266] In this embodiment of the disclosure, by displaying picture books on the user interface and receiving the user's picture book selection operation, it is beneficial for the user to accurately locate the specific picture book that needs to be continued; by receiving the continuation request information after recognizing the picture book selected by the user, it is beneficial to ensure the accurate correspondence between the continuation request and the specific picture book; thereby, it is beneficial to establish an orderly picture book continuation operation process, avoid incorrect matching between the continuation request and the picture book, and improve the accuracy of picture book continuation.

[0267] In some embodiments, after receiving the input picture book selection operation, the method further includes: responding to the picture book selection operation by identifying the picture book corresponding to the current focus; and identifying the picture book corresponding to the current focus as the selected picture book. The selected picture book can be a specific picture book selected by the user based on the current focus position. For example, if the current focus is on the picture book "Yao Yao's Adventure Journey," it is identified that the user has selected the picture book "Yao Yao's Adventure Journey."

[0268] Specifically, the user interface displays a list of picture books. When a user selects a picture book via remote control, the system responds to the selection by identifying the current focus in the user interface using the focus positioning module. Based on the current focus's position, the system identifies the picture book corresponding to the current focus from the displayed list and recognizes it as the selected picture book.

[0269] For example, when a picture book is displayed on the user interface, the system receives a continuation request for that picture book. This includes: receiving the input of a picture book selection; responding to the selection and identifying the picture book currently in focus; recognizing the picture book as the selected book; and receiving the continuation request after recognizing the selected book. By responding to the user's selection and identifying the picture book at the current focus position, a clear correspondence between the user's action and the specific picture book is established. Recognizing the picture book as the user's selected book helps accurately locate the specific picture book the user wants to continue writing about, thus improving the accuracy of the user's selection. Receiving the continuation request after recognizing the selected book ensures an accurate correspondence between the continuation request and the specific picture book, thereby establishing an orderly continuation process, avoiding mismatches between continuation requests and picture books, and improving the accuracy of picture book continuation.

[0270] In this embodiment of the disclosure, by responding to the user's picture book selection operation and identifying the picture book corresponding to the current focus position, it is beneficial to establish a clear correspondence between the user's operation and the specific picture book; by identifying the picture book corresponding to the current focus as the picture book selected by the user, it is beneficial to accurately locate the specific picture book that the user wants to continue writing; thereby improving the accuracy of the user's picture book selection operation.

[0271] In some embodiments, in the continuation position of the picture book corresponding to the picture book identifier, continuation image data is added, and the corresponding audio data and background audio data are configured to obtain the continued picture book. This includes: when the continuation position is at the end of the picture book corresponding to the picture book identifier, adding continuation image data after the last page of the picture book corresponding to the picture book identifier, and configuring the corresponding audio data and background audio data to obtain the continued picture book; when the continuation position is at the beginning of the picture book corresponding to the picture book identifier, adding continuation image data before the first page of the picture book corresponding to the picture book identifier, and configuring the corresponding audio data and background audio data to obtain the continued picture book.

[0272] Specifically, the system retrieves character information, style information, and story theme from the database based on the picture book identifier, and determines the content addition method according to the continuation position. When the continuation position is at the end of the picture book, continuation image data is added after the last page; when the continuation position is at the beginning of the picture book, continuation image data is added before the first page. During the generation of continuation image data, a consistency attention mechanism is used to maintain the consistency of character images. Based on the story theme, a speech synthesis module is invoked to generate narration audio data, and based on the picture book style, a music generation module is invoked to generate background audio data. Finally, the continuation image data, narration audio data, and background audio data are integrated to form the continued picture book. In this embodiment, flexibly adding continuation image data at the beginning or end of the picture book allows users to choose the appropriate continuation position according to their actual needs. Configuring corresponding narration audio data and background audio data for the continuation image data helps maintain consistency between the continued content and the original picture book in terms of visual and auditory experience.

[0273] This disclosure also provides a computer device, which can be a terminal, and its internal structure is shown in Figure 14. The computer device includes a processor, memory, input / output interface, communication interface, display unit, and input device. The processor, memory, and input / output interface are connected via a system bus, and the communication interface, display unit, and input device are also connected to the system bus via the input / output interface. The processor provides computing and control capabilities. The memory includes a non-volatile storage medium and internal memory. The non-volatile storage medium stores the operating system and computer programs. The internal memory provides an environment for the operation of the operating system and computer programs in the non-volatile storage medium. The input / output interface is used for exchanging information between the processor and external devices. The communication interface is used for wired or wireless communication with external terminals; wireless communication can be achieved through Wi-Fi, mobile cellular networks, Near Field Communication (NFC), or other technologies. The display unit is used to form a visually visible image and can be a display screen, a projection device, or a virtual reality imaging device. The display screen can be an LCD screen or an e-ink screen. The input device of the computer device can be a touch layer covering the display screen, or buttons, trackballs, or touchpads set on the casing of the computer device, or external keyboards, touchpads, or mice, etc.

Claims

1. A display device, comprising: The display is configured to show content from a broadcast system or network and / or a user interface; At least one processor, connected to the display, is configured to: In response to an instruction to open a picture book application, the function page of the picture book application is displayed on the display, and the function page displays a prompt indicating that a picture book can be generated through voice interaction; In response to a voice interaction command, a picture book generation waiting page is displayed on the display, and the picture book generation waiting page displays the progress of picture book generation. After the picture book is generated, a picture book playback page is displayed on the monitor. The picture book playback page includes an image display area and a text display area. The generated picture book includes at least one story segment and an image corresponding to each story segment. The story segments are arranged in a corresponding order. In the order described, each segment of the at least one story content is selected as the target story content to be displayed in the text display area, and the target story content is read aloud via voice. Additionally, the image corresponding to the target story content is displayed in the image display area.

2. The display device according to claim 1, wherein the function page further displays at least one historically generated picture book, and the processor is further configured to: In response to a triggering operation of the first picture book, a picture book playback page is displayed on the monitor, and the first picture book is played on the picture book playback page; In response to the deletion operation of the second picture book, the second picture book is deleted from the at least one picture book; the first picture book and the second picture book are either of the at least one picture book.

3. The display device according to claim 1, wherein the processor is further configured to: After the target story text is generated based on the voice interaction command, the target story text is displayed on the picture book generation waiting page.

4. The display device according to claim 1, wherein the processor is further configured to: A static virtual image is displayed on the picture book generation waiting page; The process of broadcasting the target story content displayed in the text display area via voice, and controlling the virtual character to perform corresponding actions in response to the voice broadcast.

5. The display device according to claim 1, wherein the processor is further configured to: Receive interactive voice input from the user, and generate the voice interaction command based on the interactive voice; In response to the voice interaction command, the interactive voice is converted into target text; Based on the target text and a pre-set first prompt word, a first input text is assembled. The first prompt word is used to guide the language model to generate character information and story theme that meet the first target requirements. The first input text is then input into the language model to obtain the target character information and target story theme. Based on the target character information, the target story theme, and the pre-set second prompt words, a second input text is assembled. The second prompt words are used to guide the language model to generate a story title, story synopsis, and story content that meet the second target requirements. The second input text is then input into the language model to obtain the target story text, which includes the target story title, target story synopsis, and target story content. Based on the target character information, the target story theme, the target story text, and the pre-set third prompt words, a third input text is assembled. The third prompt words are used to guide the language model to generate a visual description text that meets the third target requirements. The third input text is then input into the language model to obtain the target description text. Based on the keywords in the target text, the picture book style of the picture book to be generated is determined. The style prompt words corresponding to the picture book style of the picture book to be generated are obtained from the preset style library. The target description text and the style prompt words are concatenated to obtain the concatenated text. The concatenated text is input into the text-to-image model to generate the picture book.

6. The display device according to claim 5, wherein the processor is further configured to: A training set was constructed based on the style of the picture book to be adjusted; among which... The picture book style to be adjusted represents the picture book style with unsatisfactory generation effect; the training set includes multiple training samples, each training sample includes a sample image under the picture book style to be adjusted, and the content description text corresponding to the sample image; For each sample image in the training set, the content description text corresponding to the sample image is input into the text-generated graph network in the text-generated graph model; and, from other sample images besides the sample image, one image is selected as the target sample image; The target sample image is input into the adapter layer of the Wensheng image model, so that the adapter layer performs feature extraction on the target sample image to obtain the feature representation of the target sample image; The feature representation is output to the text-based graph network, so that the text-based graph network processes the received content description text and feature representation, and outputs a predicted image; The model loss is calculated based on the current sample image and the predicted image. The parameters of the adapter layer are adjusted based on the model loss, and training continues until the model loss meets the preset conditions.

7. A display device, comprising: The display is configured to show content from a broadcast system or network and / or a user interface; At least one processor, connected to the display, is configured to: In response to an instruction to open the picture book application, the function page of the picture book application is displayed on the display. In response to a picture book generation instruction triggered on the function page, a target picture book is generated, the target picture book containing at least one picture book page with a page order, each picture book page containing picture book text and picture book image; A picture book playback page is displayed on the monitor. The picture book playback page includes a picture book display area and a picture book editing control. The picture book display area is used to display at least one page of the picture book according to the page order. In response to a trigger operation on the picture book editing control, the system obtains new text content input by the user, generates a target image corresponding to the new text content based on the new text content and the picture book style corresponding to the target picture book, combines the new text content and the target image to form a new picture book page, inserts the new picture book page into the target picture book to obtain an updated picture book, and displays the updated picture book in the picture book display area.

8. The display device according to claim 7, wherein the picture book display area includes a text display area and an image display area, the text display area being used to display picture book text in the picture book page, and the image display area being used to display picture book images in the picture book page; the at least one processor performs an operation in response to a triggering operation of the picture book editing control to obtain newly added text content input by the user, specifically configured as follows: In response to a trigger operation on the picture book editing control, the target picture book text currently displayed in the text display area is cleared; Receive text content entered by the user in the text display area and use the text content as new text content.

9. The display device according to claim 7, wherein when the at least one processor performs an operation in response to a triggering operation on the picture book editing control to obtain newly added text content input by the user, it is further configured to: In response to a trigger operation on the picture book editing control, a voice input prompt is displayed on the picture book playback page; Receive interactive voice input from the user, and use the text content converted from the interactive voice as new text content.

10. The display device according to claim 7, wherein the picture book playback page further includes a forward expansion control; and the at least one processor, when executing the insertion of the new picture book page into the target picture book, is further configured to: In response to the triggering operation of the forward expansion control, the page number of the target picture book currently displayed in the picture book display area is used as the page number of the newly added picture book page, and the page number of the target picture book page and each picture book page after the target picture book page is incremented by one.

11. The display device according to claim 7, wherein the picture book playback page further includes a backward expansion control; and the at least one processor, when executing the insertion of the new picture book page into the target picture book, is further configured to: In response to the triggering operation of the backward expansion control, the page number of the next page of the target picture book currently displayed in the picture book display area is used as the page number of the newly added picture book page, and the page number of the next picture book page and each picture book page after the next picture book page is incremented by one.

12. The display device according to any one of claims 7 to 11, wherein the at least one processor performs the operation of generating a target image corresponding to the newly added text content based on the newly added text content and the picture book style corresponding to the target picture book, specifically configured to: Based on the newly added text content and the picture book style corresponding to the target picture book, a picture book editing instruction is constructed; the picture book editing instruction carries the newly added text content, the picture book style corresponding to the target picture book, and the identification data of the target picture book; The picture book editing instructions are sent to the server to instruct the server to generate the target image corresponding to the newly added text content based on the picture book editing instructions.

Images